improve numerical stability of vector.rb

[alshedivat]

when qrot[r, r] is close to 0, ruby might represent it with a small negative value (e.g., -2.8475347e-16), which results in error when passed to Math.sqrt. adding a small epsilon constant to qrot[r, r] inside Math.sqrt solves the issue.