Issue with larger scales?

[uklimaschewski]

I'm currently working on using your excellent library for my EvalEx project.
While porting the acos() function, I came to some kind of rounding problems, I guess.
Given this code:

    MathContext mathContext = new MathContext(68, RoundingMode.HALF_EVEN);
    BigDecimal x = new BigDecimal("1", mathContext);
    BigDecimal acos = BigDecimalMath.acos(x, mathContext);

    System.out.println(acos.toPlainString());

it will not print 0 as expected, but it prints out 0.0000000000000000000000000000000000000000000000000000000000000000000000001

This only happens, when I use a scale >= 60 in the MathContext.

Is this a problem of big-math or BigDecimal?

[uklimaschewski]

When I debug into the function, the result is calculated as follows:

MathContext mc = new MathContext(mathContext.getPrecision() + 6, mathContext.getRoundingMode());

BigDecimal result = pi(mc).divide(TWO, mc).subtract(asin(x, mc));

pi(mc).divide(TWO, mc) results to 1.5707963267948966192313216916397514420985846996875529104874722961539082032
asin(x, mc) results to 1.5707963267948966192313216916397514420985846996875529104874722961539082031

doing the calculation without raising the precision by 6 digits, I get the correct result 0.
I also get a correct result when raising the precision by any number between 1 and 12, only the 6 will cause the above error.

[eobermuhlner]

I think this is a problem of the standard BigDecimal.round() method that I use to round the result to the desired precision.

I noticed that it does not round values that are closer to zero than the desired precision down to zero but leaves them as they are.
Notice that in your example the value
0.0000000000000000000000000000000000000000000000000000000000000000000000001
has a significant digit in the 74 position so in my opinion it should be rounded to zero.

I am considering to change my private round() method (which currently only wraps the call to standard round()).

    public static BigDecimal round(BigDecimal value, MathContext mathContext) {
	if (value.abs().compareTo(ONE.scaleByPowerOfTen(-mathContext.getPrecision())) < 0) {
		return ZERO;
	}
	return value.round(mathContext);
    }

This should fix your issue.
Need to check a bit more for side effects (and run the full set of unit tests on it).

[uklimaschewski]

Thank you for the quick reply and the workaround. Seems to work for me.