# is it a bug of Math library?

### Question

• Hi all,

In short Math.Acos(1.0000000000000002) returns NaN instead of 0 or some very small number. I run the testing on .Net Framework 3.5SP1. I tried on both Win7 32bit and Win2003 32bit.

Here is the complete context and method. The function I created is

```public static double CalculateDistanceByLatitudeLongitude(double latitude1, double longitude1,

double latitude2, double longitude2) {

// ACOS(SIN(lat1)*SIN(lat2)+COS(lat1)*COS(lat2)*COS(lon2-lon1))*6371

double var1 = Math.Sin(latitude1 * Math.PI / 180);

double var2 = Math.Sin(latitude2 * Math.PI / 180);

double var3 = Math.Cos(latitude1 * Math.PI / 180);

double var4 = Math.Cos(latitude2 * Math.PI / 180);

double var5 = Math.Cos(longitude2 * Math.PI / 180 - longitude1 * Math.PI / 180);

double var6 = var1 * var2 + var3 * var4 * var5;

double var7 = Math.Acos(var6);

double var8 = var7 * 6378.137;

return var7;

}

```

And I call the function by passing parameters:

```double distance = CalculateDistanceByLatitudeLongitude(38.41646, -81.82033, 38.41646, -81.82033);

```

I set break point on var7 and got NaN.

Any clue?

Thanks

Hardy

Welcome to help me with my open source project at http://code.google.com/p/batch-image-watermark-processor/
Thursday, November 04, 2010 1:51 PM

• According to Math.Acos it returns "NaN if d < -1 or d > 1." (d is the value passed.)
MCTS, CodeProject MVP 2008
Thursday, November 04, 2010 2:13 PM

### All replies

• According to Math.Acos it returns "NaN if d < -1 or d > 1." (d is the value passed.)
MCTS, CodeProject MVP 2008
Thursday, November 04, 2010 2:13 PM
• So is there a way I can make the calculation more precisely? Really the result from var6 should be 1.

Welcome to help me with my open source project at http://code.google.com/p/batch-image-watermark-processor/
Thursday, November 04, 2010 5:27 PM
• Actually, sounds like you want to make it work less precisely.  Test if the argument is within some tolerance greater than 1 and force the answer to be 0 instead of calling Math.Acos in this case.  If the argument is within some tolerance less than -1, force the answer to Math.Pi.

EDIT:  If you know that you never really go out of range, just clamp your input to [-1, 1].

Friday, November 05, 2010 12:13 AM
• Decimal rather than Double!!!

Should still check for valid range.

David Marso/TrancePlant
Friday, November 05, 2010 6:14 AM
• http://en.wikipedia.org/wiki/Significant_figures#Rounding
http://en.wikipedia.org/wiki/Significance_arithmetic

Your computation error is within the range of your significant digits.  It is easier to identify significant digits when the numerical values are written in scientific notation.  The cumulative effects of all of the sequential calculations can be estimated far more easily.  The number of  leading zeros is irrelevant, and the number of digits in the original operands are easily counted.

Dim PI as Double = 3.14259 x 10^0   ' 6 significant digits.

Also, I do not think that defining your type as decimal would gain you much.  Decimal variables are great for financial calculations and are quite accurate, but it comes at the price of speed.  When all of the calculations are performed as decimal aritmetic, then no rounding is introduced.  However, you are calling trigonometric functions which are using floating point math-or table lookups which approximate the values, too---for the calculations.  The trig functions will invariably introduce round-off errors of their own, anyway, even if you use decimal types.

http://en.wikipedia.org/wiki/Propagation_of_uncertainty

Here's a real world example.  Most people of heard of those GPS devices that you can put in your car, or even have built into your phone.  Those devices are rated to place your location within a certain range of error, which is commonly stated as shown at the links as plus/minus factor.  The more precise the original input measurements, the more precise the output calculation.

Based upon your method signature, I would think your calculated result has no more 7 significant digits, which puts that number 2 way out there in right field well within the margin of error..  But what about the accuracy of original paramters to the method?  I based my estimate of 7 significant digits on your sample method call.

Mark the best replies as answers. "Fooling computers since 1971."

http://rudedog2.spaces.live.com/default.aspx

Friday, November 05, 2010 9:33 PM
• Math.Acos doesn't have an overload that takes Decimal, so that idea is non starter.  Even if it did, that would not necessarily help here because the approximation error in this case is probably not isolated to base-10 to base-2 conversions.

The problem here is that Math.Acos needs to draw the line between returning a number close to or equal to 0 or Math.Pi versus returning Double.NaN.  It draws it strictly at the bounds of acos's domain of [-1, 1].  The actual argument could easily go slightly out of bounds, due to accumulated approximation errors, but the implementation of Math.Acos shows no compassion for this.  Sounds like the original poster needs to loosen this up, as I suggested earlier.

Saturday, November 06, 2010 12:08 AM
• Math.Acos doesn't have an overload that takes Decimal, so that idea is non starter.  Even if it did, that would not necessarily help here because the approximation error in this case is probably not isolated to base-10 to base-2 conversions.

The problem here is that Math.Acos needs to draw the line between returning a number close to or equal to 0 or Math.Pi versus returning Double.NaN.  It draws it strictly at the bounds of acos's domain of [-1, 1].  The actual argument could easily go slightly out of bounds, due to accumulated approximation errors, but the implementation of Math.Acos shows no compassion for this.  Sounds like the original poster needs to loosen this up, as I suggested earlier.

Actually I added some detection before callin Acos, if the value is > 1 (e.g. in this case value is 1.0000000000000002) I force the value to be 1.
Welcome to help me with my open source project at http://code.google.com/p/batch-image-watermark-processor/
Monday, November 08, 2010 2:03 PM
• Try to use correct literals.

I mean that 180 should be written as 180d and 6378.137 should be written as 6378.137d. It helped in my case.

Monday, November 08, 2010 4:21 PM