Answered by:
What is wrong with my math?
Question

Answers

A common issue with floatingpoint arithmetic, that is not as precise as you might think. 'What Every Computer Scientist Should Know About FloatingPoint Arithmetic" is a good read if you're looking for explanations.
wizend
 Marked as answer by Arne Garvan Thursday, May 31, 2018 11:18 PM
All replies

A common issue with floatingpoint arithmetic, that is not as precise as you might think. 'What Every Computer Scientist Should Know About FloatingPoint Arithmetic" is a good read if you're looking for explanations.
wizend
 Marked as answer by Arne Garvan Thursday, May 31, 2018 11:18 PM

Greetings Arne.
In case you don't want to go through the whole article in Wizend's link, here's the Readers' Digest version.
It's impossible for a computer to store floating point numbers (numbers with a decimal part, or to put it another way, not integers) to infinite accuracy. This means that when you perform mathematical operations on floating point numbers, you often get teensyweensy errors, even when it seems obvious to you what the result should be.
So in your example above, (a+b) is probably 0.29999999999 or 3.00000001 or something like that. This means that when comparing floats, you need to use a tolerance, rather than test that they are exactly equal.
public void Test() { var b = .1; var a = .2; bool c = NearEqual((a + b), 0.3, 0.001); Assert.IsTrue(c, "No math"); } bool NearEqual(double x, double y, float tolerance) { return ((x > y  tolerance) && (x < y + tolerance)); }


I want to pipe in too, because this is an interesting topic.
We see 0.1 and 0.2 as terminating decimals, but that's only because we are trained to use base 10. In base 2, the only "terminating decimals" are the powers of 2. So, 0.5 and 0.75 can be represented exactly in binary, but representing 1/10 in base 2 is like representing 1/3 in base 10. With a fixed number of digits, you cannot represent it exactly. Your computation gets a different rounding error than the compiler's constant for 3/10.
Tim Roberts, Driver MVP Providenza & Boekelheide, Inc.