Math not working

    General discussion

  • Hi All I have strange problem. In Visual Studio 2010 when I am doing math operation where I have 2 numbers like 2,53+2 its giving me results=4.52999973.

    Do you know why this is happening?

    My Code:

    mk.ValueP=2.53 as Single

    mk.MaxVal = mk.ValueP + 2

    mk.MaxVal =4.52999973

    Tuesday, July 12, 2016 3:21 PM

All replies

  • Not all decimal fractions can be represented by floating point registers. Instead, you will get a close approximation.

    This is basic computer science.

    To understand why, read What Every Computer Scientist Should Know About Floating-Point Arithmetic.

    • Edited by Brian Muth Tuesday, July 12, 2016 3:24 PM
    Tuesday, July 12, 2016 3:23 PM
  • If it is VB.NET, then check maybe Decimal type is more appropriate for your program:

      mk.ValueP =2.53D ‘ fields declared “As Decimal”
      mk.MaxVal = mk.ValueP + 2 ‘ result: 4.53.

    Tuesday, July 12, 2016 6:26 PM
  • Allow me to expand on what Brian wrote.

    This seems to blow a lot of folks' minds.  I'm not sure what will make this click for you.  At some point in our careers we come across this ugly truth about floating point numbers.  Brian Muth's link is as good a place to learn that truth as any.  But I'm not sure at what point in the 30+ pages of reading you will have your "a-ha" moment.  It's the kind of thing I usually talk about one-on-one with people so that I can tease out their misconceptions.  But if I taught a group class on this, easily half the people wouldn't get it.  So don't feel bad if you find this confusing.  It's hard to know what preconception you have that needs dispelling.

    I'll fire a few ideas out like buckshot with hopes that something sticks.

    What I was doing when the lights came on for me was trying to write code to output the base-10 string representation of a floating point number.  The first time I did long division in binary also helped this make sense too.  I encourage you to try one or both of those things.  And ultimately I came away with this interesting conclusion that didn't seem obvious:

    Just because a number has a terminating decimal expansion doesn't mean it has a terminating expansion in base 2 (binary).

    Most of us get that you can't represent some fractions like 1/3 with a finite number of decimal digits because they have repeating digits  0.333333....  But the same problem occurs in other bases other than base 10.  Binary has this problem too.  But we don't have an infinite number of bits, and the floating point format can't capture "repeating sequences of digits" (other than the final digit, which is effectively like repeating 0, as you would expect)  So sadly, the binary representation for exactly 2.53 requires repeating digits to specify the fractional part of the number, but the floating point format is limited to a finite number of bits.  (It would be a repeating number in binary radix notation and would require an infinite number of repeating binary digits after the "radix point" to represent precisely).

    When you type 2.53 in your source code, the compiler generates the closest floating point value to 2.53 using the precision of the format (single or double) and it stores and uses that value.  You usually don't see the effect of this until you try to convert the number back to a string in base 10 after having performed some arithmetic on it.  The code that formats floating point numbers as decimal strings cheats to round off numbers, very often giving you the same thing you typed into your source code originally.  Many numbers with a modest number of digits will get formatted as exactly what you typed in originally in decimal.  That's intentional, but only to placate obsessive humans, it doesn't really have anything to do with precision.

    Anyway, I'm not the first and won't be the last to try to try to "show you the light" about floating point.  But I hope you manage to stay awake as you read at least some of the page that Brian Muth linked to.  It's worth it.

    Tuesday, July 12, 2016 8:27 PM
  • Thx for you help. I sorted this by changing single variant to decimal, and all is counting well. Truely I do not understand why is this variant used if its not giving proper numbers.
    Wednesday, July 13, 2016 11:13 AM
  • They both have similar problems, it's just that Decimal uses base 10 and float and double use base 2.  Neither can represent one third, for example.  But Decimal can precisely represent one tenth, whereas float and double cannot.  Because decimal uses base 10 internally, it's ideal for currency calculations that require a couple of decimal digits of accuracy after the decimal point.  The 128-bit Decimal type takes up more space and is a little less efficient than the 64-bit double or 32-bit float types - but you get higher precision.
    Thursday, July 14, 2016 1:01 AM