none
Calculations in floating point comparison RRS feed

  • Question

  • I have encountered the issue a couple of times now, and wondered if it is a deliberate implementation "feature":

    When comparing floating points and doing an inline calculation in the comparison, the outcome is very unpredicatable (yet luckily consistent ... ).
    For example the comparison between 3 and 10 * 0.3; the outcome should obviously be false since the result is equal:

    3 < 10F * 3
        true
    3 < 10 * 3F
        true
    3 < 10F * 0.3F
        true

    3 < 0.3F * 10
        true
    3 < 0.3 * 10F
        false
    3 < 0.3F * 10F
        true

    3 < 10 * 0.3
        false

    float result = 10 * 0.3F;
    3 < result
        false

    What is the logic behind this?

    // Rasmus
    Wednesday, November 12, 2008 2:46 PM

Answers

  • It is the same kind of logic that's in your desktop calculator.  Type 1 / 3 = * 3 =.  Why doesn't it produce "1"?  Use the System.Decimal type if you want to reproduce the same kind of calculation errors as your desktop calculator.  The float type cannot represent decimal fractions accurately, only binary fractions.
    Hans Passant.
    • Marked as answer by Zhi-Xin Ye Tuesday, November 18, 2008 1:58 PM
    Wednesday, November 12, 2008 3:40 PM
    Moderator
  • Read Jon Skeet's article on how floating-point arithmetic works:

     http://www.yoda.arachsys.com/csharp/floatingpoint.html

    • Marked as answer by Zhi-Xin Ye Tuesday, November 18, 2008 1:58 PM
    Thursday, November 13, 2008 8:32 AM

All replies

  • It is the same kind of logic that's in your desktop calculator.  Type 1 / 3 = * 3 =.  Why doesn't it produce "1"?  Use the System.Decimal type if you want to reproduce the same kind of calculation errors as your desktop calculator.  The float type cannot represent decimal fractions accurately, only binary fractions.
    Hans Passant.
    • Marked as answer by Zhi-Xin Ye Tuesday, November 18, 2008 1:58 PM
    Wednesday, November 12, 2008 3:40 PM
    Moderator
  • Read Jon Skeet's article on how floating-point arithmetic works:

     http://www.yoda.arachsys.com/csharp/floatingpoint.html

    • Marked as answer by Zhi-Xin Ye Tuesday, November 18, 2008 1:58 PM
    Thursday, November 13, 2008 8:32 AM