none
14.0 / (14 * 0.01) = ? RRS feed

  • Question

  • Hello everyone.

    Got issue with calculation items success ratio. Here's source code:

    List<ResponseMail> conversationsForCurrentPeriod = _responsesForCurrentPeriod;
    
    if (conversationsForCurrentMonth.Count <= 0)
    {
       return 0;
    }
    
    double successCount = conversationsForCurrentMonth.Count(obj => obj.ResponseInterval > 0);
    double result = successCount / (conversationsForCurrentMonth.Count * 0.01);
    
    return result;

    So in case I got 14 records in '_responsesForCurrentPeriod' collection and all of them are successful (their ResponseInterval is greater then 0) I got successCount = 14. So the 'result' variable is calculated by operation 14.0 / (14 * 0.01). The result of this operation should be 100 but in my particular case I'm resulted with 99.999999999999986.

    Please tell me whether this is a bug or what ?
    Thanks in advance.


    • Edited by eks77ern Saturday, December 20, 2014 5:51 PM
    Saturday, December 20, 2014 5:48 PM

Answers

  • this is a standard (IEEE 754) spec not a bug. Most numerical values cannot be stored in a finite amount of space thus data are truncated when storing, causing rounding errors that you must take into consideration when writing code.



    Visual C++ MVP


    Saturday, December 20, 2014 10:00 PM
  • If acceptable, consider decimal type instead of double:

    int successCount = 14;

    decimal result = successCount / ( 14 * 0.01m );

    • Proposed as answer by Blackwood Thursday, December 25, 2014 3:07 PM
    • Marked as answer by Fred BaoModerator Monday, December 29, 2014 9:11 AM
    Thursday, December 25, 2014 9:13 AM

All replies

  • this is a standard (IEEE 754) spec not a bug. Most numerical values cannot be stored in a finite amount of space thus data are truncated when storing, causing rounding errors that you must take into consideration when writing code.



    Visual C++ MVP


    Saturday, December 20, 2014 10:00 PM
  • If acceptable, consider decimal type instead of double:

    int successCount = 14;

    decimal result = successCount / ( 14 * 0.01m );

    • Proposed as answer by Blackwood Thursday, December 25, 2014 3:07 PM
    • Marked as answer by Fred BaoModerator Monday, December 29, 2014 9:11 AM
    Thursday, December 25, 2014 9:13 AM