none
Why I do not get the exact operation RRS feed

  • Question

  • Hi folks,

    I am transforming feet-in to cm, but I have a case where I am getting a lot of decimals in the transformed value, although using the calculator, it gets an exact value. Here is the function: 

     

    public double ConvertFTLevelToCM(int ftLevel, intinLevel)

    {

     

      double levelInCM;

      levelInCM = ftLevel * 30.48f + inLevel * 2.54f;

    }

    If I set ftLevel = 22, inLevel = 6, using a calculator it gets an exact value of 685.8, but using the function it gets 685.79998970031738

    The problem I am facing with this is that I have to truncate the result. It should truncate from the exact value of 685.8, which is 685.8 exactly and correct, but using the result value of 685.79998970031738, the trunc value is 685.7, which is not correct.

    I am using Microsoft Visual c# 2008 and wonder why c# does not drop the exact value of 685.8 and looking for a general workaround when I should have exact results in the conversions.

    Thanks,

    Marco

    Monday, July 4, 2011 11:35 PM

Answers

  • This is standard behavior - see the Floating Point Guide for details: http://floating-point-gui.de/

     

    There are a few things you could do to help.  I'd recommend not using floats (use 30.48 instead of 30.48f), since your math will cause a double conversion anyways.  This will allow things to be more accurate.

     

    If you need the most accuracy, try using decimal (instead of double) for all of your math.  For example, try this:

    public decimal ConvertFTLevelToCM(int ftLevel, int inLevel)
    {
     decimal levelInCM;
     levelInCM = ftLevel * 30.48m + inLevel * 2.54m;
    
    }
    


     

    That being said, double will give you a LOT of precision - it's more a matter of how you use/display the result.  Instead of truncating, you should always round as needed, etc.

     

     


    Reed Copsey, Jr. - http://reedcopsey.com
    If a post answers your question, please click "Mark As Answer" on that post and "Mark as Helpful".
    • Proposed as answer by Paul Zhou Tuesday, July 5, 2011 4:15 AM
    • Marked as answer by MarcoViY Tuesday, July 12, 2011 2:21 PM
    Tuesday, July 5, 2011 12:09 AM
    Moderator

All replies

  • This is standard behavior - see the Floating Point Guide for details: http://floating-point-gui.de/

     

    There are a few things you could do to help.  I'd recommend not using floats (use 30.48 instead of 30.48f), since your math will cause a double conversion anyways.  This will allow things to be more accurate.

     

    If you need the most accuracy, try using decimal (instead of double) for all of your math.  For example, try this:

    public decimal ConvertFTLevelToCM(int ftLevel, int inLevel)
    {
     decimal levelInCM;
     levelInCM = ftLevel * 30.48m + inLevel * 2.54m;
    
    }
    


     

    That being said, double will give you a LOT of precision - it's more a matter of how you use/display the result.  Instead of truncating, you should always round as needed, etc.

     

     


    Reed Copsey, Jr. - http://reedcopsey.com
    If a post answers your question, please click "Mark As Answer" on that post and "Mark as Helpful".
    • Proposed as answer by Paul Zhou Tuesday, July 5, 2011 4:15 AM
    • Marked as answer by MarcoViY Tuesday, July 12, 2011 2:21 PM
    Tuesday, July 5, 2011 12:09 AM
    Moderator
  •  

    Hi,

     

    I am writing to check the status of the issue on your side. Would you mind letting us know the result of the suggestions?

     

    If you have got answers, please remember to mark answer and close this thread.

    If not, any more concerns, please feel free to let us know.

     

    Have a nice day!


    Paul Zhou [MSFT]
    MSDN Community Support | Feedback to us
    Get or Request Code Sample from Microsoft
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Tuesday, July 12, 2011 5:40 AM
  • Everything worked fine and the solution is as expected. Marked as Answered.

    Thanks,

    Marco

    Tuesday, July 12, 2011 2:22 PM