Answered by:
Why I do not get the exact operation
Question

Hi folks,
I am transforming feetin to cm, but I have a case where I am getting a lot of decimals in the transformed value, although using the calculator, it gets an exact value. Here is the function:
{
levelInCM = ftLevel * 30.48f + inLevel * 2.54f;
}
If I set ftLevel = 22, inLevel = 6, using a calculator it gets an exact value of 685.8, but using the function it gets 685.79998970031738
The problem I am facing with this is that I have to truncate the result. It should truncate from the exact value of 685.8, which is 685.8 exactly and correct, but using the result value of 685.79998970031738, the trunc value is 685.7, which is not correct.
I am using Microsoft Visual c# 2008 and wonder why c# does not drop the exact value of 685.8 and looking for a general workaround when I should have exact results in the conversions.
Thanks,
Marco
Answers

This is standard behavior  see the Floating Point Guide for details: http://floatingpointgui.de/
There are a few things you could do to help. I'd recommend not using floats (use 30.48 instead of 30.48f), since your math will cause a double conversion anyways. This will allow things to be more accurate.
If you need the most accuracy, try using decimal (instead of double) for all of your math. For example, try this:
public decimal ConvertFTLevelToCM(int ftLevel, int inLevel) { decimal levelInCM; levelInCM = ftLevel * 30.48m + inLevel * 2.54m; }
That being said, double will give you a LOT of precision  it's more a matter of how you use/display the result. Instead of truncating, you should always round as needed, etc.
Reed Copsey, Jr.  http://reedcopsey.com
If a post answers your question, please click "Mark As Answer" on that post and "Mark as Helpful".
All replies

This is standard behavior  see the Floating Point Guide for details: http://floatingpointgui.de/
There are a few things you could do to help. I'd recommend not using floats (use 30.48 instead of 30.48f), since your math will cause a double conversion anyways. This will allow things to be more accurate.
If you need the most accuracy, try using decimal (instead of double) for all of your math. For example, try this:
public decimal ConvertFTLevelToCM(int ftLevel, int inLevel) { decimal levelInCM; levelInCM = ftLevel * 30.48m + inLevel * 2.54m; }
That being said, double will give you a LOT of precision  it's more a matter of how you use/display the result. Instead of truncating, you should always round as needed, etc.
Reed Copsey, Jr.  http://reedcopsey.com
If a post answers your question, please click "Mark As Answer" on that post and "Mark as Helpful". 
Hi,
I am writing to check the status of the issue on your side. Would you mind letting us know the result of the suggestions?
If you have got answers, please remember to mark answer and close this thread.
If not, any more concerns, please feel free to let us know.
Have a nice day!
Paul Zhou [MSFT]
MSDN Community Support  Feedback to us
Get or Request Code Sample from Microsoft
Please remember to mark the replies as answers if they help and unmark them if they provide no help.
