none
Precision in Double data type, I need 8 digit precision RRS feed

  • Question

  • I an working on a University project, where I have to calculate a ratio, a typical value is 0.00017493xxx

    xxx means three non significant digits. 

    8 Digits are to be calculated that there must not be an error. I have learned that Double data type accumulates error and is approximate in its way of calculation. 
    I have been suggested to use decimal data type but it is too heavy on RAM and furthermore it is not very vast in terms of how much >1 numbers it can store say no 10^56 

    My question is, can I TRUST double data type for precision up to 8 digits. That there must not be loss due to any algorithmic approximation only exception is rounding (optional) of the 8th digit.

    If double has more than that what I asked then what is that limit and if it is lesser than that, what is that??? 

    Please clarify :) Love you all :) 

    Tuesday, November 28, 2017 1:36 PM

All replies

  • One way to find out is to setup a test with values that you know what the result should be and run code such as this to see if you get the expected result. Unit test are great for this.

    Dim number As Double = 0.00017493999
    Console.WriteLine(Math.Round(number, 8, MidpointRounding.AwayFromZero))


    Please remember to mark the replies as answers if they help and unmark them if they provide no help, this will help others who are looking for solutions to the same or similar problem. Contact via my Twitter (Karen Payne) or Facebook (Karen Payne) via my MSDN profile but will not answer coding question on either.
    VB Forums - moderator
    profile for Karen Payne on Stack Exchange, a network of free, community-driven Q&A sites

    Tuesday, November 28, 2017 1:48 PM
    Moderator
  • Warsi,

    Yes you can trust double precision.

    The problem comes from storing irrational numbers like 1/3. Since the value is .33333333333 sooner of later its .33333333333332 or .3333333333334. Or 4.00000000000000 is 3.99999999999999 (I am just making up numbers).

    So you have to make your code deal with it. The only place it will make any difference if you just need 8 digit precision is when the values are shown to the user. Then instead of 4.0 in the textbox it is 3.999999999. Thats where decimal does a bit better. So you can calc double precision and the present the answer in decimal so you can format it better for display, but keep the double value for calcs.

    I would try it, check the answers and see where my problems are if any. Then I have to fix them. Thats about the only way I think.

    Tuesday, November 28, 2017 1:53 PM
  • Warsi,

    The single and double are both approximations. If you want to absolutely maintain that sort of precision, consider using a decimal type which isn't floating point at all. From that document (please read them all):

    "Precision. Decimal is not a floating-point data type. The Decimal structure holds a binary integer value, together with a sign bit and an integer scaling factor that specifies what portion of the value is a decimal fraction. Because of this, Decimal numbers have a more precise representation in memory than floating-point types (Single and Double). "


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Tuesday, November 28, 2017 2:39 PM