locked
Why does Wf(i,1) not = 0.6? RRS feed

  • General discussion

  • OK, I read the values for an array in from a data file. It reads in 0.6 for Wf(6,1).

    I can inspect the value for Wf(6,1), and it says it is 0.6.

    Then, when I step through the array to use the value and check for WF(i,1)=0.6 it doesn't?  The Wf(x,y) array is declared as a Single type, but this doesn't work.  It worked fine in VB6.

    If Wf(i, 1)= 0.6 And chk10D.Checked = True Then  WPi = 1.0 * Fwgt

    I have to rewrite it to be:

    If Math.Round(Wf(i, 1), 1) = 0.6 And chk10D.Checked = True Then  WPi = 1.0 * Fwgt



    Allen Johnson

    Tuesday, December 17, 2013 2:46 PM

All replies

  • Hi

    Put    OPTION STRICT ON    at the top of your code, this will pinpoint and explain the reason for you. And, you could add    OPTION INFER OFF    too, which is a preference of mine.  Then Build the solution and see what is reported.


    Regards Les, Livingston, Scotland


    • Edited by leshay Tuesday, December 17, 2013 2:52 PM
    Tuesday, December 17, 2013 2:50 PM
  • Are all discrete real numbers, like "0.6", assumed to be type DOUBLE then?

    Allen Johnson


    • Edited by AJohnson212 Tuesday, December 17, 2013 2:56 PM
    Tuesday, December 17, 2013 2:55 PM
  • Hello Allen,

    You have created this as a discussion topic and not as a question. 

    But what do you want to discuss. 

    If VB6 did it well and VBNet wrong. 

    That discussion took place already a decenia ago.


    Success
    Cor


    Tuesday, December 17, 2013 5:48 PM
  • Because it's not a question on how to fix it. I figured that out already. It's why it occurs.

    Allen Johnson

    Tuesday, December 17, 2013 5:53 PM
  • It's why it occurs.

    That's a question.

    See here for an answer:
    http://msdn.microsoft.com/en-us/library/c151dt3s.aspx

    Tuesday, December 17, 2013 9:43 PM
  • Aren't we picky.  

    Your answer does not answer the question then either.

    If you want to get to the crux of the "question" then I guess I'm asking "If you type in an exact real number in Code, is it a single or double precision number?"



    • Edited by AJohnson212 Tuesday, December 17, 2013 9:55 PM
    Tuesday, December 17, 2013 9:52 PM
  • The answer is they are double precision numbers. So, a better version would be to use the ! suffix on single precision numbers to make sure it IS single precision.

    If Wf(i, 1) = 0.6! And chk10D.Checked = True Then WPi = 1.0 * Fwgt


    Allen Johnson

    Tuesday, December 17, 2013 10:02 PM
  • If you want to get to the crux of the "question" then I guess I'm asking "If you type in an exact real number in Code, is it a single or double precision number?"

    It's a string - the only thing you can 'type' is a string.

    When you convert it to a numeric variable it will be the Type of that variable - double, single, decimal, integer, whatever.

    However your original post indicated that the data item is being read in from a file, not typed in.  In that case it will be the Type of the variable that the file read statement reads the data into.  If that involves a conversion then additional things might be happening.

    • Edited by Acamar Tuesday, December 17, 2013 10:37 PM sp
    Tuesday, December 17, 2013 10:34 PM
  • You don't get it, do you?

    Allen Johnson

    Tuesday, December 17, 2013 10:38 PM
  • You don't get it, do you?

    Allen Johnson

    I think you don't get it, you can use this forum to make discussions and to ask questions. 

    If you look most is question. However, you started a discussion, so if it is not a discussion change it in your original message to question.


    Success
    Cor

    Tuesday, December 17, 2013 11:44 PM
  • You don't get it, do you?

    What part of the responses don't you get?

    You have already determined that the Type of a literal can be indicated using the type suffixes such as ! for Single or # for Double (although declaring a constant using the As clause is probably a better way to do it).   That exact same consideration must apply to your variables.  That applies whether or not they are initialised from a literal, a constant, or somewhere else.   Unless you have included Option Strict On in your code then the system will read a file variable stored as a Double (for instance) into a Single.   If that happens the single might not be 0.6 at all, even though a cursory look at the variable value indicates it is.  Or, if you are actually reading it into a double but comparing it to a single, then the comparison may fail, as per your original post.  What is actually happening is impossible to say without knowing the source of your original value and the Type of the variable (Wf(,)) that you are using. The point is, there is no exact binary representation of that value, so you need to be very careful about how you handle it.

    Wednesday, December 18, 2013 1:43 AM
  • OK, I read the values for an array in from a data file. It reads in 0.6 for Wf(6,1).

    I can inspect the value for Wf(6,1), and it says it is 0.6.

    Then, when I step through the array to use the value and check for WF(i,1)=0.6 it doesn't?  The Wf(x,y) array is declared as a Single type, but this doesn't work.  It worked fine in VB6.

    If Wf(i, 1)= 0.6 And chk10D.Checked = True Then  WPi = 1.0 * Fwgt

    I have to rewrite it to be:

    If Math.Round(Wf(i, 1), 1) = 0.6 And chk10D.Checked = True Then  WPi = 1.0 * Fwgt



    Allen Johnson

    Acamar is giving you the correct answer. You may not understand it though. Not saying I do. :)

    However you may be missing the changes in the, whats it called, how about precision between the versions? I found when I converted from vb6 to .net, I had to change my vb6 declares for anything read from a file made in vb6 using vb.net. Variables that were as double in vb6 went to as single in vb.net. Integers became shorts. Longs became integers. I forget exactly. I don't know what the exact values are but it is something like in vb.net a single can have 14 decimal places (or maybe more) where in vb6 it was 7. In vb.net all the types can be much longer on the left side of the ".".

    So you need to read up on the differences between single and double etc. from vb6 to vb.net.

    Wednesday, December 18, 2013 2:42 AM
  • The Wf(x,y) array is declared as a Single type 

    What is actually happening is impossible to say without knowing the source of your original value and the Type of the variable (Wf(,)) that you are using.

    Maybe you should read the post first then.

    Allen Johnson

    Thursday, December 19, 2013 2:52 PM
  • In your line

    If Wf(i, 1)= 0.6 And chk10D.Checked = True Then  WPi = 1.0 * Fwgt

    you can hover over the 0.6 and VS will tell you that its type is Double. Now, when you compare it to the Single value Wf(i, 1), it "sees" that the types are different and so it cannot directly compare them. It looks to see if there is some conversion it can do without losing data (a "widening" conversion) to make them of the same type and finds that it can use a widening conversion from Single to Double. Unfortunately, 0.6 is a decimal fraction with no exact representation in binary, so the conversion to a Double is not as you might hope [1].

    There is any easy way to get round this: use the same type on both sides of the comparison. You could either declare Fw() as Double, or write 0.6 as 0.6F to tell the compiler it is a Single value. I'd go with using Double instead of Single everywhere you can: with only seven or eight digits of precision, Singles can soon build up significant error after only a few calculations.

    [1] What Every Computer Scientist Should Know About Floating-Point Arithmetic

    HTH,

    Andrew

    Thursday, December 19, 2013 7:01 PM
  • just to complete Andrew post

    The compiler will look for a widening conversion to make the comparaison between the single and the double

     --- Converting the double to single is not widening ... therefore it will convert the single to a double. This is a widening conversion

    so, W(i,1)  which is 0.6 as single will convert to double.

    this converstion will give as result 0.600000023841858 ... which is not equal to 0.6

    therefore W(i,1) = 0.6 returns false

    ----------------------

    Conclusion: never mix the type of the values when working with floating point values. Floating point conversion mostly never works. (either the conversion is narrowing or the result of the conversion is not exact)

    ----------------------

    If you know the assember language, check the documentation from Intel/AMD about your processor, you will find the exact reason why this is happening





    • Edited by Crazypennie Friday, December 20, 2013 12:36 AM 1234
    Thursday, December 19, 2013 11:59 PM
  • Aha!

    I was expecting it to be more a "narrowing" than a "widening". Your explanation clears it up a lot. 

    I figured (from the beginning!) that it was a precision issue, which is the reason for using the Round function and then the If wf(i,j) = 0.6!... method to restrict the precision, but I didn't know about widening rather than narrowing the precision. 

    Thanks, your reply is what I was looking for, not a lecture on precision.


    Allen Johnson

    Friday, December 20, 2013 3:15 PM