none
weird math error in VB2010 RRS feed

  • Question

  • the following code:

    x1 = 2305 / 4096

    x5 = 4096

    r = 2025

    x1 = r / x5

    results in both instances is that x1 ends up being 1, and not 0 as it should.

    r, x1 and x5 are defined as int32.

    solution anyone?


    developer of embedded apps using VB as complement

    Wednesday, April 18, 2018 9:12 PM

All replies

  • It is rounding up.  2025/4096 is greater than .5 so it becomes 1.

    You will want to use Math.Floor(r/x5)


    Lloyd Sheen


    • Edited by sqlguy Wednesday, April 18, 2018 9:49 PM
    • Proposed as answer by Mr. Monkeyboy Wednesday, April 18, 2018 11:27 PM
    Wednesday, April 18, 2018 9:47 PM
  • Yes, this is the rule. Rounding from a fractional or decimal number to an Integer is either up or down to the closest integer value. If the value is in middle then it's rounded up or down to the closest even integer value.


    Paul ~~~~ Microsoft MVP (Visual Basic)


    Wednesday, April 18, 2018 10:10 PM
  • x1 = 2305/4095 makes x1 = 0.562744140625 so rounded up = 1

    x1 = r / x5 makes x1 = 2025 / 4096 or 0.494384765625 which rounded down = 0.

    I suppose you could use type Double or Decimal as they will contain decimal places then use Math.Floor Math.Ceiling depending on the result and how you would like to round it.

    Integer data type (Visual Basic)

    "Range

    If you try to set a variable of an integral type to a number outside the range for that type, an error occurs. If you try to set it to a fraction, the number is rounded up or down to the nearest integer value. If the number is equally close to two integer values, the value is rounded to the nearest even integer. This behavior minimizes rounding errors that result from consistently rounding a midpoint value in a single direction. The following code shows examples of rounding."

    From code "Note, Visual Basic uses banker’s rounding (toward nearest even number)"

    ' The valid range of an Integer variable is -2147483648 through +2147483647.  
    Dim k As Integer  
    ' The following statement causes an error because the value is too large.  
    k = 2147483648  
    ' The following statement sets k to 6.  
    k = 5.9  
    ' The following statement sets k to 4  
    k = 4.5  
    ' The following statement sets k to 6  
    ' Note, Visual Basic uses banker’s rounding (toward nearest even number)  
    k = 5.5  

    Maybe this Function can assist you.

    Option Strict On
    
    Public Class Form1
    
        Dim x1 As Double = 0
        Dim r As Double = 2025
        Dim x5 As Double = 4096
    
        Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
            Me.Location = New Point(CInt((Screen.PrimaryScreen.WorkingArea.Width / 2) - (Me.Width / 2)), CInt((Screen.PrimaryScreen.WorkingArea.Height / 2) - (Me.Height / 2)))
        End Sub
    
    
        Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
            x1 = 2305 / 4096
            Dim i As Integer = RoundUpOrDown(x1)
            MessageBox.Show(i.ToString)
        End Sub
    
        Private Sub Button2_Click(sender As Object, e As EventArgs) Handles Button2.Click
            x1 = r / x5
            Dim i As Integer = RoundUpOrDown(x1)
            MessageBox.Show(i.ToString)
        End Sub
    
        Private Function RoundUpOrDown(ByVal Input As Double) As Integer
            Dim s As String = Input.ToString
            Dim Find As Integer = 0
            Dim Result As Integer = 0
            If s.Contains(".") Then
                Find = s.IndexOf(".") + 1
                s = s.Substring(Find, 1)
                MessageBox.Show(s)
                Result = CInt(s)
                If Result >= 5 Then
                    Result = CInt(Math.Ceiling(Input))
                Else
                    Result = CInt(Math.Floor(Input))
                End If
            Else
                Result = CInt(Input)
            End If
            Return Result
        End Function
    
    End Class


    La vida loca

    Wednesday, April 18, 2018 11:17 PM
  • the following code:

    x1 = 2305 / 4096

    x5 = 4096

    r = 2025

    x1 = r / x5

    results in both instances is that x1 ends up being 1, and not 0 as it should.

    r, x1 and x5 are defined as int32.

    solution anyone?


    developer of embedded apps using VB as complement

    Hi zecharia12,

    I get x1=0 according to the following code, because 2025/4096=0.49,rounding 0.

    Dim x5 As Int32 = 4096
            Dim r As Int32 = 2025
            Dim x1 As Int32 = r / x5
            Console.WriteLine(x1)

    But x1=2305/4096=0.56, rounding 1

     Dim x1 As Int32 = 2305 / 4096

    Please check Which execution result is x1.

    Best Regards,

    Cherry


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Thursday, April 19, 2018 7:51 AM
    Moderator