locked
Image Recognition between 2 images vb.net Lockbits RRS feed

  • Question

  • I was in the process of using Pixel to do my search between a large image and a small image. I got that one to successfully work but am having trouble getting the lockbits version to do what I want it to do.

    Fist I want to loop through the large image finding the first pixel of the smaller image inside it. Then once finding the first pixel to do a math equation to tell me what percentage of accuracy it got off it. In doing so if it meets the criteria to move my mouse to that location on the screen at the first pixel (top most left) of that small image inside the large image. Which works with a great rate of success and is extremely fast compared to GetPixel format.

    Problem Cannot get location on the large image of where the beginning Pixel of the small image is and move the mouse to that location. Below is the code:

            Try
                Dim ifnd As Boolean = False
                Dim PreviousX As Integer
                Dim PreviousY As Integer
                PreviousX = MousePosition.X
                PreviousY = MousePosition.Y
                Dim MatchCount As Integer = 0
    
                Dim oX As Integer = 0
                Dim oi As Integer = 0
                Dim iX As Integer = 0
                Dim iY As Integer = 0
                Dim bmp_original As Bitmap
                Dim ImG As Bitmap
                ImG = PictureBox2.BackgroundImage
                bmp_original = ImG
                Dim bmp_large As Bitmap
                Dim SmG As Image
                SmG = PictureBox1.BackgroundImage
                bmp_large = SmG
                Dim bmg As Bitmap
                'large image
                ImG = BBt
                'small image
                bmg = Abt
                Thread.Sleep(1000)
                'large image
                If BBt Is Nothing Then
                    Dim xbit As Bitmap = New Bitmap(Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height)
    
                    Dim g As Graphics = Graphics.FromImage(xbit)
                    BBt = xbit
                    g.CopyFromScreen(0, 0, 0, 0, Screen.PrimaryScreen.Bounds.Size)
                    g.Dispose()
                    Thread.Sleep(2000)
                End If
    
                'small image
                PictureBox5.BackgroundImage = Abt
                'large image
                PictureBox6.BackgroundImage = BBt
                'For value As Integer = 0 To 5
                'For value As Integer = 10 To 0 Step -2
                Thread.Sleep(1000)
                'original image
                Dim oRect As New Rectangle(0, 0, bmg.Width, bmg.Height)
                Dim oBmpData As BitmapData = bmg.LockBits(oRect, ImageLockMode.ReadWrite, System.Drawing.Imaging.PixelFormat.Format24bppRgb)
                Dim oPtr As IntPtr = oBmpData.Scan0
                Dim oPixels(99) As Integer
                Dim oMaxPix As Integer = bmg.Width + bmg.Height
                Marshal.Copy(oPtr, oPixels, 0, oMaxPix)
                Dim smWidth As Integer
                smWidth = bmg.Width - 1
                'small image
                PictureBox3.BackgroundImage = bmg
                'large image
                Dim lRect As Rectangle = New Rectangle(0, 0, bmp_large.Width, bmp_large.Height)
                Dim lBmpData As BitmapData = ImG.LockBits(lRect, ImageLockMode.ReadWrite, System.Drawing.Imaging.PixelFormat.Format24bppRgb)
                Dim lPtr As IntPtr = lBmpData.Scan0
                Dim PixelCount As Integer = ImG.Width * ImG.Height
                Dim lPixels(PixelCount - 1) As Integer
                Marshal.Copy(lPtr, lPixels, 0, PixelCount)
                'large image
                PictureBox4.BackgroundImage = ImG
    
                Dim MathScore As Integer
                Dim MaxScore As Integer = bmg.Height
    
                'beginning of Marshal Loop
                For i As Integer = 0 To lPixels.GetUpperBound(0)
    
                    If oPixels(0) = lPixels(i) Then
                        'we have a match for top left pixel - so compare the other pixels
                        Dim PixelsToLeft As Integer = (i Mod ImG.Width) - 1 'pixels to left of 10by10 section of large image
                        Dim PixelsToRight As Integer = ImG.Width - PixelsToLeft - smWidth 'pixels to right of 10by10 section of large image
                        Dim d As Integer = PixelsToLeft + PixelsToRight 'array distance between right edge of 10by10 section and left edge of next row
    
                        For y As Integer = 0 To 9
                            For x As Integer = 0 To 9
                                Dim oIndex As Integer = (y * 10) + x
                                Dim lIndex As Integer = (i + (y * (d + smWidth))) + x
                                If oPixels(oIndex) = lPixels(lIndex) Then
                                    MathScore = MathScore + 1
                                    xx = oPixels(0) + 2
                                    yy = lPixels(i) + 3
                                    SetCursorPos(xx, yy)
                                End If
    
                            Next
    
    
    
                        Next
                        If MathScore >= Val(MaxScore / 2.5) Then
                            SetCursorPos(xx, yy)
                            Dim percent As String
                            Dim myDec As Decimal
                            'inttemp = (intData2 * 100) / intData1
                            myDec = Val((MathScore * 100) / MaxScore)
                            myDec = FormatNumber(myDec, 0)
                            percent = myDec & "%"
                            Label16.Text = "Match Score: " & percent
                            Label17.Text = "Math Score: " & MathScore & " out of " & MaxScore
                            Me.ToolStripStatusLabel2.Text = "Completed"
                            Me.Button4.Enabled = True
                            GoTo Foundit
                        End If
                    End If
                Next
                PictureBox1.Image = (Abt)
                PictureBox2.Image = (BBt)
                ImG.UnlockBits(oBmpData)
                bmg.UnlockBits(lBmpData)
    Foundit:
                mouse_event(MOUSEEVENTF_LEFTDOWN, 0, 0, 0, 0)
                mouse_event(MOUSEEVENTF_LEFTUP, 0, 0, 0, 0)
            Catch
            End Try

    Now if I could figure out how to get the mouse to move on the screen then I would have figured it out. Unfortunately I have been working on this for several days without any success. If you could help I would greatly appreciate it. Thank you in advance.

    Monday, August 27, 2012 6:03 PM

Answers

  • The Try/Catch issue was incidental to your problem. It prevented you from seeing that the simplified example would not work.  It is not related to your original problem.  The actual problem I was addressing was this one:

    "Problem Cannot get location on the large image of where the beginning Pixel of the small image is and move the mouse to that location. Below is the code:"

    The procedure I provided will solve that problem for you, although it is presented at a fairly high level and there is still a certain amount of work to be done.  I cannot see how the comment you have provided assists with implementing that solution.

    • Marked as answer by SChurchel Wednesday, August 29, 2012 12:52 AM
    Wednesday, August 29, 2012 12:37 AM
  • Yes Acamar,

    Removing the Try, Catch and End Try will tell me error on the lockbits but that was not what I was trying to figure out here though.  I was trying to simplify the whole project and some how it all stopped working on the example.  So I went ahead and scrapped that one and went back to the version that was working. When doing so I did give you a point for the catch and try comments.

    • Marked as answer by SChurchel Tuesday, August 28, 2012 9:49 PM
    Tuesday, August 28, 2012 9:49 PM

All replies

  • I don't know if this is what you want but setting "Cursor.Position = New Point(X, Y)" will move the cursor around the screen where X = integer and Y = integer.


    You've taught me everything I know but not everything you know.

    • Proposed as answer by Paul Ishak Monday, August 27, 2012 7:08 PM
    Monday, August 27, 2012 7:04 PM
  • Monkeyboy,

    I tried to add that to the above text but it comes up with error

    Error    1    Form1.Point' has no constructors.   

    I am not sure how to fix that Point error

    Monday, August 27, 2012 7:39 PM
  • Monkeyboy,

    I tried to add that to the above text but it comes up with error

    Error    1    Form1.Point' has no constructors.   

    I am not sure how to fix that Point error

    You are not accessing "Form1.Cursor".... You are accessing "Cursor.Position". Monkeyboys method works just fine.

    Option Strict On
    Public Class Form1
        Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
            Dim Y As Integer
            For X As Integer = 0 To My.Computer.Screen.Bounds.Width - 1
                Cursor.Position = New Point(X, Y)
                Wait(3)
                Y = Y + 1
            Next
        End Sub
        Sub Wait(ByVal Milliseconds As Integer)
            Dim Sw As Stopwatch = Stopwatch.StartNew
            Do : Loop Until Sw.ElapsedMilliseconds >= Milliseconds
        End Sub
    End Class


    If you want something you've never had, you need to do something you've never done.



    • Edited by Paul Ishak Monday, August 27, 2012 7:56 PM
    Monday, August 27, 2012 7:49 PM
  • Paul,

    I put in the code that he suggested but I get the error above.  Here is just a portion of it to see that I did add it.  But unfortunately I am getting an error in the error window.

    All I want to do is be able to move the cursor to the position on the large image where the small image is and click there to focus on that window.

    If oPixels(oIndex) = lPixels(lIndex) Then
                                    MathScore = MathScore + 1
                                    xx = oPixels(0) + 2
                                    yy = lPixels(i) + 3
                                    Cursor.Position = New Point(x, y)
                                End If

    Monday, August 27, 2012 7:58 PM
  • I don't know. In the code below, 1366 x 768 monitor, it bounces the cursor around to all corners and the center of the screen with no problem. But your code is pretty advanced and I can't follow it too well.

    Public Class Form1
    
        Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
            Timer1.Interval = 5000
            Timer2.Interval = 5000
            Timer3.Interval = 5000
            Timer4.Interval = 5000
            Timer5.Interval = 5000
            Timer1.Start()
            
        End Sub
    
        Private Sub Form1_MouseMove(sender As Object, e As System.Windows.Forms.MouseEventArgs) Handles Me.MouseMove
            Label1.Text = Cursor.Position.ToString
        End Sub
    
        Private Sub Timer1_Tick(sender As Object, e As EventArgs) Handles Timer1.Tick
            Cursor.Position = New Point(0, 0)
            Timer2.Start()
            Timer1.Stop()
        End Sub
    
        Private Sub Timer2_Tick(sender As Object, e As EventArgs) Handles Timer2.Tick
            Cursor.Position = New Point(0, 768)
            Timer3.Start()
            Timer2.Stop()
        End Sub
    
        Private Sub Timer3_Tick(sender As Object, e As EventArgs) Handles Timer3.Tick
            Cursor.Position = New Point(1366, 0)
            Timer4.Start()
            Timer3.Stop()
        End Sub
    
        Private Sub Timer4_Tick(sender As Object, e As EventArgs) Handles Timer4.Tick
            Cursor.Position = New Point(1366, 768)
            Timer5.Start()
            Timer4.Stop()
        End Sub
    
        Private Sub Timer5_Tick(sender As Object, e As EventArgs) Handles Timer5.Tick
            Cursor.Position = New Point(683, 384)
            Timer1.Start()
            Timer5.Stop()
        End Sub
    End Class
    


    You've taught me everything I know but not everything you know.

    Monday, August 27, 2012 7:59 PM
  • Cursor.Position = New Point(x, y) End If

    Should this be New Point(xx,yy)?

    Do me a favor and test this following example(in a seperate test project). This will help you confirm that monkeyboys answer to your original question(how to move the mouse on the screen) is valid. If you are having a new issue, you should create a new question for that issue in a new thread.

    Option Strict On
    Public Class Form1
        Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
            Dim X As Integer = -1
            Dim Y As Integer = 0
            Dim CountPixels As Integer = 0
            Do
                CountPixels = CountPixels + 1
                X += 1
                If X Mod (My.Computer.Screen.Bounds.Width - 1) = 0 Then
                    X = 0
                    Y = Y + 1
                End If
                Cursor.Position = New Point(X, Y)
            Loop Until Y >= My.Computer.Screen.Bounds.Height - 1
            MsgBox(CountPixels.ToString)
        End Sub
    End Class
    


    If you want something you've never had, you need to do something you've never done.

    Monday, August 27, 2012 8:07 PM
  • Monkey,

    The whole point is to find the point on the large image which is in the first "for loop" which starts in the top x=0 to y=0

            Try
                Dim ifnd As Boolean = False
                Dim PreviousX As Integer
                Dim PreviousY As Integer
                PreviousX = MousePosition.X
                PreviousY = MousePosition.Y
                Dim MatchCount As Integer = 0

                Dim oX As Integer = 0
                Dim oi As Integer = 0
                Dim iX As Integer = 0
                Dim iY As Integer = 0
                Dim bmp_original As Bitmap
                Dim ImG As Bitmap
                ImG = PictureBox2.BackgroundImage
                bmp_original = ImG
                Dim bmp_large As Bitmap
                Dim SmG As Image
                SmG = PictureBox1.BackgroundImage
                bmp_large = SmG
                Dim bmg As Bitmap
                'large image
                ImG = BBt
                'small image
                bmg = Abt
                Thread.Sleep(1000)
                'large image
                If BBt Is Nothing Then
                    Dim xbit As Bitmap = New Bitmap(Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height)

                    Dim g As Graphics = Graphics.FromImage(xbit)
                    BBt = xbit
                    g.CopyFromScreen(0, 0, 0, 0, Screen.PrimaryScreen.Bounds.Size)
                    g.Dispose()
                    Thread.Sleep(2000)
                End If

                'small image
                PictureBox5.BackgroundImage = Abt
                'large image
                PictureBox6.BackgroundImage = BBt
                'For value As Integer = 0 To 5
                'For value As Integer = 10 To 0 Step -2
                Thread.Sleep(1000)
                'original image
                Dim oRect As New Rectangle(0, 0, bmg.Width, bmg.Height)
                Dim oBmpData As BitmapData = bmg.LockBits(oRect, ImageLockMode.ReadWrite, System.Drawing.Imaging.PixelFormat.Format24bppRgb)
                Dim oPtr As IntPtr = oBmpData.Scan0
                Dim oPixels(99) As Integer
                Dim oMaxPix As Integer = bmg.Width + bmg.Height
                Marshal.Copy(oPtr, oPixels, 0, oMaxPix)
                Dim smWidth As Integer
                smWidth = bmg.Width - 1
                'small image
                PictureBox3.BackgroundImage = bmg
                'large image
                Dim lRect As Rectangle = New Rectangle(0, 0, bmp_large.Width, bmp_large.Height)
                Dim lBmpData As BitmapData = ImG.LockBits(lRect, ImageLockMode.ReadWrite, System.Drawing.Imaging.PixelFormat.Format24bppRgb)
                Dim lPtr As IntPtr = lBmpData.Scan0
                Dim PixelCount As Integer = ImG.Width * ImG.Height
                Dim lPixels(PixelCount - 1) As Integer
                Marshal.Copy(lPtr, lPixels, 0, PixelCount)
                'large image
                PictureBox4.BackgroundImage = ImG

                Dim MathScore As Integer
                Dim MaxScore As Integer = bmg.Height

                'beginning of Marshal Loop  (Large Image)
                For i As Integer = 0 To lPixels.GetUpperBound(0)

                    If oPixels(0) = lPixels(i) Then
                        'we have a match for top left pixel - so compare the other pixels (of the Small Image)
                        Dim PixelsToLeft As Integer = (i Mod ImG.Width) - 1 'pixels to left of 10by10 section of large image
                        Dim PixelsToRight As Integer = ImG.Width - PixelsToLeft - smWidth 'pixels to right of 10by10 section of large image
                        Dim d As Integer = PixelsToLeft + PixelsToRight 'array distance between right edge of 10by10 section and left edge of next row

                        For y As Integer = 0 To 9
                            For x As Integer = 0 To 9
                                Dim oIndex As Integer = (y * 10) + x
                                Dim lIndex As Integer = (i + (y * (d + smWidth))) + x
                                If oPixels(oIndex) = lPixels(lIndex) Then
                                    MathScore = MathScore + 1
                                    xx = oPixels(0) + 2
                                    yy = lPixels(i) + 3
                                    Cursor.Position = New Point(x, y) <======= Here is where I want it to get the location on the larger image where the first x and y of the small image shows up.
                                End If

                            Next



                        Next
                        If MathScore >= Val(MaxScore / 2.5) Then

                            Dim percent As String
                            Dim myDec As Decimal
                            'inttemp = (intData2 * 100) / intData1
                            myDec = Val((MathScore * 100) / MaxScore)
                            myDec = FormatNumber(myDec, 0)
                            percent = myDec & "%"
                            Label16.Text = "Match Score: " & percent
                            Label17.Text = "Math Score: " & MathScore & " out of " & MaxScore
                            Me.ToolStripStatusLabel2.Text = "Completed"
                            Me.Button4.Enabled = True
                            GoTo Foundit
                        End If
                    End If
                Next
                PictureBox1.Image = (Abt)
                PictureBox2.Image = (BBt)
                ImG.UnlockBits(oBmpData)
                bmg.UnlockBits(lBmpData)
    Foundit:
                mouse_event(MOUSEEVENTF_LEFTDOWN, 0, 0, 0, 0)
                mouse_event(MOUSEEVENTF_LEFTUP, 0, 0, 0, 0)
            Catch
            End Try

    Monday, August 27, 2012 8:22 PM
  • I don't need cursor position.  I want to get the pixel position of the smaller image that is within the larger image and move the mouse to the point in the larger image where the smaller image begins.

    Kind of like facial recognition where a persons face is on file (smaller image) but then take a large image with the guys face in it and have the mouse move on the screen to that image(large image) where it is on the screen.  I have all the functionality down and can get good results as far as accuracy and speed with lockbits is great.  Only problem I am having is pulling out the location on the large image the start pixel of the smaller image.  Hence wanting to move the mouse to that position. all done with a stored image (and a screen capture of the main screen) and then finding the image inside the screen capture)  can find it it no problem.  but I want to move my mouse to the position of the first pixel in the large image where that small image is located.  That is what I am having trouble with. 

    Monday, August 27, 2012 8:42 PM
  • You're beyond my scope. I'm sorry I can't help you any further.

    You've taught me everything I know but not everything you know.

    Monday, August 27, 2012 8:44 PM
  • The X,Y position in the orignal image will be derived from the index of the current byte in the loop:

    Y = Math.Floor(index / (ImageWidth * BytesPerPixel))

    X = (index / BytesPerPixel) - Y


    Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

    Monday, August 27, 2012 8:48 PM
  • The X,Y position in the orignal image will be derived from the index of the current byte in the loop:

    Y = Math.Floor(index / (ImageWidth * BytesPerPixel))

    X = (index / BytesPerPixel) - Y


    Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

    Reed, 

    I understand your concept but how would I implement it within the code that exists already?  in the For next loops.  I am not very good with lockbits and marshal this project is actually my first attempt at it.  I can do it with GetPixel but unfortunately am having trouble with the lockbit. For next loop since it is a 2 dimensional array.

    For i As Integer = 0 To lPixels.GetUpperBound(0)
    
                    If oPixels(0) = lPixels(i) Then
                        'we have a match for top left pixel - so compare the other pixels
                        Dim PixelsToLeft As Integer = (i Mod ImG.Width) - 1 'pixels to left of 10by10 section of large image
                        Dim PixelsToRight As Integer = ImG.Width - PixelsToLeft - smWidth 'pixels to right of 10by10 section of large image
                        Dim d As Integer = PixelsToLeft + PixelsToRight 'array distance between right edge of 10by10 section and left edge of next row
    
                        For y As Integer = 0 To 9
                            For x As Integer = 0 To 9
                                Dim oIndex As Integer = (y * 10) + x
                                Dim lIndex As Integer = (i + (y * (d + smWidth))) + x
                                If oPixels(oIndex) = lPixels(lIndex) Then
                                    MathScore = MathScore + 1
                                    xx = oPixels(0) + 2
                                    yy = lPixels(i) + 3
                                    Cursor.Position = New Point(x, y)
                                End If
    
                            Next
    
    
    
                        Next
                        If MathScore >= Val(MaxScore / 2.5) Then
    
                            Dim percent As String
                            Dim myDec As Decimal
                            'inttemp = (intData2 * 100) / intData1
                            myDec = Val((MathScore * 100) / MaxScore)
                            myDec = FormatNumber(myDec, 0)
                            percent = myDec & "%"
                            Label16.Text = "Match Score: " & percent
                            Label17.Text = "Math Score: " & MathScore & " out of " & MaxScore
                            Me.ToolStripStatusLabel2.Text = "Completed"
                            Me.Button4.Enabled = True
                            GoTo Foundit
                        End If
                    End If
                Next
    Thank you ahead of time.

    Monday, August 27, 2012 8:54 PM
  • Well, I find your code a bit hard to follow, but it would appear from this snippit that the variable "i" represents your index within the source image byte array.  So the value of "i" at the time you want to determine the location would be your source of "index" for the calculation.

    Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

    Monday, August 27, 2012 8:58 PM
  • Keep in mind that the byte array you copied from unmanaged memory (marshalled) is a list of all bytes representing the color data for each pixel.  So if you have 24bbp image, you have three bytes for each pixel.  So starting at lPixels(0) you have the blue byte of pixel 1, lPixels(1) the green byte of pixel 1, lPixels(2) the red byte of pixel 1, lPixels(3) the blue byte of Pixel 2, etc.

    Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

    Monday, August 27, 2012 9:01 PM
  • Give me a moment and let me see if I can simplify the code above
    Monday, August 27, 2012 9:08 PM
  • Its not that the code is overly complex, it is that many of the variable names are arbitrary so the logic is not easy to follow.

    Try the calculation using "i" for "index" and see if that gives you the X,Y Point that you desire.


    Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

    Monday, August 27, 2012 9:12 PM
  • Reed,

    I went ahead and made a basic one that you can just copy paste the code into a new vb.net form

    need 2 buttons and 4 picture boxes

    Unfortunately now its not even giving any results but figured you might be able to find the error quicker if you had all the code.

    Imports System
    Imports System.Drawing
    Imports System.Drawing.Graphics
    Imports System.Drawing.Imaging
    Imports System.Runtime
    Imports System.Runtime.InteropServices
    Imports System.Drawing.Drawing2D
    Imports System.Windows.Forms
    Imports System.Threading
    Imports System.Drawing.Bitmap
    Public Class Form1
        Public Declare Sub mouse_event Lib "user32" Alias "mouse_event" (ByVal dwFlags As Long, ByVal dx As Long, ByVal dy As Long, ByVal cButtons As Long, ByVal dwExtraInfo As Long)
        Public Declare Auto Function SetCursorPos Lib "User32.dll" (ByVal X As Integer, ByVal Y As Integer) As Long
        Public Const MOUSEEVENTF_LEFTDOWN = &H2 ' left button down
        Public Const MOUSEEVENTF_LEFTUP = &H4 ' left button up
        Public Const MOUSEEVENTF_MIDDLEDOWN = &H20 ' middle button down
        Public Const MOUSEEVENTF_MIDDLEUP = &H40 ' middle button up
        Public Const MOUSEEVENTF_RIGHTDOWN = &H8 ' right button down
        Public Const MOUSEEVENTF_RIGHTUP = &H10 ' right button up
    
        'Public Bitmap Variables for Both Images Original and SceenCaputre(large)
        Dim Original_Bitmap As Bitmap
        Dim Large_Bitmap As Bitmap
    
    
    
        Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
    
    
    
            Try
                Dim Fild As New OpenFileDialog
                Fild.Filter = ""
                Fild.Multiselect = False
    
                If Fild.ShowDialog() = System.Windows.Forms.DialogResult.OK Then
                    'Original Image
                    Dim Img As Bitmap = Image.FromFile(Fild.FileName)
                    PictureBox1.BackgroundImage = (Img)
                    'large image
                    Dim Bmg As Bitmap = New Bitmap(Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height)
                    Dim g As Graphics = Graphics.FromImage(Bmg)
                    g.CopyFromScreen(0, 0, 0, 0, Screen.PrimaryScreen.Bounds.Size)
                    g.Dispose()
                    PictureBox2.BackgroundImage = Bmg
                    'assigning Public Bitmaps
                    Original_Bitmap = Img
                    Large_Bitmap = Bmg
    
    
                End If
            Catch
            End Try
    
        End Sub
    
        Private Sub FindImage_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles FindImage.Click
            Try
                'Small Image
                Dim Img As Bitmap = Original_Bitmap
    
                Dim MatchCount As Integer = 0
                'Large Image 
                Dim Bmg As Bitmap = Large_Bitmap
    
                'to check to make sure both pictures are showing up
                PictureBox3.BackgroundImage = Img
                PictureBox4.BackgroundImage = Bmg
    
                Thread.Sleep(2000)
                'original image is img array
                Dim oRect As New Rectangle(0, 0, Bmg.Width, Bmg.Height)
                Dim oBmpData As BitmapData = Bmg.LockBits(oRect, ImageLockMode.ReadWrite, System.Drawing.Imaging.PixelFormat.Format24bppRgb)
                Dim oPtr As IntPtr = oBmpData.Scan0
                Dim oPixels(99) As Integer
                Dim oMaxPix As Integer = Bmg.Width + Bmg.Height
                Marshal.Copy(oPtr, oPixels, 0, oMaxPix)
                Dim smWidth As Integer
                smWidth = Bmg.Width - 1
                'Large image array
                Dim lRect As Rectangle = New Rectangle(0, 0, Img.Width, Img.Height)
                Dim lBmpData As BitmapData = Img.LockBits(lRect, ImageLockMode.ReadWrite, System.Drawing.Imaging.PixelFormat.Format24bppRgb)
                Dim lPtr As IntPtr = lBmpData.Scan0
                Dim PixelCount As Integer = Img.Width * Img.Height
                Dim lPixels(PixelCount - 1) As Integer
                Marshal.Copy(lPtr, lPixels, 0, PixelCount)
    
                'Math variables
                Dim MathScore As Integer
                Dim MaxScore As Integer = bmg.Height
    
                'beginning of Marshal Loop
                For i As Integer = 0 To lPixels.GetUpperBound(0)
    
                    If oPixels(0) = lPixels(i) Then
                        'we have a match for top left pixel - so compare the other pixels
                        Dim PixelsToLeft As Integer = (i Mod Img.Width) - 1
                        Dim PixelsToRight As Integer = Img.Width - PixelsToLeft - smWidth
                        Dim d As Integer = PixelsToLeft + PixelsToRight
    
                        For y As Integer = 0 To 9
                            For x As Integer = 0 To 9
                                Dim oIndex As Integer = (y * 10) + x
                                Dim lIndex As Integer = (i + (y * (d + smWidth))) + x
                                If oPixels(oIndex) = lPixels(lIndex) Then
                                    MathScore = MathScore + 1
                                    'This is where I cannot find the Match
                                End If
    
                            Next
    
    
    
                        Next
                        If MathScore >= Val(MaxScore / 2.5) Then
    
                            Dim percent As String
                            Dim myDec As Decimal
                            'inttemp = (intData2 * 100) / intData1
                            myDec = Val((MathScore * 100) / MaxScore)
                            myDec = FormatNumber(myDec, 0)
                            percent = myDec & "%"
                            Label1.Text = "Match Score: " & percent
                            Label2.Text = "Math Score: " & MathScore & " out of " & MaxScore
                            Me.ToolStripStatusLabel2.Text = "Completed"
                            Me.Button1.Enabled = True
                            GoTo Foundit
                        End If
                    End If
                Next
    
    
                Img.UnlockBits(oBmpData)
                bmg.UnlockBits(lBmpData)
    Foundit:
                mouse_event(MOUSEEVENTF_LEFTDOWN, 0, 0, 0, 0)
                mouse_event(MOUSEEVENTF_LEFTUP, 0, 0, 0, 0)
    
            Catch
            End Try

    Monday, August 27, 2012 10:27 PM
  • You have three separate steps that you are trying to do at once.  Break the process down into its stages and it should be clear what's needed.

    I am assuming that you are examining bytes, not pixels, because that's the array that lockbits gives you.  If it's an 8-bit image, then they are the same - otherwise bytes per pixel is greater than one, and you comparison is bytes, not pixels.  With that proviso -

    The index value of the array at the 'found' position is i in this line
        For i As Integer = 0 To lPixels.GetUpperBound(0)
    when Foundit is executed.

    You need to convert that array index to a pixel position (x,y) within the image.  You will do that by using the bytes per pixel and pixels per row calculation to work out the row (y) and the column (x).  Divide by bytes per pixel to get the sequential pixel position.  Take that number Mod pixels per row to get the column.  Take the integer divide of that pixel index by pixels per row to get the row.

    Now you have to convert a pixel position (x,y) into a mouse position.  To do that you need to know how the image is displayed in the picture box, and the position of the picture box in the display.  If the image is not resized at all in the picture box then the pixel position is the offset within the picture box.   If it is resized, then you need to map the pixel position in the image to a pixel position within the picture box using the implied scale of the resizing.  There are routines available to do this, but it would be simplest if you display the image in the picture box without any resizing, so no scaling is needed.

    Finally you need to convert the offset in the picture box to a mouse position on the display.  The picturebox PointToScreen method does that for you.  Now you can move the cursor to the calculated screen positiion.

    • Edited by Acamar Tuesday, August 28, 2012 1:05 AM sp
    Tuesday, August 28, 2012 12:59 AM
  • Unfortunately now its not even giving any results but figured you might be able to find the error quicker if you had all the code.

    because of this:

            Catch
            End Try

    Remove your Try/Catch handling and you will be able to see the error.  Currently you have the equivalent of "On Error Resume Next" which makes debugging almost impossible.  You want to see those errors so that you can fix them.

    Tuesday, August 28, 2012 1:03 AM
  • Yes Acamar,

    Removing the Try, Catch and End Try will tell me error on the lockbits but that was not what I was trying to figure out here though.  I was trying to simplify the whole project and some how it all stopped working on the example.  So I went ahead and scrapped that one and went back to the version that was working. When doing so I did give you a point for the catch and try comments.

    • Marked as answer by SChurchel Tuesday, August 28, 2012 9:49 PM
    Tuesday, August 28, 2012 9:49 PM
  • The Try/Catch issue was incidental to your problem. It prevented you from seeing that the simplified example would not work.  It is not related to your original problem.  The actual problem I was addressing was this one:

    "Problem Cannot get location on the large image of where the beginning Pixel of the small image is and move the mouse to that location. Below is the code:"

    The procedure I provided will solve that problem for you, although it is presented at a fairly high level and there is still a certain amount of work to be done.  I cannot see how the comment you have provided assists with implementing that solution.

    • Marked as answer by SChurchel Wednesday, August 29, 2012 12:52 AM
    Wednesday, August 29, 2012 12:37 AM
  • Acamar,

    Correct I tried to simplify it but in the process it stopped working all together, so I scrapped that one and went back to working with the original. although in both incidents the pictureboxes both are x'ed out. After the for loops finish.

    That's the problem I am not sure how to fix is why they keep x'ing out the picture boxes.

    Wednesday, August 29, 2012 12:55 AM