Quickest way to grab frame / synchronize multiple cameras

Question

I'm working on some computer vision for a school project.  For my application, I need to grab a single frame from eight USB cameras simultaneously, on command.  

I started off using OpenCV, and with the 3 cameras I currently have, I was able to grab a frame from each within 0.1s.  The problem with OpenCV is that it is not really possible to control the camera settings, and so I've been exploring DirectShow.

With DirectShow, I have finally figured out how to set up a capture and control all its possible settings, but I am not sure how to quickly grab frames.  Apparently in OpenCV, you could essentially turn on a camera and have it continuously sending frames, but not actually grab any of these frames until you sent the command.  I know how to capture (grab every frame) in DirectShow and I've read about commands to turn on and then snap a picture, but I don't know how to turn a camera on, leave it on, and grab frames on command. 

I've also tried simply altering the camera settings with DirectShow and then use OpenCV to grab frames, but that doesn't appear to work.  I would greatly appreciate any help.

Answer 1

Kevin,

In Graph Edit program build a simple graph like the following:
   [Camera Source] --> [Sample Grabber] --> [Color Space Converter] --> [Video Renderer]

If this graph works, then you will have to write the code to connect each piece together in your program then start the graph. I'll leave this up to you.

...The sample code below ASSUMES that you will be capturing 24 bit bitmap images...

Concerning the Picture Capture (Sample Grabber):

   1. Use something like the following in the header file of your program:

         IBaseFilter *m_pGrabber;
         ISampleGrabber *m_pGrabberSettings;


   2. include reference to qedit.h which uses these values:

         // Sample Grabber CLSID: {C1F400A0-3F08-11D3-9F0B-006008039E37}
         CLSID_SampleGrabber

         // Sample Grabber IID: {6B652FFF-11FE-4FCE-92AD-0266B5D7C78F}
         IID_ISampleGrabber


   3. After initializing m_pGrabber when adding CLSID_SampleGrabber item to your graph then
       use something like the following to setup the Sample Grabber:

         m_pGrabber->QueryInterface(IID_ISampleGrabber, (void**)&m_pGrabberSettings);
         AM_MEDIA_TYPE mt;
         ZeroMemory(&mt, sizeof(AM_MEDIA_TYPE));
         mt.majortype = MEDIATYPE_Video;
         mt.subtype = MEDIASUBTYPE_RGB24;
         hr = m_pGrabberSettings->SetMediaType(&mt);
         if (FAILED(hr))
         {
            return hr;
         }
         hr = m_pGrabberSettings->SetOneShot(FALSE);
         hr = m_pGrabberSettings->SetBufferSamples(TRUE);


   4. Next connect all pieces of the Graph together then start the graph.


   5. include a reference to winerror.h


   6. Add a button on your Form Dialog to cause a picture to be captured, then call a function
       similar to the following from the button capture function to save your picture:

         // Pass Unique Filename as well as the Width and Height of the Video Window
         HRESULT Snapshot(CString wcsFileName,int lWidth,int lHeight)
         {
            CString SnapshotFileName; // For Example: "C:\\MyDirectory\\Camera-01-YYYYMMDD-HHMMSS.BMP"

            // Use the Path and Filename that was passed to this routine!
            SnapshotFileName = wcsFileName;

            // *** NOTE *** Make sure that OurFileName is a valid pointer and m_pGrabber was properly initialized.

            HRESULT hr = S_OK;

            long pBufferSize = 0;
            unsigned char* pBuffer = 0;
            long Size = 0;
            hr = m_pGrabberSettings->GetCurrentBuffer(&Size, NULL);

            if (FAILED(hr))
            {
               // hr = E_FAIL;
               goto Cleanup;
            }
            else if (Size != pBufferSize)
            {
               pBufferSize = Size;

               if (pBuffer != 0)
               {
                  delete[] pBuffer;
               }

               pBuffer = new unsigned char[pBufferSize];
            }

            hr = m_pGrabberSettings->GetCurrentBuffer(&pBufferSize, (long*)pBuffer);

            // write out our BMP file using the Filename that we passed to this routine!
            HANDLE hf = CreateFile(OurFileName, GENERIC_WRITE, FILE_SHARE_READ, NULL,
                                   CREATE_ALWAYS, NULL, NULL );

            if( hf == INVALID_HANDLE_VALUE )
            {
               hr = E_FAIL;
               goto Cleanup;
            }

            // write out the file header
            BITMAPFILEHEADER bfh;
            memset( &bfh, 0, sizeof( bfh ) );
            bfh.bfType = 'MB';
            bfh.bfSize = sizeof( bfh ) + pBufferSize + sizeof( BITMAPINFOHEADER );
            bfh.bfOffBits = sizeof( BITMAPINFOHEADER ) + sizeof( BITMAPFILEHEADER );

            DWORD dwWritten = 0;
            WriteFile( hf, &bfh, sizeof( bfh ), &dwWritten, NULL );

            // and the bitmap information header format
            BITMAPINFOHEADER bih;
            memset( &bih, 0, sizeof( bih ) );
            bih.biSize = sizeof( bih );
            bih.biWidth = lWidth;
            bih.biHeight = lHeight;
            bih.biPlanes = 1;
            bih.biBitCount = 24;

            dwWritten = 0;
            WriteFile( hf, &bih, sizeof( bih ), &dwWritten, NULL );

            // and the bits themselves
            dwWritten = 0;
            WriteFile( hf, pBuffer, pBufferSize, &dwWritten, NULL );
            CloseHandle( hf );

         Cleanup:
            return hr;
         }

Hope this helps!

Once you get this to work for the first video graph then you do the same for each camera that you will use.

You might prefer to do a Timer to snap a picture say every 15 or 30 minutes.

BillNew

---

DirectShow Filter Graph Spy: http://alax.info/blog/777 Extremely helpful for finding deep details for DirectShow Graphs.

Answer 2

Kevin H M wrote:

>

>I'm working on some computer vision for a school project.  For my

>application, I need to grab a single frame from eight USB cameras

>simultaneously, on command.

 

That may not be possible.  To do that, you will have to have all eight

cameras streaming.  A USB camera reserves part of the USB bandwidth when it

starts.  There is usually not enough for 8 cameras to get their

reservations.

 

>With DirectShow, I have finally figured out how to set up a capture

>and control all its possible settings, but I am not sure how to quickly

>grab frames.  ... I know how to capture (grab every frame) in DirectShow

>and I've read about commands to turn on and then snap a picture, but I

>don't know how to turn a camera on, leave it on, and grab frames on

>command. 

 

Build your graphs with the "sample grabber", CLSID_SampleGrabber.  The

sample grabber will call a call back every time a frame arrives.  When you

want to capture, set flag that you check in the callbacks.  If the flag is

set, copy the next frame to a safe place.  Eventually, you'll get a frame

on all 8 cameras.

 

The code you posted shows that you have a lot of learning to do.  You wrote

a single-threaded console application, then used a MessageBox to prevent

the app from ending so that the graph's threads could run in the

background.  You need to write a GUI application with a message loop.  Then

the graphs can run while the app waits for messages, until you actually

close it.

--

Tim Roberts, [e-mail]

Providenza & Boekelheide, Inc.

 

---

Tim Roberts, VC++ MVP Providenza & Boekelheide, Inc.

Answer 3

Tim and Bill, thanks very much to both of you for your replies.  I should be able to dive in this stuff tonight and work heavily on it this week.

Tim,

As far as the bandwidth goes, I was hoping to alleviate that problem with PCI USB controllers.  I have some PCI cards that each have 4 USB 2.0 compliant ports, in addition to the ports I have on my motherboard.  I have four cameras at the moment and I can get them all to stream video at the same time, but not at their full 30 fps (at 720p).  I can only get 3 of them to stream at 30 fps and the 4th to stream at about 15 fps.  I don't think the PCI cards are the bottleneck, as even if I only use two cameras per PCI card, one camera is limited to 15 fps.  So must the bottleneck be in my processor or memory?  This is all just capturing, no rendering, by the way.  

As far as learning goes, yeah, I definitely am lacking in my knowledge.  I forgot to mention that my background is not in programming, so DirectShow is quite a hurdle for me.  

Thanks again,

Kevin

Answer 4

Kevin H M wrote:

>

>As far as the bandwidth goes, I was hoping to alleviate that problem

>with PCI USB controllers.  I have some PCI cards that each have 4 USB

>2.0 compliant ports, in addition to the ports I have on my motherboard.

>I have four cameras at the moment and I can get them all to stream

>video at the same time, but not at their full 30 fps (at 720p).  I can

>only get 3 of them to stream at 30 fps and the 4th to stream at about

>15 fps.

 

Are all four cameras identical?

 

Are you really talking about PCI, or do you mean PCI Express?  It only

takes two USB buses running at full speed to fill a PCI bus.  USB 2.0 is

speced at 60 MB/s, although the most you can really achieve is about 50

MB/s.  A single PCI bus has a theoretical limit of 133 MB/s, with a

practical limit closer to 100.

 

>As far as learning goes, yeah, I definitely am lacking in my knowledge.

>I forgot to mention that my background is not in programming, so

>DirectShow is quite a hurdle for me.

 

Yes, you picked a heck of a place to start.  I love DirectShow, but there's

a fair learning curve to climb.

--

Tim Roberts, [e-mail]

Providenza & Boekelheide, Inc.

 

---

Tim Roberts, VC++ MVP Providenza & Boekelheide, Inc.

Answer 5

The four cameras I tested on were not identical, but all were capable of, and set to, 30 fps (at varying resolutions).  I now have 4 cameras that are identical (Microsoft HD-5000) and will have 4 more soon, but I haven't tested them together yet.

Yep, I am talking about conventional PCI, but you are right about the bus speeds.  For some reason I thought I had the 64 bit, 66 MHz PCI, but apparently I only have the 32 bit, 33 MHz version.  However, I did try using just 2 cameras on both of the PCI-USB controllers I have, and still had one camera only getting 15 fps.  I'll do some more testing.

I did finally get the sample grabber working.  That was an enormous relief to me...I briefly considered cutting my losses and just going back to LabVIEW (which worked, but slowly).  I would not have been able to do it without help.  Thanks again.

Answer 6

Kevin H M wrote:

>

>However, I did try using just 2 cameras on both of the PCI-USB controllers

>I have, and still had one camera only getting 15 fps.  I'll do some more

>testing.

 

It could be a flawed design.  USB cameras use "isochronous pipes".  An

isochronous pipe reserves a fraction of the USB bandwidth for its own

exclusive use.  Even if it doesn't use all of that all the time, the

bandwidth is still reserved.  Such a device usually has a number of

"alternate settings" for different bandwidths.

 

An intelligent driver will look at the resolution and frame rate, figure

out the bandwidth it needs, and select the lowest alternate setting that

meets the requirement.  A stupid driver will grab the largest alternate

setting that it can, and then limit the frame rate based on what it got.

 

You could be seeing the latter: the first driver grabs the largest

reservation, and that fits 30fps.  The second driver find that there isn't

enough room, so it scales back to a lower bandwidth setting, and adjusts

its frame rate to compensate.

--

Tim Roberts, [e-mail]

Providenza & Boekelheide, Inc.

 

---

Tim Roberts, VC++ MVP Providenza & Boekelheide, Inc.

Answer 7

Success!

I have all 8 cameras hooked up (2 on my motherboard's USB ports and 2 on three different PCI USB controllers) and can grab a frame from each of them in what appears to be 1/30th of a second.  I'm not sure why I had trouble doing that before, but I suspect it was LabVIEW related.

I am quite pleased (and thankful) about where I am right now and am glad that I decided to go with DirectShow/OpenCV rather than LabVIEW.  I think the level of synchronization I have right now will provide very good results.  

I don't want to be too greedy, but I wonder if it can be improved even further.  I saw something about setting the Graph Clock and wondered if that could be useful for my application...specifically, whether or not it can be used to force all the cameras to send their frames at the same time.  If anyone has any ideas, please let me know. 

Answer 8

Kevin H M wrote:

>

>I don't want to be too greedy, but I wonder if it can be improved even

>further.  I saw something about setting the Graph Clock and wondered

>if that could be useful for my application...specifically, whether or

>not it can be used to force all the cameras to send their frames at

>the same time.

 

No, as a few minutes thought should reveal.  The graph clock covers the

timestamps of buffers in the graph, and that's it.  Each camera has its own

pixel clock, and frames are generated whenever that pixel clock comes

around to the top of the sensor again.

 

And, in the end, it doesn't matter.  Web cam pictures are not taken

instantaneously.  The surface of the sensor is exposed over a period of

many milliseconds or tens of milliseconds, and it takes some milliseconds

to dump the contents of the sensor once you start reading it.

 

So, aligning the various cameras is just not that important.

--

Tim Roberts, [e-mail]

Providenza & Boekelheide, Inc.

 

---

Tim Roberts, VC++ MVP Providenza & Boekelheide, Inc.

Answer 9

And, in the end, it doesn't matter.  Web cam pictures are not taken

instantaneously.  The surface of the sensor is exposed over a period of

many milliseconds or tens of milliseconds, and it takes some milliseconds

to dump the contents of the sensor once you start reading it.

If the exposure time is the same (as well as resolution, color mode, etc) and the cameras are all the same model, shouldn't it be possible to synchronize?  If all the cameras were turned on at exactly the same time, with the exact same settings, I would think their frames would be dumped at the exact same time, is that not the case?  I was hoping that I could somehow use the graph clock to turn on all the cameras at the same time, but I guess that's not possible (I had a hard time understanding the functionality of the graph clock).  Do you think there is any possible way to do this, or would my only option basically be to buy raw image sensors and hook them up to an FPGA or something?

Answer 10

Kevin H M wrote:

>

>

>If the exposure time is the same (as well as resolution, color mode,

>etc) and the cameras are all the same model, shouldn't it be possible

>to synchronize?

 

No.  Each board has its own crystal or PLL generating its master clock.

Those crystals are not infinitely accurate, and they drift over time and

over temperature.  Plus, the clocks did not all start at the same time.

 

>If all the cameras were turned on at exactly the same time, with the

>exact same settings, I would think their frames would be dumped at

>the exact same time, is that not the case?  I was hoping that I could

>somehow use the graph clock to turn on all the cameras at the same

>time, but I guess that's not possible...

 

It's not physically possible.  You can't send commands to multiple cameras

at exactly the same time.

 

>(I had a hard time understanding the functionality of the graph clock)

 

The graph clock stamps the time that the frames arrive in the processor,

which is in turn used to decide when they should be presented in the

renderer.  The graph clock has nothing to do with hardware.

 

>Do you think

>there is any possible way to do this, or would my only option

>basically be to buy raw image sensors and hook them up to an FPGA or

>something?

 

Even then, I think you're chasing rainbows.  What would be the point?  The

top lines of an image are often 10 milliseconds earlier than the bottom

scanlines.  What does it matter if the images from two sensors are a few

milliseconds apart?

--

Tim Roberts, [e-mail]

Providenza & Boekelheide, Inc.

 

---

Tim Roberts, VC++ MVP Providenza & Boekelheide, Inc.

Answer 11

The application I'm working on is similar to computer stereo vision.  Pictures will be taken on a moving platform, and even just a few milliseconds difference in capture time would not be acceptable on an actual product.  But that's ok for now.  It'll work while immobile and will likely even produce good results while moving fairly straight/slowly, so the proof of concept will be there.

Tim, you really know your stuff.  Thanks again for your help and patience. 

Answer 12

Kevin H M wrote:

>

>The application I'm working on is similar to computer stereo vision.

>Pictures will be taken on a moving platform, and even just a few

>milliseconds difference in capture time would not be acceptable on

>an actual product.

 

Ah, I can understand your requirement now.  Honestly, I think you will end

up with custom hardware for this.  Most sensor chips have some kind of a

reset or hold signal, but it's a hardware signal.  You'd want have to have

some custom circuit that triggers them all at once.  That could be done.

--

Tim Roberts, [e-mail]

Providenza & Boekelheide, Inc.

 

---

Tim Roberts, VC++ MVP Providenza & Boekelheide, Inc.