none
depth stream FPS also decreases as image stream FPS decreases RRS feed

  • Question

  • Hello, I'm getting 30 FPS for both depth and color stream but when I added an opencv algorithm (SURF Descriptors/ FLANN matching) which is for recognizing object, color stream gets very slow fps, screen get freezes. I know it is because of the SURF algo which slows the performance of the color stream.

    but the question is why is it also affects the depth stream? It also slows down its fps. Any idea?

    THanks

    Monday, July 22, 2013 1:51 PM

Answers

  • Have a look at the Kinect Bridge with OpenCV sample from the Toolkit. Keep in mind, VC++ with /clr has specific rules for being managed vs unmanaged. Obviously, if you use any of the .Net objects those will be managed regardless. To avoid confusion, you should not be using /clr.

    http://msdn.microsoft.com/en-US/library/0adb9zxe(v=vs.100).aspx

    When calling GetNextFrame, you need to minimize the amount of time before releasing the frame. Any delays will cause the runtime to drop frames in the processing. You have less than 33ms (for 30fps) to do any work. If the GC decides to run because your function is managed, this is going to cause delays. 

    HRESULT hr = m_pNuiSensor->NuiImageStreamGetNextFrame(
    	m_hColorStreamHandle,
    	waitMillis,
    	&imageFrame);
    if (FAILED(hr))
    {
    	return hr;
    }
    // keep this block of code minimal to prevent 
    // blocking the Kinect runtime from losing frames
    // Release image stream frame
    hr = m_pNuiSensor->NuiImageStreamReleaseFrame(m_hColorStreamHandle, &imageFrame);

    Therefore, move the code for converting to openCV outside of that block of code. As with all the samples, the minimal amount of code you need is:

    NUI_LOCKED_RECT lockedRect; pTexture->LockRect(0, &lockedRect, NULL, 0); // Check if image is valid if (lockedRect.Pitch != 0) { ... memcpy_s(m_pColorBuffer, size, pBuffer, size); ... } // Unlock texture pTexture

    ->UnlockRect(0);

    As I suggested above, there are other advanced topics, beyond this forum, that you can employ to improve performance and keep your UI, Kinect Data, and Processing threads separate. These include, but not limited to:

    Processes and Threads
    http://msdn.microsoft.com/en-us/library/windows/desktop/ms684841(v=vs.85).aspx

    Low Fragmentation Heap
    http://msdn.microsoft.com/en-us/library/windows/desktop/aa366750(v=vs.85).aspx

    Multimedia Class Scheduler
    http://msdn.microsoft.com/en-us/library/windows/desktop/ms684247(v=vs.85).aspx


    Carmine Sirignano - MSFT


    Tuesday, July 23, 2013 6:44 PM

All replies

  • Are you using the polling method or event based? Is this a .Net managed application or unmanaged c++?

    You should not be doing any intensive data processing in you main loop that is getting the data from the sensor. Once the data is copied into your application level variables, you can spin up thread and do processing on these.


    Carmine Sirignano - MSFT

    Monday, July 22, 2013 10:46 PM
  • I am using polling method and its a VC++ CLR project(unmanaged c++).

    so here is a glimpse of my code.

    int drawColor(HANDLE h, Mat color)
    {
    	const NUI_IMAGE_FRAME * pImageFrame = NULL;
    
    	HRESULT hr = NuiImageStreamGetNextFrame( h, 0, &pImageFrame );
    
    	if( FAILED( hr ) )
    	{
    		cout<<"Get Image Frame Failed"<<endl;
    		return -1;
    	}
    
    	INuiFrameTexture * pTexture = pImageFrame->pFrameTexture;
    	NUI_LOCKED_RECT LockedRect;
    	pTexture->LockRect( 0, &LockedRect, NULL, 0 );
    
    
    
    if( LockedRect.Pitch != 0 )
    	{
    		BYTE * pBuffer = (BYTE*) LockedRect.pBits;
    		Mat kinColorImg2(COLOR_HEIGHT,COLOR_WIDTH, CV_8UC4, pBuffer);
    
    
    		/// convert kinect image to OpenCV MAT
    		uchar* kinColorImg_Ptr;
    		uchar* kinColorImg2_Ptr;
    		for(int j= 0 ; j< COLOR_HEIGHT; j++)
    		{
    			kinColorImg_Ptr = (uchar*) kinColorImg.ptr(j);
    			kinColorImg2_Ptr = (uchar*) kinColorImg2.ptr(j);
    			
    			for(int i=0; i< COLOR_WIDTH; i++)
    			{
    			kinColorImg_Ptr[i*3] = kinColorImg2_Ptr[i*4];
    			kinColorImg_Ptr[i*3+1] = kinColorImg2_Ptr[i*4+1];
    			kinColorImg_Ptr[i*3+2] = kinColorImg2_Ptr[i*4+2];
    			}
    		} 
    
    	/*
    
    	....codes for SURF descriptor(OpenCv processing)
    
    	*/
    
    	imshow("Color Image", kinColorImg);
    	NuiImageStreamReleaseFrame( h, pImageFrame );
    
    return 0;
    
    }
    
    
    
    int main(array<System::String ^> ^args)
    {
    
    Mat color(COLOR_HEIGHT, COLOR_WIDTH, CV_8UC4);
    
    HRESULT hr = NuiInitialize(NUI_INITIALIZE_FLAG_USES_DEPTH|NUI_INITIALIZE_FLAG_USES_COLOR);
    
    	if( hr != S_OK )
    	{
    		cout<<"NuiInitialize failed"<<endl;
    		return hr;
    	}
    
    	HANDLE h1 = CreateEvent( NULL, TRUE, FALSE, NULL );
    	HANDLE h2 = NULL;
    
    	hr = NuiImageStreamOpen(NUI_IMAGE_TYPE_COLOR,NUI_IMAGE_RESOLUTION_640x480, 0, 2, h1, &h2);
    
    	if( FAILED( hr ) )
    	{
    		cout<<"Could not open image stream video"<<endl;
    		return hr;
    	}
    
    	while(1)
    	{
    	WaitForSingleObject(h1,INFINITE);
    	drawColor(h2,color);
    
    	}
    
    	NuiShutdown();
    
    	return 0;
    
    }


    Tuesday, July 23, 2013 1:44 PM
  • Have a look at the Kinect Bridge with OpenCV sample from the Toolkit. Keep in mind, VC++ with /clr has specific rules for being managed vs unmanaged. Obviously, if you use any of the .Net objects those will be managed regardless. To avoid confusion, you should not be using /clr.

    http://msdn.microsoft.com/en-US/library/0adb9zxe(v=vs.100).aspx

    When calling GetNextFrame, you need to minimize the amount of time before releasing the frame. Any delays will cause the runtime to drop frames in the processing. You have less than 33ms (for 30fps) to do any work. If the GC decides to run because your function is managed, this is going to cause delays. 

    HRESULT hr = m_pNuiSensor->NuiImageStreamGetNextFrame(
    	m_hColorStreamHandle,
    	waitMillis,
    	&imageFrame);
    if (FAILED(hr))
    {
    	return hr;
    }
    // keep this block of code minimal to prevent 
    // blocking the Kinect runtime from losing frames
    // Release image stream frame
    hr = m_pNuiSensor->NuiImageStreamReleaseFrame(m_hColorStreamHandle, &imageFrame);

    Therefore, move the code for converting to openCV outside of that block of code. As with all the samples, the minimal amount of code you need is:

    NUI_LOCKED_RECT lockedRect; pTexture->LockRect(0, &lockedRect, NULL, 0); // Check if image is valid if (lockedRect.Pitch != 0) { ... memcpy_s(m_pColorBuffer, size, pBuffer, size); ... } // Unlock texture pTexture

    ->UnlockRect(0);

    As I suggested above, there are other advanced topics, beyond this forum, that you can employ to improve performance and keep your UI, Kinect Data, and Processing threads separate. These include, but not limited to:

    Processes and Threads
    http://msdn.microsoft.com/en-us/library/windows/desktop/ms684841(v=vs.85).aspx

    Low Fragmentation Heap
    http://msdn.microsoft.com/en-us/library/windows/desktop/aa366750(v=vs.85).aspx

    Multimedia Class Scheduler
    http://msdn.microsoft.com/en-us/library/windows/desktop/ms684247(v=vs.85).aspx


    Carmine Sirignano - MSFT


    Tuesday, July 23, 2013 6:44 PM
  • Well explained thanks!.

    The reason why I am using CLR is because I need to access using namespace System::IO::Ports;

    which I think won't work on Win32 projects?

    Wednesday, July 24, 2013 2:26 AM