locked
Program with Gesture and Speech Recognition RRS feed

  • Question

  • Hello,

    I need to develop a C++ program with Kinect  gesture and speech recognition. I tried to do that by myself but I can never have the two at same time... I am new with Windows programming and that's probably the problem :p

    I'm using Kinect SDK 1.5 and Speech SDK 11. I changed the examples SkeletonBasics and SpeechBasics, and then I tried to merge them into one, but I can never run the two things at same time. On my main.cpp, inside my main function I have the following:

    KinectGestures gesturesRecognizer;
    KinectSpeech speechRecognizer;
    
    HRESULT hr = CoInitializeEx(NULL, COINIT_MULTITHREADED);
    
    if (SUCCEEDED(hr))
    {
       {	
    	//Start Gestures Recognizer
    	gesturesRecognizer.init();
    	gesturesRecognizer.HandleEvents();
       }
       {
    	//Start Speech Recognizer
    	speechRecognizer.init();
    	speechRecognizer.HandleEvents();
       }
       CoUninitialize();
    }

    If I run it like that, I'll get only gestures recognition, it seems it never gets to speechRecognizer... Those functions are similar to the ones in the examples that come with kinect:

    - init(): is used to initialize Kinect and Speech variables;

    - HandleEvents() is like the Run() function, it is waiting for events;

    Those functions work. I think I need to create two threads to run the two things in simultaneously, but I'm not sure. Or is it possible to use NuiInitialize() to set Gesture and Speech Recognition on same INuiSensor object?

    Thank you very much!

    Tuesday, May 29, 2012 11:07 PM

Answers

  • What I did is to use just one thread:

    - Follow the way C++ samples like SkeletalViewer access the video/skeleton/depth data

    - In the Nui_ProcessThread(), right after the index = WaitForMultipleObjects(...) call, I call the ProcessSpeech() function which has the

    speechContext->GetEvents(1, &curEvent, &fetched); while (fetched > 0) {...} as one of the first lines of code -> thus it will simply return and do nothing in case there is no speech event for you.

    So essentially what I'm suggesting is to NOT wait for speech events, but simply poll for new data.


    Wednesday, June 6, 2012 10:37 AM

All replies

  • Yes, creating two threads should work.

    Wednesday, May 30, 2012 5:29 AM
  • Ok, I'm trying to that but I have a very strange error -.- I'm following the example here. And currently I have the following functions inside my KinectSpeech class:

    /// <summary>
    /// Start Speech Recognition Thread
    /// </summary>
    void KinectSpeech::Start()
    {
    	HANDLE thread;
    	int threadData = 0;
    	DWORD   threadId;
    
    	thread = CreateThread( 
            NULL,			// default security attributes
            0,				// use default stack size  
            HandleEvents,	// thread function name
            &threadData,			// argument to thread function 
            0,				// use default creation flags 
            &threadId		// returns the thread identifier
    	);
    
        // Check the return value for success.
        // If CreateThread fails, terminate execution. 
        // This will automatically clean up threads and memory. 
        if (thread == NULL) 
        {
            std::cout << "Error Starting Speech Thread." << std::endl;
            ExitProcess(3);
        }
    }
    
    /// <summary>
    /// Creates the main window and begins processing
    /// </summary>
    DWORD WINAPI KinectSpeech::HandleEvents(LPVOID lpParam)
    {
    	MSG       msg = {0};
    	const int eventCount = 1;
    	HANDLE hEvents[eventCount];
    
        // Main message loop
        while (WM_QUIT != msg.message)
        {
    		hEvents[0] = m_hSpeechEvent;
    
    
    		// Check to see if we have either a message (by passing in QS_ALLEVENTS)
            // Or a Kinect Speech Event (hEvents)
            DWORD dwEvent = MsgWaitForMultipleObjects(eventCount, hEvents, FALSE, INFINITE, QS_ALLINPUT);
    
            // Check if this is an event we're waiting on and not a timeout or message
            if (WAIT_OBJECT_0 == dwEvent)
            {
                if (NULL == m_pNuiSensor)
    			{
    				return -1;
    			}
    
    			if (WAIT_OBJECT_0 == dwEvent)
    			{
    				ProcessAction();
    			}
    		}
    	}
    	return 0;
    }

    The problem is in third argument of CreateThread. It gives me the following error: "argument of type DWORD is incompatible with parameter of type LPTHREAD_START_ROUTINE". I searched around and didn't find any solution, I don't understand what's wrong... Could you help me please?

    C++ is always lovely....

    Thanks.

    Wednesday, May 30, 2012 10:07 PM
  • 0
     Already Voted 
    
    	thread = CreateThread( 
            NULL,			// default security attributes
            0,				// use default stack size  
            (LPTHREAD_START_ROUTINE) HandleEvents,	// thread function name
            &threadData,			// argument to thread function 
            0,				// use default creation flags 
            &threadId		// returns the thread identifier
    	);
    

    Just cast the function pointer to type LPTHREAD_START_ROUTINE.

    So, the code above should work. :)

    Thursday, May 31, 2012 5:34 AM
  • Ok I was able to create the threads now, but I still have the same problem about running gestures and speech recognition at same time :/

    Each time speech recognition starts in its thread, the gestures recognition stops on other thread :/ And some time I get this error "Some other application is streaming from the same Kinect sensor".

    Any ideas?

    Thursday, May 31, 2012 10:29 PM
  • What I did is to use just one thread:

    - Follow the way C++ samples like SkeletalViewer access the video/skeleton/depth data

    - In the Nui_ProcessThread(), right after the index = WaitForMultipleObjects(...) call, I call the ProcessSpeech() function which has the

    speechContext->GetEvents(1, &curEvent, &fetched); while (fetched > 0) {...} as one of the first lines of code -> thus it will simply return and do nothing in case there is no speech event for you.

    So essentially what I'm suggesting is to NOT wait for speech events, but simply poll for new data.


    Wednesday, June 6, 2012 10:37 AM
  • Thank you very much, that was the solution!! I finally did it ^^
    Friday, June 8, 2012 12:32 AM