none
speech in game RRS feed

  • Question

  • hey im trying to make an rpg what i want is to with my voice be able to speak to the characters in the game using the kinect mic  ask them questions and have them respond im torn between unity or the udk as the engine i have limited coding experience but i want to learn please help 
    Monday, June 20, 2011 1:32 AM

Answers

  • My advice: forget about that idea.

    For the speech recognition you need the SpeechAPI, that is part of .NET and doesn't work with Unity.. don't know about UDK, but with limited coding experience I wouldn't recommend trying to find a way to get the SpeechAPI to work in there. I heard about a guy running CMU Sphinx (another speech recognition engine) in Unity. That might help, if you get the code. However there is still the problem of connecting to Kinect then. I guess it needs .NET4, so it might be a problem.

    Even if you did somehow manage to do it, you still need a good grammar and AI of some sort to process the natural language input and turn it into something a machine can work with. That AI would then need a big knowledge base to understand questions and generate answers from. After that they need to be rephrased as natural language output.... Alternatively it might work like a chat bot, using pattern matching and responding with pre-written answers. Then it still needs a lot of well-picked patterns and answers - per character, if you don't want them all to sound a alike.

    If you want them to say it out loud, it needs speech synthesis. So an engine for that (like in Speech API) is needed, which has to be compatible with your game engine of choice. A bunch of different voices would be necessary, to make them not sound all the same, too.

    In short: it is an aweful lot of work and even with state-of-the-art tools not a task I would expect any satisfying outcome from. The AI part is the biggest problem and that is still an active topic of research...

     

    But, everybody, feel free to correct me if I'm mistaken. Because I like the idea, too.

    Wednesday, June 22, 2011 9:26 PM

All replies

  • help?
    Wednesday, June 22, 2011 4:31 PM
  • My advice: forget about that idea.

    For the speech recognition you need the SpeechAPI, that is part of .NET and doesn't work with Unity.. don't know about UDK, but with limited coding experience I wouldn't recommend trying to find a way to get the SpeechAPI to work in there. I heard about a guy running CMU Sphinx (another speech recognition engine) in Unity. That might help, if you get the code. However there is still the problem of connecting to Kinect then. I guess it needs .NET4, so it might be a problem.

    Even if you did somehow manage to do it, you still need a good grammar and AI of some sort to process the natural language input and turn it into something a machine can work with. That AI would then need a big knowledge base to understand questions and generate answers from. After that they need to be rephrased as natural language output.... Alternatively it might work like a chat bot, using pattern matching and responding with pre-written answers. Then it still needs a lot of well-picked patterns and answers - per character, if you don't want them all to sound a alike.

    If you want them to say it out loud, it needs speech synthesis. So an engine for that (like in Speech API) is needed, which has to be compatible with your game engine of choice. A bunch of different voices would be necessary, to make them not sound all the same, too.

    In short: it is an aweful lot of work and even with state-of-the-art tools not a task I would expect any satisfying outcome from. The AI part is the biggest problem and that is still an active topic of research...

     

    But, everybody, feel free to correct me if I'm mistaken. Because I like the idea, too.

    Wednesday, June 22, 2011 9:26 PM