Disabling Speech Recognition during Text to speech output

Question

I've written a "quick and dirty" GUI interface for a CLI chat program, and I've been trying to integrate TTS and Speech Recognition into it, so that the user can "have a conversation" with a chat bot, rather than just "type and read". The TTS portion works well (barring some small pronunciation issues - I have yet to find a TTS voice that can properly pronounce "Geek" without my having to mangle the spelling), and the SR module works well IF I have the TTS module disabled. The problem I'm having is that the SR module is processing the output from the TTS, and causing issues where an ever-increasing string of TTS output occurs, causing undesirable results.

 

I've tried several methods of temporarily disabling Speech recognition, none of which have proved useful. I'm hoping that, in my general inexperience, I've simply missed a solution. Below are some short code snippets that I've tried to use:

Imports SpeechLib
Imports System.Drawing
Imports System.Net.Sockets


Public Class Form1
  Public voice As New SpVoice
  Public recog As New SpSharedRecoContext
  Public Grammar As ISpeechRecoGrammar

  Private Sub Form1_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
    loadSettings()
    If (RecoContext Is Nothing) Then
      RecoContext = New SpSharedRecoContextClass    'Create a new Reco Context Class
      Grammar = RecoContext.CreateGrammar(1)      'Setup the Grammar
      Grammar.DictationLoad()             'Load the Grammar
    End If
    Grammar.DictationSetState(SpeechRuleState.SGDSActive)
    Timer1.Start()
    talkToMe("Welcome to chat script voice. Please tell me your name.")
    Me.txtName.Focus()
  End Sub


'This is the most recent attempt, and the one that looked the most promising
  Public Sub talkToMe(ByVal message As String)
    If UseTTS = True Then
      Grammar.DictationSetState(SpeechRuleState.SGDSInactive)
      voice.Speak(message, SpeechVoiceSpeakFlags.SVSFlagsAsync)
      Grammar.DictationSetState(SpeechRuleState.SGDSActive)
    End If
  End Sub

'This is an earlier attempt to disable SR during TTS output
  Public Sub talkToMe(ByVal message As String)
  recog.pause()
      voice.Speak(message, SpeechVoiceSpeakFlags.SVSFlagsAsync)
  recog.resume()
  End Sub

End Class

I've omitted sections that seem irrelevant. If more code is needed, then I'll be happy to provide it.

---

Safe, Reliable Insanity, Since 1961!

Answer 1

Hi Dave,

Welcome to the MSDN Forum.

According to your discription, it seems that your project cannot recognize the input voice while the the "Voice.speak".

If you can provide more code about how to handle the input voice, for example, where do you handle the Recognition event, and how to?

If I have misunderstood anything, please feel free to let me know.

Best regards,

---

Mike Feng [MSFT]
MSDN Community Support | Feedback to us
Get or Request Code Sample from Microsoft
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 2

Hi, Mike, and thanks for the welcome.

 

The only 2 subs that I had left out that dealt with the speech recognition portion of the app are as follows:

 Private Sub OnHypo(ByVal StreamNumber As Integer, ByVal StreamPosition As Object, ByVal Result _
As ISpeechRecoResult) Handles RecoContext.Hypothesis
  lblStatus.Text = "Receiving..."
 End Sub

 Private Sub OnReco(ByVal StreamNumber As Integer, ByVal StreamPosition As Object, ByVal RecognitionType _
As SpeechRecognitionType, ByVal Result As ISpeechRecoResult) Handles RecoContext.Recognition
  Dim recoResult As String = Result.PhraseInfo.GetText 'Create a new string, and assign the recognized text to it.
  Select Case Me.ActiveControl.Name
   Case "txtName"
    txtName.Text = recoResult
    txtInput.Focus()
   Case "txtInput"
    If recoResult = "enter" Then
     sendChat()
    Else
     txtInput.SelectionStart = Len(txtInput.Text)
     txtInput.Text = txtInput.Text & recoResult & " "
     txtInput.ScrollToCaret()
    End If
   Case Else
    txtInput.Text = ""
    txtInput.Focus()
  End Select
  lblStatus.Text = "Listening..."
  'Uncomment the next line if you want to send the text to the selected window rather than constrain it to the textbox.
  'SendKeys.Send(recoResult & " ") 'This line sends the result via SendKeys to the top window.

 End Sub

The first sub, dealing with recoContext.Hypothesis, is just used as a visual cue that SR is processing input. I expect that this isn't the best way to go about it, but it works for what I need. The second sub is what deals with filling the textual "input buffer" that gets sent to the chatbot portion of the app. My problem isn't that the SR "module" doesn't recognize the input (user's) voice. It's that it also registers and processes the computer's (TTS) voice, and processes that, too. This is something that I want to avoid, so I've been trying several different possible methods of disabling Speech Recognition, but only while the TTS is "speaking". So far, nothing has worked, so the Speech Recognition is also processing the TTS output, as if it were the user that was speaking.

 

Speech Recognition and TTS are new areas of study for me, and all of my experience with Visual Basic is self taught, so there are massive, gaping holes in my experience. Thus, I often tend to have code that is (at least to my eye) somewhat ugly. :) I hope that won't make it more difficult for you, when reading my code.

---

Safe, Reliable Insanity, Since 1961!

Answer 3

Well, I THINK I've found a solution to the problem. I tried the whole "use a timer" trick, which only partially worked, so I started poking around, investigating the spVoice object, and it's various properties and methods. What I found was spVoice.StartStream and spVoice.EndStream, which gave me an idea that resulted in the following code:

 Dim WithEvents voice As New SpVoice

 Private Sub startListening() Handles voice.EndStream
  Grammar.DictationSetState(SpeechRuleState.SGDSActive)
 End Sub

 Private Sub stopListening() Handles voice.StartStream
  Grammar.DictationSetState(SpeechRuleState.SGDSInactive)
 End Sub

This seems to have fixed the entire problem, so far. I still want to test it a bit more, just to be certain, but it looks very promising. Of course, in order for this to work, the voice object has to be declared using WithEvents (as shown above), but that goes without saying, one would hope. :)

---

Safe, Reliable Insanity, Since 1961!

Answer 4

 Hi Dave,

Sorry for late response since the weekend.

Congratulations.

Thank you very much for sharing your solution here, it will be very benefit to othere community members which has similar question.

Best regards,

---

Mike Feng [MSFT]
MSDN Community Support | Feedback to us
Get or Request Code Sample from Microsoft
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 5

Thanks, Mike. I had researched this particular problem not only here, but all across the web, and never found anything that helped, although I read a lot of posts that made other suggestions. I, too, hope that this will help others in the future.

---

Safe, Reliable Insanity, Since 1961!