none
Speech Recognition Carry Conversation RRS feed

  • Question

  • Hello Everyone,  I have created basic speech recognition programs in the past and want to create something a little more advanced, that being said I want to have an actual conversation with my computer. Here is an example of what would be said:

    Me: Create a new folder

    Computer: What would like to name this folder?

    Me: (Insert Name)

    Computer: Where would you like to create (insert name)?

    Me: (Location such as Documents)

    Computer: Folder created

    I know how to create the folder and put it in a destination its the carrying of the conversation I don't how to do.

    Monday, August 27, 2018 7:52 PM

All replies

  • Hi Mr.Developer144,

    Thank you for posting here.

    For your question, you could try the code below.

    using System;
    using System.Collections.Generic;
    using System.ComponentModel;
    using System.Data;
    using System.Drawing;
    using System.Linq;
    using System.Speech.Recognition;
    using System.Speech.Synthesis;
    using System.Text;
    using System.Threading.Tasks;
    using System.Windows.Forms;
    
    namespace WindowsForms
    {
        public partial class Speech_chat : Form
        {
            public Speech_chat()
            {
                InitializeComponent();
            }
            SpeechSynthesizer speechSynthesizerObj;
            SpeechRecognitionEngine recognizer = new SpeechRecognitionEngine();
            Grammar dictationGrammar = new DictationGrammar();
            RecognitionResult result;
            private void BtnSend_Click(object sender, EventArgs e)
            {
    
                recognizer.LoadGrammar(dictationGrammar);
                try
                {
                    recognizer.SetInputToDefaultAudioDevice();
                    result = recognizer.Recognize();
                    textBox1.Text += "Me: " + result.Text + Environment.NewLine;
    
                }
                catch (InvalidOperationException exception)
                {
                    btnSend.Text = String.Format("Could not recognize input from default aduio device. Is a microphone or sound card available?\r\n{0} - {1}.", exception.Source, exception.Message);
                }
                finally
                {
                    recognizer.UnloadAllGrammars();
                }
            }
    
            private void BtnReceive_Click(object sender, EventArgs e)
            {
                if (result.Text.Contains("create") && result.Text.Contains("new folder"))
                {
                    speechSynthesizerObj = new SpeechSynthesizer();
                    speechSynthesizerObj.SpeakAsync("What would like to name this folder?");
                    textBox1.Text += "Computer: What would like to name this folder?" + Environment.NewLine;
                    
                }
                else if (result.Text.Contains("Insert Name"))
                {
                    speechSynthesizerObj = new SpeechSynthesizer();
                    speechSynthesizerObj.SpeakAsync("Where would you like to create (insert name)?");
                    textBox1.Text += "Computer: Where would you like to create (insert name)?" + Environment.NewLine;
                             
                }
    
                else if (result.Text.Contains("Location"))
                {
                    //create the folder
                    if (true)//check the foler exist or not, if exists, return true
                    {
                        speechSynthesizerObj = new SpeechSynthesizer();
                        speechSynthesizerObj.SpeakAsync("Folder created");
                        textBox1.Text += "Computer: Folder created" + Environment.NewLine;
                     
                    }
                }
            }
    
        }
    }
     


    When you speak, click Speak Send button. If you want to get the feedback from Computer Click Receive button. But the code I provided is confirm with the text from speech, if the speech could not be recognized, it could not back what you want.

    Best Regards,

    Wendy


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.


    Wednesday, August 29, 2018 12:57 PM
    Moderator
  •  That doesn't really do that I want. I want to be able to actually speak the commands. This is what the layout of my speech recognition program is:

    using System;
    using System.Collections.Generic;
    using System.ComponentModel;
    using System.Data;
    using System.Drawing;
    using System.Linq;
    using System.Text;
    using System.Threading.Tasks;
    using System.Windows.Forms;
    using System.IO;
    using System.Speech.Recognition;
    using System.Speech.Synthesis;
    using System.Runtime.InteropServices;
    using System.Runtime;
    using System.Net.NetworkInformation;
    using System.Diagnostics;
    using System.Threading;
    
    
    
    namespace Jarvis
    {
    
        public partial class Jarvis : Form
        {
            [DllImport("Shell32.dll")]
            static extern int SHEmptyRecycleBin(IntPtr hwnd, string pszRootPath, RecycleFlag dwFlags);
            SpeechRecognitionEngine Speechreco = new SpeechRecognitionEngine();
            SpeechSynthesizer jarvis = new SpeechSynthesizer();
            //private bool listening;
           // iTunesApp itunes = new iTunesApp();
            DateTime dateTime = DateTime.UtcNow.Date;
            public Jarvis()
            {
                Speechreco.SpeechRecognized += new EventHandler<SpeechRecognizedEventArgs>(Speechreco_SpeechRecognized);
                LoadGrammar();
                Speechreco.SetInputToDefaultAudioDevice();
                Speechreco.RecognizeAsync(RecognizeMode.Multiple);
                InitializeComponent();
            }
    
            enum RecycleFlag : int
            {
                SHERB_NOCONFIRMATION = 0x00000001, // No confirmation, when emptying
                SHERB_NOPROGRESSUI = 0x00000001, // No progress tracking window during the emptying of the recycle bin
                SHERB_NOSOUND = 0x00000004 // No sound when the emptying of the recycle bin is complete
            }
    
            private void LoadGrammar()
            {
                Choices texts = new Choices();
                string[] lines = File.ReadAllLines(Environment.CurrentDirectory + "\\commands.txt");
                texts.Add(lines);
                Grammar wordlist = new Grammar(new GrammarBuilder(texts));
                Speechreco.LoadGrammar(wordlist);
            }
    
    
    
    
            private void Speechreco_SpeechRecognized(object sender, SpeechRecognizedEventArgs e)
            {
                //bool listening = false;
                richTextBox1.Text = e.Result.Text;
                string speech = e.Result.Text;
                switch (speech)
                {
                    //Basic Commands
                    case "what is the date":
                        jarvis.Speak("Today is" + dateTime.ToString("MM/dd/yyyy"));
                        break;
                    case "what is the current time":
                        jarvis.Speak("Is is currently" + DateTime.Now.ToString("HH:mm:ss"));
                        break;
                    case "say something":
                        jarvis.Speak("A new guest has arrived.");
                        break;
                    //Basic Conversation
                    case "hello jarvis":
                        jarvis.Speak("hello sir");
                        break;
                    case "open disk":
                    case "open optical drive":
                        jarvis.Speak("Opening Optical Drive");
                        ExecuteCommand("C:/nircmd.exe cdrom open d:");
                        break;
                    case "driver status":
                        jarvis.Speak("All drivers are working at 100% efficency.");
                        break;
                    case "are you up":
                    case "jarvis are you up":
                        jarvis.Speak("For you sir, always.");
                        break;
                    //Volume
                    case "max volume":
                        jarvis.Speak("Volume is now at max");
                        ExecuteCommand("C:/nircmd.exe setsysvolume 65535");
                            break;
                    case "volume up by 20":
                        jarvis.Speak("increasing volume by 20");
                        ExecuteCommand("C:/nircmd.exe changesysvolume 13107");
                            break;
                    case "mute":
                        ExecuteCommand("C:/nircmd.exe mutesysvolume 1");
                        break;
    
                    case "unmute":
                        ExecuteCommand("C:/nircmd.exe mutesysvolume 0");
                        break;
                    //Internet
                    case "restart IIS":
                        jarvis.Speak("Restarting IIS Service please wait one moment.");
                        ExecuteCommand("C:/nircmd.exe service restart w3svc");
                        break;
                    case "check internet":
                        bool Connection = NetworkInterface.GetIsNetworkAvailable();
                        if (Connection == true)
                        {
                            jarvis.Speak("Internet Connection is Secured");
                        }
                        else
                        {
                            jarvis.Speak("There is No Internet");
                        }
    
                            break;
                    
                    //Open Programs
                    case "open visual studio":
                        String ProcessName = "devenv";
                        if (Process.GetProcessesByName(ProcessName).Length > 0)
                        {
                            jarvis.Speak("The Process Visual Studio Community 2017 is running");
                            
                        }
                        else
                        {
                            Process process = new Process();
                            // Configure the process using the StartInfo properties.
                            process.StartInfo.FileName = ProcessName;
                            process.Start();
                        }
                        break;
                    case "open itunes":
                        String ProcessName1 = "itunes";
                        if (Process.GetProcessesByName(ProcessName1).Length > 0)
                        {
                            jarvis.Speak("The Process Itunes is running");
                        }
                        else
                        {
                            Process process = new Process();
                            // Configure the process using the StartInfo properties.
                            process.StartInfo.FileName = ProcessName1;
                            process.Start();
                        }
                        break;
    
    
                    //Close Programs
                    case "close itunes":
                        // Start notepad.
                        Process proc = Process.Start("itunes.exe");
                        // Wait one second.
                        Thread.Sleep(1000);
                        // End notepad.
                        proc.Kill();
                        break;
                    //Exit
                    case "exit jarvis":
                    case "exit":
                    case "goodbye":
                    
                        jarvis.Speak("goodbye sir.");
                        Application.Exit();
                        break;
    
                    //Minimize and maximize
                    case "minimize":
                    case "leave me":
                        this.WindowState = FormWindowState.Minimized;
                        break;
                    case "maximize":
                    case "come back":
                        this.WindowState = FormWindowState.Maximized;
                        break;
                }
    
            }
                public static void ExecuteCommand(string Command)
            {
                System.Diagnostics.ProcessStartInfo procStartInfo =
                 new System.Diagnostics.ProcessStartInfo("cmd", "/c " + Command);
    
                // The following commands are needed to redirect the standard output.
                // This means that it will be redirected to the Process.StandardOutput StreamReader.
                procStartInfo.RedirectStandardOutput = true;
                procStartInfo.UseShellExecute = false;
                // Do not create the black window  .
                procStartInfo.CreateNoWindow = true;
                // Now we create a process, assign its ProcessStartInfo and start it
                System.Diagnostics.Process proc = new System.Diagnostics.Process();
                proc.StartInfo = procStartInfo;
                proc.Start();
                proc.Close();
            }
            private void button1_Click(object sender, EventArgs e)
            {
                Application.Exit();
            }
        }
    }

    Giving my way of creating this program, how can I do the conversation carry like I asked above?

    Wednesday, August 29, 2018 5:26 PM
  • is there a way to do this with my program's framework? And if so how?
    Monday, September 3, 2018 6:35 PM
  • Is this even possible with my code?
    Saturday, September 29, 2018 5:07 PM
  • Yes, what you're talking about is a state machine.  When you detect the "create a new folder" command, you just need to set a variable somewhere that tells you you're in the "waiting for folder name" state.  If you're in that state, the only thing you'll accept are folder names.  At that point, you'll go into the "waiting for destination" state.  When you get the destination, that's when you execute the command and go into the normal "waiting for command" state.

    Tim Roberts | Driver MVP Emeritus | Providenza &amp; Boekelheide, Inc.


    • Edited by Tim Roberts Sunday, September 30, 2018 6:20 AM
    Sunday, September 30, 2018 6:19 AM
  • Yes, what you're talking about is a state machine. When you detect the "create a new folder" command, you just need to set a variable somewhere that tells you you're in the "waiting for folder name" state.  If you're in that state, the only thing you'll accept are folder names.  At that point, you'll go into the "waiting for destination" state.  When you get the destination, that's when you execute the command and go into the normal "waiting for command" state.

    While this is perfectly true, the OP's intention seems, at least to me, go more in a direction where an AI approach might be fit in better. Even then it'll be a state machine of some kind, but let's assume a conversation like this:

    He: Jarvis, create a folder!
    Jarvis: What would you like to name the new folder?
    He: Wait, Jarvis, what's the current date?
    Jarvis: Today is September 30th, 2018.
    He: Call the new folder ''Images of September 30th"!
    Jarvis: Where do you like to create the folder ''Images of September 30th"?
    He: Create it below ''Images''!
    Jarvis: The folder ''Images of September 30th" has been created as a sub-folder of ''Images''.

    So far, so nice. Let's expand this example a bit (quite a bit ^^)...

    He: Jarvis, create a folder!
    Jarvis: What would you like to name the new folder?
    He: Wait, Jarvis, what's the current date?
    Jarvis: Today is September 30th, 2018.
    He: Erm, is there already a folder "2018" in the "Images" folder?"
    Jarvis: Yes, there is.
    He: And below that a folder "September"?
    Jarvis: No, there isn't any such folder. Do you want me to list the sub-folders of the folder "2018"?
    He: No, wait... Jarvis, what's John Doe's phone number?
    Jarvis: Okay. It's 555-1234.
    He: Wait... I've forgotten the name of that lady...
    Jarvis: Okay.
    He (calls John Doe)
    He: Jarvis, create a subfolder "Helen" below the "2018" folder, then "September"!
    Jarvis: Where do you like me to create the new folders?
    He: Create it below ''Images''!
    Jarvis: The folder ''Helen" has been created as a sub-folder of ''Images, 2018'', and the folder "September" has been created as a sub-folder of "Helen".

    What I've tried to sketch here is that a "natural" converation can have multiple "states" or better "contexts". Every time the user says "Jarvis", a new context would be created or returned to the most recent context. Furthermore, an AI approach seems "natural" to me to "know" whether the folder should be named "Helenbelowthetwothousandeighteenfolderthenseptember" or the system should create a folder "Helen" below the "2018" folder, and then another sub-folder "September" below the "Helen" folder.

    Apart from this, you would an application like this recognize "Jarvin, under ''Images'' and ''2018'', create a new folder called ''Tom'', and a folder ''September'' below that!", wouldn't you?

    Perhaps we sould either wait for "Data" or for the final integration of the Tensorflow stuff into the ML lib ;)

    Sunday, September 30, 2018 5:49 PM
  • Thank you so much for that response, can you point me in a direction to start my code, like a foundation of the code?
    Tuesday, October 2, 2018 3:09 PM