none
[UWP][Desktop Bridge]Send speech recognition args.Result as parameter in UWP desktop-bridge package RRS feed

  • Question

  • I'm trying to figure out, if it is possible send using Windows.Media.SpeechRecognition; output args.Result.Text as parameter from UWP to Console application.

    For example by following scenario I'm sending TextToSpeech(args.Result.Text); with args.Result.Text; value, where using Windows.Media.SpeechSynthesis; text-to-speech pronounces the recognition result each time when args.Result.Text; appears. textBlock2.Text = args.Result.Text; also displays result:

     private async void ContinuousRecognitionSession_ResultGenerated(
                SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)
            {
                await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
                {               
                    textBlock1.Text = args.Result.Text;    
    
                    TextToSpeech(args.Result.Text);               
                });
            }

    but if I'm trying to send args.Result.Text; as parameter to console application, included with UWP in Desktop-Bridge package:

        private async void ContinuousRecognitionSession_ResultGenerated(
            SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)
        {
            await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
            {                            
              textBlock1.Text = args.Result.Text; 
    
              SendTTSResult(args.Result.Text);
            });
        }

    to the requested function:

     private async void SendTTSResult(string res)
        {
            await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
            {
                if (ApiInformation.IsApiContractPresent("Windows.ApplicationModel.FullTrustAppContract", 1, 0))
                {
                    ApplicationData.Current.LocalSettings.Values["parameters"] = res;
                    await FullTrustProcessLauncher.LaunchFullTrustProcessForCurrentAppAsync("Parameters");
                }
            });
        }

    Behavior of failure looks to me not really clear:

    With first recognition result, it sends parameter to console application, which successfully loads, gets and displays this parameter. But with second request the problem backs away from this processing level, even if parameters sending function is unambiguously the cause of a failure SendTTSResult(args.Result.Text); function does not receives args.Result.Text but this happens already before function will comes in action, because preceding output display textBlock1.Text = args.Result.Text; also does not receives event anymore.

    With async() => behavior is a bit different, it successfully receives event and sends value as parameter to console, in this case it happens 2-3 times from beginning of execution and voice requesting, then event disappears, when it is not even passed through SendTTSResult(string res), to imagine that the something in SendTTSResult(string res) does not allows pass string from recognition, but just stops, even if I put it in the end of TextToSpeech(string text) function, text to speech also stops receive event:

    private async void SendTTSResult(string res)
            {
                await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, async() =>
                {
                    if (ApiInformation.IsApiContractPresent("Windows.ApplicationModel.FullTrustAppContract", 1, 0))
                    {
                        ApplicationData.Current.LocalSettings.Values["parameters"] = res;
                        await FullTrustProcessLauncher.LaunchFullTrustProcessForCurrentAppAsync("Parameters");
                    }
                });
            }

    It looks like sending of args.Result.Text value as parameter with SendTTSResult(string res) function works fine, sends string successfully, but at the same time presence of this function in ContinuousRecognitionSession_ResultGenerated somehow affects receiving of event inside of it.

    At the same time behavior of ContSpeechRecognizer_HypothesisGenerated looks completely different, args.Hypothesis.Text event appears each time and result successfully passes as parameter with same SendTTSResult(string res).

    What can prevent an event from being executed when the function of sending a parameter is involved in its process, and how to fix it if possible?

    EDIT 1: **************************************************************************************************

    I've tried SendTTSResult(SpeechRecogVal); with change of variable value:

     private async void ContinuousRecognitionSession_ResultGenerated(
            SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)
        {
            await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
            {               
                SpeechRecogVal = args.Result.Text;
            });
        }

    but it is same behavior tbRec.Text = SpeechRecogVal; shows output successfully until I add SendTTSResult(SpeechRecogVal);,

       private string _srVal;
    
        public string SpeechRecogVal
        {
            get
            {
                return _srVal;
            }
    
            set
            {
                _srVal = value;
                ValueChanged();
            }
        }
    
        void ValueChanged()
        {
            tbRec.Text = SpeechRecogVal;
            // SendTTSResult(SpeechRecogVal);
        }

    So, seems like problem is something between await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, async () => and if (ApiInformation.IsApiContractPresent("Windows.ApplicationModel.FullTrustAppContract", 1, 0)) and await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () => of private async voidContinuousRecognitionSession_ResultGenerated(SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)

    Also, I've tried:

    private async void ContinuousRecognitionSession_ResultGenerated(
        SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)
    {
           await  SendTTSResult(args.Result.Text);        
    }

    as Task:

    async Task SendTTSResult(string res)
        {
            await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, async () =>
            {
                if (ApiInformation.IsApiContractPresent("Windows.ApplicationModel.FullTrustAppContract", 1, 0))
                {
                    ApplicationData.Current.LocalSettings.Values["parameters"] = res;
                    await FullTrustProcessLauncher.LaunchFullTrustProcessForCurrentAppAsync("Parameters");
                }
            });
        }

    And it is also successful only with first request event instance response, then goes quite. So seems like ContinuousRecognitionSession_ResultGenerated is someway different from other options in Windows.Media.SpeechRecognition Namespace and not compatible with something in async Task SendTTSResult(string res) or rather with this code lines content:

     ApplicationData.Current.LocalSettings.Values["parameters"] = args.Result.Text;
     await FullTrustProcessLauncher.LaunchFullTrustProcessForCurrentAppAsync("Parameters");











    Wednesday, July 10, 2019 12:08 AM

All replies

  • Hi,

    Could you please offer a repro demo about this? A simple demo that could reproduce the issue you said will be enough. Please delete the sensitive information about yourself and give us a clean sample. This could help to find the reason about this.

    Best regards,

    Roy


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Wednesday, July 10, 2019 3:30 AM
    Moderator
  • Could you please offer a repro demo about this? A simple demo that could reproduce the issue you said will be enough. Please delete the sensitive information about yourself and give us a clean sample. This could help to find the reason about this.

    Hello,

    Thank your your your feedback!

    As you have said in my previous question: "I made a sample here myself following the blog. All is working well in my side.", so, if you already created package, then it is easy to reproduce this code and failure:

    Windows Application Package Project includes UWP with general using Windows.Media.SpeechRecognition; events in MainPage.xaml.cs, with AudioCapturePermissions:

    SpeechRecognizer contSpeechRecognizer = new SpeechRecognizer(); private CoreDispatcher dispatcher; protected async override void OnNavigatedTo(NavigationEventArgs e) { bool permissionGained = await AudioCapturePermissions.RequestMicrophonePermission(); if (permissionGained) { dispatcher = CoreWindow.GetForCurrentThread().Dispatcher; contSpeechRecognizer = new Windows.Media.SpeechRecognition.SpeechRecognizer(); await contSpeechRecognizer.CompileConstraintsAsync(); contSpeechRecognizer.HypothesisGenerated += ContSpeechRecognizer_HypothesisGenerated; contSpeechRecognizer.ContinuousRecognitionSession.ResultGenerated += ContinuousRecognitionSession_ResultGenerated; contSpeechRecognizer.ContinuousRecognitionSession.Completed += ContinuousRecognitionSession_Completed; await contSpeechRecognizer.ContinuousRecognitionSession.StartAsync(); } else { this.textBlock1.Text = "Permission to access capture resources was not given by the user, reset the application setting in Settings->Privacy->Microphone."; } } private async void ContinuousRecognitionSession_Completed(SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionCompletedEventArgs args) { await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () => { textBlock1.Text = "Timeout."; }); await contSpeechRecognizer.ContinuousRecognitionSession.StartAsync(); } private async void ContSpeechRecognizer_HypothesisGenerated( SpeechRecognizer sender, SpeechRecognitionHypothesisGeneratedEventArgs args) { await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () => { textBlock2.Text = args.Hypothesis.Text; }); } private async void ContinuousRecognitionSession_ResultGenerated( SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args) { await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () => { textBlock1.Text = args.Result.Text; SendTTSResult(args.Result.Text); }); }

    AudioCapturePermission:
    public class AudioCapturePermissions
            {
                private static int NoCaptureDevicesHResult = -1072845856;   
                public async static Task<bool> RequestMicrophonePermission()
                {
                    try
                    {
                        MediaCaptureInitializationSettings settings = new MediaCaptureInitializationSettings(); 
                        settings.StreamingCaptureMode = StreamingCaptureMode.Audio;
                        settings.MediaCategory = MediaCategory.Speech;
                        MediaCapture capture = new MediaCapture();
                        await capture.InitializeAsync(settings);
                    }
                    catch (TypeLoadException)
                    {
                        var messageDialog = new Windows.UI.Popups.MessageDialog("Media player components are unavailable."); 
                        await messageDialog.ShowAsync();
                        return false;
                    }
                    catch (UnauthorizedAccessException)
                    {
                        return false; 
                    }
                    catch (Exception exception)
                    {
                        if (exception.HResult == NoCaptureDevicesHResult)  
                        {
                            var messageDialog = new Windows.UI.Popups.MessageDialog("No Audio Capture devices are present on this system.");
                            await messageDialog.ShowAsync();
                            return false;
                        }
                        else { throw; }
                    }
                    return true;
                }
            }

    ContinuousRecognitionSession_ResultGenerated will show you recognition result each time event fires,

    but including of parameter sender function, will show how recognition event disappears after one or two successful response,

    textBlock1.Text included to ContinuousRecognitionSession_ResultGenerated clearly shows how it happens:
    private async void SendTTSResult(string res)
            {
                await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, async () =>
                {
                    if (ApiInformation.IsApiContractPresent("Windows.ApplicationModel.FullTrustAppContract", 1, 0))
                    {
                        ApplicationData.Current.LocalSettings.Values["parameters"] = res;
                        await FullTrustProcessLauncher.LaunchFullTrustProcessForCurrentAppAsync("Parameters");
                    }
                });

    MainPage.xaml UI consist of:

     <TextBlock x:Name="textBlock1" HorizontalAlignment="Left" Margin="7,3,0,0" TextWrapping="Wrap" Text="Recognition state" VerticalAlignment="Top" FontSize="12"/>
            <TextBlock x:Name="textBlock3" HorizontalAlignment="Left" Margin="8,39,0,0" TextWrapping="Wrap" Text="Hypothesis" VerticalAlignment="Top" FontSize="27"/>
            <TextBlock x:Name="textBlock2" HorizontalAlignment="Left" Margin="8,80,0,0" TextWrapping="Wrap" Text="Result" VerticalAlignment="Top" FontSize="27"/>
            <MediaElement x:Name="textToSpeech" Volume="0.9" CurrentStateChanged="Media_State_Changed"/>

    behind parameter function, Console Connector.exe only shows parameter without running of any app or anything else:

      static void Main(string[] args)
            {
                string result = Assembly.GetExecutingAssembly().Location;
                int index = result.LastIndexOf("\\");
                string rootPath = $"{result.Substring(0, index)}\\..\\";
                if (args.Length > 2)
                {
                    switch (args[2])
                    {
                        case "/parameters":
                            string parameters = ApplicationData.Current.LocalSettings.Values["parameters"] as string;
                            Console.WriteLine("Parameter: " + parameters);
                            Console.ReadLine();
                            break;
                    }
                }
            }

    Packeage.appxmanifest:

    <uap:Extension Category="windows.appService">
      <uap:AppService Name="SampleInteropService" />
    </uap:Extension>
    
    <desktop:Extension Category="windows.fullTrustProcess" Executable="Connector\Connector.exe">
      <desktop:FullTrustProcess>
        <desktop:ParameterGroup GroupId="Parameters" Parameters="/parameters" />
      </desktop:FullTrustProcess>
    </desktop:Extension>









    • Edited by lado Oniani Wednesday, July 10, 2019 11:30 AM
    Wednesday, July 10, 2019 11:16 AM
  • Hello, please check my EDIT 1
    Thursday, July 11, 2019 4:34 PM
  • Hi,

    I'm sorry to say I could not reproduce the problem with the code snippet. Could you please give a sample package for this? Please delete the sensitive information about yourself and give us a clean sample.

    Best regards,

    Roy


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.


    Friday, July 12, 2019 2:28 AM
    Moderator
  • I'm sorry to say I could not reproduce the problem with the code snippet. Could you please give a sample package for this? Please delete the sensitive information about yourself and give us a clean sample.

    Hello,

    Sure! Here is a complete project UWP_SR_desktop_bridge, which reproduces  successful continuous speech recognition process (including AudioCapturePermissions and pronouncing  of recognized string  with TTS), and  will show main failure as described above with uncomment of line 100: // SendTTSResult(args.Result.Text); in  UWP_SR_desktop_bridge/UWP/MainPage.xaml.cs. With uncomment of line 87 // SendTTSResult(args.Hypothesis); same namespace event appears, and successfully received as parameter  in console UWP_SR_desktop_bridge/Connector/Program.cs

    Main failure indicator is successful textBlock2.Text = args.Result.Text; result display and TextToSpeech(args.Result.Text); recognized string output, each time voice recognition requested, and disappearance of args.Result.Textwith both this outputs, if I include SendTTSResult(args.Result.Text);, even with variable change event, or rather to say non-direct call of function void ValueChanged() shown in EDIT 1 above, where I've also noted my guess, that there is a some kind of conflict between specifically  ContinuousRecognitionSession_ResultGenerated and ApplicationData.Current.LocalSettings.Values["parameters"] = args.Result.Text; await FullTrustProcessLauncher.LaunchFullTrustProcessForCurrentAppAsync("Parameters");

    As short guide to redirect the functional code to your Windows Application Package Project, add  to UWP References/Add References/Universal Windows/Extensions/Windows Desktop Extension for the UWP, set UWP As Entry Point, Package as SturtUp Project and replace content:

    1. UWP_SR_desktop_bridge/UWP/MainPage.xaml.cs

    2.  UWP_SR_desktop_bridge/UWP/MainPage.xaml

    3.  UWP_SR_desktop_bridge/Connector/Program.cs

    4.  UWP_SR_desktop_bridge/Package/Package.appxmanifest (xmlns and <Extensions>)

    Configuration Properties:

    Thank you


    Friday, July 12, 2019 10:04 AM
  • Hi,

    I'll ask another engineer to take a look at this. There might be some time delay. 

    Best regards,

    Roy


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Tuesday, July 16, 2019 1:58 AM
    Moderator
  • I'll ask another engineer to take a look at this. There might be some time delay.

    Hello,

    Thank you for your attentive support and sorry for late replay

    So, question is still open as well as in topic on stackoverflow Send speech recognition args.Result as parameter in UWP desktop-bridge package. Answer given by Nico Zhu - MSFT related to another linked question System.NullReferenceException in AppServiceResponse and System.BadImageFormatException of UWP with WPF desktop bridge package, which particular System.BadImageFormatException problem solved with answer on the similar stackoverflow topic System.BadImageFormatException in UWP with WPF desktop bridge package by Stefan Wick MSFT

    But current task with  using Windows.Media.SpeechRecognition; and sending args.Result.Text as parameter  from UWP to Console is still open, because this question not fully provoked by BadImage problem with creating of app connection in Package, but to Speech recognition event and parameters function interaction, even if app connection with System.BadImageFormatException  problem solved and works well. Because, if I will get event result to console successfully with parameter sender function included, then I've alternative idea with sending of value from UWP through Console to external app outside of Windows Application Package Project, what seems to me very useful in number of cases.



    Sunday, July 21, 2019 12:28 PM
  • Hi lado_O,

    I was able to reproduce and resolve the problem using the following steps to correct the issue.

    Modify signature of SendTTSResult to be async Task. 
    Modify the delegates to be async and then await the SendTTSResult.

    From your sample, I modified the delegates in Hypothesis and Result generated methods, as well as the SendTTSResult signature shown in the following snippets. I bolded the modifications.

    private async Task SendTTSResultAsync(string ttsdata)

           private async void ContSpeechRecognizer_HypothesisGenerated(
               SpeechRecognizer sender, SpeechRecognitionHypothesisGeneratedEventArgs args)
           {
               await dispatcher.RunAsync(CoreDispatcherPriority.Normal, async () =>
               {
                   textBlock3.Text = args.Hypothesis.Text;
    
                   await SendTTSResultAsync(args.Hypothesis.Text.ToLower());  
               });          
           }
    
           private async void ContinuousRecognitionSession_ResultGenerated(
               SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)
           {
               await dispatcher.RunAsync(CoreDispatcherPriority.Normal, async () =>
               {               
                   textBlock1.Text = "Waiting ...";
                   textBlock2.Text = args.Result.Text; /// textBlock2.Text += args.Result.Text + "。\n";
                   TextToSpeech(args.Result.Text); /// TTS  
    
                   await SendTTSResultAsync(args.Result.Text); // SpeechRecogVal = args.Result.Text;
               });
           }

    FYI: Developers should not write an async void method except when overriding event handlers from a parent class where that handler has async void signature requirement. https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/async/async-return-types


    -David Hollowell (MSFT) Please remember to click &amp;amp;quot;Mark as Answer&amp;amp;quot; the responses that resolved your issue, and to click &amp;amp;quot;Unmark as Answer&amp;amp;quot; if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.



    Thursday, August 1, 2019 6:19 PM
    Moderator

  • I was able to reproduce and resolve the problem using the following steps to correct the issue.

    Modify signature of SendTTSResult to be async Task. 
    Modify the delegates to be async and then await the SendTTSResult.

    From your sample, I modified the delegates in Hypothesis and Result generated methods, as well as the SendTTSResult signature shown in the following snippets. I bolded the modifications.

    FYI: Developers should not write an async void method except when overriding event handlers from a parent class where that handler has async void signature requirement. https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/async/async-return-types


    Hello, David Hollowell

    Thank you for your feedback and sorry for late replay

    My package, which is UWP Speech Recognition and WPF TTS application instead of Console launcher and using parameter, now just with app connection to WPF included in package also to another external (out of package) WinForms application, both receives message from UWP Speech Recognition result as internally to WPF, also  through the same WPF to external app accessed with named pipe, but the failure still exist for any scenario including completely independent UWP Speech Recognition app testing:

    I want to use continuous speech recognition in UWP application, but each time after number of successful results in very different random moments during processing about 1-3 minutes from start of program ContinuousRecognitionSession_ResultGenerated just stops receive recognition event:

    SpeechRecognizer contSpeechRecognizer = new SpeechRecognizer();
            private CoreDispatcher dispatcher;
    
            protected async override void OnNavigatedTo(NavigationEventArgs e)
            {           
                bool permissionGained = await AudioCapturePermissions.RequestMicrophonePermission();
                if (permissionGained)
                {
                    dispatcher = CoreWindow.GetForCurrentThread().Dispatcher;
                    contSpeechRecognizer = new Windows.Media.SpeechRecognition.SpeechRecognizer();
                    await contSpeechRecognizer.CompileConstraintsAsync();
                    contSpeechRecognizer.HypothesisGenerated += ContSpeechRecognizer_HypothesisGenerated;
                    contSpeechRecognizer.ContinuousRecognitionSession.ResultGenerated += ContinuousRecognitionSession_ResultGenerated;
                    contSpeechRecognizer.ContinuousRecognitionSession.Completed += ContinuousRecognitionSession_Completed;
                    await contSpeechRecognizer.ContinuousRecognitionSession.StartAsync();
                }
                else
                {
                    this.textBlock1.Text = "Permission to access capture resources was not given by the user, reset the application setting in Settings->Privacy->Microphone.";
                }
            }
    
            private async void ContinuousRecognitionSession_Completed(SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionCompletedEventArgs args)
            {
                await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
                {
                    textBlock1.Text = "Timeout.";
                });
                await contSpeechRecognizer.ContinuousRecognitionSession.StartAsync();
            }
    
            private async void ContSpeechRecognizer_HypothesisGenerated(
                SpeechRecognizer sender, SpeechRecognitionHypothesisGeneratedEventArgs args)
            {
                await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
                {
                    textBlock3.Text = args.Hypothesis.Text;
                });
            }
    
            private async void ContinuousRecognitionSession_ResultGenerated(
                SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)
            {
                await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
                {
                    textBlock1.Text = "Waiting ...";
                    string r = args.Result.Text();
                    TTS(r);               
                });
            }

    I found two types of behavior during testing:

    1. It goes in Timeout with ContinuousRecognitionSession_Completed and remains in this state, in some rare cases resumes the process, otherwise only after restart. I'm not sure how to control this condition.

    2. Just stops receive args.Result.Text in ContinuousRecognitionSession_ResultGenerated, when unlike the first option ContSpeechRecognizer_HypothesisGenerated continuous receive args.Hypothesis.Text; successfully

    In both cases process interrupted. Same problem described in this question Why does UWP continuous speech recognition stop, but does not have marked or shown answer, as well as in my linked question Send speech recognition args.Result as parameter in UWP desktop-bridge package, and same UWP speech recognition failure requires restart with foreground and timeout, but now it is clear that the problem is not connected with Package desktop applications (Desktop Bridge) or calling of TTS or any other function inside, it is independent problem and occurs with args.Result.Text(); event same way outside the package without app connection, parameter or calling of TTS function.

    I have tried change it to async and await with private async Task TTS(string r) instead of private async void TTS(string r) as you suggested, but I also did the same before and result is same. After 1-2 minutes of testing about 30 recognition transaction requests, I got the same failure:

    private async void ContinuousRecognitionSession_ResultGenerated(
            SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)
        {
            await dispatcher.RunAsync(CoreDispatcherPriority.Normal, async () =>
            {
                textBlock1.Text = "Waiting ...";
                string r = args.Result.Text();
                await TTS(r);
            });
        }

     It seems like, event disappears before and independently of any included function like given UWP native TTS or sending directly to WPF with AppServiceResponse response = await App.Connection.SendMessageAsync(request); same way as described in this topic above with only difference that when accessing the Console with parameter failure happened immediately after the first successful request, and to WPF with app connection and included TTS or any other function,  or even just separate UWP Speech recognition app without inter-process communication or calling of some function, just receiving  recognition event, args.Result.Text(); event disappearing failure or uncontrollable Timeout state appears after 20-30 requests. Now, I'm trying to figure out, how to restart the ContSpeechRecognizer when it goes into the foreground if it can be a reason of failure, as suggested in this answer







    • Edited by lado Oniani Saturday, August 17, 2019 11:54 PM
    Saturday, August 17, 2019 10:13 PM
  • I was able to reproduce and resolve the problem using the following steps to correct the issue.

    Modify signature of SendTTSResult to be async Task. 
    Modify the delegates to be async and then await the SendTTSResult.

    From your sample, I modified the delegates in Hypothesis and Result generated methods, as well as the SendTTSResult signature shown in the following snippets. I bolded the modifications.

    FYI: Developers should not write an async void method except when overriding event handlers from a parent class where that handler has async void signature requirement. https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/async/async-return-types


    Hello, David Hollowell

    Thank you for your feedback and sorry for late replay

    My package, which is UWP Speech Recognition and WPF TTS application instead of Console launcher and using parameter, now just with app connection to WPF included in package also to another external (out of package) WinForms application, both receives message from UWP Speech Recognition result as internally to WPF, also  through the same WPF to external app accessed with named pipe, but the failure still exist for any scenario including completely independent UWP Speech Recognition app testing:

    I want to use continuous speech recognition in UWP application, but each time after number of successful results in very different random moments during processing about 1-3 minutes from start of program ContinuousRecognitionSession_ResultGenerated just stops receive recognition event:

    SpeechRecognizer contSpeechRecognizer = new SpeechRecognizer();
            private CoreDispatcher dispatcher;
    
            protected async override void OnNavigatedTo(NavigationEventArgs e)
            {           
                bool permissionGained = await AudioCapturePermissions.RequestMicrophonePermission();
                if (permissionGained)
                {
                    dispatcher = CoreWindow.GetForCurrentThread().Dispatcher;
                    contSpeechRecognizer = new Windows.Media.SpeechRecognition.SpeechRecognizer();
                    await contSpeechRecognizer.CompileConstraintsAsync();
                    contSpeechRecognizer.HypothesisGenerated += ContSpeechRecognizer_HypothesisGenerated;
                    contSpeechRecognizer.ContinuousRecognitionSession.ResultGenerated += ContinuousRecognitionSession_ResultGenerated;
                    contSpeechRecognizer.ContinuousRecognitionSession.Completed += ContinuousRecognitionSession_Completed;
                    await contSpeechRecognizer.ContinuousRecognitionSession.StartAsync();
                }
                else
                {
                    this.textBlock1.Text = "Permission to access capture resources was not given by the user, reset the application setting in Settings->Privacy->Microphone.";
                }
            }
    
            private async void ContinuousRecognitionSession_Completed(SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionCompletedEventArgs args)
            {
                await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
                {
                    textBlock1.Text = "Timeout.";
                });
                await contSpeechRecognizer.ContinuousRecognitionSession.StartAsync();
            }
    
            private async void ContSpeechRecognizer_HypothesisGenerated(
                SpeechRecognizer sender, SpeechRecognitionHypothesisGeneratedEventArgs args)
            {
                await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
                {
                    textBlock3.Text = args.Hypothesis.Text;
                });
            }
    
            private async void ContinuousRecognitionSession_ResultGenerated(
                SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)
            {
                await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
                {
                    textBlock1.Text = "Waiting ...";
                    string r = args.Result.Text();
                    TTS(r);               
                });
            }

    I found two types of behavior during testing:

    1. It goes in Timeout with ContinuousRecognitionSession_Completed and remains in this state, in some rare cases resumes the process, otherwise only after restart. I'm not sure how to control this condition.

    2. Just stops receive args.Result.Text in ContinuousRecognitionSession_ResultGenerated, when unlike the first option ContSpeechRecognizer_HypothesisGenerated continuous receive args.Hypothesis.Text; successfully

    In both cases process interrupted. Same problem described in this question Why does UWP continuous speech recognition stop, but does not have marked or shown answer, as well as in my linked question Send speech recognition args.Result as parameter in UWP desktop-bridge package, and same UWP speech recognition failure requires restart with foreground and timeout, but now it is clear that the problem is not connected with Package desktop applications (Desktop Bridge) or calling of TTS or any other function inside, it is independent problem and occurs with args.Result.Text(); event same way outside the package without app connection, parameter or calling of TTS function.

    I have tried change it to async and await with private async Task TTS(string r) instead of private async void TTS(string r) as you suggested, but I also did the same before and result is same. After 1-2 minutes of testing about 30 recognition transaction requests, I got the same failure:

    private async void ContinuousRecognitionSession_ResultGenerated(
            SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)
        {
            await dispatcher.RunAsync(CoreDispatcherPriority.Normal, async () =>
            {
                textBlock1.Text = "Waiting ...";
                string r = args.Result.Text();
                await TTS(r);
            });
        }

     It seems like, event disappears before and independently of any included function like given UWP native TTS or sending directly to WPF with AppServiceResponse response = await App.Connection.SendMessageAsync(request); same way as described in this topic above with only difference that when accessing the Console with parameter failure happened immediately after the first successful request, and to WPF with app connection and included TTS or any other function,  or even just separate UWP Speech recognition app without inter-process communication or calling of some function, just receiving  recognition event, args.Result.Text(); event disappearing failure or uncontrollable Timeout state appears after 20-30 requests. Now, I'm trying to figure out, how to restart the ContSpeechRecognizer when it goes into the foreground if it can be a reason of failure, as suggested in this answer

    Also I've tried this code suggested by Richard Zhang - MSFT on same question UWP speech recognition failure requires restart with foreground and timeout:

    private async void ContinuousRecognitionSession_Completed(SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionCompletedEventArgs args)
    {
        await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
        {
            textBlock1.Text = "Timeout.";
        });
        sender.ResultGenerated -= ContinuousRecognitionSession_ResultGenerated;
        sender.ResultGenerated += ContinuousRecognitionSession_ResultGenerated;
        await sender.StartAsync();
    }

    but it is same behavior: 1. args.Result.Text event loss after 20-30 successful requests, then after some time in same session args.Hypothesis.Text event also stops to work. 2. Irrevocable entering into to Timeout state.

    I've tried different microphones including condenser. Recording quality is good, including headphones, and also I've checked  quiet speech, works same way until a certain moment, so seems like pronouncing with low sound does not provokes failure, until this happens on its own, sometime when there is no speech at all


    • Edited by lado Oniani Monday, August 19, 2019 7:11 PM
    Monday, August 19, 2019 7:09 PM
  • Hi lado_O,

    Thanks for the update. I'm sorry to hear this issue is still occurring. I hope you're doing well, despite the trouble. Can you create a support ticket for us to look further into this? 

    You can do this from http://aka.ms/storesupport

    Click “contact us” tab near the top of the page.

    Select “windows 10 uwp development”

    Click “submit an incident”

    The first few fields on the following form are filled out already, but for the "category" dropdown, select “graphics and multimedia”

    Select “speech synthesis” from the “problem” dropdown.

    Fill in the remainder of the forms.

    If this issue is a defect in the Windows 10 SDK you will not be changed for the incident.


    -David Hollowell (MSFT) Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to others in our community. members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Wednesday, August 21, 2019 5:58 PM
    Moderator