none
Send speech recognition args.Result as parameter in UWP desktop-bridge package RRS feed

  • Question

  • I'm trying to figure out, if it is possible send using Windows.Media.SpeechRecognition; output args.Result.Text as parameter from UWP to Console application.

    For example by following scenario I'm sending TextToSpeech(args.Result.Text); with args.Result.Text; value, where using Windows.Media.SpeechSynthesis; text-to-speech pronounces the recognition result each time when args.Result.Text; appears. textBlock2.Text = args.Result.Text; also displays result:

     private async void ContinuousRecognitionSession_ResultGenerated(
                SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)
            {
                await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
                {               
                    textBlock1.Text = args.Result.Text;    
    
                    TextToSpeech(args.Result.Text);               
                });
            }

    but if I'm trying to send args.Result.Text; as parameter to console application, included with UWP in Desktop-Bridge package:

        private async void ContinuousRecognitionSession_ResultGenerated(
            SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)
        {
            await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
            {                            
              textBlock1.Text = args.Result.Text; 
    
              SendTTSResult(args.Result.Text);
            });
        }

    to the requested function:

     private async void SendTTSResult(string res)
        {
            await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
            {
                if (ApiInformation.IsApiContractPresent("Windows.ApplicationModel.FullTrustAppContract", 1, 0))
                {
                    ApplicationData.Current.LocalSettings.Values["parameters"] = res;
                    await FullTrustProcessLauncher.LaunchFullTrustProcessForCurrentAppAsync("Parameters");
                }
            });
        }

    Behavior of failure looks to me not really clear:

    With first recognition result, it sends parameter to console application, which successfully loads, gets and displays this parameter. But with second request the problem backs away from this processing level, even if parameters sending function is unambiguously the cause of a failure SendTTSResult(args.Result.Text); function does not receives args.Result.Text but this happens already before function will comes in action, because preceding output display textBlock1.Text = args.Result.Text; also does not receives event anymore.

    With async() => behavior is a bit different, it successfully receives event and sends value as parameter to console, in this case it happens 2-3 times from beginning of execution and voice requesting, then event disappears, when it is not even passed through SendTTSResult(string res), to imagine that the something in SendTTSResult(string res) does not allows pass string from recognition, but just stops, even if I put it in the end of TextToSpeech(string text) function, text to speech also stops receive event:

    private async void SendTTSResult(string res)
            {
                await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, async() =>
                {
                    if (ApiInformation.IsApiContractPresent("Windows.ApplicationModel.FullTrustAppContract", 1, 0))
                    {
                        ApplicationData.Current.LocalSettings.Values["parameters"] = res;
                        await FullTrustProcessLauncher.LaunchFullTrustProcessForCurrentAppAsync("Parameters");
                    }
                });
            }

    It looks like sending of args.Result.Text value as parameter with SendTTSResult(string res) function works fine, sends string successfully, but at the same time presence of this function in ContinuousRecognitionSession_ResultGenerated somehow affects receiving of event inside of it.

    At the same time behavior of ContSpeechRecognizer_HypothesisGenerated looks completely different, args.Hypothesis.Text event appears each time and result successfully passes as parameter with same SendTTSResult(string res).

    What can prevent an event from being executed when the function of sending a parameter is involved in its process, and how to fix it if possible?

    EDIT 1: **************************************************************************************************

    I've tried SendTTSResult(SpeechRecogVal); with change of variable value:

     private async void ContinuousRecognitionSession_ResultGenerated(
            SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)
        {
            await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
            {               
                SpeechRecogVal = args.Result.Text;
            });
        }

    but it is same behavior tbRec.Text = SpeechRecogVal; shows output successfully until I add SendTTSResult(SpeechRecogVal);,

       private string _srVal;
    
        public string SpeechRecogVal
        {
            get
            {
                return _srVal;
            }
    
            set
            {
                _srVal = value;
                ValueChanged();
            }
        }
    
        void ValueChanged()
        {
            tbRec.Text = SpeechRecogVal;
            // SendTTSResult(SpeechRecogVal);
        }

    So, seems like problem is something between await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, async () => and if (ApiInformation.IsApiContractPresent("Windows.ApplicationModel.FullTrustAppContract", 1, 0)) and await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () => of private async voidContinuousRecognitionSession_ResultGenerated(SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)

    Also, I've tried:

    private async void ContinuousRecognitionSession_ResultGenerated(
        SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args)
    {
           await  SendTTSResult(args.Result.Text);        
    }

    as Task:

    async Task SendTTSResult(string res)
        {
            await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, async () =>
            {
                if (ApiInformation.IsApiContractPresent("Windows.ApplicationModel.FullTrustAppContract", 1, 0))
                {
                    ApplicationData.Current.LocalSettings.Values["parameters"] = res;
                    await FullTrustProcessLauncher.LaunchFullTrustProcessForCurrentAppAsync("Parameters");
                }
            });
        }

    And it is also successful only with first request event instance response, then goes quite. So seems like ContinuousRecognitionSession_ResultGenerated is someway different from other options in Windows.Media.SpeechRecognition Namespace and not compatible with something in async Task SendTTSResult(string res) or rather with this code lines content:

     ApplicationData.Current.LocalSettings.Values["parameters"] = args.Result.Text;
     await FullTrustProcessLauncher.LaunchFullTrustProcessForCurrentAppAsync("Parameters");










    • Edited by lado_O Thursday, July 11, 2019 4:44 PM
    Wednesday, July 10, 2019 12:08 AM

All replies

  • Hi,

    Could you please offer a repro demo about this? A simple demo that could reproduce the issue you said will be enough. Please delete the sensitive information about yourself and give us a clean sample. This could help to find the reason about this.

    Best regards,

    Roy


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Wednesday, July 10, 2019 3:30 AM
    Moderator
  • Could you please offer a repro demo about this? A simple demo that could reproduce the issue you said will be enough. Please delete the sensitive information about yourself and give us a clean sample. This could help to find the reason about this.

    Hello,

    Thank your your your feedback!

    As you have said in my previous question: "I made a sample here myself following the blog. All is working well in my side.", so, if you already created package, then it is easy to reproduce this code and failure:

    Windows Application Package Project includes UWP with general using Windows.Media.SpeechRecognition; events in MainPage.xaml.cs, with AudioCapturePermissions:

    SpeechRecognizer contSpeechRecognizer = new SpeechRecognizer(); private CoreDispatcher dispatcher; protected async override void OnNavigatedTo(NavigationEventArgs e) { bool permissionGained = await AudioCapturePermissions.RequestMicrophonePermission(); if (permissionGained) { dispatcher = CoreWindow.GetForCurrentThread().Dispatcher; contSpeechRecognizer = new Windows.Media.SpeechRecognition.SpeechRecognizer(); await contSpeechRecognizer.CompileConstraintsAsync(); contSpeechRecognizer.HypothesisGenerated += ContSpeechRecognizer_HypothesisGenerated; contSpeechRecognizer.ContinuousRecognitionSession.ResultGenerated += ContinuousRecognitionSession_ResultGenerated; contSpeechRecognizer.ContinuousRecognitionSession.Completed += ContinuousRecognitionSession_Completed; await contSpeechRecognizer.ContinuousRecognitionSession.StartAsync(); } else { this.textBlock1.Text = "Permission to access capture resources was not given by the user, reset the application setting in Settings->Privacy->Microphone."; } } private async void ContinuousRecognitionSession_Completed(SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionCompletedEventArgs args) { await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () => { textBlock1.Text = "Timeout."; }); await contSpeechRecognizer.ContinuousRecognitionSession.StartAsync(); } private async void ContSpeechRecognizer_HypothesisGenerated( SpeechRecognizer sender, SpeechRecognitionHypothesisGeneratedEventArgs args) { await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () => { textBlock2.Text = args.Hypothesis.Text; }); } private async void ContinuousRecognitionSession_ResultGenerated( SpeechContinuousRecognitionSession sender, SpeechContinuousRecognitionResultGeneratedEventArgs args) { await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () => { textBlock1.Text = args.Result.Text; SendTTSResult(args.Result.Text); }); }

    AudioCapturePermission:
    public class AudioCapturePermissions
            {
                private static int NoCaptureDevicesHResult = -1072845856;   
                public async static Task<bool> RequestMicrophonePermission()
                {
                    try
                    {
                        MediaCaptureInitializationSettings settings = new MediaCaptureInitializationSettings(); 
                        settings.StreamingCaptureMode = StreamingCaptureMode.Audio;
                        settings.MediaCategory = MediaCategory.Speech;
                        MediaCapture capture = new MediaCapture();
                        await capture.InitializeAsync(settings);
                    }
                    catch (TypeLoadException)
                    {
                        var messageDialog = new Windows.UI.Popups.MessageDialog("Media player components are unavailable."); 
                        await messageDialog.ShowAsync();
                        return false;
                    }
                    catch (UnauthorizedAccessException)
                    {
                        return false; 
                    }
                    catch (Exception exception)
                    {
                        if (exception.HResult == NoCaptureDevicesHResult)  
                        {
                            var messageDialog = new Windows.UI.Popups.MessageDialog("No Audio Capture devices are present on this system.");
                            await messageDialog.ShowAsync();
                            return false;
                        }
                        else { throw; }
                    }
                    return true;
                }
            }

    ContinuousRecognitionSession_ResultGenerated will show you recognition result each time event fires,

    but including of parameter sender function, will show how recognition event disappears after one or two successful response,

    textBlock1.Text included to ContinuousRecognitionSession_ResultGenerated clearly shows how it happens:
    private async void SendTTSResult(string res)
            {
                await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, async () =>
                {
                    if (ApiInformation.IsApiContractPresent("Windows.ApplicationModel.FullTrustAppContract", 1, 0))
                    {
                        ApplicationData.Current.LocalSettings.Values["parameters"] = res;
                        await FullTrustProcessLauncher.LaunchFullTrustProcessForCurrentAppAsync("Parameters");
                    }
                });

    MainPage.xaml UI consist of:

     <TextBlock x:Name="textBlock1" HorizontalAlignment="Left" Margin="7,3,0,0" TextWrapping="Wrap" Text="Recognition state" VerticalAlignment="Top" FontSize="12"/>
            <TextBlock x:Name="textBlock3" HorizontalAlignment="Left" Margin="8,39,0,0" TextWrapping="Wrap" Text="Hypothesis" VerticalAlignment="Top" FontSize="27"/>
            <TextBlock x:Name="textBlock2" HorizontalAlignment="Left" Margin="8,80,0,0" TextWrapping="Wrap" Text="Result" VerticalAlignment="Top" FontSize="27"/>
            <MediaElement x:Name="textToSpeech" Volume="0.9" CurrentStateChanged="Media_State_Changed"/>

    behind parameter function, Console Connector.exe only shows parameter without running of any app or anything else:

      static void Main(string[] args)
            {
                string result = Assembly.GetExecutingAssembly().Location;
                int index = result.LastIndexOf("\\");
                string rootPath = $"{result.Substring(0, index)}\\..\\";
                if (args.Length > 2)
                {
                    switch (args[2])
                    {
                        case "/parameters":
                            string parameters = ApplicationData.Current.LocalSettings.Values["parameters"] as string;
                            Console.WriteLine("Parameter: " + parameters);
                            Console.ReadLine();
                            break;
                    }
                }
            }

    Packeage.appxmanifest:

    <uap:Extension Category="windows.appService">
      <uap:AppService Name="SampleInteropService" />
    </uap:Extension>
    
    <desktop:Extension Category="windows.fullTrustProcess" Executable="Connector\Connector.exe">
      <desktop:FullTrustProcess>
        <desktop:ParameterGroup GroupId="Parameters" Parameters="/parameters" />
      </desktop:FullTrustProcess>
    </desktop:Extension>









    • Edited by lado_O Wednesday, July 10, 2019 11:30 AM
    Wednesday, July 10, 2019 11:16 AM
  • Hello, please check my EDIT 1
    Thursday, July 11, 2019 4:34 PM
  • Hi,

    I'm sorry to say I could not reproduce the problem with the code snippet. Could you please give a sample package for this? Please delete the sensitive information about yourself and give us a clean sample.

    Best regards,

    Roy


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.


    Friday, July 12, 2019 2:28 AM
    Moderator
  • I'm sorry to say I could not reproduce the problem with the code snippet. Could you please give a sample package for this? Please delete the sensitive information about yourself and give us a clean sample.

    Hello,

    Sure! Here is a complete project UWP_SR_desktop_bridge, which reproduces  successful continuous speech recognition process (including AudioCapturePermissions and pronouncing  of recognized string  with TTS), and  will show main failure as described above with uncomment of line 100: // SendTTSResult(args.Result.Text); in  UWP_SR_desktop_bridge/UWP/MainPage.xaml.cs. With uncomment of line 87 // SendTTSResult(args.Hypothesis); same namespace event appears, and successfully received as parameter  in console UWP_SR_desktop_bridge/Connector/Program.cs

    Main failure indicator is successful textBlock2.Text = args.Result.Text; result display and TextToSpeech(args.Result.Text); recognized string output, each time voice recognition requested, and disappearance of args.Result.Textwith both this outputs, if I include SendTTSResult(args.Result.Text);, even with variable change event, or rather to say non-direct call of function void ValueChanged() shown in EDIT 1 above, where I've also noted my guess, that there is a some kind of conflict between specifically  ContinuousRecognitionSession_ResultGenerated and ApplicationData.Current.LocalSettings.Values["parameters"] = args.Result.Text; await FullTrustProcessLauncher.LaunchFullTrustProcessForCurrentAppAsync("Parameters");

    As short guide to redirect the functional code to your Windows Application Package Project, add  to UWP References/Add References/Universal Windows/Extensions/Windows Desktop Extension for the UWP, set UWP As Entry Point, Package as SturtUp Project and replace content:

    1. UWP_SR_desktop_bridge/UWP/MainPage.xaml.cs

    2.  UWP_SR_desktop_bridge/UWP/MainPage.xaml

    3.  UWP_SR_desktop_bridge/Connector/Program.cs

    4.  UWP_SR_desktop_bridge/Package/Package.appxmanifest (xmlns and <Extensions>)

    Configuration Properties:

    Thank you


    • Edited by lado_O Friday, July 12, 2019 11:17 AM
    Friday, July 12, 2019 10:04 AM
  • Hi,

    I'll ask another engineer to take a look at this. There might be some time delay. 

    Best regards,

    Roy


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Tuesday, July 16, 2019 1:58 AM
    Moderator
  • I'll ask another engineer to take a look at this. There might be some time delay.

    Hello,

    Thank you for your attentive support and sorry for late replay

    So, question is still open as well as in topic on stackoverflow Send speech recognition args.Result as parameter in UWP desktop-bridge package. Answer given by Nico Zhu - MSFT related to another linked question System.NullReferenceException in AppServiceResponse and System.BadImageFormatException of UWP with WPF desktop bridge package, which particular System.BadImageFormatException problem solved with answer on the similar stackoverflow topic System.BadImageFormatException in UWP with WPF desktop bridge package by Stefan Wick MSFT

    But current task with  using Windows.Media.SpeechRecognition; and sending args.Result.Text as parameter  from UWP to Console is still open, because this question not fully provoked by BadImage problem with creating of app connection in Package, but to Speech recognition event and parameters function interaction, even if app connection with System.BadImageFormatException  problem solved and works well. Because, if I will get event result to console successfully with parameter sender function included, then I've alternative idea with sending of value from UWP through Console to external app outside of Windows Application Package Project, what seems to me very useful in number of cases.



    • Edited by lado_O 5 hours 53 minutes ago
    5 hours 57 minutes ago