none
[C#] TaskCanceledException when appending content concurrent using ConcurrentAppendAsync

    Question

  • I have written a log system that writes logentries in json format to a file in Azure Data Lake. Multiple threads are writing to that file so I use the `ConcurrentAppendAsync` method.

    I initialize the access like this:

    var clientCredential = new ClientCredential(ClientId, ClientSecret);

    var creds = ApplicationTokenProvider.LoginSilentAsync(Domain, clientCredential).GetAwaiter().GetResult();   

    fileSystemClient = new DataLakeStoreFileSystemManagementClient(creds);   

    fileSystem = fileSystemClient.FileSystem; 


    and I write the content like this:
           
    private async Task WriteToDataLakeAsync(IGrouping<string, LogEvent> logEvents)
            {
                var logEvent = logEvents.FirstOrDefault();
                if (logEvent == null)
                    return;
                var filename = string.Format(@"Logs\{0:yyyy}\{0:MM}\{0:dd}\{1}\{2}.json", logEvent.Session.Timestamp, logEvent.Session.Origin, logEvent.Session.SessionId);
                
                var jsonBuilder = new StringBuilder();
                foreach (var @event in logEvents)
                {
                    jsonBuilder.AppendLine(JsonConvert.SerializeObject(@event.SimplifyForBlob(), Formatting.None));
                }
                try
                {
                    await fileSystem.ConcurrentAppendAsync(Account, filename, new MemoryStream(Encoding.UTF8.GetBytes(jsonBuilder.ToString())), AppendModeType.Autocreate, cancellationToken: CancellationToken.None);
                }
                catch (Exception ex)
                {
                    telemetryClient.TrackException(new ExceptionTelemetry(new Exception($"{nameof(DataLake)}: Failed to write {filename}", ex)));
                }
            }

    The problem is, although I pass `CancellationToken.None` to `fileSystem.ConcurrentAppendAsync(...)` I still get a `TaskCanceledException` in my catch block:

     System.Exception:
    System.Threading.Tasks.TaskCanceledException:
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
       at Microsoft.Rest.RetryDelegatingHandler+<>c__DisplayClass11_0+<<SendAsync>b__1>d.MoveNext (Microsoft.Rest.ClientRuntime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
       at Microsoft.Rest.RetryDelegatingHandler+<SendAsync>d__11.MoveNext (Microsoft.Rest.ClientRuntime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
       at System.Net.Http.HttpClient+<FinishSendAsync>d__58.MoveNext (System.Net.Http, Version=4.1.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
       at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
       at Microsoft.Azure.Management.DataLake.Store.FileSystemOperations+<ConcurrentAppendWithHttpMessagesAsync>d__10.MoveNext (Microsoft.Azure.Management.DataLake.Store, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
       at Microsoft.Azure.Management.DataLake.Store.FileSystemOperationsExtensions+<ConcurrentAppendAsync>d__3.MoveNext (Microsoft.Azure.Management.DataLake.Store, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
       at DHS.PlanCare.CentralLogging.EventHub.Processors.StreamProcessors.DataLake+<WriteToDataLakeAsync>d__15.MoveNext (DHS.PlanCare.CentralLogging.EventHub.Processors, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\Sources\DHS\DHS.PlanCare.ManagementPortal\DHS.PlanCare.ManagementPortal.EventHub.Processors\StreamProcessors\DataLake.cs: 95)

    (Line 95 is the call to `ConcurrentAppendAsync`)

    What could be happening here? It does not help that [the docs](https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.management.datalake.store.filesystemoperationsextensions.concurrentappendasync?view=azuremgmtdatalakestore-2.2.0) don't say anything about this exception. There is even a mistake in the documentation since it states that when a CancellationToken is not passed the value will be `null`, which is impossible since CancellationToken is a value type and therefor cannot be `null`.
     
    So I do know for a fact that I do not initiate any kind of cancellation. The question is, can I safely ignore this exception or won't the data be written at all. Due to the large amount of data ingested and the fact that it happens on occasion it is hard to check if the data is actually written or not.


    • Edited by Expecho Monday, July 31, 2017 5:15 PM
    Monday, July 31, 2017 5:12 PM

All replies

  • Hi Expecho,

    It's possible that the SDK client reached its timeout limit when waiting for a response from the server for the ConcurrentAppend call. You can configure this timeout when creating the DataLakeStoreFileSystemManagementClient.

    If you're able to provide some more info to help us reproduce the issue that you're experiencing, such as a more complete solution, we can try to reproduce the issue locally to determine what's happening.

    Also, if you have any other debug information, such as any missing/delayed responses that you're able to see through a tool like Fiddler, that would also be very helpful from our point of view. (As always, if there's any sensitive information like passwords / keys / tokens / etc., please remove sensitive information before sharing.)

    Best,

    Matthew

    Monday, August 7, 2017 9:02 PM