locked
Suddenly started getting "Unable to read data from the transport connection" and " Unable to write data to the transport connection" exceptions when uploading batches of documents to indexes RRS feed

  • Question

  •  

    Hi

    We have been receiving exceptions at random points every time we upload around 400k documents, each of 500 bytes, in batches of 100 or 200 documents. We notice that the exceptions occur at different points in the input, so do not seem to be caused by the input.  Each batch is uploaded to 1 of several indexes. 

    We are using a 1 replica, 1 partition Standard Azure Search Service which was created around 20 dec 2015. We are using Microsoft.Azure.Search SDK 0.12.0-preview which targets API version 2015-02-28. The upload client code runs on a website: the problem occurs both when the website is run locally in our office and when deployed on Azure in the same ResourceGroup and Region as the Service.

    These errors started yesterday after 6 or 8 weeks of testing on other similar Services with the same and other similar data and the same volumes and upload rates resulting in 10m+ successful uploads during which this error was never seen. We have not made any changes to the batch upload functionality in weeks.

    When we created a new Service and re-uploaded the data, it worked first time. 

    Has anyone else had this problem or have any insights about the causes? 

    To preserve evidence we have not yet tried Stopping and Starting the Service. What other workarounds or solutions are likely?



    The 2 exceptions we have been getting are:

    System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> 
    System.IO.IOException: Unable to read data from the transport connection: The connection was closed. 
    at System.Net.ConnectStream.EndWrite(IAsyncResult asyncResult) 
    at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization) 
    at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
    at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task) 
    at Microsoft.Azure.Search.DocumentOperationsExtensions.Index(IDocumentOperations operations, IndexBatch batch)

    System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> 
    System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> 
    System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.EndSend(IAsyncResult asyncResult) 
    at System.Net.Sockets.NetworkStream.EndWrite(IAsyncResult asyncResult) 
    at System.Net.ConnectStream.EndWrite(IAsyncResult asyncResult) 
    at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization) 
    at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
    at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task) 
    at Microsoft.Azure.Search.DocumentOperationsExtensions.Index(IDocumentOperations operations, IndexBatch batch) at 


    The exception occurs on the first of these 2 statements :

                Microsoft.Azure.Search.Models.IndexBatch batch = new IndexBatch(actions);
                DocumentIndexResponse response = client.Documents.Index(batch);

    Thanks

    Thursday, January 7, 2016 12:56 PM

Answers

  • We followed up offline with Mark.

    The transport connection errors are the result of a known issue in Azure Search that we are investigating. Sorry for the inconvenience.

    The errors are transient, so retrying is an effective mitigation until we can solve the issue.

    -Bruce

    Thursday, January 14, 2016 10:45 PM
    Moderator

All replies

  • Hi Mark,

    Please e-mail me at bruce dot johnston at the usual domain. Please include your service name, region, and the approximate date and time of some of these failed requests. This will help us investigate.

    Thanks,

    -Bruce

    Thursday, January 7, 2016 6:59 PM
    Moderator
  • We followed up offline with Mark.

    The transport connection errors are the result of a known issue in Azure Search that we are investigating. Sorry for the inconvenience.

    The errors are transient, so retrying is an effective mitigation until we can solve the issue.

    -Bruce

    Thursday, January 14, 2016 10:45 PM
    Moderator