locked
Gateway unreachable - The socket connection has been disposed RRS feed

  • Question

  • Hi,

    I'm working on a integration solution with Logic Apps. It is a very simple scenario:

    1. Get access token from endpoint
    2. In a until loop of n iterations:
    3.      Use access token to hit data endpoint
    4.      Parse json returned from data endpoint
    5.      Save json to a on prem SQL Server instance (stored proc)

    The first few iterations (4-5) work fine but then it starts failing with the following error:

    {
      "status"404,
      "message""The gateway is either offline or could not be reached.\r\n     inner exception: Gateway on endpoint '<ii>sb://wabi-us-east2-relay11.servicebus.windows.net/ba63cf64-a53c-4af2-a126-31ac3cb44772/</ii>' is unreachable.\r\n         inner exception: The socket connection has been disposed. TrackingId:af7a806b-3539-4f95-a862-79628ccf97d4, SystemTracker:wabi-us-east2-relay11.servicebus.windows.net:ba63cf64-a53c-4af2-a126-31ac3cb44772, Timestamp:2020-05-26T20:00:48\r\n             inner exception: The write operation failed, see inner exception.\r\n                 inner exception: The socket connection has been disposed. TrackingId:af7a806b-3539-4f95-a862-79628ccf97d4, SystemTracker:wabi-us-east2-relay11.servicebus.windows.net:ba63cf64-a53c-4af2-a126-31ac3cb44772, Timestamp:2020-05-26T20:00:48\r\n                     inner exception: Cannot access a disposed object.\r\nObject name: 'System.Net.Sockets.Socket'.\r\nclientRequestId: ab436d0b-e5ed-4043-8596-ccf7fb813a1a",
      "error": {
        "message""The gateway is either offline or could not be reached.\r\n     inner exception: Gateway on endpoint '<ii>sb://wabi-us-east2-relay11.servicebus.windows.net/ba63cf64-a53c-4af2-a126-31ac3cb44772/</ii>' is unreachable.\r\n         inner exception: The socket connection has been disposed. TrackingId:af7a806b-3539-4f95-a862-79628ccf97d4, SystemTracker:wabi-us-east2-relay11.servicebus.windows.net:ba63cf64-a53c-4af2-a126-31ac3cb44772, Timestamp:2020-05-26T20:00:48\r\n             inner exception: The write operation failed, see inner exception.\r\n                 inner exception: The socket connection has been disposed. TrackingId:af7a806b-3539-4f95-a862-79628ccf97d4, SystemTracker:wabi-us-east2-relay11.servicebus.windows.net:ba63cf64-a53c-4af2-a126-31ac3cb44772, Timestamp:2020-05-26T20:00:48\r\n                     inner exception: Cannot access a disposed object.\r\nObject name: 'System.Net.Sockets.Socket'."
      },
      "source""sql-eus2.azconn-eus2.p.azurewebsites.net"
    }

    Is there anything I'm missing? Could someone please provide guidance on how to solve/troubleshoot this?

    Please find below details about everything I tried and the information about the error I was able to collect.

    I tried rewriting the app as two apps, where the second one would be invoked inside the loop and would contain the database call to save the data, but the behavior is basically the same.

    I tried using different on prem data gateways (including a dedicated one), recreating the SQL Server Stored Proc actions, recreating the gateway connections, reinstalling the local On-premises data gateway application, making sure the on prem gateway service was running, and restarting the service. All without any change to outcome.

    I ran the network port test and got 100% success rate with partial results below:

         General Information
         Service Bus Namespace : WABI-US-EAST2-relay11.servicebus.windows.net
         Host Name : ns-sb2-prod-bn3-005.cloudapp.net
         Host IP Address : 104.46.98.9
         Overall Results : 32/32 tests succeeded.

    I looked at the Event Viewer logs for the On-premises data gateway and found no errors in the logs.

    I enabled the Additional logging option and was able to find this in one of the GatewayNetwork logs:

    DM.EnterpriseGateway Information: 1 : 2020-05-26T18:23:35.8744785Z DM.EnterpriseGateway 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 0000 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 33B47475 [DM.NetworkCore] ServiceEndpoint Connection status: Online, LastError: ''
    DM.EnterpriseGateway Information: 1 : 2020-05-26T18:31:47.6381163Z DM.EnterpriseGateway 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 0000 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 8DC38AE2 [DM.NetworkCore] ServiceEndpoint Connection status: Connecting, LastError: 'System.ServiceModel.CommunicationException: More data was expected, but EOF was reached. ---> System.IO.IOException: More data was expected, but EOF was reached. ---> System.IO.InvalidDataException: ServiceBusClientWebSocket was expecting more bytes
       --- End of inner exception stack trace ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.ServiceBusClientWebSocket.EndReceive(IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Amqp.Transport.ClientWebSocketTransport.HandleReadComplete(IAsyncResult result)
       --- End of inner exception stack trace ---'
    DM.EnterpriseGateway Information: 1 : 2020-05-26T18:31:48.4532812Z DM.EnterpriseGateway 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 0000 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 A423E68D [DM.NetworkCore] ServiceEndpoint Connection status: Connecting, LastError: 'System.ServiceModel.CommunicationException: The remote name could not be resolved: 'wabi-us-east2-relay11.servicebus.windows.net' ---> System.IO.IOException: The remote name could not be resolved: 'wabi-us-east2-relay11.servicebus.windows.net' ---> System.Net.WebException: The remote name could not be resolved: 'wabi-us-east2-relay11.servicebus.windows.net'
       at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
       at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.ServiceBus.ServiceBusClientWebSocket.<ConnectAsync>d__54.MoveNext()
       --- End of inner exception stack trace ---
       at Microsoft.ServiceBus.ServiceBusClientWebSocket.<ConnectAsync>d__54.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.ServiceBus.Common.TaskHelpers.EndAsyncResult(IAsyncResult asyncResult)
       at Microsoft.ServiceBus.WebSocketConnectAsyncResult.<GetAsyncSteps>b__19_1(WebSocketConnectAsyncResult thisPtr, IAsyncResult r)
       at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
       --- End of inner exception stack trace ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Amqp.Transport.WebSocketTransportInitiator.Complete(IAsyncResult asyncResult, Boolean completeSynchronously)
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext()'
    DM.EnterpriseGateway Information: 1 : 2020-05-26T18:31:49.5177819Z DM.EnterpriseGateway 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 0000 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 A423E68D [DM.NetworkCore] ServiceEndpoint Connection status: Connecting, LastError: 'System.ServiceModel.CommunicationException: The remote name could not be resolved: 'wabi-us-east2-relay11.servicebus.windows.net' ---> System.IO.IOException: The remote name could not be resolved: 'wabi-us-east2-relay11.servicebus.windows.net' ---> System.Net.WebException: The remote name could not be resolved: 'wabi-us-east2-relay11.servicebus.windows.net'
       at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
       at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.ServiceBus.ServiceBusClientWebSocket.<ConnectAsync>d__54.MoveNext()
       --- End of inner exception stack trace ---
       at Microsoft.ServiceBus.ServiceBusClientWebSocket.<ConnectAsync>d__54.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.ServiceBus.Common.TaskHelpers.EndAsyncResult(IAsyncResult asyncResult)
       at Microsoft.ServiceBus.WebSocketConnectAsyncResult.<GetAsyncSteps>b__19_1(WebSocketConnectAsyncResult thisPtr, IAsyncResult r)
       at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
       --- End of inner exception stack trace ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Amqp.Transport.WebSocketTransportInitiator.Complete(IAsyncResult asyncResult, Boolean completeSynchronously)
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext()'
    DM.EnterpriseGateway Information: 1 : 2020-05-26T18:31:51.5229122Z DM.EnterpriseGateway 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 0000 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 A423E68D [DM.NetworkCore] ServiceEndpoint Connection status: Connecting, LastError: 'System.ServiceModel.CommunicationException: The remote name could not be resolved: 'wabi-us-east2-relay11.servicebus.windows.net' ---> System.IO.IOException: The remote name could not be resolved: 'wabi-us-east2-relay11.servicebus.windows.net' ---> System.Net.WebException: The remote name could not be resolved: 'wabi-us-east2-relay11.servicebus.windows.net'
       at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
       at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.ServiceBus.ServiceBusClientWebSocket.<ConnectAsync>d__54.MoveNext()
       --- End of inner exception stack trace ---
       at Microsoft.ServiceBus.ServiceBusClientWebSocket.<ConnectAsync>d__54.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.ServiceBus.Common.TaskHelpers.EndAsyncResult(IAsyncResult asyncResult)
       at Microsoft.ServiceBus.WebSocketConnectAsyncResult.<GetAsyncSteps>b__19_1(WebSocketConnectAsyncResult thisPtr, IAsyncResult r)
       at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
       --- End of inner exception stack trace ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at Microsoft.ServiceBus.Messaging.Amqp.Transport.WebSocketTransportInitiator.Complete(IAsyncResult asyncResult, Boolean completeSynchronously)
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.ServiceBus.AmqpRelay.ConnectTask.<Start>d__4.MoveNext()'
    DM.EnterpriseGateway Information: 1 : 2020-05-26T18:31:59.0454993Z DM.EnterpriseGateway 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 0000 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 33B47475 [DM.NetworkCore] ServiceEndpoint Connection status: Online, LastError: ''
    DM.EnterpriseGateway Information: 1 : 2020-05-26T18:46:35.8877851Z DM.EnterpriseGateway 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 0000 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 8F53A177 [DM.NetworkCore] ServiceEndpoint Connection status: Offline, LastError: ''

    Find below a couple of screen shots from Azure showing the errors:


    Wednesday, May 27, 2020 3:25 PM

All replies

  • Hi EduardoCMB - In the past, we've had customers successfully overcome this issue by upgrading their gateway to the latest version. Could you try that out? If that doesn't work, I recommend you create a support case and work with a support engineer who has deeper insight to your environment to troubleshoot the root cause.
    Wednesday, June 3, 2020 5:51 AM