none
Outbound TCP Connection issue cause be sending data to event hub and data lake from azure function RRS feed

  • Question

  • I'm working on a Azure function with http POST trigger, once client call it and post a json data, I will send it to event hub and save to data lake. once it got hitted by the high traffic, 20k/hour, azure functino will generate high outbound TCP connection, which will exceed the limitation (1920) of the plan.

    1. does high outbound TCP connection cause by writing to event hub, data lake, or both?
    2. is there a way to reduce it so I don't have to pay more to upgrade our plan?
    3. how to debug it to trouble shooting the problem?

    here is the code of send data to event hub:

    EventHubClient ehc = EventHubClient.CreateFromConnectionString(cn);
    
    try
    {
      log.LogInformation($"{CogniPointListener.LogPrefix}Sending {batch.Count} Events: {DateTime.UtcNow}");
    
      await ehc.SendAsync(batch);
    
      await ehc.CloseAsync();
    }
    catch (Exception exception)
    {
      log.LogError($"{CogniPointListener.LogPrefix}SendingMessages: {DateTime.UtcNow} > Exception: {exception.Message}");
      throw;
    }
    

    here is the send data to data lake:

    var creds = new ClientCredential(clientId, clientSecret);
    var clientCreds = ApplicationTokenProvider.LoginSilentAsync(tenantId, creds).GetAwaiter().GetResult();
    
    // Create ADLS client object
    AdlsClient client = AdlsClient.CreateClient(adlsAccountFQDN, clientCreds);
    
    try
    {
        using (var stream = client.CreateFile(fileName, IfExists.Overwrite))
        {
            byte[] textByteArray = Encoding.UTF8.GetBytes(str);
            stream.Write(textByteArray, 0, textByteArray.Length);
        }
    
        // Debug
        log.LogInformation($"{CogniPointListener.LogPrefix}SaveDataLake saved ");
    }
    catch (System.Exception caught)
    {
        string err = $"{caught.Message}Environment.NewLine{caught.StackTrace}Environment.NewLine";
    log.LogError(err, $"{CogniPointListener.LogPrefix}SaveDataLake");
        throw;
    }
    

    Thanks,


    Wes

    Tuesday, October 9, 2018 6:44 PM

All replies

  • Hey Wes

    It looks like it's been some time since you have posted this question here. Even though you didn't get a resolution here yet, were you able to get this issue resolved through different means/methods? If so, would you kindly share the solution with us here so that others visiting this post can learn from it?
    Thursday, October 18, 2018 6:29 PM
    Moderator
  • Your code is creating a client at each invocation and will quickly run out of connections. If you check your Functions or Application Insights logs, there should be an error message indicating that has the pattern "Host thresholds exceeded [Connections]…" In that error message, there should be the following link

    https://github.com/Azure/azure-functions-host/wiki/Managing-Connections

    that will give you some guidance on how to manage your connections.

    Thursday, October 18, 2018 8:43 PM