none
HTTP-triggered Azure Function (C#) not called and servers scalated with no (apparent) reason RRS feed

  • Question

  • Hello,

    As reported to @AzureSupport on Twitter (https://twitter.com/jaimeml/status/1021393642001494016), on an HTTP-triggered Azure Function we missed some calls this morning. Almost at the same time of the missed calls we see a "Starting Host" message, with no other call involved. Why the scalatation? Why did we miss at least one call?

    At 1:01:12 am (all times GMT +2) we got a valid call.

    After that, a new call (notification from a third-party service) was supposed to arrive at 1:02:49, but it didn't arrive. Our partner reports a "Read timed out", as if it couldn't reach the Azure Function endpoint or it didn't reponded on time.

    In application insights we cannot see any error, but the logged information reports the following:

    - Call at 1:01:12 am has been received and properly managed.

    - Call at 1:02:49 does not appear, nor an error either, in the application insights data.

    - At 1:03:57 a new cloud instance (Host) seems to be created. Looks like an scalation, but why? There seems not be high CPU usage, no new calls... But we see a message "Starting Host", and the its setup, that lasts 6 seconds aprox.

    Could you please help?

    Thanks.


    Monday, July 23, 2018 2:08 PM

All replies

  • Can you provide the following information about your Function app?

    Monday, July 23, 2018 7:41 PM
    Moderator
  • Can you provide the following information about your Function app?

    Hello,

    • The region is West Europe.
    • The name of the function is redsys-notification-webhook.

    This strange behaviour is happening to us from time to time: missing calls -> function not triggered -> scalation at the same time (cannot say which one is the cause and which one the consequence here), thus we are missing important notifications from a payment gateway.

    Thanks for helping, we appreciate that.

    Tuesday, July 24, 2018 6:19 AM
  • Thanks for that! We are investigating and will update you with any findings. 
    Tuesday, July 24, 2018 8:41 PM
    Moderator
  • With regards to your HTTP trigger Function named reds***-***-***, we looked at our logs for the timeframe of 2018-07-22 01:00 to 2018-07-22 02:00 UTC and located that 2 HTTP requests came in at the following times:

    Our Functions logs also indicated that the 2 requests were picked up and logged by your Function when they arrived.

    We do not see any observable dropped requests. For the requests that were not triggered, how are they being sent? Is it possible that there was an issue on the sender client side?  Or do we need to search for a different UTC time and date?




    • Edited by Ling Toh Friday, July 27, 2018 10:40 PM
    Thursday, July 26, 2018 6:51 PM
  • Hello Ling,

    I'm afraid dates and times are not correct. Sorry for misleading you.

    The right ones can be observed in the attached image (2018-07-22T23:01:12.915):

    

    A yellow mark has been added to show separation between the last received call and the server scalation. The missing call / dropped request should be there.

    Thanks again.

    Friday, July 27, 2018 10:14 AM
  • It looks like there were 2 requests that came in right after the successful execution of your Function at 2018-07-22T23:01:12.915 but they both failed with a 502.

    Could the client that is sending the request implement a retry? 


    • Edited by Ling Toh Friday, July 27, 2018 10:40 PM
    Friday, July 27, 2018 10:40 PM
  • Any update on this? 
    Monday, July 30, 2018 11:19 PM
    Moderator
  • Hello Ling and Micah,

    Sorry for the delay on the answer. I didn't notice yours and have been out of the office for a couple of weeks.

    We are afraid to say that the client sending the request has no intention of implementing a retry. We wish they would make it, but for the moment it's not an option.

    Is there anything else that could be done?

    On the other hand, why don't we see those failing requests? Or, can we check them anywhere?

    And, why was a new instance instantiated?

    Thanks again,

    Jaime

    Thursday, August 16, 2018 7:41 AM