locked
In ADF Azure Function returns valid JObject and still fails with 'errorCode 3610' : 'There was an error while calling endpoint' RRS feed

  • Question

  • I am trying to run a python HttpRequest Azure Function in Azure Data Factory. The app when tested locally successfully returns valid JSON. When deployed, the result is the same. When I try to call it in Azure Data Factory, the following error is returned. 

    {
        "errorCode": "3610",
        "message": "There was an error while calling endpoint.",
        "failureType": "UserError",
        "target": "fme-activity",
        "details": []
    }

    I have followed the steps in the guide below. I adjusted my code so that my function returns a json object as ADF expects.  Still, I run into the same issue.

    https://microsoft-bitools.blogspot.com/2019/01/introducing-azure-function-activity-to.html

    My code looks like the following:

    __init__.py

    import logging
    import json
    
    import azure.functions as func
    
    def main(req: func.HttpRequest) -> func.HttpResponse:
        logging.info('Python HTTP trigger function processed a request.')
    
        name = req.params.get('name')
        if not name:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                name = req_body.get('name')
    
        if name:
            myobj = {"testsubject":name}
            return func.HttpResponse(
                json.dumps(myobj),
                mimetype="application/json",
            )
        else:
            return func.HttpResponse(
                 "Please pass a name on the query string or in the request body",
                 status_code=400
            )

    function.json

    {
      "scriptFile": "__init__.py",
      "bindings": [
        {
          "authLevel": "anonymous",
          "type": "httpTrigger",
          "direction": "in",
          "name": "req",
          "methods": [
            "get",
            "post"
          ]
        },
        {
          "type": "http",
          "direction": "out",
          "name": "$return"
        }
      ]
    }

    In ADF I connect the azure function via secrets and key vaults. My application has access permissions to use the key vault as well. Below are the details from the last run.

    Run Id : a0b4235d-d19c-4a7d-a5aa-01cd1d2f270d

    Pipeline run ID: b12f2709-26ba-4630-961d-134bc25915d3

    Any help is much appreciated,

    Thank You!

    Sam



    • Edited by samkg Friday, February 21, 2020 9:24 PM Spelling
    Friday, February 21, 2020 9:20 PM

Answers

  • Hello Sam Guerrero, I noticed one difference in our setups.

    Your URL: https://fme-trasform.cloudgis-ase-ilb.appserviceenvironment.net/

    My URL: https://functionfactory.azurewebsites.net

    The suffix is different.  AppServiceEnvironment seems to belong to Azure Web App.  I see this may come into play if you are using Private Site Access or Load Balancer.  However I am not certain that is the appropriate endpoint to use.  Can you try using the URL generated by the function app service?  (See below)

    • Marked as answer by samkg Friday, February 28, 2020 7:47 PM
    Thursday, February 27, 2020 6:25 AM
  • Hello,

    You were right, the website was not accessible. It was to be designed  to be inaccessible. The remedy was to use another azure function that is public to expose the private one to the Azure Data Factory. 


    Sam Guerrero

    Friday, February 28, 2020 7:50 PM

All replies

  • Hello,

    This article explores common troubleshooting methods for external control activities in Azure Data Factory.

    Error code: 3610

    Message: There was an error while calling endpoint.

    Cause: Function URL may be incorrect.

    Recommendation: Please make sure the value for 'functionAppUrl' in the activity JSON is correct and try again.

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    Monday, February 24, 2020 6:08 AM
  • Hello,

    I have reviewed your article and investigated the JSON for this application. Everything is spelled correctly. The JSON is cdorrect when the ADF validator is run. I am still getting the same error.

    {
        "name": "test-fme-transform-trigger",
        "properties": {
            "activities": [
                {
                    "name": "fme-activity",
                    "type": "AzureFunctionActivity",
                    "dependsOn": [],
                    "policy": {
                        "timeout": "7.00:00:00",
                        "retry": 0,
                        "retryIntervalInSeconds": 30,
                        "secureOutput": false,
                        "secureInput": false
                    },
                    "userProperties": [],
                    "typeProperties": {
                        "functionName": "Transform",
                        "method": "POST",
                        "body": {
                            "name": "Sam"
                        }
                    },
                    "linkedServiceName": {
                        "referenceName": "fme",
                        "type": "LinkedServiceReference"
                    }
                }
            ],
            "folder": {
                "name": "PipelineTesting"
            },
            "annotations": []
        }
    }
    Is there a way to check if the Azure Data Factory has the correct access or permissions? The function app is "Anonymous", would that be enough to stop it from being accessible to the ADF?


    Sam Guerrero


    • Edited by samkg Monday, February 24, 2020 11:49 PM spelling
    Monday, February 24, 2020 11:48 PM
  • Hello Sam Guerro, I will be taking a second look into your issue.  Thank you for the detailed repro information.  I will let you know when I have more information.
    Tuesday, February 25, 2020 7:24 PM
  • Hello Sam Guerro, let me share my findings.

    First I put together the app using your code and deployed from VSCode, and set it up in Data Factory.  All went well.  Since I didn't run into problems there, I figured I should try changing things until I got your error.  Here is what I found:

    • Typo in Function App URL causes Error Code 3610.
    • Typo in Function Key does not cause error when Function is set to Anonymous.  
    • Typo in Function Name causes Error Code 3608.
    • Using HTTP Method not coded for causes Error Code 3608
    • Forgetting to give Key Vault permissions to Data Factory results in ErrorCode 2011

    Could you please share the Linked Service?  Are you using a Self-Hosted Integration Runtime, or are you using the Default IR?

    Tuesday, February 25, 2020 9:06 PM
    • Typo in Function App URL causes Error Code 3610.

             - I have tested the URL in postman and in a google chrome browser, the URL is accessible via these clients. 

    • Typo in Function Key does not cause error when Function is set to Anonymous.  

             - I have set the function to 'Anonymous' as shown below in the attached images. Still the function fails with code 3610.

    • Typo in Function Name causes Error Code 3608.

             - I have provided an image below that shows what I have been debugging, In this image you can see that the function name is correct.

    • Using HTTP Method not coded for causes Error Code 3608

             - The function is coded for "httpTrigger", It supports GET and POST methods. You can see this in the attached function.json below I have attached.

    • Forgetting to give Key Vault permissions to Data Factory results in ErrorCode 2011

             - I will see how to implement this, I am not sure, this function specifically has access to the key vault. I am not sure if the data factory itself has access to the key vault.


    My function json updated to be 'anonymous' as per your observation mentioned above.

    {
      "scriptFile": "__init__.py",
      "bindings": [
        {
          "authLevel": "Anonymous",
          "type": "httpTrigger",
          "direction": "in",
          "name": "req",
          "methods": [
            "get",
            "post"
          ]
        },
        {
          "type": "http",
          "direction": "out",
          "name": "$return"
        }
      ]
    }

    My linked service as it is represented in the ADF.


    Below is what the activity looks like in the ADF pipeline. It is what I have been debugging.


    Thank you very much,

    Sam Guerrero


    • Edited by samkg Wednesday, February 26, 2020 6:58 PM
    Wednesday, February 26, 2020 6:55 PM
  • Hello Sam Guerrero, I noticed one difference in our setups.

    Your URL: https://fme-trasform.cloudgis-ase-ilb.appserviceenvironment.net/

    My URL: https://functionfactory.azurewebsites.net

    The suffix is different.  AppServiceEnvironment seems to belong to Azure Web App.  I see this may come into play if you are using Private Site Access or Load Balancer.  However I am not certain that is the appropriate endpoint to use.  Can you try using the URL generated by the function app service?  (See below)

    • Marked as answer by samkg Friday, February 28, 2020 7:47 PM
    Thursday, February 27, 2020 6:25 AM
  • Hello,

    I believe I am using the URL that is generated by the function app service. In my image below you can see that the url in my data factory linked service is the same as my function app overview.


    If I am using Private Site Access or a Load Balancer, how do I give my data factory access to use this URL? The function application and the data factory already uses 'access policies' to retrieve secrets from the key fault. They have 'GET' and 'LIST' permissions. Is there a further step I am missing to let my data factory access this URL? How is it that my data factory can not access the azure function even thought they are both on the same VNET? 


    Sam Guerrero

    Thursday, February 27, 2020 6:33 PM
  • Hello Sam Guerrero,

    Marin has a point here , when i just did a simple ping , we have a resolution for Martins azure function , but none for your . Can you please make the changes which Martin suggested and see if that helps ?

    ping fme-trasform.cloudgis-ase-ilb.appserviceenvironment.net
    Ping request could not find host fme-trasform.cloudgis-ase-ilb.appserviceenvironment.net. Please check the name and try again.

    ping functionfactory.azurewebsites.net

    Pinging waws-prod-bay-063.cloudapp.net [13.93.220.109] with 32 bytes of data:
    Request timed out.
    Request timed out.
    Request timed out.

    Ping statistics for 13.93.220.109:
        Packets: Sent = 3, Received = 0, Lost = 3 (100% loss),


    Thanks Himanshu

    Thursday, February 27, 2020 7:12 PM
  • Hello,

    Thank you very much for your help. The issue was that it is a private site. I have used an azure function to expose the URL in a proxy, so that Azure Data Factory can use this api.


    Sam Guerrero

    Friday, February 28, 2020 7:48 PM
  • Hello,

    You were right, the website was not accessible. It was to be designed  to be inaccessible. The remedy was to use another azure function that is public to expose the private one to the Azure Data Factory. 


    Sam Guerrero

    Friday, February 28, 2020 7:50 PM