none
Copy activity, HTTP with no Base URL RRS feed

  • Question

  • I'm using the Enterprise Billing API to get usage detail with the asynchronous call (https://docs.microsoft.com/en-us/rest/api/billing/enterprise/billing-enterprise-api-usage-detail). When the data is ready, the polling response returns a URL where I can retrieve the data. The linked service requires a Base URL, but unfortunately the hostname of the URL is not constant, for example it could be 

    https://ccmreportstorageeastus3.blob.core.windows.net/ or 

    https://ccmreportstorageeastus2.blob.core.windows.net/ ...

    I've tried making the Base URL "https://" and the relative URL "ccmreport...../....", and "https://ccm" + "reportstorage..../...."

    but the activity fails with "The remote name could not be resolved".

    Is there a way for a linked service to be used that has a dynamic host name?

    Friday, February 8, 2019 9:16 PM

Answers

  • Hi Matti71,

    Yes, there's a way to make host name of the http linked service dynamically determined at copy runtime, which is known as linked service parameterization in ADF. See the steps below:

    1. Instead of using visual experience, manually typing in json definition when creating linked service as below:

    The below is an example json format for you to take a reference, the bold part is defining the parameter and passing parameter to 'url' property

    {
        "name": "HttpServer1",
        "type": "Microsoft.DataFactory/factories/linkedservices",
        "properties": {
            "parameters": {
                "baseUrl": {
                    "type": "String"
                }
            },
            "type": "HttpServer",
            "typeProperties": {
                "url": "@{linkedService().baseUrl}",
                "enableServerCertificateValidation": true,
                "authenticationType": "Basic",
                "userName": "***",
                "password": "***"
            }
        }
    }

    2. Reference the linked service created in dataset, define a dataset parameter and pass to 'baseUrl' linked service.

    3. Then in activity level, pass the dynamic value to dataset parameter 'hostName'. The dynamic host name will be pass down to linked service.

    Hope this helps.

    • Marked as answer by Matti71 Monday, February 11, 2019 5:22 PM
    Saturday, February 9, 2019 9:25 AM

All replies

  • Hi Matti71,

    Yes, there's a way to make host name of the http linked service dynamically determined at copy runtime, which is known as linked service parameterization in ADF. See the steps below:

    1. Instead of using visual experience, manually typing in json definition when creating linked service as below:

    The below is an example json format for you to take a reference, the bold part is defining the parameter and passing parameter to 'url' property

    {
        "name": "HttpServer1",
        "type": "Microsoft.DataFactory/factories/linkedservices",
        "properties": {
            "parameters": {
                "baseUrl": {
                    "type": "String"
                }
            },
            "type": "HttpServer",
            "typeProperties": {
                "url": "@{linkedService().baseUrl}",
                "enableServerCertificateValidation": true,
                "authenticationType": "Basic",
                "userName": "***",
                "password": "***"
            }
        }
    }

    2. Reference the linked service created in dataset, define a dataset parameter and pass to 'baseUrl' linked service.

    3. Then in activity level, pass the dynamic value to dataset parameter 'hostName'. The dynamic host name will be pass down to linked service.

    Hope this helps.

    • Marked as answer by Matti71 Monday, February 11, 2019 5:22 PM
    Saturday, February 9, 2019 9:25 AM
  • Excellent, thank you.
    Monday, February 11, 2019 5:23 PM