none
Data Move : UserErrorInvalidHttpUrlAddress , error in passing parameter values to URL RRS feed

  • Question

  • Error UserErrorInvalidHttpUrlAddress

    I'm trying to pass pipelines parameter values into a Relative Uri , from the error message the Uri seems ok except from the two ' wrapping it . 

    Is there anything I'm missing ?

    { "errorCode": "2200", "message": "ErrorCode=UserErrorInvalidHttpUrlAddress, 'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException, Message=Http Url Address 'http://www.mysite.it%2Fportal%2Fdocuments%2F20182%2F241820%2Farchive.csv' is invalid., Source=Microsoft.DataTransfer.ClientLibrary,'", "failureType": "UserError", "target": "Data Move HTTP" }

    Following the guides I've tested both cases :

    (1) entering parameter expression in Relative URI ,

    (2) entering it in Base URI , the result is the same


    Saturday, July 13, 2019 9:26 AM

All replies

  • We are assuming that you are using a web activity and we did tried the browse a yahoo article ( keep the yahoo.com as the base ) and the rest as parameter and it worked fine . We used concatenate function for appending the urls . 

    Let us know how it goes .

    {
        "name": "pipeline2",
        "properties": {
            "activities": [
                {
                    "name": "Web1",
                    "type": "WebActivity",
                    "dependsOn": [],
                    "policy": {
                        "timeout": "7.00:00:00",
                        "retry": 0,
                        "retryIntervalInSeconds": 30,
                        "secureOutput": false,
                        "secureInput": false
                    },
                    "userProperties": [],
                    "typeProperties": {
                        "url": {
                            "value": "@concat('https://news.yahoo.com/',pipeline().parameters.someurl)",
                            "type": "Expression"
                        },
                        "method": "GET"
                    }
                }
            ],
            "parameters": {
                "someurl": {
                    "type": "string",
                    "defaultValue": "/trump-says-congresswomen-he-insulted-should-apologize-to-him-132250027.html"
                }
            },
            "annotations": []
        }
    }


    Thanks Himanshu

    Monday, July 15, 2019 7:31 PM
    Moderator
  • Just wanted to check , if you had a resolution to the issue which you were facing . If you had a different resolution , we request you to share  that with the community here .

    Thanks Himanshu

    Tuesday, July 16, 2019 8:42 PM
    Moderator
  • Thanks Himanshu for the reply.

    Actually I wasn't able to post the pictures and file as attachment, so I'm posting them again.

    I'm not using the web activity but the copy activity.

    The pipeline is a simple "Copy" activity, where I pass as parameter the full URI of the file I want to get into ADLS Gen2 in parquet format. So I used a Generic CSV datasource, vith a Generic HTTP connection where I'm trying to pass the pipeline parameter "URI" in cascade down to the datasource and the connection

    Here the three pieces of code .

     

    Pipeline code , only one block : Data Move of t

    {
        "name": "HTTP_Generic_Ingestion",
        "properties": {
            "description": "Generic Ingestion Pipeline with HTTP URI via parameters",
            "activities": [
                {
                    "name": "Data Move HTTP",
                    "type": "Copy",
                    "dependsOn": [],
                    "policy": {
                        "timeout": "7.00:00:00",
                        "retry": 0,
                        "retryIntervalInSeconds": 30,
                        "secureOutput": false,
                        "secureInput": false
                    },
                    "userProperties": [],
                    "typeProperties": {
                        "source": {
                            "type": "DelimitedTextSource",
                            "storeSettings": {
                                "type": "HttpReadSetting",
                                "requestMethod": "GET"
                            },
                            "formatSettings": {
                                "type": "DelimitedTextReadSetting"
                            }
                        },
                        "sink": {
                            "type": "ParquetSink",
                            "storeSettings": {
                                "type": "AzureBlobFSWriteSetting"
                            }
                        },
                        "enableStaging": false
                    },
                    "inputs": [
                        {
                            "referenceName": "HTTP_Generic_Source_csv",
                            "type": "DatasetReference",
                            "parameters": {
                                "uri": "@pipeline().parameters.uri"
                            }
                        }
                    ],
                    "outputs": [
                        {
                            "referenceName": "Generic_Parquet",
                            "type": "DatasetReference",
                            "parameters": {
                                "organization": "@pipeline().parameters.organization",
                                "filename": "@pipeline().parameters.filename"
                            }
                        }
                    ]
                }
            ],
            "parameters": {
                "filename": {
                    "type": "string"
                },
                "uri": {
                    "type": "string"
                },
                "organization": {
                    "type": "string"
                }
            },
            "annotations": []
        },
        "type": "Microsoft.DataFactory/factories/pipelines"
    }

    Here the HTTP_Generic_Source_csv

    {
        "name": "HTTP_Generic_Source_csv",
        "properties": {
            "description": "HTTP Generic Source CSV format , uri passed via parameters",
            "linkedServiceName": {
                "referenceName": "GenericHTTP",
                "type": "LinkedServiceReference"
            },
            "parameters": {
                "uri": {
                    "type": "string",
                    "defaultValue": "@pipeline().parameters.uri"
                }
            },
            "annotations": [],
            "type": "DelimitedText",
            "typeProperties": {
                "location": {
                    "type": "HttpServerLocation"
                },
                "columnDelimiter": ",",
                "escapeChar": "\\",
                "firstRowAsHeader": true,
                "quoteChar": "\""
            },
            "schema": []
        },
        "type": "Microsoft.DataFactory/factories/datasets"
    }

    And the GenericHTTP linked service

    {
        "name": "GenericHTTP",
        "type": "Microsoft.DataFactory/factories/linkedservices",
        "properties": {
            "description": "Generic HTTP Get for Open Data ( Anonymous Connection )",
            "annotations": [
                "This is a generic http connect via params"
            ],
            "type": "HttpServer",
            "typeProperties": {
                "url": "@dataset().uri",
                "enableServerCertificateValidation": false,
                "authenticationType": "Anonymous"
            }
        }
    }

    Sunday, July 21, 2019 12:32 AM
  • Thanks for the reply , I see that you are already following the doc https://docs.microsoft.com/en-us/azure/data-factory/connector-http .

    I am assuming that you are running a IR ( intermediate run-time) , can you cross check and confirm that ADF is able to access the url  ?

    To check that you can just pull an web activity on the canvas and pass the full url and check that .  


    Thanks Himanshu

    Tuesday, July 23, 2019 12:54 AM
    Moderator
  • Yes is an IR

    The URI is accessible for sure, as if I use it as default value in the Connection URI it works

    No problems with the "Web" Activity , the problem seems to be with my specific use case where the parameter is passed down to the DataSet definition.


    You can try using this public open data file as parameter for my specific use case posted in the previous post

    https://www.anpr.interno.it/portale/documents/20182/241820/ANPR_archivio_comuni.csv


    Tuesday, July 23, 2019 3:10 PM
  • Thanks Andy for sharing the  , I was able to make it work ( I had the sick as blob , but that should not matter ) . When I see my JSON I see the below think which is different and that may be the reason . Can you please check at your end . 

                    "inputs": [

                        {
                            "referenceName": "DelimitedText3",
                            "type": "DatasetReference",
                            "parameters": {
                                "url1": {
                                    "value": "@pipeline().parameters.uri",
                                    "type": "Expression"
                                }
                            }
                        }
                    ],
        
        
        this is what you have
        
         "inputs": [
                        {
                            "referenceName": "HTTP_Generic_Source_csv",
                            "type": "DatasetReference",
                            "parameters": {
                                "uri": "@pipeline().parameters.uri"
                            }
                        }
                    ],
        
        I am adding the screenshot also hopefully this may help you . Let me know how this goes .


        
        


    Thanks Himanshu

    Tuesday, July 23, 2019 7:37 PM
    Moderator
  • Thanks ,

    Unfortunately, even if I'm now using the Relative Url correctly the get operation fails

    {
        "errorCode": "2200",
        "message": "ErrorCode=UserErrorInvalidHttpUrlAddress,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Http Url Address 'https://www.anpr.interno.it/portale/documents/20182/241820/ANPR_archivio_comuni.csv' is invalid.,Source=Microsoft.DataTransfer.ClientLibrary,'",
        "failureType": "UserError",
        "target": "Ingest HTTP"
    }

    The construct of the Url is correct as you can see in the above error response

    However here the code of the HTTP_Generic_Source_csv DataSet definition

        "name": "HTTP_Generic_Source_csv",
        "properties": {
            "description": "HTTP Generic Source CSV format , uri passed via parameters",
            "linkedServiceName": {
                "referenceName": "GenericHTTP",
                "type": "LinkedServiceReference"
            },
            "parameters": {
                "uri": {
                    "type": "string",
                    "defaultValue": "@pipeline().parameters.uri"
                }
            },
            "annotations": [],
            "type": "DelimitedText",
            "typeProperties": {
                "location": {
                    "type": "HttpServerLocation",
                    "relativeUrl": {
                        "value": "@dataset().uri",
                        "type": "Expression"
                    }
                },
                "columnDelimiter": ",",
                "escapeChar": "\\",
                "firstRowAsHeader": true,
                "quoteChar": "\""
            },
            "schema": []
        },
        "type": "Microsoft.DataFactory/factories/datasets"
    }

    The way I pass parameters is exactly as per the tutorial




    Tuesday, July 23, 2019 11:23 PM
  • We did redo the thing again and it always works for me , at this time wethink somebody needs to look into the ADF and figure out whats not right .
    Can you please send an email to azcommunity@microsoft.com  with the following details .

    1. Your subscription id
    2. Link to this thread .


    Thanks Himanshu

    Tuesday, July 30, 2019 10:28 PM
    Moderator
  • Just wanted to know if the issue is resolved and if yes , can you please let the community know  the resolution ? Otherwise do send an email to us and we will try to help you with this .

    Thanks Himanshu

    Wednesday, July 31, 2019 4:54 PM
    Moderator