none
Get Meta data Structure issue RRS feed

  • Question

  • Hi Team,

    We get the structure within the file in Get meta data activity whenever the dataset is a defined dataset: meaning the file path and the file is specified explicitly.

    But when we make the dataset dynamic as shown below,we do not get the file structure.

    Issue:

    So is this a known issue or do I need to make some additional modifications to get the structure?


    Wednesday, May 15, 2019 7:30 AM

All replies

  • Hi Nandan,

    Structure parameter is used to auto-detect the columns in a particular dataset. Hence, irrespective of whether you use a fixed path or a dynamic path, you should ideally see something like this :

    "structure": [ { "name": "description", "type": "String" }, { "name": "industry", "type": "String" }, { "name": "level", "type": "String" }, { "name": "size", "type": "String" }, { "name": "linecode", "type": "String" }, { "name": "value", "type": "String" } ]

    The only things that affect the output of get metadata activity lie within the dataset. Some of the factors include :

    • First row of the dataset(file in your case).
    • Delimiters used to separate the columns.
    • Delimiters used to separate rows.

    Hence, to get consistent output, I would recommend you to configure these settings in your datasets. An easy way to do so is by clicking on "Detect Text Format" button which automatically detects all the settings and then by clicking "Preview Data" to make sure the settings are fine. Please see below screenshot for reference :

    Hope this helps.

    Wednesday, May 15, 2019 10:21 AM
    Owner
  • Hello Chirag,

    Since the dataset is dynamic,the same dataset is being used for different files like:

    Source file path and file name parameters in the dynamic query can have different values pertaining to 20190228-product.csv or 20190331-Account.csv

    You were able to detect the text format since your file is fixed but we cannot use that option for the dynamic dataset.

    By dynamic dataset and static dataset we are able to get all the values same except for the structure.

    So wanted the ask whether that is a bug or I am missing out something.

    Wednesday, May 15, 2019 11:23 AM
  • Hi Nandan,

    I understood that. If you have a fixed structure in terms of column and row delimiters, you can manually also configure these settings. Once that's done, you should be good.

    Wednesday, May 15, 2019 3:29 PM
    Owner
  • Hello Chirag,

    The files are manually uploaded by the users.So we have to validate the headers before reading the file via polybase in Azure data warehouse.

    So in case if I manually input the structure,then even if the file has some other structure ; my validation would be successful which shouldnt be the case.

    My use case as below:

    We have a configuration table wherein we have the headers that our file should have.

    Then based on the metadata of the actual file,I need to compare the metadata and the table output.

    In case if it is same ,then proceed with the dataflow else throw an error stating that the file provided in in correct.

    So how do you foresee I proceed?

    Thursday, May 16, 2019 3:28 AM
  • Hi Nandan,

    Thanks for providing more info. I would recommend you to have a Get Metadata to read the columns of the file. Make sure you check the structure checkbox in the field list setting of the dataset as shown in the screenshot below.

    The output of this get metadata activity would be in the form below :

    { "structure": [ { "name": "description", "type": "String" }, { "name": "industry", "type": "String" }, { "name": "level", "type": "String" }, { "name": "size", "type": "String" }, { "name": "linecode", "type": "String" }, { "name": "value", "type": "String" } ], "effectiveIntegrationRuntime": "DefaultIntegrationRuntime (East US)", "executionDuration": 75 }

    You can chain an if condition activity to use the result of this get metadata activity to compare the structure against the one from your configuration table. Based on the if condition, you can have activities as desired.

    Hope this helps.

    Monday, May 20, 2019 8:49 AM
    Owner
  • Hello Chirag,

    That's what I have done but the point is we get the structure value only if the dataset is hardcoded to a file.

    As shown above,if the dataset is a dynamic one ; then the structure value is not populated properly.

    Please see above the 2 images w.r.t getmetadata values (one for hardcoded dataset and another for dynamic dataset :both mapping to the same files).

    So in the dynamic dataset scenario: what should be my approach?

    Wednesday, May 22, 2019 4:20 AM
  • Hi Nandan,

    In that case, better to use a custom activity, which offers much more flexibility. To read more about it, please refer this doc.

    Wednesday, May 22, 2019 6:50 AM
    Owner