locked
Data Factory DataFlow source DataSets? RRS feed

  • Question

  • Why can I only find/select SQL datasets for my source?   Am I missing something fundamental?  I am trying to select an Azure Storage Blob file I have defined as a DataSet.

    

    Thursday, January 23, 2020 6:01 PM

All replies

  • Hi Derek Horrall,

    Thanks for reaching out. Could you please try refreshing your ADF UI? Also please try below steps.

    • Please make sure that your Azure blob dataset exists in your Data factory.
    • Please try creating a dummy Copy activity and try if you were able to see that Azure blob data set listed in your Copy activity. If it appears, then try to create a new Data flow activity and search for the blob dataset in your Data flow activity.
    • If you still notice the same issue, please do let us know (It would be great if you can share the JSON code of your Azure Blob dataset that you are unable to list in Data flow).

    Note: ORC and Binary data format are not supported in Data flow. Please make sure your blob dataset is not of this type.

    

    Here is the doc link: Supported source connectors in mapping data flow


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered"Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.


    Thursday, January 23, 2020 8:45 PM
  • Thanks for the reply.  None of the suggestions worked.  I can see the datasets without issue using a regular copy activity in DataFactory.  However, I still only see SQL SERVER data sets in the DataFlow Source dropdown.  One thing I also tried was changing my dataset to reference a regular text file instead of a compressed file, and I still see the same phenomenon.  The DataFlow 'source' dropdown is simply not allowing me to select/find anything in Blob Storage.  Could it be some sort of access/connectivity issue between DataFlow and my Blob Storage account?

    Here is the JSON of the dataset.  I deleted most of the column names for brevity....

    {
    "name": "aa_Temp_SF_CASE",
    "properties": {
    "linkedServiceName": {
    "referenceName": "LS_ABLB_BITEAM",
    "type": "LinkedServiceReference"
    },
    "parameters": {
    "fileName": {
    "type": "String"
    }
    },
    "annotations": [],
    "type": "AzureBlob",
    "structure": [
    {
    "name": "Id",
    "type": "String"
    },

    /*...

    deleted column names for brevity

    ...*/
    {
    "name": "Pipeline_Name",
    "type": "String"
    }
    ],
    "typeProperties": {
    "format": {
    "type": "TextFormat",
    "columnDelimiter": "|",
    "rowDelimiter": "",
    "quoteChar": "\"",
    "nullValue": "\\N",
    "treatEmptyAsNull": true,
    "skipLineCount": 0,
    "firstRowAsHeader": true
    },
    "fileName": {
    "value": "tstcasetst",
    "type": "Expression"
    },
    "folderPath": "salesforce"
    }
    },
    "type": "Microsoft.DataFactory/factories/datasets"
    }


    Friday, January 24, 2020 2:51 PM
  • Hi Derek Horrall,

    Thanks much for verifying all the steps and providing the requested details. I am unable to reproduce your issue. I tried creating a Azure Blob dataset and it was listed in my Data flow activity source options. 

    For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket, else could you please send an email to AzCommunity@Microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter. 

    Subject of the email: <ATTENTION : Kranthi - MSDN Thread title>

    Thread URL: 

    Subscription ID:

    Data Factory Name: 

    Data Factory Region: 

    Please let me know here once you have done the same.


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Saturday, January 25, 2020 12:12 AM
  • Try creating a new dataset from data flow:

    1. Add a source transformation

    2. Click New Dataset

    3. Choose Blob

    ...

    Copy Activity has access to older dataset variations from the old ADF dataset model definitions that data flow cannot read.

    Saturday, January 25, 2020 1:31 AM
  • Hi Derek,

    Following up to see if you have got a chance to try the suggestion provided by mkro? Please let us know if you still notice the same issue with new dataset as well.

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Monday, January 27, 2020 5:02 PM
  • Hi Derek Horrall,

    Following up to see if your issue was resolved? In case if you have found a different solution would you please share it here with the community as it would be beneficial for other users. 

    Or if you still need assistance, do let us know. Will be happy to assist. 

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Wednesday, January 29, 2020 7:24 AM