none
DELETE activity no longer supporting delimited text files on Blob RRS feed

  • Question

  • When the DELETE activity was first released I started using it to delete delimited csv and text files from Blob storage after having copied them to an Azure SQL Database. Now that the datasets that reference delimited files are no longer classified as 'Blob' datasets but are instead being called 'Delimited' datasets, the DELETE activity doesn't recognise them as a valid source. Is there any way around this?
    Wednesday, May 15, 2019 3:34 PM

Answers

  • Hi Matt,

    DelimitedText is not a supported dataset type for Delete Activity. In your scenario, you need to:

    1. Create dataset from Delete activity dataset selection tab

    2. Select a data store type, select Binary format type

    3. Go to the newly created dataset, uncheck the 'Binary copy' checkbox, then you could specify any format as you want in this dataset.

    Please note that we have upgraded dataset with format setting to DelimitedText dataset, while Delete activity hasn't support DelimitedText (which is under development). This is why we can't find any delimited text dataset in the Delete activity dataset selection. 

    Thanks.

    Wednesday, May 22, 2019 2:43 AM

All replies

  • Hello Matt , 

    The delete activity is deleting the whole file . We understand that the file initially existed and then it was deleted at some point . Please try to validate the pipeline and it should work fine . You can also select the binary copy and it should work fine . 

    Let us know how it goes .



    Thanks Himanshu

    Thursday, May 16, 2019 8:14 PM
    Moderator
  • The problem was occurring when selecting the dataset from the Delete activity; the dataset was simply not showing in the drop down. For some reason the dataset in question is now showing fine, however, when I create a new dataset I have the same problem. Here are the steps I am following:

    • Create new dataset: Azure Blob Storage
    • Select format: DelimitedText
    • I then select a csv file on blob storage

    In the properties of this new dataset I have the following code:

    "type": "DelimitedText",

    Whereas in my older Blob datasets I have this code:

    "type": "AzureBlob",

    Friday, May 17, 2019 2:27 PM
  • As suggested , can you please let us know as to selected the format as "Binary" helps , we understand it should.

    Thanks Himanshu

    Monday, May 20, 2019 10:26 PM
    Moderator
  • I take it when you say "select the binary copy" that you mean I should be selecting the format of the Blob dataset as being 'Binary'? As the file is a csv I should be selecting 'CSV' so I take it this is some kind of temporary workaround?

    Regardless, I have tried selecting the format of the dataset as 'Binary' and, even when published, I cannot select this dataset from the drop-down in the Delete activity. The dataset type JSON for this dataset is showing as:

    "type": "DelimitedTextSource",

    Clearly the improvements made in dataset creation have resulted in new dataset types such as 'DelimitedText' and 'DelimitedTextSource' that aren't being shown in the Delete activity because their type is not 'AzureBlob'

    Tuesday, May 21, 2019 7:57 AM
  • Hi Matt,

    DelimitedText is not a supported dataset type for Delete Activity. In your scenario, you need to:

    1. Create dataset from Delete activity dataset selection tab

    2. Select a data store type, select Binary format type

    3. Go to the newly created dataset, uncheck the 'Binary copy' checkbox, then you could specify any format as you want in this dataset.

    Please note that we have upgraded dataset with format setting to DelimitedText dataset, while Delete activity hasn't support DelimitedText (which is under development). This is why we can't find any delimited text dataset in the Delete activity dataset selection. 

    Thanks.

    Wednesday, May 22, 2019 2:43 AM
  • Thank you, this workaround has been successful, I hope proper functionality will be restored soon.

    It is somewhat frustrating that the development cycle for Azure products means upgrades are being made to different areas seemingly without thinking of how this will affect things as a whole; for 'Delete' to be added as an activity in March and work perfectly with csv files on Blob only for it to cease working without a workaround just two months later is pretty poor.

    Friday, May 24, 2019 7:53 AM
  • Hi Matt,

    I'm so sorry about the confusion caused by a improvement of Dataset. We have received similar feedback and are actively working in progress to resolve this design flaw.

    You may or may not have noticed that ADF portal underlying generates a slightly different Dataset JSON payload for CSV for new features, like Mapping Data Flow (Public Preview), hive-like partition folder (backlog) and rich data type support. If CSV Dataset could describe partition folder, it may look less like a collection of folders / files than a partitioned table, so that we intentionally removed its support from Delete activity. Hope this explains.

    Thanks for your feedback and continuous investment on ADF. Your feedback is definitely critical, so please don't hesitate to let us know anything not making sense.

    Thanks
    Keynes

    Sunday, June 2, 2019 3:56 AM