ADF Copy to Blob is ok, but how to copy to File Directory?

Question

I try to Bulk Copy from SQL Server to Data Lake gen2.

I have succesfully bulk copied to Block Blob, then I tried to modify data set to copy to File System. Data Set still copy files to Blob. What need to be done that ADF copy to File System so that I get files in CSV format, I preserve Hierarchy and I can open them easily?

I can see that "filesystem" is missing from location of dataset. I cannot see such setting in Azure Portal UI. How to define?

https://docs.microsoft.com/en-us/azure/data-factory/tutorial-bulk-copy-portal

{

"name": "Data_Lake_D_Storage",

"type": "Microsoft.DataFactory/factories/linkedservices",

"properties": {

"type": "AzureBlobFS",

"typeProperties": {

"url": "https://mydatalaked012345.dfs.core.windows.net",

"encryptedCredential": "111111111111111111111111111"

}

}

}

{

"name": "DataLake_DelimitedText1",

"properties": {

"linkedServiceName": {

"referenceName": "Data_Lake_Storage",

"type": "LinkedServiceReference"

},

"type": "DelimitedText",

"typeProperties": {

"location": {

"type": "AzureBlobFSLocation",

"folderPath": "mydirectory",

"container": "mycont"

},

"columnDelimiter": ",",

"escapeChar": "\\",

"firstRowAsHeader": true,

"quoteChar": "\""

},

"schema": []

}

}

---

Kenny_I

Answer 1

Hi Kenny,

As discussed on the other thread opened by you(link), you can store csv files in the blob storage (ADLS Gen2). This will make it easy to read in ADF as well.

This way you will be able to preserve hierarchy as well.

However, you can copy to Azure File Storage as well. Please refer this doc for more detalis.

Answer 2

I have issues with copy to File Storage and I need advice.

Is this possible to UI? Please advice? I cannot make working. Data is copied always to Block Blob.

---

Kenny_I

Answer 3

Hi Kenny,

You can do it easily from the UI. Please follow the following steps (Assuming you have a file share created already):

1. Create a new Connection(Linked Service) of type - Azure File Storage in ADF as shown in the below screenshot :
   
2. In the Host field, specify :
   \\<storage name>.file.core.windows.net\<file service name>
   As shown below :
   
3. Hit on Test Connection to verify that everything is working fine.
4. Create a new dataset of type Azure File Storage in the ADF Portal and select the format you want to copy to (Parquet, JSON, DelimitedText or csv, Avro, etc.)
5. In the Linked Service field, select the linked service created in the previous steps.
6. Once the dataset is created, you can use it as a source or sink(in your case) in your copy activities.

Hope this helps.