locked
MD5 is missing when creating csv file from azure data factory copy pipe line? RRS feed

  • Question

  • I have created a pipe line where source is synapse sp and sink in blob storage csv file. this pipe line is working fine but some how is not creating ContentMD5 on every csv. that is why I am not able to download csv from blob container using C#.

    1) how to set MD5 content header from ADF pipe line 

    2) how to download file without MD5 header from C#

    Tuesday, April 14, 2020 3:51 AM

All replies

  • Hi there,

    You can download a file by skipping MD5 validation in C#. To do so, you can set the BlobRequestOptions.DisableContentMD5Validation Property.

    Eg - 

    // Instruct the Storage Client to skip validating the MD5 hash of the content,
    BlobRequestOptions optionsWithDisableContentMD5Validation = new BlobRequestOptions() { DisableContentMD5Validation = true };
    
    using (targetStream = new MemoryStream())
    {
        testBlob.DownloadToStream(targetStream, accessCondition: null, options: optionsWithDisableContentMD5Validation);
    }

    Please have a look at the below doc for details :

    https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.storage.blob.blobrequestoptions.disablecontentmd5validation?view=azure-dotnet-legacy

    Hope this helps.


    Tuesday, April 14, 2020 2:43 PM
  • thanks for your valuable answer, what about the point number 1. why copy pipe line in adf is not creating MD5 property on csv file in blob storage. when using sink as csv in blob container in adf pipeline. 
    Wednesday, April 15, 2020 6:47 AM
  • Hi Eathan,

    I have reached out to the Team internally to gather the right info for you on the 1st question. I will update you as soon as I hear back.

    Meanwhile, were you unblocked ? Were you able to use the suggestion above to download blobs by skipping MD5 validation?

    Thursday, April 16, 2020 5:36 AM
  • Hi Eathan,

    I just heard back. In the current ADF Copy activity, we don't set ContentMD5 header now. (An exception is for small files <4MB, Storage service would set the MD5 automatically). 

    We have a feature coming up for Binary copy (with no format changes) using which you would be able to set the header as a part of data consistency.
    Thursday, April 16, 2020 5:50 AM
  • Hi there,

    Just wanted to check - was the above suggestion helpful to you? If yes, please consider upvoting and/or marking it as answer. This would help other community members reading this thread.
    Wednesday, April 22, 2020 11:30 AM