locked
Azure ADF - ContentMD5 field in Get Metadata activity is always null RRS feed

  • Question

  • If I manually upload txt or csv files in azure blob storage, when using the get metadata activity, I always get "contentMD5": null, while the other fields of the output are always populated. I am using ADF v2

    Does anyone have any ideas? Thanks



    Tuesday, March 10, 2020 4:02 PM

Answers

  • This doesn't look like an issue with ADF but could be an issue with the client/way the file is uploaded to storage. To solve your problem, please try below:

    1. Download the file(s) which is missing the ContentMD5 value to your local machine,  then delete it from your Azure Blob Storage container (don't forget to select Delete blob snapshots)
    2.  Then re-upload the downloaded file to your Azure Blob Storage container from Azure Portal.
    3. Then open the file/blob properties and you should be able see the ContentMD5 value populated.
    4. Now try using GetMetaData activity in your ADF pipeline, and you should see the ContentMD5 value in the output of GetMetaData activity output.

    Hope this helps. Let us know if you have any further query. 


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Wednesday, March 11, 2020 3:21 PM

All replies

  • Hi Alina Ciorici,

    Thanks for reaching out. Have you noticed this behavior ("contentMD5":"null") to all your files in the storage or only particular set of files? When you mention you uploaded manually - Have you uploaded from Azure Portal? 

    Could you please verify if contentMD5 value exist in file properties from blob storage as below? If the value didn't exist then your getMetadata will return "null" since it is null in file metadata.




    Could you please try creating a new .txt file of small size and upload it to your storage from Azure Portal and try using GetMetadata to get 'contentMD5' value and let us know if the behavior is same or different? 



    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered"Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.



    Tuesday, March 10, 2020 11:40 PM
  • This doesn't look like an issue with ADF but could be an issue with the client/way the file is uploaded to storage. To solve your problem, please try below:

    1. Download the file(s) which is missing the ContentMD5 value to your local machine,  then delete it from your Azure Blob Storage container (don't forget to select Delete blob snapshots)
    2.  Then re-upload the downloaded file to your Azure Blob Storage container from Azure Portal.
    3. Then open the file/blob properties and you should be able see the ContentMD5 value populated.
    4. Now try using GetMetaData activity in your ADF pipeline, and you should see the ContentMD5 value in the output of GetMetaData activity output.

    Hope this helps. Let us know if you have any further query. 


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Wednesday, March 11, 2020 3:21 PM
  • Hi Alina Ciorici,

    Just checking in to see if the above suggestion was helpful. If this answers your query, please do click “Mark as Answer” and Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Thursday, March 12, 2020 5:31 PM
  • The issue was happening with all the files that I manually uploaded through the portal. Indeed, the blobs' property showed a null md5. deleting and re-uploading worked fine, but I don;t really understand the difference between the 2 circumstances.

    Glad it now works ok, thank you!

    Thursday, March 12, 2020 6:26 PM
  • Hi Alina Ciorici,

    Glad to be of help and thanks much for confirming that the above workaround resolved your issue. 

    I tried to reproduce your issue, but no luck. In-order to to determine why the contentMD5 value didn't populate for the first time when you uploaded through portal, I would recommend you to perform below steps to analysis if it was a one-time/transient issue when you uploaded the files. If it is consistent then a deeper investigation is needed to figure out the root cause.

    1. Please try to upload couple of original files (the files which didn't have contentMD5 value populated for the first time) again to your blob storage (probably to a new folder) from portal and check if the contentMD5 value is populated or not.
    2. If the value is populated, then it concludes that there might be a transient issue with the backend storage service that populates contentMD5 when you uploaded them for the first time.
    3. If the value is not populated, then issue remains consistent and a deeper investigation is required to analysis the root cause. If you notice consistency in this behavior then you may file a support ticket if you have a support plan, else please send an email to AzCommunity@Microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter.

      Email subject: <Attn - Kranthi : MSDN Thread title>
      Thread URL: <MSDN Thread>
      Subscription ID:  <your subscription id>

    Please keep me posted here about your findings. 


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Thursday, March 12, 2020 7:21 PM
  • I tried several times to re-upload the original files, some of the times with overwrite, some with delete and upload, i can always see the content md5 in the property now, so I can;t reproduce the error.

    If it will happen again, I will email you.

    Thanks a lot for the help!

    Friday, March 13, 2020 8:47 AM
  • Hi Alina Ciorici,

    Thanks much for your response. So this confirms that it could be a transient issue with blob backend service while you uploaded the files for the first time. 

    But feel free to reach out, if you notice this behavior again. We will be happy to assist.



    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered"Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.



    Friday, March 13, 2020 7:43 PM