none
fast way to upload files on data lake store using .net SDK

    Question

  • Hi,

    I am using Azure .net SDK to upload files on Azure Data lake store.

    Its working fine. But its working very slow, as i have to upload around 10,000 images to data lake store. So it's taking very long time to upload files on Data lake store.

    Can you please suggest any alternate way or method to upload huge number of files on data lake store with fast performance.

    Also please suggest if there is any other out of box feature to ingest images (specifically) into ADLS. I tried Event hub but this seems to be for messages only. Azure Data factory seems to be working for SQL Data type of ingesting.

    Please suggest on this.

    Thanks

    Rajni

    Friday, April 28, 2017 11:07 AM

All replies

  • Hi Rajni,

    Sorry for the delay in responding. 

    I am not clear on where the messages are located. Assuming they are on your local computer, you could use PowerShell or CLI v2.0 which will help parallelize the uploads for you.

    If they are located in Azure Storage or other object store, Azure Data Factory will be the best tool.

    The pointers can be found on https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-data-scenarios 

    If you can please provide more details, I can guide better.

    Thanks,
    Sachin Sheth
    Program Manager,
    Azure Data Lake.

    Tuesday, June 13, 2017 6:44 PM