Parallel loading of log files into BLOB storage


  • Here are some more details on the data we need to get into BLOB storage. there can be 200-300 log files per test being receiving in parallel from multiple test machines. The logs are semi-structured and vary in size from 50 - 200 mb per file. A received an answer to another question terms of "chunking" and moving blocks of each individual file in parallel. However, my question now is, we will start multiple instances of the multi-threaded program loading the files, is there anything in the azure ecosystem that will "manage" the multiple "loaders" in terms of partitioning the files to be loaded across the multiple loaders to avoid duplication/conflicts/etc or do we need to write software to manage this ?

    Tuesday, November 24, 2015 2:46 PM


All replies