none
Too Many Files

    Question

  • We are getting a compile error when trying to process 6000+ files using a custom extractor and script.  The compile error indicates that the resulting script is too complex for the compiler and so it shuts down with a time out.  Surely ADLA can handle more than 6000 files, can't it?  I processed half that many with no problem other than it took about 2 hours to run.  Why is 6000 too many?

    Brian L Custer

    Thursday, August 10, 2017 8:26 PM

Answers

  • Dear Brian

    The current version is indeed having a limit of about 3000 to 5000 files using filesets. However, we have a feature in preview that will allow you to scale this to several 10ks of files. The details are provided here on how to enable the preview feature. The feature will soon be available in released form as well.

    In short, the preview feature can be turned on by adding the following statement to your script:

    SET @@FeaturePreviews = "FileSetV2Dot5:on";
    

    I hope this helps!


    Michael Rys

    Wednesday, August 16, 2017 11:16 PM
    Moderator

All replies

  • Dear Brian

    The current version is indeed having a limit of about 3000 to 5000 files using filesets. However, we have a feature in preview that will allow you to scale this to several 10ks of files. The details are provided here on how to enable the preview feature. The feature will soon be available in released form as well.

    In short, the preview feature can be turned on by adding the following statement to your script:

    SET @@FeaturePreviews = "FileSetV2Dot5:on";
    

    I hope this helps!


    Michael Rys

    Wednesday, August 16, 2017 11:16 PM
    Moderator
  • Thanks Michael.  We also encountered long processing times for more than a few files but was told by Microsoft to send our largest file to ADL storage instead of blob storage.  This makes a lot of sense now that I think about it since ADL has compute resources behind its storage where blob store does not.  Thanks again for your answer.

    Brian L Custer

    Wednesday, August 16, 2017 11:45 PM