none
Large number of files in system folder

    Question

  • Hello,

    I use Data Factory to run Data Lake Analytics jobs on my Data Lake Store. I have to give read/write/execute permissions to all files in the system folder to the user otherwise Data Factory / Data Lake Analytics throws an error about needing to access a system file. While updating the Data Lake Store access permissions for the system folder I see over a 100,000 files and the assign permissions operation takes a very long time to complete (about 30 minutes). I only have around 10,000-20,000 files in non-system folders. I noticed that there are a lot of system files when first setting up the data lake, and the number of files keeps growing over time. I'm worried that eventually I won't be able to assign permissions if there are too many, and I'm worried about paying for the storage of all of these.

    Is it normal to have so many system files? Do I need to manually cleanup system files?
    Friday, July 21, 2017 8:46 PM

All replies