none
WEC7 have limitation on number of files in one drive RRS feed

  • Question

  • i have FATFS drive in WEC7. I need to store a lot of files under this drive. When the file count reach around 22K files, the file system is stopped creating any new files. It throwing error saying Disk Full. But i am sure the disk is not full yet. The total disk space is 280MB and the files only 1KB per file.

    Is there any limitation on file count for WEC7 file system? Thanks!

    Wednesday, September 4, 2019 9:21 AM

Answers

  • in terms of FAT, your 8byte and 1k files are pretty close in size - try 8byte and 10k and I think your results will change.

    The problem is that the File Allocation Table is fixed sized, so once you fill it up, the folder is full.  As you found, you can create other folders which create new FATs.

    Maybe we should take a step back and address the elephant in the room - why does your app create so many files?  Isn't there a better way to manage that data?  Like a database?  I ask this because you maybe trying to solve the wrong problem.


    Bruce Eitman
    Senior Enginer
    Bruce.Eitman AT Synopsys DOT com
    My BLOG http://geekswithblogs.net/bruceeitman
    I work for
    Synopsys

    • Marked as answer by YLCheng Wednesday, September 18, 2019 9:03 AM
    Friday, September 6, 2019 12:51 PM
    Moderator
  • So, Bruce was correct; you are reading it wrong.

    The site you mentioned says "Max Files on Volume", not "per Folder".

    Wikipedia is better in explaining the maximum number of files per folder:

    Directory file set

    exFAT and the rest of the FAT family of file systems does not use indexes for file names, unlike NTFS which uses B-trees for file searching. When a file is accessed, the directory must be sequentially searched until a match is found. For file names shorter than 16 characters in length, one file name record is required but the entire file is represented by three 32-byte directory records. This is called a directory file set, and a 256 MiB sub-directory can hold up to 2,796,202 file sets. (If files have longer names, this number will decrease but this is the maximum based on the minimum three-record file set.) To help improve the sequential searching of the directories (including the root) a hash value of the file name is derived for each file and stored in the directory record. When searching for a file, the file name is first converted to upper case using the upcase table (file names are case insensitive) and then hashed using a proprietary patented algorithm into a 16-bit (2 byte) hash value. Each record in the directory is searched by comparing the hash value. When a match is found, the file names are compared to ensure that the proper file was located in case of hash collisions. This improves performance because only 2 bytes have to be compared for each file. This significantly reduces the CPU cycles because most file names are more than 2 characters (bytes) in size and virtually every comparison is performed on only 2 bytes at a time until the intended file is located.

    As you can read, there is no "hard maximum". It all depends on the filename size and probably implementation as well.

    How did you determine the file system of your partition is in fact exFAT? You may think you have exFAT, but in fact only have FAT32. Verify you really have formatted your store as exFAT.

    I fully agree with Bruce; it is just a very bad implementation that creates >22,000 files in a folder, so change that and kill the problem that way.


    Good luck,

    Michel Verhagen, eMVP
    Check out my blog: https://guruce.com/blog

    GuruCE
    Microsoft Embedded Partner
    NXP Proven Partner
    https://guruce.com
    Consultancy, training and development services.

    Interested in WEC on i.MX6?
    Get the only 100% stable and best performing i.MX6 BSP for WEC7 and WEC2013 here: https://guruce.com/imx6



    Thursday, September 12, 2019 8:03 PM
    Moderator

All replies

  • Wednesday, September 4, 2019 2:30 PM
    Moderator
  • Hi Bruce, 

    Thanks for your reply and the link. That's triggered me to dig in further on how to determine the file number limit.

    Found that the file count is limited by the cluster size.
    Rough calculation: Total Drive Size / cluster size (provided each file size is less than cluster size)

    That also depends if the file name is too long then might took 2 clusters.
    Directory also considered as a file. So, creating too much directories also use up some clusters.

    Somehow, I am not able to figure out why my WEC7 device stopped at 22K file count under a 210MB disk drive.
    I have created 1k file and 8 byte file, both stopped around 22k file count. short file name (8.3) and under root directory. But, if I create another folder, then i can continue to create files.

    Looks like there is limitation on the number of files under root directory or maybe same limitation also apply for each directory.


    Thursday, September 5, 2019 10:07 AM
  • in terms of FAT, your 8byte and 1k files are pretty close in size - try 8byte and 10k and I think your results will change.

    The problem is that the File Allocation Table is fixed sized, so once you fill it up, the folder is full.  As you found, you can create other folders which create new FATs.

    Maybe we should take a step back and address the elephant in the room - why does your app create so many files?  Isn't there a better way to manage that data?  Like a database?  I ask this because you maybe trying to solve the wrong problem.


    Bruce Eitman
    Senior Enginer
    Bruce.Eitman AT Synopsys DOT com
    My BLOG http://geekswithblogs.net/bruceeitman
    I work for
    Synopsys

    • Marked as answer by YLCheng Wednesday, September 18, 2019 9:03 AM
    Friday, September 6, 2019 12:51 PM
    Moderator
  • Hi Bruce,

    Actually the file system used in WEC7 is exFAT. My understanding is that the max number of files that can be stored in exFAT is "nearly unlimited". However, what I found is that it hit the limit just below 22K files even though there is still plenty of storage left. However, if I create a sub-folder within the main folder, I can store many more additional files but still it will also hit the same limit of 22K (even though I checked there is still plenty of storage left). Is there any explanation why does it behave this way?

    rgds

    Wednesday, September 11, 2019 6:14 AM
  • CH CHow

    Are you the original poster, YLCheng, or working with YLCheng?


    Bruce Eitman
    Senior Enginer
    Bruce.Eitman AT Synopsys DOT com
    My BLOG http://geekswithblogs.net/bruceeitman
    I work for
    Synopsys

    Wednesday, September 11, 2019 2:56 PM
    Moderator
  • CH Chow

    Please provide a link that show the "facts" like "unlimited"  - I suspect that you are misreading it.  I see things like "up to" which of course really means "less than".  I also see "per directory".

    Again, I think that you are focused on the wrong problem.  The problem is that someone implemented a bad solution to a problem that has better known solutions.


    Bruce Eitman
    Senior Enginer
    Bruce.Eitman AT Synopsys DOT com
    My BLOG http://geekswithblogs.net/bruceeitman
    I work for
    Synopsys

    Wednesday, September 11, 2019 3:03 PM
    Moderator
  • I am a colleague of YLCheng and we are working on the same issue.

    The link is in ntfs.com/ntfs_vs_fat.htm 

    Thursday, September 12, 2019 12:04 AM
  • So, Bruce was correct; you are reading it wrong.

    The site you mentioned says "Max Files on Volume", not "per Folder".

    Wikipedia is better in explaining the maximum number of files per folder:

    Directory file set

    exFAT and the rest of the FAT family of file systems does not use indexes for file names, unlike NTFS which uses B-trees for file searching. When a file is accessed, the directory must be sequentially searched until a match is found. For file names shorter than 16 characters in length, one file name record is required but the entire file is represented by three 32-byte directory records. This is called a directory file set, and a 256 MiB sub-directory can hold up to 2,796,202 file sets. (If files have longer names, this number will decrease but this is the maximum based on the minimum three-record file set.) To help improve the sequential searching of the directories (including the root) a hash value of the file name is derived for each file and stored in the directory record. When searching for a file, the file name is first converted to upper case using the upcase table (file names are case insensitive) and then hashed using a proprietary patented algorithm into a 16-bit (2 byte) hash value. Each record in the directory is searched by comparing the hash value. When a match is found, the file names are compared to ensure that the proper file was located in case of hash collisions. This improves performance because only 2 bytes have to be compared for each file. This significantly reduces the CPU cycles because most file names are more than 2 characters (bytes) in size and virtually every comparison is performed on only 2 bytes at a time until the intended file is located.

    As you can read, there is no "hard maximum". It all depends on the filename size and probably implementation as well.

    How did you determine the file system of your partition is in fact exFAT? You may think you have exFAT, but in fact only have FAT32. Verify you really have formatted your store as exFAT.

    I fully agree with Bruce; it is just a very bad implementation that creates >22,000 files in a folder, so change that and kill the problem that way.


    Good luck,

    Michel Verhagen, eMVP
    Check out my blog: https://guruce.com/blog

    GuruCE
    Microsoft Embedded Partner
    NXP Proven Partner
    https://guruce.com
    Consultancy, training and development services.

    Interested in WEC on i.MX6?
    Get the only 100% stable and best performing i.MX6 BSP for WEC7 and WEC2013 here: https://guruce.com/imx6



    Thursday, September 12, 2019 8:03 PM
    Moderator
  • Hi Bruce and Michel,

    Thanks for your replies and explanation. 

    So, there is a limitation for the number of files for each directory due to the fix size of FAT table and also the 16-bit search algorithm (this limit to 65536). Sure there is other limitation that set the real limit to around 22k in our case.

    Yes. I agreed storing files into folders will resolve the problem. Some how our partner who is using our platform want to store that much of files for some reason. Let see what we can do with Microsoft support. We know they not likely want to support WEC7 anymore. sigh....

    Wednesday, September 18, 2019 9:02 AM