locked
Enumerating a large set of files drives RuntimeBroker nuts and maxes out disk I/O

    Question

  • I have this folder:

    The folder contains PDFs. When I FolderPick this folder and call getFilesAsync on it all hell breaks loose on the system. The RuntimeBroker and Anti Maleware grabbing nearly 100% of CPU together and maxing out the Disk I/O at 99% with 10-12 MB/S. The RuntimeBroker process allocates 5.6 GB of RAM (from 8 GB available). This decreases over time (I guess when the Broker goes through the files) to 2 GB. Fetching all those files takes more than 25 Minutes with the system being practically unusable.

    Additionally the JS console emits this error:

    SCRIPT14: Exception is about to be caught by JavaScript library code at line 313, column 9 in ms-appx://myapp.app/js/data/import.js
    0x8007000e - JavaScript runtime error: Not enough storage is available to complete this operation.
    File: import.js, Line: 313, Column: 9

    The line that contains the "promises.push(sourceFile.copyAsync(folder, "" + timestamp + "-" + sourceFile.name, NameCollisionOption.replaceExisting)" method call.

    return getImportFolderAsync().then(function(folder) {
          var filesInImportFolder, progress, promises, start;
          promises = [];
          progress = new ui.progress.Progress(storageFiles.length);
          filesInImportFolder = [];
          storageFiles.forEach(function(sourceFile, i) {
            var timestamp;
            timestamp = asDemoData ? DEMO_DATA_TIMESTAMP : "" + (Date.now());
            assert(timestamp.length === TIMESTAMP_LENGTH);
            return promises.push(sourceFile.copyAsync(folder, "" + timestamp + "-" + sourceFile.name, NameCollisionOption.replaceExisting).then(function(copiedFile) {
              filesInImportFolder.push(copiedFile);
              return progress.progress();
            }));
          });
          start = new Date().getTime();
          return WinJS.Promise.join(promises).then(function() {
            var end;
            end = new Date().getTime();
            log.debug("Copyied " + storageFiles.length + " files in " + (end - start) + " ms");
            return processImportFilesAsync(filesInImportFolder);
          }, function(error) {
            return debug.error(error);
          });
        });

    Is there another, more resource friendly way to enumerate files in folders than fetching them all into a big array? Something like an enumerator we had in the good old COM times?

    Does the FolderQuery object work equally bad on such folder Structures or is better more optimized?



    • Edited by phil_ke Thursday, November 22, 2012 1:50 PM
    Thursday, November 22, 2012 1:34 PM