locked
Downloading Thousands of files in Metro App - what's the best way

    Question

  • Currently I am using the background transfer api's to download anywhere from 100-10000 files/images they range from 5kb to 5mb. On some runs the program works fine, usually when it's less then? 1,000/ a few hundred. Sometimes it flakes out completely. So the question is should I really be using the webclient API's? and if I should be using the backgroundtransfer API's which seemed at the time at least that it'd be the easier and best option when I first wrote it. I can provide the code but I thought I'd start with am I even approaching this correctly.

    It seems like the OS is killing the app which I've heard it'll do if it thinks the app isn't behaving correctly. I think I may be overloading/having it do too much at once. I'm more then happy to share all of my code with someone if they'll help me figure it out.


    Kyle Weir

    Sunday, April 05, 2015 3:51 PM

Answers

  • You have a few options for non-background-capable downloads in a Windows Store app. The most straightforward is Windows.Web.Http.HttpClient - here's a good primer on how to use it. However, it's only available in Windows 8.1 apps, not Windows 8.0. If you need Windows 8.0 compatibility, the distinct but similarly named System.Net.Http.HttpClient is probably the way to go. Here is a link to the relevant Quickstart documentation.

    Best Regards,
    Please remember to mark the replies as answers if they help

    • Marked as answer by Kyle Weir Friday, April 10, 2015 7:20 PM
    Monday, April 06, 2015 9:49 AM

All replies

  • You have a few options for non-background-capable downloads in a Windows Store app. The most straightforward is Windows.Web.Http.HttpClient - here's a good primer on how to use it. However, it's only available in Windows 8.1 apps, not Windows 8.0. If you need Windows 8.0 compatibility, the distinct but similarly named System.Net.Http.HttpClient is probably the way to go. Here is a link to the relevant Quickstart documentation.

    Best Regards,
    Please remember to mark the replies as answers if they help

    • Marked as answer by Kyle Weir Friday, April 10, 2015 7:20 PM
    Monday, April 06, 2015 9:49 AM
  • Hello,

    Sounds like you are blowing over your quota. Windows enforces a limit in terms of CPU and network I/O resources an app can use during background tasks. Each app gets a quota of these resources on a periodic basis, and exhausting this quota suspends your app's background task. An app must be resilient to being suspended due to sandboxing.

    Being productive in the background – background tasks

    I hope this helps,

    James

     

    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Tuesday, April 07, 2015 12:31 AM
    Moderator
  • It is a windows 8.1 app, and I was going the way of using the BackgroundDownloader API's which if I understood it correctly passes the download process to the system. https://msdn.microsoft.com/en-us/library/windows/apps/windows.networking.backgroundtransfer.backgrounddownloader.aspx?f=255&MSPPError=-2147217396

    I'm a bit new to the metro programming, so let me know if I'm doing something wrong.

    Basically I have another process running which is adding items to be downloaded to an observable collection. While the downloading process this below is removing items from that same collection.

    async Task DownloadManager()
            {
                try
                {
                    //await Task.Delay(tumbgconfig.delaybeforedownload);
                    // Delay this task for a set amount of time (5-10 seconds so that some items can be queued before it runs)
                    if (_queued.Count <= 10)
                    {
                        await Task.Delay(tumbgconfig.delaybeforedownload);
     
                    }
                    // Runs async and schedules downloads to run in background and to dequeue the items from the queue. 
                    List<Task> downloadCompletionTasks = new List<Task>();
                    queueinfo todlqueueinfo; // Queueinfo instance for download manager
     
                    DownloadbgImagesRing.IsActive = true;
     
                    // We loop until queue is empty
                    string savefolder = "";
                    // Check to see if folder exists if not then create Folder
                    try
                    {
                        if (tumbgconfig.downloadlikes)
                        {
                            savefolder = tumbgconfig.LikesFolder;
                        }
                        else
                        {
                            Uri bgurl = new Uri(tumbgconfig.SelectedtbUrl);
                            savefolder = bgurl.Host.ToString();
                        }
                        StorageFolder appFolder = await Windows.Storage.KnownFolders.PicturesLibrary.CreateFolderAsync(savefolder, CreationCollisionOption.OpenIfExists);
                        // move items from _queue to downloader
                        tumbgconfig.SelectedtbDLFolder = appFolder;
     
                        // string source;
                        // Setup dowloader 
                        BackgroundDownloader downloader = new BackgroundDownloader();
                        BackgroundTransferGroup tbSFTransferGroup = BackgroundTransferGroup.CreateGroup("XXXXXXXXXXXXXXXXXXXXXXXXXXX");
                        // Set Transfer preferences
     
                        try
                        {
                            downloader.CostPolicy = BackgroundTransferCostPolicy.Always; // Always transfer regardless of wifi/cellular - cost
     
                            if (SettingsStore.ParallelBKGTransfer == true)
                            {
                                if (tbSFTransferGroup.TransferBehavior != BackgroundTransferBehavior.Parallel)
                                    tbSFTransferGroup.TransferBehavior = BackgroundTransferBehavior.Parallel;
                            }
                            else
                                tbSFTransferGroup.TransferBehavior = BackgroundTransferBehavior.Serialized;
                        }
                        catch (Exception ex)
                        {
                            System.Diagnostics.Debug.WriteLine(ex);
                            if (SettingsStore.Debug == "True")
                                ReportErrorMessage(ex);
                            downloader.CostPolicy = BackgroundTransferCostPolicy.Always; // Always transfer regardless of wifi/cellular - cost
                            tbSFTransferGroup.TransferBehavior = BackgroundTransferBehavior.Serialized;
     
                        }
                        downloader.TransferGroup = tbSFTransferGroup;
                        
                        //         var BKGRDownloads = await BackgroundDownloader.GetCurrentDownloadsForTransferGroupAsync(downloader.TransferGroup);
     
                        do
                        {
                            // Dequeue source url
                            if ((tumbgconfig.totalposts == 0 | _queued.Count == 0) & ParsebgRing.IsActive == false)// Reset - -- Is currently running twice needs to run only once. 
                            {
                                break;
                            }
                            _queued.TryDequeue(out todlqueueinfo);
                            // If requested we cancel the downloads. This will call a cancelation function that will bring up the tb following
     
                            while ((tumbgconfig.currentpost < tumbgconfig.totalposts) & (tumbgconfig.currentpost <= tumbgconfig.MaxDLimit) & (_queued.Count == 0))
                            {
                                if (m_cancelationTokenSource.IsCancellationRequested)
                                    break;
                                Sleep(1000);
                                if (tumbgconfig.currentpost == tumbgconfig.MaxDLimit)
                                    break;
                            }
                            if (m_cancelationTokenSource.IsCancellationRequested)
                                break;
                            try
                            {
                                StorageFile destinationFile = await appFolder.CreateFileAsync(todlqueueinfo.destinationfilename.ToString(), CreationCollisionOption.FailIfExists); // skip if they are already existing 
                                // Setup downloader backgroundDownloader to download the file and put into queue.
     
                                DownloadOperation download = downloader.CreateDownload(new Uri(todlqueueinfo.sourcefilename.ToString()), destinationFile);
     
                                while ((DLoperations.Count - (DLoperations.FindAll((e) => { return (e.Progress.BytesReceived.Equals(e.Progress.TotalBytesToReceive)); }).Count)) > tumbgconfig.concurrentdownloads)
                                {
                                    Sleep(5);
                                }
     
                                // var operation = await Task.Run(() => { return download.StartAsync(); });
                                Task downloadCompletionTask = StartDownloadAsync(download);  // here we have it start the download and tell it to continue without waiting for it to finish.
     
                                tumbgconfig.activedownloads++;
                                downloadCompletionTasks.Add(downloadCompletionTask);
                                DLoperations.Add(download);
     
                                // this keeps us to only download (tumbgconfig.concurrentdownloads) number of downloads at a time
                                
                                if (tumbgconfig.QueuedDownloads != 0)
                                    tumbgconfig.QueuedDownloads--; // This is so we tell the progressbar that we are removing items from the queue
                                // Updates the stats on the main screen.
                            }
                            catch (Exception ex)
                            {
                                System.Diagnostics.Debug.WriteLine(ex.Message);
                                if (SettingsStore.Debug == "True")
                                    ReportErrorMessage(ex);
     
                                if (ex.HResult == -2147024713)
                                {
                                    // If this is a duplicate load image
                                    tumbgconfig.skippedimages++;
                                }
                                else
                                {
                                    //        tumbgconfig.duplicatefilecount++;
                                }
                                // Log("Skipped" + destinationfilename.ToString());
                            }
                            await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, updatestats);
     
                            if (m_cancelationTokenSource.IsCancellationRequested)
                                break;
     
                        } while ((!_queued.IsEmpty) | (tumbgconfig.activedownloads != tumbgconfig.completeddownloads));
                    }
                    catch (Exception ex)
                    {
                        System.Diagnostics.Debug.WriteLine(ex);
                        if (SettingsStore.Debug == "True")
                            ReportErrorMessage(ex);
     
     
                    }
                    if (tumbgconfig.activedownloads != tumbgconfig.completeddownloads)
                        tumbgconfig.finshprocessingdownloads = true;
                    while (!Task.WhenAll(downloadCompletionTasks).IsCompleted)
                    {
                        // Here we check to see if the downloadCompletionTasks are still running, or have failed out, for some reason.
                        //foreach(downloadtask in downloadCompletionTasks)
                        //{
     
     
                        //}
     
                    }
                    //      await Task.WhenAll();
     
     
                    // Save/flush logging to file
     
     
                    tumbgconfig.downloadlikes = false;
                    reset2following();
                }
                catch (Exception ex)
                {
                    System.Diagnostics.Debug.WriteLine(ex);
                    if (SettingsStore.Debug == "True")
                        ReportErrorMessage(ex);
                    tumbgconfig.downloadlikes = false;
                    reset2following();
                }
            }
    
    
    
    
    private async Task StartDownloadAsync(DownloadOperation downloadOperation)
    {
        try
        {
            _activeDownload = downloadOperation;
            //   var progress = new Progress<downloadOperation>(ProgressCallback);
     
            Progress<DownloadOperation> progressCallback = new Progress<DownloadOperation>(DownloadProgress);
     
            var dlop = await downloadOperation.StartAsync().AsTask(m_cancelationTokenSource.Token);
     
            Debug.WriteLine("Downloading: " + downloadOperation.Guid);
     
     
            if (dlop != null)
            {
                //do
                //{
                //    Sleep(50);
                //} while (dlop.Progress.Status != BackgroundTransferStatus.Completed);
                // here we wait for the image to be finished downloading since it's async and then add the image to the observable collection
                if (dlop.Progress.Status != BackgroundTransferStatus.Completed)
                {
     
                }
                else
                {
                    addimageobservcoll(dlop.ResultFile);
     
                    await Dispatcher.RunAsync(CoreDispatcherPriority.Normal, updatestats);
                }
     
            }
        }
        catch (TaskCanceledException)
        {
            _activeDownload.ResultFile.DeleteAsync(); // This will delete the file if it was canceled or had an issue downloading this way we avoid the 0kb files
            _activeDownload = null;
     
        }
        catch (Exception ex)
        {
            _activeDownload.ResultFile.DeleteAsync(); // This will delete the file if it was canceled or had an issue downloading this way we avoid the 0kb files
            _activeDownload = null;
            System.Diagnostics.Debug.WriteLine(ex);
            if (SettingsStore.Debug == "True")
                ReportErrorMessage(ex);
        }
     
    }
    


    Kyle Weir

    Tuesday, April 07, 2015 2:24 AM
  • I didn't think that there were quota's when using the windows 8.1 backgrounddownloader as I am. Since essentially the system is doing the downloading. 

    Kyle Weir

    Tuesday, April 07, 2015 4:05 AM
  • The system is doing the downloading on your behalf but you are still bound to your task's quota. The only background task that gets a quota deferral is a media playback background task (and even then we can stop playback if too much bandwidth is being used). All other task types are bound to the quota.

    -James


    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Tuesday, April 07, 2015 10:13 PM
    Moderator
  • So I'm looking at the web.http.httpclient and I'm sorting out how to get it to download files, and save it to a folder. And part of the reason I'd gone with the background downloader was because there wasn't a size limit per file. The default size of a response is 64k so I'd assume that to download files larger I'd need to increase the HttpClient.MaxResponseContentBufferSize but since some of the files could be 5-10mb I'd need to increase this a lot, if I understand it. Some files may be larger since it will sometimes download videos which could be between 5mb and 50mb or so. Which the httpclient wouldn't be able to handle.

    So do I end up using both? which throws simplicity out the window... lol...

    Use the httpclient for images, and background downloader for videos? how do I find the size of the files before I download them?  Or am I confusing this?


    Kyle Weir

    Wednesday, April 08, 2015 4:13 AM
  • Why are you trying to download a large number of very large files? What is your scenario?

    -James


    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Wednesday, April 08, 2015 10:02 PM
    Moderator
  • Basically I have a windows store app, which I'm working on, which downloads primarily images, but videos as well from several blogs, The only part I'm having issues with at the moment is the downloads which apparently I'm maxing out the backgrounddownloads on, and I have an issue with the app states as well (which I believe I know what I need to do to resolve that.)

    Kyle Weir

    Wednesday, April 08, 2015 10:52 PM
  • Hello Kyle,

    I think I understand what you are trying to do but it is not clear why you want to do this. Can you provide more information about why your app needs to transfer large numbers of files?

    -James


    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Thursday, April 09, 2015 6:14 PM
    Moderator
  • It allows the user to images/videos for archival purposes. And also goes into building a offline diary of sorts.


    Kyle Weir

    Thursday, April 09, 2015 6:30 PM
  • Hello Kyle,

    Windows Store apps are not designed to enable file management capabilities. I would recommend that you either create a desktop app or use an architecture more like OneDrive where the files are stored in the cloud and only pulled down to the local device if necessary. I don't believe that we have a good solution for your on Windows 8.1 to enable the functionality that you are looking for. Windows Store apps are designed to transfer small amounts of data but be always connected and available.

    -James


    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Friday, April 10, 2015 6:19 PM
    Moderator
  • It's a archiver ie take blog from online and archive parts of it. Not a desktop file management system. And I thought Microsoft was going for universal? all inclusive type apps and not to limit metro/store apps to a subset of programs. My App is mainly used on windows 8.1 desktops/laptops, I don't have a phone version and I outright state that the app should be used on wifi or wired connections only (that'd ridiculous, although I have thought of having a phone app part that would only do minimal things, ie select which things to download and then when at a desktop again they'd be selected for the user to download.)

    James in any case I believe I've resolved the issue by going with the web http client thanks to issuekiller.


    Kyle Weir

    Friday, April 10, 2015 7:20 PM