locked
File upload on blob storage in chunks with silverlight wcf RRS feed

  • Question

  • Hello All,

    I am facing an issue while uploading file with big size(more than 2 or 3 MB).

    logic has been written in a such way that it takes small chunk of file and upload it on blob. After that it takes few other chunk of the file. but the content should be appended to previous content which is not happening.

    for ex. i am uploading excel/csv file, application will take some row from the file and upload that file on blob. now when it comes next time, it will take next few rows. but this time, when file is uploaded to blob storage, file is getting over written.

    is there any provision to append data to the existing(uploaded) file on azure blob storage ? 

    BR,

    Jonyy

    Tuesday, December 11, 2012 12:00 PM

Answers

  • Steve,

    You're correct (As always :)). I just tried it with the code below (it uses storage client library 2.0) and was able to modify the blob.

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using Microsoft.WindowsAzure.Storage;
    using Microsoft.WindowsAzure.Storage.Blob;
    using System.IO;
    
    namespace ConsoleApplication18
    {
        class Program
        {
            static void Main(string[] args)
            {
                var storageAccount = CloudStorageAccount.DevelopmentStorageAccount;
                var blobContainer = storageAccount.CreateCloudBlobClient().GetContainerReference("container");
                blobContainer.CreateIfNotExists();
                var blockBlob = blobContainer.GetBlockBlobReference("test.txt");
                byte[] firstLine = Encoding.UTF8.GetBytes("This is line 1.");
                List<string> blockIds = new List<string>();
                blockIds.Add(Convert.ToBase64String(Encoding.UTF8.GetBytes("Block-001")));
                using (MemoryStream ms1 = new MemoryStream(firstLine))
                {
                    blockBlob.PutBlock(Convert.ToBase64String(Encoding.UTF8.GetBytes("Block-001")), ms1, null);
                    blockBlob.PutBlockList(blockIds);
                }
                byte[] secondLine = Encoding.UTF8.GetBytes("\r\nThis is line 2.");
                blockIds.Add(Convert.ToBase64String(Encoding.UTF8.GetBytes("Block-002")));
                using (MemoryStream ms2 = new MemoryStream(secondLine))
                {
                    blockBlob.PutBlock(Convert.ToBase64String(Encoding.UTF8.GetBytes("Block-002")), ms2, null);
                    blockBlob.PutBlockList(blockIds);
                }
            }
        }
    }
    

    I'll update my previous response.

    Wednesday, December 12, 2012 5:16 AM
  • Basically if the file you're uploading is big in size, Windows Azure Storage allows you to upload this file in chunks (called blocks) (PutBlock function in the code above) and then later you can tell Windows Azure Storage to combine these chunks and commit the blob in storage (PutBlockList function in the code above).

    You may find these links helpful:

    http://msdn.microsoft.com/en-us/library/windowsazure/ee691964.aspx

    http://msdn.microsoft.com/en-us/library/windowsazure/dd135726.aspx

    http://msdn.microsoft.com/en-us/library/windowsazure/dd179467.aspx

    Hope this helps.

    Wednesday, December 12, 2012 9:10 AM
  • That's correct. Also a few more things:

    1. All blocks in a single blob upload must have block ids of same length.

    2. Maximum length of a block id is 50 characters.

    3. When splitting a blob into blocks, there can be a maximum of 50,000 blocks.

    4. You can upload blocks in any order however you must pass the block ids in PutBlock operation in the order in which you want to have these blocks ordered and constructed into a blob.

    Hope this helps.

    Thursday, December 13, 2012 9:20 AM

All replies

  • Please see response below from Steve. I've removed my original response because it is possible to update the blob using the approach outlined by Steve.


    • Edited by Gaurav Mantri Wednesday, December 12, 2012 5:18 AM Incorrect information provided earlier.
    Tuesday, December 11, 2012 2:40 PM
  • Gaurav, couldn't he use blocks? Just keep uploading new blocks and committing the new block list.

    In other words, upload block 1 and then commit the block list [1].

    Later, upload block 2 and then commit the block list [1,2].

    Right?

    Wednesday, December 12, 2012 4:53 AM
  • smarx,

    can you please provide an example if possible ?

    Wednesday, December 12, 2012 4:58 AM
  • Steve,

    You're correct (As always :)). I just tried it with the code below (it uses storage client library 2.0) and was able to modify the blob.

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using Microsoft.WindowsAzure.Storage;
    using Microsoft.WindowsAzure.Storage.Blob;
    using System.IO;
    
    namespace ConsoleApplication18
    {
        class Program
        {
            static void Main(string[] args)
            {
                var storageAccount = CloudStorageAccount.DevelopmentStorageAccount;
                var blobContainer = storageAccount.CreateCloudBlobClient().GetContainerReference("container");
                blobContainer.CreateIfNotExists();
                var blockBlob = blobContainer.GetBlockBlobReference("test.txt");
                byte[] firstLine = Encoding.UTF8.GetBytes("This is line 1.");
                List<string> blockIds = new List<string>();
                blockIds.Add(Convert.ToBase64String(Encoding.UTF8.GetBytes("Block-001")));
                using (MemoryStream ms1 = new MemoryStream(firstLine))
                {
                    blockBlob.PutBlock(Convert.ToBase64String(Encoding.UTF8.GetBytes("Block-001")), ms1, null);
                    blockBlob.PutBlockList(blockIds);
                }
                byte[] secondLine = Encoding.UTF8.GetBytes("\r\nThis is line 2.");
                blockIds.Add(Convert.ToBase64String(Encoding.UTF8.GetBytes("Block-002")));
                using (MemoryStream ms2 = new MemoryStream(secondLine))
                {
                    blockBlob.PutBlock(Convert.ToBase64String(Encoding.UTF8.GetBytes("Block-002")), ms2, null);
                    blockBlob.PutBlockList(blockIds);
                }
            }
        }
    }
    

    I'll update my previous response.

    Wednesday, December 12, 2012 5:16 AM
  • thanks Gaurav, Steve

    I have tried your code and its working fine on emulator. i am little bit confused about using "blockIds".

    can you please explain me why we are using that and why you have passed it into PutBlockList method ?

    BR,

    Jonyy

    Wednesday, December 12, 2012 8:54 AM
  • Basically if the file you're uploading is big in size, Windows Azure Storage allows you to upload this file in chunks (called blocks) (PutBlock function in the code above) and then later you can tell Windows Azure Storage to combine these chunks and commit the blob in storage (PutBlockList function in the code above).

    You may find these links helpful:

    http://msdn.microsoft.com/en-us/library/windowsazure/ee691964.aspx

    http://msdn.microsoft.com/en-us/library/windowsazure/dd135726.aspx

    http://msdn.microsoft.com/en-us/library/windowsazure/dd179467.aspx

    Hope this helps.

    Wednesday, December 12, 2012 9:10 AM
  • Gaurav,

    each block has unique blockid, right ? 

    Thursday, December 13, 2012 9:16 AM
  • That's correct. Also a few more things:

    1. All blocks in a single blob upload must have block ids of same length.

    2. Maximum length of a block id is 50 characters.

    3. When splitting a blob into blocks, there can be a maximum of 50,000 blocks.

    4. You can upload blocks in any order however you must pass the block ids in PutBlock operation in the order in which you want to have these blocks ordered and constructed into a blob.

    Hope this helps.

    Thursday, December 13, 2012 9:20 AM