locked
How to deal with temporary files? (Blob or Table Service) RRS feed

  • Question

  • Hi,

    I want users to upload image files. The originals will be saved as also 1 resized versions. If there are more than 1000 resized images (or the total size is more than 400MB) then I want to delete the last 200 resized versions.

    Which storage type is best for me and why?

    Thanks!

    Sunday, April 24, 2011 8:47 PM

Answers

  • The appropriate storage for images is Azure Blob. Within pretty high limits there is no restriction on the size of entities. There is, however, no ability to query for blobs - you can list them with some primitive name filtering. Consequently, if you need more sophisticated indexing capability you might want to use Azure Table as an index on the blobs you store.
    • Marked as answer by Arjen Monday, April 25, 2011 8:04 PM
    Sunday, April 24, 2011 10:01 PM
    Answerer
  • Yes, blobs for the actuall images and tables for the meta data.

    For the temporary files, that's your choice based on the costs associated with storing them ($0.15/gb per month). Compute time shouldn't figure into your equation because you're charged for the amount of time the application is deployed, not the amount of work it is doing. If its sitting idle or deleting temporary files the charge is the same.

    The bigger question I have is if your temporary files are stored within the role instance, if so, please keep in mind that if the role instance is restarted, its highly likely that anything you have stored on the local file system may be lost.

    • Marked as answer by Arjen Monday, April 25, 2011 8:04 PM
    Monday, April 25, 2011 1:05 PM
  • -- Last question, images that are part of the webapp can be stored on the local system safely?

    When you deploy an application into a hosted service, (i.e. upload your solution to Windows Azure), the Azure fabric controller stores a copy of the uploaded package. It deploys the appropriate parts of each package to the various role instances you have specified in the Service Definition and Service Configuration files. This deployment includes any images you have inserted in the application. If the fabric controller has to reimage or move any of your instances it gets the information out of the saved copy of the uploaded package. Consequently, reimaged instances have precisely the same content as the original image.

    However, any change you have made to an instance do not survive a reimage. If you have configured the Local Storage resource appropriately in the Service Definition file then it will probably survive an instance recycle and may, if you are lucky, survive a reimage that does not move the instance.

    The bottom line is that there is no guarantee of durability for any file you put onto the local filesystem of an instance. The deployed application is durable because the fabric controller maintains a copy.

    --  And disk space is charged by average. Thus, store 500MB at the beginning of the month and delete this at the end of the month makes a bill based on 250MB? ($0.15 / 4).

    Disk space is billed by average amount stored per day. The following is documented (measuring consumption tab):

    -- Storage is metered in units of the average daily amount of data stored (in GB) over a monthly period. For example, if a user uploaded 30GB of data and stored it on Windows Azure for a day, her monthly billed storage would be 1 GB. If the same user uploaded 30GB of data and stored it on Windows Azure for an entire billing period, her monthly billed storage would be 30GB. Storage is also metered in terms of storage transactions used to add, update, read and delete storage data. These are billed at a rate of $0.01 for 10,000 (10k) transaction requests

    • Marked as answer by Arjen Monday, April 25, 2011 8:04 PM
    Monday, April 25, 2011 5:05 PM
    Answerer
  • Last question, images that are part of the webapp can be stored on the local system safely?


    If by safely you mean "they are stored there temporarially but not persisted or shared between role instances, then yes. But if you mean what you put on the local file system stays there until you delete it, then no. You can work around this by leveraging Azure Drive, but even then you can only have one instance write to an Azure Drive (unless you jump through some shared network drive hoops) at a time. So its not a good place to save files.

    I'd recommend using Azure Storage blobs for all your file stores. If the application has properly implemented a adapter/connectory pattern, it should be a fairly simple matter of swapping out the current one for one that's more Azure friendly.

    • Marked as answer by Arjen Monday, April 25, 2011 8:04 PM
    Monday, April 25, 2011 5:19 PM
  • Files like image, css etc. that are deployed along with your cloud solution package (.cfpkg) would be there for you even when your role instance is restarted for whatever reason. I guess, you are not trying to change these files at run time. Do you?

     

    HTH.


    Please mark it as answer by clicking on "Propose As Answer", if it helps. My Blog : http://dotnetizen.blogspot.com
    • Marked as answer by Arjen Monday, April 25, 2011 8:04 PM
    Monday, April 25, 2011 7:35 PM

All replies

  • The appropriate storage for images is Azure Blob. Within pretty high limits there is no restriction on the size of entities. There is, however, no ability to query for blobs - you can list them with some primitive name filtering. Consequently, if you need more sophisticated indexing capability you might want to use Azure Table as an index on the blobs you store.
    • Marked as answer by Arjen Monday, April 25, 2011 8:04 PM
    Sunday, April 24, 2011 10:01 PM
    Answerer
  • Thanks for your answer.

    So you suggest using both blobs (for images) and Tables (for meta data)?

    Another question: should we delete temporary files? If we create them we use compute time, when we delete them we use compute time, and when we need the files again we use copmpute time + (again) disk space. So, what is more cost effective? Keeping temporary files or clean up and create again when needed?

    Monday, April 25, 2011 12:06 PM
  • Yes, blobs for the actuall images and tables for the meta data.

    For the temporary files, that's your choice based on the costs associated with storing them ($0.15/gb per month). Compute time shouldn't figure into your equation because you're charged for the amount of time the application is deployed, not the amount of work it is doing. If its sitting idle or deleting temporary files the charge is the same.

    The bigger question I have is if your temporary files are stored within the role instance, if so, please keep in mind that if the role instance is restarted, its highly likely that anything you have stored on the local file system may be lost.

    • Marked as answer by Arjen Monday, April 25, 2011 8:04 PM
    Monday, April 25, 2011 1:05 PM
  • Yes, blobs for the actuall images and tables for the meta data.

    Clear.

    For the temporary files, that's your choice based on the costs associated with storing them ($0.15/gb per month). Compute time shouldn't figure into your equation because you're charged for the amount of time the application is deployed, not the amount of work it is doing. If its sitting idle or deleting temporary files the charge is the same.

    Okay, thus "750 hours of an Extra Small Compute Instance" just means 24hr x ~30days? I did not know that.

    And disk space is charged by average. Thus, store 500MB at the beginning of the month and delete this at the end of the month makes a bill based on 250MB? ($0.15 / 4).

    The bigger question I have is if your temporary files are stored within the role instance, if so, please keep in mind that if the role instance is restarted, its highly likely that anything you have stored on the local file system may be lost.

    The idea is to store each file in the blob storage.
    Last question, images that are part of the webapp can be stored on the local system safely?

    Thank you very much!


    Monday, April 25, 2011 3:48 PM
  • -- Last question, images that are part of the webapp can be stored on the local system safely?

    When you deploy an application into a hosted service, (i.e. upload your solution to Windows Azure), the Azure fabric controller stores a copy of the uploaded package. It deploys the appropriate parts of each package to the various role instances you have specified in the Service Definition and Service Configuration files. This deployment includes any images you have inserted in the application. If the fabric controller has to reimage or move any of your instances it gets the information out of the saved copy of the uploaded package. Consequently, reimaged instances have precisely the same content as the original image.

    However, any change you have made to an instance do not survive a reimage. If you have configured the Local Storage resource appropriately in the Service Definition file then it will probably survive an instance recycle and may, if you are lucky, survive a reimage that does not move the instance.

    The bottom line is that there is no guarantee of durability for any file you put onto the local filesystem of an instance. The deployed application is durable because the fabric controller maintains a copy.

    --  And disk space is charged by average. Thus, store 500MB at the beginning of the month and delete this at the end of the month makes a bill based on 250MB? ($0.15 / 4).

    Disk space is billed by average amount stored per day. The following is documented (measuring consumption tab):

    -- Storage is metered in units of the average daily amount of data stored (in GB) over a monthly period. For example, if a user uploaded 30GB of data and stored it on Windows Azure for a day, her monthly billed storage would be 1 GB. If the same user uploaded 30GB of data and stored it on Windows Azure for an entire billing period, her monthly billed storage would be 30GB. Storage is also metered in terms of storage transactions used to add, update, read and delete storage data. These are billed at a rate of $0.01 for 10,000 (10k) transaction requests

    • Marked as answer by Arjen Monday, April 25, 2011 8:04 PM
    Monday, April 25, 2011 5:05 PM
    Answerer
  • Last question, images that are part of the webapp can be stored on the local system safely?


    If by safely you mean "they are stored there temporarially but not persisted or shared between role instances, then yes. But if you mean what you put on the local file system stays there until you delete it, then no. You can work around this by leveraging Azure Drive, but even then you can only have one instance write to an Azure Drive (unless you jump through some shared network drive hoops) at a time. So its not a good place to save files.

    I'd recommend using Azure Storage blobs for all your file stores. If the application has properly implemented a adapter/connectory pattern, it should be a fairly simple matter of swapping out the current one for one that's more Azure friendly.

    • Marked as answer by Arjen Monday, April 25, 2011 8:04 PM
    Monday, April 25, 2011 5:19 PM
  • I'd recommend using Azure Storage blobs for all your file stores. If the application has properly implemented a adapter/connectory pattern, it should be a fairly simple matter of swapping out the current one for one that's more Azure friendly.


    Well, I mean with webapp files such as images (logo, button icons), css (layout), js (like jQuery). Now, I have a webrole (my webapp) with a directory that includes such files... do you mean I should add these files as blob too?

     

    Monday, April 25, 2011 6:58 PM
  • Files like image, css etc. that are deployed along with your cloud solution package (.cfpkg) would be there for you even when your role instance is restarted for whatever reason. I guess, you are not trying to change these files at run time. Do you?

     

    HTH.


    Please mark it as answer by clicking on "Propose As Answer", if it helps. My Blog : http://dotnetizen.blogspot.com
    • Marked as answer by Arjen Monday, April 25, 2011 8:04 PM
    Monday, April 25, 2011 7:35 PM
  • Files like image, css etc. that are deployed along with your cloud solution package (.cfpkg) would be there for you even when your role instance is restarted for whatever reason. I guess, you are not trying to change these files at run time. Do you?

     

    HTH.


    Please mark it as answer by clicking on "Propose As Answer", if it helps. My Blog : http://dotnetizen.blogspot.com

    No, these files are forever. Untill an update. :)
    Monday, April 25, 2011 8:04 PM