none
Cold Storage Early Deletion Queries

    Întrebare

  • Hi,

    I am new to this cold storage i just need some help,

    1.) Could you please brief out about the cold storage Early Deletion period

    2.) Could you please tell us which would be best way of storing the data lake data

    Regards,

    Sudarson Krishnamoorthy

    4 iunie 2018 16:39

Toate mesajele

  • Hello Sudarson,

    1) in short, selecting cold or cool tier is a commitment for the period of at least 180 days and 30 days (respectively) - if you change your mind before that and decide to move your blobs to the hot tier, there is an extra charge. Details at https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-storage-tiers

    2) Azure Data lake is intended for certain types of scenarios (typically involving big data). For comparison between Azure Storage and Azure Data Lake, refer to https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-comparison-with-blob-storage . To ingest data into Data Lake Store, you can use a variety of methods, including Data Factory, Apache DstCp, AdlCopy, etc. (more at https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-data-scenarios#ingest-data-into-data-lake-store )

    hth
    Marcin

    4 iunie 2018 19:37
  • Hi, Marcin

    Thanks for the update. I need some more clarifications,

    1.) You said "selecting cold or cool tier is a commitment for the period of at least 180 days and 30 days (respectively) - if you change your mind before that and decide to move your blobs to the hot tier" but this Early Deletion period cost will also inclusive for just downloading the blob files from the cold storage..?

    2.) Which would be the best way for storing the data is it hot/cold/archive?

    Regards,Sudarson Krishnamoorthy

    5 iunie 2018 08:12
  • Hi Sudarson,

    1) in order for you to download a blob which resides in the cold/archive tier, that tier of that blob must be changed to cool/hot - so yes, in case of cold/archive storage, this will effectively result in the early deletion period charges (if the request to download happens within the 180 day period). This does not apply to cool tier, but in this case, there are charges for data access

    2) There is no single answer to this - it depends on the data usage patterns and the intended data use. For starters, refer to the link I provided earlier that compares Azure Storage with Azure Data Lake

    hth
    Marcin

    5 iunie 2018 10:46
  • Hi, Marcin

    Thanks for the reply.

    The actual requirement is we are moving the data from data lake which are older than 3 months due to cost reduction so we decided to have a back up of all those data on the blob storage the data volume is pretty much high since all are log files so could you recommend which would be the best way for storing the data is it hot/cold/archive

    Regards,

    Sudarson Krishnamoorthy

    7 iunie 2018 10:17
  • It is actually pivot on the data usage patterns and intended data use as marcin said. How frequently you planning to access the logs, because archive storage is unique because it does not offer read performance – you cannot download or directly access blobs (files) from archive storage. You can only send items from hot/cool storage to archive storage, and then “rehydrate” the blobs again by restoring them to hot/cool storage – then you can download or read the blobs. Hot and cool storage have a read latency of milliseconds, but rehydrating a blob from archive storage can take up to 15 hours. 15 hours is alright because these are files that aren’t even cool any more – they’re files that you’re keeping for legal reasons. In a legal scenario, a retrieval isn’t a rush operation because you’ll have days/weeks to comply with requests/orders.

    • Hot: The cheapest to access, but the most expensive to store.
    • Cool: Medium price storage, but expensive to access.
    • Archive: Extremely cheap per GB storage (~$2.05 per TB per month)

    Cool and archive storage both have minimum storage durations. For example, if you place a file into cool storage, Azure expects you to keep that file there for a minimum of 30 days. If you retrieve it after 5 days, then there’s a pro-rated minimum storage charge of 25 days (30-5). Archive storage expects you to keep files in that tier for at least 180 days. If I retrieve a file after 5 days, then there is a pro-rated charge of 175 days (180-5). In other words, only put things into cool or archive storage if they are either being used infrequently (cool) or not at all (archive). See this helps.

    --------------------------------------------------------------------------------------------------

    If this answer was helpful, click “Mark as Answer” or Up-Vote. To provide additional feedback on  your forum experience, click here

    9 iunie 2018 03:23
    Moderator
  • Hey ,

    Just checking in to see if the above suggestions helped or you need further assistance on this issue.

    14 iunie 2018 19:10
    Moderator