persistent storage in azure kubernetes services RRS feed

  • Question

  • We are trying to deploy an application on AKS and the application is to use persistent volume. If we use azure disk , we have noticed if the node having the pod running the application container is stopped / not working , another pod from another node is spinned up but it is no longer accessing the persistent volume.
    As per documentation ,azure disk is mapped to a particular node and file share is shared across nodes. What is the way to ensure that a application running on aks using persistent volume is not lost if a pod/node does not work ? 
    Tuesday, August 27, 2019 9:04 AM

All replies

  • Hi,

    The type of persistent volume to choose is based on the type of application which is going to use that storage.

    Azure Disk:

    Azure disk can be attached to onyone node at a time. So one pod at a time. We can configure the azure disk to be created dynamically as per the need and the can be used with the deployment.

    We can create a disk manually(static disk) and then use that with the pod. 

    In both the cases data will be persisted and data will not be cleared unless the "Reclaim policy" is delete.

    Azure disk has more IOPS and less latency than Azure files.

    If the node where the disk is attached to fails , Then AKS should detach the disk from the failed node and apply it on the other healthy node. Till that time, Our app will be down. Becasue of the node failure sometimes detaching disk may take longer time. redeployment helps in that case.

    All these happens only when the node fails. Normal redeployment wont take much time.
    This is a github issue for this

    Azure File:

    Files can be attached to more than one node at a time. So more pods at a time.

    But files have less IOPS and more latency when compared to disks. Currently its recommended to use file share as the persistent volume for the workloads.

    If multi containers needs to read and write data from a PV then go for Azure files.

    If you are using PV as a place to do temporary storage you can go with Dynamic Azure disks.

    Please let me know your comments

    Tuesday, August 27, 2019 10:22 AM
  • We are assessing the possibility of using persistent storage for kafka and redis.
    So as per the information , we need to use the azure file share instead of managed azure disk since it might be a case where pods running on different nodes for kafka will access the same persistent storage. 
    Please correct if this understanding is wrong.
    Also can you let me know where does the image get stored? As per understanding , managed os disk is used for storing images. What is the use of data disk? 

    Wednesday, September 4, 2019 12:33 PM
  • HI,

    Yes. If two or more pods needs to access the same data, Azure Files is the option. Its an easy option and performance will be low.  You need to load test your environment to check if it fits your requirement or not.

    If you are planning to deploy the Kafka as a stateful set, Then you can use Azure disk as well. Because statefull sets ties the pod with the spec and the persistent storage. In this case is you have if you are running 3 kafka pods, you will be creating the 3 pvc (backed by 3 Azure disks) and attach it to the 3 pods respectively.

    Only risk here is that in case of unexpected node failure, sometimes the disk may fail to get detached.

    But this is manageable if you are running more replicas. 

    Since we are using disks here, Performance will be high. Also note that Kafka is memory intensive, So make sure to assign enough memory and apply memory limits as well.

    I found a nice article in Medium which discusses about the points to take care when doing a kafka deployment in kubernetes.

    Disclaimer: This response contains a reference to a third party World Wide Web site. 

    Microsoft is providing this information as a convenience to you. Microsoft does not control these sites and has not tested any software or information found on these sites; therefore, Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. 

    Thursday, September 5, 2019 11:48 AM