Ten minutes to deploy a container RRS feed

  • Question

  • Azure Container Instances is taking way too long to deploy a Windows container. With the right situation, it can take seconds. But I cannot get it to work with ACI. Help! Some background:

    I have a Windows application that runs fine in a container. Locally it starts in seconds. I pushed it to an Azure container repository, and I can start it on a core i3  Windows server 2016 machine that never saw it before in 3 minutes including download and unpacking. But on Azure the deployment takes at least 10 minutes. I investigated this, and created a version 1803 VM with docker and it too sits there and downloads and unpacks an image, taking even longer. Indeed, I got the impression the download was throttled after a gigabyte or two, and the unpacking paused a lot despite provisioning two cores. 

    So I dug into the troubleshooting, and found the mention of the supported images. I rebuilt my container with the 1803 image, and pushed, and ran it on the VM and it took less than 10 SECONDS! Almost so fast I didn't think it had done it. Now that's the performance I want! So I then used the web interface to deploy, and it complained that it didn't support that version, and needs the ltcs version. So I build again with that, and I'm back to very slow. 

    I'm wondering if there is some extra layer that is being added here, that is adding some "non-standard" image for supporting something above the base ltcs server? But whatever it is, it makes the Azure ACI completely useless for rapid scaling of images. I want that 10 second startup, not the 10 minutes. What can I do to get that please?

    Thanks, Matthew

    Tuesday, September 11, 2018 2:35 PM

All replies

  • You said it yourself. Issue is that ACI is built on LTSC and hence long download time since LTSC images are multi GB bigger then SAC channel and hence slow down.

    Wednesday, September 12, 2018 1:29 AM
  • How can I "match" the ACI server then? Obviously the 1803 base image is able to match the 1803 server for instant run. Surely there must be an image that I can base my container on that will match the ACI service servers? Then I'd get the same instant run on them? 

    Surely it wasn't originally shipped with this poor performance?

    Wednesday, September 12, 2018 8:41 AM
  • You can not run 1803, 1709 images on LTSC. The only image you will be able to run on ACI is LTSC version. So your base image shall be `FROM microsoft/windowsservercore:ltsc2016`
    Wednesday, September 12, 2018 11:35 AM
  • Thanks for the confirmation. We shall not use Azure for Windows containers then - I am amazed that a ten minute start time is considered acceptable. That's twice the time to start a Windows VM!
    Thursday, September 13, 2018 7:57 AM
  • I'm not sure Microsoft positioned ACI for this type of workloads in a first place. If your top priority is fast start up times then ACI probably will never be a great choice, it will be always lag behind bare metal solution since ACI is always started from scratch and no caching of any previous layers present
    Thursday, September 13, 2018 9:49 PM
  • OMG. Today, in going about documenting this for the project, ten minutes not being a viable start time, I find that the ltsc2016 build I do will start in under a minute. Sometimes just over a minute. This is now viable! But how can we depend on this when "nothing has changed"? And if it works for a month, and then goes killer-slow again, how would I know how to fix it? Did the ltsc2016 image get updated or something? Not on my build PC. 

    How I wish for a decent contact at Microsoft who can answer these key questions. We need to know if we can quickly deploy 30-ish containers rapidly. They may live for a few hours each, and then stop. But start-time is crucial. ACI is the preferred option, but we need certainty. Thanks for anyone providing solid answers!

    Monday, September 24, 2018 12:29 PM
  • Windows base images are updated every month with patches. So every month you start up times will be slower since new layer of base image needs to be downloaded. Once you have image on server start up of base image shall be around 10-15 seconds or a little higher for Hyper-V isolation. 
    Monday, September 24, 2018 1:02 PM
  • How can we get notification of these updates? A rebuild would be possible, but I don't want to have 30 people sitting waiting for over ten minutes for the code to start working for them "just randomly". Also the server image I am using was downloaded 12 days ago, and is fast now. If the server has been updated, why such a time difference between the image and the server update? Is this all documented somewhere?
    Monday, September 24, 2018 1:08 PM
  • You monitor when this image ( was last pushed in docker hub to find out when it was last updated. 

    If your image is already downloaded though it shall not require redownloading it again even if new version of base image is released. You image is hard set to original image which was used when you built it. So I'm not sure what exactly you are seeing since not fully understanding I guess your pipeline.

    Monday, September 24, 2018 1:13 PM
  • Okay, thanks for that. Parsing a web page doesn't seem ideal, but I can work out how to make that part of a build system. 

    However, it does indeed take me back to uncertainty, as that is the image that I was using when I started all of this thread - so it went from being over ten minutes to just under a minute with no apparent changes.

    I am though going to be doing some new work on another Linux based container project, so will start by testing Windows containers for the first development part, and perhaps gain more timing details over time. 

    Monday, September 24, 2018 1:19 PM