locked
getting "No space left on device" error, new images not pulling as a result RRS feed

  • Question

  • I've been iterating on some changes on noticed my Web App stopped pulling new images and responding to requests.

    My FTP Error Logs report the following:

    2017-03-24T02:40:23.243446165Z groupadd: /etc/group.5: No space left on device
    2017-03-24T02:40:23.245858675Z groupadd: cannot lock /etc/group; try again later.
    2017-03-24T02:40:23.250684595Z useradd: group '8b370433317866d90a16aa5a' does not exist
    2017-03-24T02:40:23.252956774Z chown: invalid user: '8b370433317866d90a16aa5a:8b370433317866d90a16aa5a'
    2017-03-24T02:40:23.254765183Z chown: invalid user: '8b370433317866d90a16aa5a:8b370433317866d90a16aa5a'
    2017-03-24T02:40:23.262692973Z chown: invalid user: '8b370433317866d90a16aa5a:8b370433317866d90a16aa5a'
    2017-03-24T02:40:23.262720459Z touch: cannot touch '/var/log/apache2/kudu-error.log': No space left on device
    2017-03-24T02:40:23.265123974Z touch: cannot touch '/var/log/apache2/kudu-access.log': No space left on device
    2017-03-24T02:40:23.273815987Z mkdir: cannot create directory '/var/lock/apache2': No space left on device
    2017-03-24T02:40:23.273846272Z mkdir: cannot create directory '/var/run/apache2': No space left on device
    2017-03-24T02:40:23.273853168Z chown: invalid user: '8b370433317866d90a16aa5a:8b370433317866d90a16aa5a'
    2017-03-24T02:40:23.273859665Z chown: invalid user: '8b370433317866d90a16aa5a:8b370433317866d90a16aa5a'
    2017-03-24T02:40:23.278011118Z sed: couldn't open temporary file /etc/apache2/sites-available/sedd4Q9U8: No space left on device
    2017-03-24T02:40:23.282908103Z sed: couldn't open temporary file /etc/apache2/sed2Vh2p7: No space left on device
    2017-03-24T02:40:23.282936089Z sed: couldn't open temporary file /etc/apache2/sedE5ykx7: No space left on device
    2017-03-24T02:40:23.292410516Z sed: couldn't open temporary file /etc/apache2/sedA45v37: No space left on device
    2017-03-24T02:40:23.292445699Z mkdir: cannot create directory '/etc/mono/registry': No space left on device
    2017-03-24T02:40:23.312047831Z chmod: cannot access '/etc/mono/registry': No such file or directory
    2017-03-24T02:40:23.312078816Z chown: invalid user: '8b370433317866d90a16aa5a:8b370433317866d90a16aa5a'
    2017-03-24T02:40:23.383440920Z Could not remove /etc/apache2/sites-enabled/000-default.conf: No space left on device
    2017-03-24T02:40:23.444920398Z Could not create /etc/apache2/sites-enabled/kudu.conf: No space left on device
    2017-03-24T02:40:23.464088745Z mkdir: cannot create directory '/var/run/apache2': No space left on device
    2017-03-24T02:40:23.464121129Z mktemp: failed to create directory via template '/var/lock/apache2.XXXXXXXXXX': No space left on device
    2017-03-24T02:40:23.467957936Z chmod: missing operand after '755'
    2017-03-24T02:40:23.467981825Z Try 'chmod --help' for more information.

    Not sure what I'm doing wrong. It was my understanding that pulling a new image would replace the existing one? Is there some way I can clear space out?

    Thanks!

    Friday, March 24, 2017 2:55 AM

Answers

  • Assuming that the app in question is the same as discussed previously, the disk is indeed full. I have cleared off old/unused instances of your image, can you try again?

    This is a known issue, and we are actively pursuing a fix, as your scenario (pulling new containers) is not uncommon.

    • Proposed as answer by Nick Walker Friday, March 24, 2017 11:20 PM
    • Marked as answer by mraumer Friday, March 24, 2017 11:30 PM
    Friday, March 24, 2017 9:46 PM

All replies

  • For clarification, are you referring to the steps outlined in the documentation on How to use a Docker image from a private image registry?

    There is a discussion on similar issue with the space limit here, kindly look into that.

    By default, on Azure WebApps, all files are stored in the file system with the application, including the media files. You may wish to know about the main types of files that are dealt on Azure WebApp (Persisted files, Temporary files and Machine level read-only files). 

    Refer to the article File structure on azure to know the sets of files & dirs on Azure WebApp, and check the directories which include the possible increasing files, such as LogFiles, site/repository, site/deployments(for deployment slots) and your directory for uploading files.


    Friday, March 24, 2017 7:07 PM
    Owner
  • Assuming that the app in question is the same as discussed previously, the disk is indeed full. I have cleared off old/unused instances of your image, can you try again?

    This is a known issue, and we are actively pursuing a fix, as your scenario (pulling new containers) is not uncommon.

    • Proposed as answer by Nick Walker Friday, March 24, 2017 11:20 PM
    • Marked as answer by mraumer Friday, March 24, 2017 11:30 PM
    Friday, March 24, 2017 9:46 PM
  • that did it! I'll be more judicious in how often I push images up to Azure until there's a fix. Is there anything I can do on my end in the meantime to clear it out built up artifacts or ensure it doesn't fill up so quickly?

    Thanks Nick!

    Friday, March 24, 2017 11:30 PM
  • There's really not; this is something we hope to abstract away completely so you don't have to worry about it.

    Friday, March 24, 2017 11:45 PM
  • Hi there Nick!

    I think I've helped you guys out w/ at least MY image size. Went from node:7.7.4 to node:7.7.4-alpine and the image size dropped from ~700MB to ~98MB.

    Was running great until i tried to clip the files that get copied to the image, fat-fingered my Dockerfile, and pushed a bunk image to Azure. Now I need a hard reboot on my web instance to pull the latest (hopefully fixed) image if I could get it. It stopped updating around 9pm EDT last night.

    Thanks!

    Monday, March 27, 2017 3:41 PM
  • You should never need a "hard reboot" unless something is really wrong.

    It looks like your app is full-stopped. Have you tried starting it via the portal or the cli and browsing it again? This should pull the newest image.

    Monday, March 27, 2017 5:12 PM
  • I had stopped it to see if a prolonged shutdown would help the situation any. I've started it up again and it does not seem to have begun pulling new images. I get a "service unavailable" error when I attempt to navigate to the URL.
    Monday, March 27, 2017 7:00 PM
  • OK - thanks for your patience. We have repaired the problem on our side. We will analyze what happened and see what we can improve.

    That said, your image immediately exits on startup. You should be able to see the errors via the debug console in your Kudu site, in /home/LogFiles/docker/*_err.log. Have you tested it locally?

    Tuesday, March 28, 2017 12:05 AM
  • The startup issue is fixed now. Sorry that got pushed, I must've mis-tagged it. I've got a decent local workflow set up to test everything before I push it with the exception of in the event I forget to re-build the dev assets for production. Need to write that script still. I'll try to be more careful.

    Despite restarting the app and making a couple of Application Settings changes, a new version hasn't been pulled for over an hour.

    Thanks!

    Tuesday, March 28, 2017 4:15 AM
  • It looks like this was resolved? :latest was pulled a couple times in the last few hours and your app is running. Did you have the wrong container name/tag configured prior to that?
    Tuesday, March 28, 2017 5:41 PM
  • I run into the same issue today, what can I do to clear the old instances of my image?
    Friday, April 7, 2017 12:29 AM
  • If you could share your app name, I can clear them off.

    Our fix for this should be out in a few days.

    Friday, April 7, 2017 12:53 AM
  • Thank you. Can I send you a private message somewhere? I don't want to post the app name in public.
    Friday, April 7, 2017 1:05 AM
  • Will this work for you?
    Friday, April 7, 2017 1:07 AM
  • I have created a site called dummy2342 in the same app service plan. The app service with the issue is the first one in alphabetical order.
    Friday, April 7, 2017 1:14 AM
  • I have cleaned off older versions of your images, you should be good to go.
    Friday, April 7, 2017 4:00 PM
  • Thank you!
    Friday, April 7, 2017 4:40 PM