locked
IO Error on Linux VM with File Storage and CIFS RRS feed

  • Question

  • Hi,

    we have connected an Azure File Storage (Standard, 500GB size, LRS) to our Linux VM (Ubuntu 16.04) and are sporadically receiving IO-Errors when accessing the mounted filestorage. 

    When trying to do a ls -la the system responds with Input/Output Error. If I run the ls command again, the directory is listed fine.

    root@blabla:/mnt/blabla# ls
    ls: cannot open directory '.': Input/output error
    

    This is how we mount the share, we followed Microsoft's instructions:

    //mycustomstorage.file.core.windows.net/myshare /mnt/myshare cifs vers=3.0,username=mycustomstorage,password=XXXX==,dir_mode=0777,file_mode=0777

    And this is some info from our logs:

    May 17 12:40:28 MYMACHINE kernel: [ 5827.834812] CIFS VFS: Autodisabling the use of server inode numbers on \\mycustomstorage.file.core.windows.net\myshare. This server doesn't seem to support them properly. Hardlinks will not be recognized on this mount. Consider mounting with the "noserverino" option to silence this message.
    May 17 13:17:01 MYMACHINE CRON[13837]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
    May 17 13:28:19 MYMACHINE kernel: [ 8698.680733] CIFS VFS: open dir failed

    If it wasn't for its unstability, the storage would run quite nice with around 60MB/s transfer rate.

    I can't provoke the IO error nor can I detect a pattern when it occours. Has anyone experienced similiar problems or an idea how to handle this?

    Thanks!


    Tuesday, May 17, 2016 2:03 PM

All replies

  • Hi,

    Are you following this document to Mount the file share?

    For Ubuntu Server 15.04 VM, you can mount the file as below:

    azureuser@azureconubuntu:~$ sudo apt-get install cifs-utils

    azureuser@azureconubuntu:~$ sudo mkdir /mnt/mountpoint

    azureuser@azureconubuntu:~$ sudo mount -t cifs //myaccountname.file.core.windows.net/mysharename /mnt/mountpoint -o vers=3.0,user=myaccountname,password=StorageAccountKeyEndingIn==,dir_mode=0777,file_mode=0777

    azureuser@azureconubuntu:~$ df -h /mnt/mountpoint

    Filesystem  Size  Used Avail Use% Mounted on

    //myaccountname.file.core.windows.net/mysharename  5.0T   64K  5.0T   1% /mnt/mountpoint

    Regards,

    Vikranth S.


    Wednesday, May 18, 2016 12:50 PM
  • Thanks for the reply.

    Yes, we followed exactly the document you linked. The mounting in itself works fine but every few minutes I/O errors occur. Running a process that tries to access many files from the share (e.g. compiling sourcecode) always fails with I/O errors.

    Are there any ways to debug this? I suppose it's a linux specific problem but we haven't tested it on Windows VMs yet nor on other linux versions.

    I'd also like to note that it's a relatively plain Ubuntu 16.04 VM with just a basic LAMP-Stack installed and it has been updated from the official repositories.


    • Edited by mhelwig Wednesday, May 18, 2016 2:49 PM
    Wednesday, May 18, 2016 2:40 PM
  • Did you manage to figure out how to rectify the problem?  We are experiencing similar issues albeit with CentOS vms.

    Cheers

    Raf

    Thursday, October 26, 2017 5:07 PM
  • Could you explain a little more about your environment and share a screenshot of the error message?

    Do click on "Mark as Answer" on the post that helps you, this can be beneficial to other community members.

    Monday, October 30, 2017 2:36 PM
  • Has this been resolved or are there thoughts on what we could check to help isolate the issue? We are running into a similar issue with our shared storage account.  We have multiple VMs that use the shared storage for shared files and data mounts.

    All of our VMs and storage account are in the same region.


    Error 
    [stagevm3 mnt]$ ls -ltr
    ls: cannot access sharedstage: Input/output error
    total 4
    ??????????? ? ?    ?       ?            ? sharedstage
    drwxr-xr-x. 3 root root 4096 Jul 10 12:48 resource
    drwxr-xr-x. 3 root root   19 Jul 10 12:48 cdrom

    [stagevm3 mnt]$ ls -ltr
    total 4
    drwxrwxrwx. 2 root aca     0 Jun 30 12:27 sharedstage
    drwxr-xr-x. 3 root root 4096 Jul 10 12:48 resource
    drwxr-xr-x. 3 root root   19 Jul 10 12:48 cdrom

    How it is mounted (slightly sanitized)

    [stagemv3 ~]$ sudo mount | grep cifs
    //sharedstage.file.core.windows.net/sharedstage on /mnt/sharedstage type cifs (rw,relatime,vers=3.0,sec=ntlmssp,cache=strict,username=sharedstage,domain=X,uid=0,forceuid,gid=1006,forcegid,addr=52.xxx.xxx.xxx,file_mode=0777,dir_mode=0777,nounix,serverino,rsize=1048576,wsize=1048576,actimeo=1)


      Static hostname: stagevm3
             Icon name: computer-vm
        Virtualization: microsoft
      Operating System: CentOS Linux 7 (Core)
           CPE OS Name: cpe:/o:centos:centos:7
                Kernel: Linux 3.10.0-327.36.3.el7.x86_64
          Architecture: x86-64

    Any thoughts on what we could check or do to resolve the issue would be most appreciated

    Thursday, November 2, 2017 4:59 PM
  • You may be running into the reconnect issue which is fixed in latest kernel. The below link has more information and intermediate workaround

    https://docs.microsoft.com/en-us/azure/storage/files/storage-troubleshoot-linux-file-connection-problems#mount-error112-host-is-down-because-of-a-reconnection-time-out

     

    Do click on "Mark as Answer" on the post that helps you, this can be beneficial to other community members.

     

    • Proposed as answer by vikranth s Monday, November 27, 2017 7:22 PM
    Tuesday, November 7, 2017 8:51 PM