Azure Linux VM Filesystem I/O error

Question

Hi all,

  We have a Linux VM(SuSE Linux Enterprise Server) on Azure that recently experienced an I/O error(dmesg output below) that caused the root filesystem to be remounted read-only, and in-effect made the VM unusable for other tasks.  Unfortunately, dmesg on SuSE doesn’t use plain timestamps, so I can’t provide an exact time when the errors occurred, but it was after 8:30 PM Pacific Time on 7/24, and before 1:30 PM Pacific Time on 7/25.  The VM was initially created on 7/24

    Furthermore, after restarting the VM via the management console, it was no longer reachable from other VMs or externally via SSH.  It also cannot be pinged from other Azure VMs(and pinging was possible before the crash).

  We also had a couple of VMs(running the Ubuntu 12.04 LTS image) go into read-only mode over the weekend(7/21-7/22), but didn't investigate further or grab log information.

Any thoughts on what might be causing the issue?

Thanks for your help,

Dan

Relevant dmesg output(I can attach the output from boot if necessary):

[102570.045689] hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x4

[102580.073255] hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x4

[102590.108656] hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x4

[102600.630709] hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x4

[102610.656294] hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x4

[102620.686627] hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x4

[102630.695776] sd 0:0:0:0: [sda] Unhandled error code

[102630.695786] sd 0:0:0:0: [sda]  Result: hostbyte=invalid driverbyte=DRIVER_OK

[102630.695796] sd 0:0:0:0: [sda] CDB: Write(10): 2a 08 01 db 5e 20 00 00 08 00

[102630.695814] end_request: critical target error, dev sda, sector 31153696

[102630.695828] end_request: critical target error, dev sda, sector 31153696

[102630.695837] Buffer I/O error on device sda1, logical block 3893956

[102630.695843] lost page write due to I/O error on sda1

[102630.696154] Aborting journal on device sda1.

[102630.696946] EXT3-fs (sda1): error: ext3_journal_start_sb: Detected aborted journal

[102630.696953] EXT3-fs (sda1): error: remounting filesystem read-only

[102630.713427] journal commit I/O error

Answer 1

Hi Dan,

Likely you are hitting this issue:

VM becomes unresponsive or disk becomes read-only

Thanks,
Craig

Answer 2

Hi Craig,

  Thanks for the response, that looks like the right issue.

Dan