I had a problem with a filesystem xfs on gluster. The
filesystem metadata was filled:
Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: space
map metadata: unable to allocate new metadata block
Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
252:2: metadata operation 'dm_thin_insert_block' failed:
error = -28
Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
252:2: aborting current metadata transaction
Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
252:2: switching pool to read-only mode
Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata
I/O error: block 0x701830 ("xfs_buf_iodone_callbacks") error
5 numblks 8
Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access
beyond end of device
Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0,
want=562056, limit=24576
Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
process_bio_read_only: dm_thin_find_block() failed: error =
-5
Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access
beyond end of device
Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0,
want=562056, limit=24576
Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
process_bio_read_only: dm_thin_find_block() failed: error =
-5
Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata
I/O error: block 0x68047c ("xlog_iodone") error 5 numblks 64
Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4):
xfs_do_force_shutdown(0x2) called from line 1170 of file
fs/xfs/xfs_log.c. Return address = 0xffffffffa012a4c1
Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): Log I/O
Error Detected. Shutting down filesystem
Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): Please
umount the filesystem and rectify the problem(s)
Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access
beyond end of device
Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0,
want=562056, limit=24576
Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
process_bio_read_only: dm_thin_find_block() failed: error =
-5
Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata
I/O error: block 0x6804bc ("xlog_iodone") error 5 numblks 64
Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4):
xfs_do_force_shutdown(0x2) called from line 1170 of file
fs/xfs/xfs_log.c. Return address = 0xffffffffa012a4c1
Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access
beyond end of device
Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4):
xfs_log_force: error 5 returned.
Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0,
want=562056, limit=24576
Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
process_bio_read_only: dm_thin_find_block() failed: error =
-5
Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access
beyond end of device
Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0,
want=562056, limit=24576
Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
process_bio_read_only: dm_thin_find_block() failed: error =
-5
Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access
beyond end of device
Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0,
want=562056, limit=24576
Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
process_bio_read_only: dm_thin_find_block() failed: error =
-5
Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata
I/O error: block 0x6804fc ("xlog_iodone") error 5 numblks 64
Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4):
xfs_do_force_shutdown(0x2) called from line 1170 of file
fs/xfs/xfs_log.c. Return address = 0xffffffffa012a4c
After that, gluster was shutdown and with it the 2 server
are shtudown too. The lvm partition was missing and so far I
haven't been able restore the file system. All data is
missing??
I don't understand the situation and I don't know if it's
due a xfs filesystem or glusterfs fail. Someone it has been
this situation?
Thanks.