Hi,
I had server failure that starts from one disk failure:
Oct 14 03:25:04 s3-10-177-64-6 kernel: [1027237.023986] sd 4:2:26:0:
[sdaa] Unhandled error code
Oct 14 03:25:04 s3-10-177-64-6 kernel: [1027237.023990] sd 4:2:26:0:
[sdaa] Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
Oct 14 03:25:04 s3-10-177-64-6 kernel: [1027237.023995] sd 4:2:26:0:
[sdaa] CDB: Read(10): 28 00 00 00 00 d0 00 00 10 00
Oct 14 03:25:04 s3-10-177-64-6 kernel: [1027237.024005] end_request:
I/O error, dev sdaa, sector 208
Oct 14 03:25:04 s3-10-177-64-6 kernel: [1027237.024744] XFS (sdaa):
metadata I/O error: block 0xd0 ("xfs_trans_read_buf") error 5 buf
count 8192
Oct 14 03:25:04 s3-10-177-64-6 kernel: [1027237.025879] XFS (sdaa):
xfs_imap_to_bp: xfs_trans_read_buf() returned error 5.
Oct 14 03:25:28 s3-10-177-64-6 kernel: [1027260.820288] XFS (sdaa):
metadata I/O error: block 0xd0 ("xfs_trans_read_buf") error 5 buf
count 8192
Oct 14 03:25:28 s3-10-177-64-6 kernel: [1027260.821194] XFS (sdaa):
xfs_imap_to_bp: xfs_trans_read_buf() returned error 5.
Oct 14 03:25:32 s3-10-177-64-6 kernel: [1027264.667851] XFS (sdaa):
metadata I/O error: block 0xd0 ("xfs_trans_read_buf") error 5 buf
count 8192
this caused that the server has been unresponsive.
After server restart 3 of 26 osd on it are down.
In ceph-osd log after "debug osd = 10" and restart is:
2013-10-14 06:21:23.141936 7fdeb4872700 -1 osd.47 43203 *** Got signal
Terminated ***
2013-10-14 06:21:23.142141 7fdeb4872700 -1 osd.47 43203 pausing thread pools
2013-10-14 06:21:23.142146 7fdeb4872700 -1 osd.47 43203 flushing io
2013-10-14 06:21:25.406187 7f02690f9780 0
filestore(/vol0/data/osd.47) mount FIEMAP ioctl is supported and
appears to work
2013-10-14 06:21:25.406204 7f02690f9780 0
filestore(/vol0/data/osd.47) mount FIEMAP ioctl is disabled via
'filestore fiemap' config option
2013-10-14 06:21:25.406557 7f02690f9780 0
filestore(/vol0/data/osd.47) mount did NOT detect btrfs
2013-10-14 06:21:25.412617 7f02690f9780 0
filestore(/vol0/data/osd.47) mount syncfs(2) syscall fully supported
(by glibc and kernel)
2013-10-14 06:21:25.412831 7f02690f9780 0
filestore(/vol0/data/osd.47) mount found snaps <>
2013-10-14 06:21:25.415798 7f02690f9780 0
filestore(/vol0/data/osd.47) mount: enabling WRITEAHEAD journal mode:
btrfs not detected
2013-10-14 06:21:26.078377 7f02690f9780 2 osd.47 0 mounting
/vol0/data/osd.47 /vol0/data/osd.47/journal
2013-10-14 06:21:26.080872 7f02690f9780 0
filestore(/vol0/data/osd.47) mount FIEMAP ioctl is supported and
appears to work
2013-10-14 06:21:26.080885 7f02690f9780 0
filestore(/vol0/data/osd.47) mount FIEMAP ioctl is disabled via
'filestore fiemap' config option
2013-10-14 06:21:26.081289 7f02690f9780 0
filestore(/vol0/data/osd.47) mount did NOT detect btrfs
2013-10-14 06:21:26.087524 7f02690f9780 0
filestore(/vol0/data/osd.47) mount syncfs(2) syscall fully supported
(by glibc and kernel)
2013-10-14 06:21:26.087582 7f02690f9780 0
filestore(/vol0/data/osd.47) mount found snaps <>
2013-10-14 06:21:26.089614 7f02690f9780 0
filestore(/vol0/data/osd.47) mount: enabling WRITEAHEAD journal mode:
btrfs not detected
2013-10-14 06:21:26.726676 7f02690f9780 2 osd.47 0 boot
2013-10-14 06:21:26.726773 7f02690f9780 10 osd.47 0 read_superblock
sb(16773c25-5054-4451-bf9f-efc1f7f21b89 osd.47
63cf7d70-99cb-0ab1-4006-00000000002f e43203 [41261,43203]
lci=[43194,43203])
2013-10-14 06:21:26.726862 7f02690f9780 10 osd.47 0 add_map_bl 43203 82622 bytes
2013-10-14 06:21:26.727184 7f02690f9780 10 osd.47 43203 load_pgs
2013-10-14 06:21:26.727643 7f02690f9780 10 osd.47 43203 load_pgs
ignoring unrecognized meta
2013-10-14 06:21:26.727681 7f02690f9780 10 osd.47 43203 load_pgs
3.df1_TEMP clearing temp
osd.47 is still down, I put it out from cluster.
47 1 osd.47 down 0
How can I check what is wrong?
ceph -v
ceph version 0.56.6 (95a0bda7f007a33b0dc7adf4b330778fa1e5d70c)
--
Pozdrawiam
Dominik
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
