Hm, so you're running OSD nodes with 2GB of RAM and 2x10TB = 20TB of storage? Literally everything posted on this list in relation to HW requirements and related problems will tell you that this simply isn't going to work. The slightest hint of a problem will simply kill the OSD nodes with OOM. Have you tried with smaller disks - like 1TB models (or even smaller like 256GB SSDs) and see if the same problem persists?
On Tue, 6 Mar 2018 at 10:51, 赵赵贺东 <zhaohedong@xxxxxxxxx> wrote:
Hello ceph-users,It is a really really Really tough problem for our team.We investigated in the problem for a long time， try a lot of efforts, but can’t solve the problem, even the concentrate cause of the problem is still unclear for us!So, Anyone give any solution/suggestion/opinion whatever will be highly highly appreciated!!!Problem Summary:When we activate osd, there will be metadata corrupttion in the activating disk, probability is 100% !Admin Nodes&MON node:Platform: X86OS: Ubuntu 16.04Kernel: 4.12.0Ceph: Luminous 12.2.2OSD nodes:Platform: armv7OS: Ubuntu 14.04Kernel: 4.4.39Ceph: Lominous 12.2.2Disk: 10T+10TMemory: 2GBDeploy log:dmesg log:(Sorry arms001-01 dmesg log has log has been lost, but error message about metadata corruption on arms003-10 are the same with arms001-01)Mar 5 11:08:49 arms003-10 kernel: [ 252.534232] XFS (sda1): Unmount and run xfs_repairMar 5 11:08:49 arms003-10 kernel: [ 252.539100] XFS (sda1): First 64 bytes of corrupted metadata buffer:Mar 5 11:08:49 arms003-10 kernel: [ 252.545504] eb82f000: 58 46 53 42 00 00 10 00 00 00 00 00 91 73 fe fb XFSB.........s..Mar 5 11:08:49 arms003-10 kernel: [ 252.553569] eb82f010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................Mar 5 11:08:49 arms003-10 kernel: [ 252.561624] eb82f020: fc 4e e3 89 50 8f 42 aa be bc 07 0c 6e fa 83 2f .N..P.B.....n../Mar 5 11:08:49 arms003-10 kernel: [ 252.569706] eb82f030: 00 00 00 00 80 00 00 07 ff ff ff ff ff ff ff ff ................Mar 5 11:08:49 arms003-10 kernel: [ 252.577778] XFS (sda1): metadata I/O error: block 0x48b9ff80 ("xfs_trans_read_buf_map") error 117 numblks 8Mar 5 11:08:49 arms003-10 kernel: [ 252.602944] XFS (sda1): Metadata corruption detected at xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0x48b9ff80Mar 5 11:08:49 arms003-10 kernel: [ 252.614170] XFS (sda1): Unmount and run xfs_repairMar 5 11:08:49 arms003-10 kernel: [ 252.619030] XFS (sda1): First 64 bytes of corrupted metadata buffer:Mar 5 11:08:49 arms003-10 kernel: [ 252.625403] eb901000: 58 46 53 42 00 00 10 00 00 00 00 00 91 73 fe fb XFSB.........s..Mar 5 11:08:49 arms003-10 kernel: [ 252.633441] eb901010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................Mar 5 11:08:49 arms003-10 kernel: [ 252.641474] eb901020: fc 4e e3 89 50 8f 42 aa be bc 07 0c 6e fa 83 2f .N..P.B.....n../Mar 5 11:08:49 arms003-10 kernel: [ 252.649519] eb901030: 00 00 00 00 80 00 00 07 ff ff ff ff ff ff ff ff ................Mar 5 11:08:49 arms003-10 kernel: [ 252.657554] XFS (sda1): metadata I/O error: block 0x48b9ff80 ("xfs_trans_read_buf_map") error 117 numblks 8Mar 5 11:08:49 arms003-10 kernel: [ 252.675056] XFS (sda1): Metadata corruption detected at xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0x48b9ff80Mar 5 11:08:49 arms003-10 kernel: [ 252.686228] XFS (sda1): Unmount and run xfs_repairMar 5 11:08:49 arms003-10 kernel: [ 252.691054] XFS (sda1): First 64 bytes of corrupted metadata buffer:Mar 5 11:08:49 arms003-10 kernel: [ 252.697425] eb901000: 58 46 53 42 00 00 10 00 00 00 00 00 91 73 fe fb XFSB.........s..Mar 5 11:08:49 arms003-10 kernel: [ 252.705459] eb901010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................Mar 5 11:08:49 arms003-10 kernel: [ 252.713489] eb901020: fc 4e e3 89 50 8f 42 aa be bc 07 0c 6e fa 83 2f .N..P.B.....n../Mar 5 11:08:49 arms003-10 kernel: [ 252.721520] eb901030: 00 00 00 00 80 00 00 07 ff ff ff ff ff ff ff ff ................Mar 5 11:08:49 arms003-10 kernel: [ 252.729558] XFS (sda1): metadata I/O error: block 0x48b9ff80 ("xfs_trans_read_buf_map") error 117 numblks 8Mar 5 11:08:49 arms003-10 kernel: [ 252.741953] XFS (sda1): Metadata corruption detected at xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0x48b9ff80Mar 5 11:08:49 arms003-10 kernel: [ 252.753139] XFS (sda1): Unmount and run xfs_repairMar 5 11:08:49 arms003-10 kernel: [ 252.757955] XFS (sda1): First 64 bytes of corrupted metadata buffer:Mar 5 11:08:49 arms003-10 kernel: [ 252.764336] eb901000: 58 46 53 42 00 00 10 00 00 00 00 00 91 73 fe fb XFSB.........s..Mar 5 11:08:49 arms003-10 kernel: [ 252.772365] eb901010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................Mar 5 11:08:49 arms003-10 kernel: [ 252.780395] eb901020: fc 4e e3 89 50 8f 42 aa be bc 07 0c 6e fa 83 2f .N..P.B.....n../Mar 5 11:08:49 arms003-10 kernel: [ 252.788417] eb901030: 00 00 00 00 80 00 00 07 ff ff ff ff ff ff ff ff ................Mar 5 11:08:49 arms003-10 kernel: [ 252.796514] XFS (sda1): metadata I/O error: block 0x48b9ff80 ("xfs_trans_read_buf_map") error 117 numblks 8Our tries for solving the problem:1.Delploy osd manually, still got the same error has been confirmed.2.Browse kernel bug fix log, but no related bug fix log has been found since kernel 4.4.39.3.Upgrade xfsprogs from 3.1.9 to 4.15.0, error number changed, still but disk will be corrupted while activating osd![2912641.987937] XFS (sda1): Metadata CRC error detected at xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0xfffffff0[2912641.999203] XFS (sda1): Unmount and run xfs_repair[2912642.004202] XFS (sda1): First 64 bytes of corrupted metadata buffer:[2912642.010759] e689a000: 58 46 53 42 00 00 10 00 00 00 00 00 91 73 fe fb XFSB.........s..[2912642.018958] e689a010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................[2912642.027177] e689a020: 61 7b 64 0d fa fe 41 14 bf ea 90 32 6c 73 e5 ad a{d...A....2ls..[2912642.035388] e689a030: 00 00 00 00 50 00 00 08 ff ff ff ff ff ff ff ff ....P...........[2912642.043630] XFS (sda1): metadata I/O error: block 0xfffffff0 ("xfs_trans_read_buf_map") error 74 numblks 8[2912642.060390] XFS (sda1): Metadata CRC error detected at xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0xfffffff0[2912642.071673] XFS (sda1): Unmount and run xfs_repair4.Use the disk as OSD node on X86 will not trigger the the problem has been confirmed.5.Use sgdisk & mkfs.xfs to format the disk, and mount do some read&write dd test then unmount, will not trigger the problem has been confirmed.6.Chang ceph version form 12.2.2 to 10.2.10, the problem still exist has been confirmed.10.2.0 Deploy log_______________________________________________Corruption error is the same as 12.2.2.
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
