from what i've heard, xfs has problems on arm. use btrfs, or (i
believe?) ext4+bluestore will work.
On Sun, Mar 11, 2018 at 9:49 PM, Christian Wuerdig
<christian.wuerdig@xxxxxxxxx> wrote:
> Hm, so you're running OSD nodes with 2GB of RAM and 2x10TB = 20TB of
> storage? Literally everything posted on this list in relation to HW
> requirements and related problems will tell you that this simply isn't going
> to work. The slightest hint of a problem will simply kill the OSD nodes with
> OOM. Have you tried with smaller disks - like 1TB models (or even smaller
> like 256GB SSDs) and see if the same problem persists?
>
>
> On Tue, 6 Mar 2018 at 10:51, 赵赵贺东 <zhaohedong@xxxxxxxxx> wrote:
>>
>> Hello ceph-users,
>>
>> It is a really really Really tough problem for our team.
>> We investigated in the problem for a long time， try a lot of efforts, but
>> can’t solve the problem, even the concentrate cause of the problem is still
>> unclear for us!
>> So, Anyone give any solution/suggestion/opinion whatever will be highly
>> highly appreciated!!!
>>
>> Problem Summary:
>> When we activate osd, there will be metadata corrupttion in the
>> activating disk, probability is 100% !
>>
>> Admin Nodes&MON node:
>> Platform: X86
>> OS: Ubuntu 16.04
>> Kernel: 4.12.0
>> Ceph: Luminous 12.2.2
>>
>> OSD nodes:
>> Platform: armv7
>> OS: Ubuntu 14.04
>> Kernel: 4.4.39
>> Ceph: Lominous 12.2.2
>> Disk: 10T+10T
>> Memory: 2GB
>>
>> Deploy log:
>>
>>
>> dmesg log:(Sorry arms001-01 dmesg log has log has been lost, but error
>> message about metadata corruption on arms003-10 are the same with
>> arms001-01)
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.534232] XFS (sda1): Unmount and
>> run xfs_repair
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.539100] XFS (sda1): First 64
>> bytes of corrupted metadata buffer:
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.545504] eb82f000: 58 46 53 42 00
>> 00 10 00 00 00 00 00 91 73 fe fb XFSB.........s..
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.553569] eb82f010: 00 00 00 00 00
>> 00 00 00 00 00 00 00 00 00 00 00 ................
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.561624] eb82f020: fc 4e e3 89 50
>> 8f 42 aa be bc 07 0c 6e fa 83 2f .N..P.B.....n../
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.569706] eb82f030: 00 00 00 00 80
>> 00 00 07 ff ff ff ff ff ff ff ff ................
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.577778] XFS (sda1): metadata I/O
>> error: block 0x48b9ff80 ("xfs_trans_read_buf_map") error 117 numblks 8
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.602944] XFS (sda1): Metadata
>> corruption detected at xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data
>> block 0x48b9ff80
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.614170] XFS (sda1): Unmount and
>> run xfs_repair
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.619030] XFS (sda1): First 64
>> bytes of corrupted metadata buffer:
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.625403] eb901000: 58 46 53 42 00
>> 00 10 00 00 00 00 00 91 73 fe fb XFSB.........s..
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.633441] eb901010: 00 00 00 00 00
>> 00 00 00 00 00 00 00 00 00 00 00 ................
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.641474] eb901020: fc 4e e3 89 50
>> 8f 42 aa be bc 07 0c 6e fa 83 2f .N..P.B.....n../
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.649519] eb901030: 00 00 00 00 80
>> 00 00 07 ff ff ff ff ff ff ff ff ................
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.657554] XFS (sda1): metadata I/O
>> error: block 0x48b9ff80 ("xfs_trans_read_buf_map") error 117 numblks 8
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.675056] XFS (sda1): Metadata
>> corruption detected at xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data
>> block 0x48b9ff80
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.686228] XFS (sda1): Unmount and
>> run xfs_repair
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.691054] XFS (sda1): First 64
>> bytes of corrupted metadata buffer:
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.697425] eb901000: 58 46 53 42 00
>> 00 10 00 00 00 00 00 91 73 fe fb XFSB.........s..
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.705459] eb901010: 00 00 00 00 00
>> 00 00 00 00 00 00 00 00 00 00 00 ................
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.713489] eb901020: fc 4e e3 89 50
>> 8f 42 aa be bc 07 0c 6e fa 83 2f .N..P.B.....n../
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.721520] eb901030: 00 00 00 00 80
>> 00 00 07 ff ff ff ff ff ff ff ff ................
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.729558] XFS (sda1): metadata I/O
>> error: block 0x48b9ff80 ("xfs_trans_read_buf_map") error 117 numblks 8
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.741953] XFS (sda1): Metadata
>> corruption detected at xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data
>> block 0x48b9ff80
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.753139] XFS (sda1): Unmount and
>> run xfs_repair
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.757955] XFS (sda1): First 64
>> bytes of corrupted metadata buffer:
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.764336] eb901000: 58 46 53 42 00
>> 00 10 00 00 00 00 00 91 73 fe fb XFSB.........s..
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.772365] eb901010: 00 00 00 00 00
>> 00 00 00 00 00 00 00 00 00 00 00 ................
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.780395] eb901020: fc 4e e3 89 50
>> 8f 42 aa be bc 07 0c 6e fa 83 2f .N..P.B.....n../
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.788417] eb901030: 00 00 00 00 80
>> 00 00 07 ff ff ff ff ff ff ff ff ................
>> Mar 5 11:08:49 arms003-10 kernel: [ 252.796514] XFS (sda1): metadata I/O
>> error: block 0x48b9ff80 ("xfs_trans_read_buf_map") error 117 numblks 8
>>
>> Our tries for solving the problem:
>> 1.Delploy osd manually, still got the same error has been confirmed.
>> 2.Browse kernel bug fix log, but no related bug fix log has been found
>> since kernel 4.4.39.
>> 3.Upgrade xfsprogs from 3.1.9 to 4.15.0, error number changed, still but
>> disk will be corrupted while activating osd!
>>
>> [2912641.987937] XFS (sda1): Metadata CRC error detected at
>> xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0xfffffff0
>> [2912641.999203] XFS (sda1): Unmount and run xfs_repair
>> [2912642.004202] XFS (sda1): First 64 bytes of corrupted metadata buffer:
>> [2912642.010759] e689a000: 58 46 53 42 00 00 10 00 00 00 00 00 91 73 fe fb
>> XFSB.........s..
>> [2912642.018958] e689a010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> [2912642.027177] e689a020: 61 7b 64 0d fa fe 41 14 bf ea 90 32 6c 73 e5 ad
>> a{d...A....2ls..
>> [2912642.035388] e689a030: 00 00 00 00 50 00 00 08 ff ff ff ff ff ff ff ff
>> ....P...........
>> [2912642.043630] XFS (sda1): metadata I/O error: block 0xfffffff0
>> ("xfs_trans_read_buf_map") error 74 numblks 8
>> [2912642.060390] XFS (sda1): Metadata CRC error detected at
>> xfs_dir3_data_read_verify+0x58/0xd0, xfs_dir3_data block 0xfffffff0
>> [2912642.071673] XFS (sda1): Unmount and run xfs_repair
>>
>> 4.Use the disk as OSD node on X86 will not trigger the the problem has
>> been confirmed.
>> 5.Use sgdisk & mkfs.xfs to format the disk, and mount do some read&write
>> dd test then unmount, will not trigger the problem has been confirmed.
>> 6.Chang ceph version form 12.2.2 to 10.2.10, the problem still exist has
>> been confirmed.
>> 10.2.0 Deploy log
>>
>> Corruption error is the same as 12.2.2.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
