Do you have osd_scrub_auto_repair set to true?
Zitat von Petr Bena <petr@bena.rocks>:
Hello,
I wanted to try out (lab ceph setup) what exactly is going to happen
when parts of data on OSD disk gets corrupted. I created a simple
test where I was going through the block device data until I found
something that resembled user data (using dd and hexdump) (/dev/sdd
is a block device that is used by OSD)
INFRA [root@ceph-vm-lab5 ~]# dd if=/dev/sdd bs=32 count=1 skip=33920
| hexdump -C
00000000 6e 20 69 64 3d 30 20 65 78 65 3d 22 2f 75 73 72 |n id=0
exe="/usr|
00000010 2f 73 62 69 6e 2f 73 73 68 64 22 20 68 6f 73 74 |/sbin/sshd" host|
Then I deliberately overwrote 32 bytes using random data:
INFRA [root@ceph-vm-lab5 ~]# dd if=/dev/urandom of=/dev/sdd bs=32
count=1 seek=33920
INFRA [root@ceph-vm-lab5 ~]# dd if=/dev/sdd bs=32 count=1 skip=33920
| hexdump -C
00000000 25 75 af 3e 87 b0 3b 04 78 ba 79 e3 64 fc 76 d2 |%u.>..;.x.y.d.v.|
00000010 9e 94 00 c2 45 a5 e1 d2 a8 86 f1 25 fc 18 07 5a |....E......%...Z|
At this point I would expect some sort of data corruption. I
restarted the OSD daemon on this host to make sure it flushes any
potentially buffered data. It restarted OK without noticing
anything, which was expected.
Then I ran
ceph osd scrub 5
ceph osd deep-scrub 5
And waiting for all scheduled scrub operations for all PGs to finish.
No inconsistency was found. No errors reported, scrubs just finished
OK, data are still visibly corrupt via hexdump.
Did I just hit some block of data that WAS used by OSD, but was
marked deleted and therefore no longer used or am I missing
something? I would expect CEPH to detect disk corruption and
automatically replace the invalid data with a valid copy?
I use only replica pools in this lab setup, for RBD and CephFS.
Thanks
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
