Hello!
Recently, we did data migration from one failure domain to another by changing the logical pool's crush_rule (ceph osd pool set <pool_name> crush_rule <rule_name>). But after the migration, we found that some objects were destroyed and their copies on different OSDs were also destroyed.
After taking some experiments, we found that for one object, the data migration uses only one OSD's data to generate three copies on new failure domain, and do not check the crc before migration by default. We tried both filestore and bluestore, and the result is the same. Once silent data corruption appears on one OSD, and when doing data migration, the data source of some objects happens to be it, the new object after the migration will be wrong on all copies.
Is there any way to ensure the correctness of data when data migration? We tried to set filestore_sloppy_crc up or do deep scrub, but the cost is too high.
The experiment we did is something like below:
First create some OSDs and devide them into two different root.
-> # ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-7 0.05699 root root1
-5 0.05699 host ceph01
3 hdd 0.01900 osd.3 up 1.00000 1.00000
4 hdd 0.01900 osd.4 up 1.00000 1.00000
5 hdd 0.01900 osd.5 up 1.00000 1.00000
-1 0.05846 root default
-3 0.05846 host ceph02
0 hdd 0.01949 osd.0 up 1.00000 1.00000
1 hdd 0.01949 osd.1 up 1.00000 1.00000
2 hdd 0.01949 osd.2 up 1.00000 1.00000
Then create a replicated pool on root default. Note we set the failure domain to OSD.
ceph osd pool create test 128 128
Next we put an object into the pool.
-> # cat txt
123
-> # rados -p test put test_copy txt
-> # rados -p test get test_copy -
123
Then we make OSD.0 down, and change its data of object test_copy.
-> # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 test_copy get-bytes
123
-> # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 test_copy set-bytes 120txt
Next we start OSD.0 and do data migration.
ceph osd pool set test crush_rule root1_rule
Finally we try to get the object by rados and ceph-objectstore-tool
-> # rados -p test get test_copy -
error getting test/test_copy: (5) Input/output error
-> # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 test_copy get-bytes
120
-> # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-4 test_copy get-bytes
120
-> # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-5 test_copy get-bytes
120
The data of test_copy on OSD.3 OSD.4 OSD.5 is from OSD.0 which has the silent data corruption.
Our config is below:
-> # cat /etc/ceph/ceph.conf
[global]
fsid = 96c5f802-ca66-4d12-974f-5b5658a18353
mon_initial_members = ceph00
mon_host = 10.18.192.27
auth_cluster_required = none
auth_service_required = none
auth_client_required = none
public_network = 10.18.192.0/24
[mon]
mon_allow_pool_delete = true
The ceph version is below
-> # ceph -v
ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable)
Thanks.
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
