Hi, I would like to report the serious problem about data loss in customer environment. An OSD data loss has been propagated to other OSDs. If backfill is performed when shard is missing in a primary OSD, the shard that is corresponding to the shard in a primary OSD is also missing in the OSD to which the backfill is directed. In case of 4+2 erasure coding, if copies are occurred against two OSDs during one backfill, three shards are missing(primary + two copies), making data recovery impossible. This data loss depends on setting of erasure coding and the number of copies during backfill. In fact, I could reproduce this situation. This is the actual data loss, and we need to fix this problem. I will verify this with the latest version of ceph, and issue a ticket to redmine later and also report detail information. In this mail, I share simple information of environment and procedure to reproduce at first. Environment: - Ceph version: Nautilus - Erasure coding: 4+2 - Type: filestore Step to Reproduce: 1. Setup more than 6 OSDs (with leaving some extra OSD out). 2. Store some object to pool. 3. Delete a file from a primary OSD in the PG. (In fact, the shard on the primary OSD was unrecognized due to medium error of the primary OSD in the customer environment. To simulate this situation, run `rm`.) e.g.) rm -f /var/lib/ceph/osd/ceph-7/current/1.0s0_head/<some file>.04.21.09\:55\:* 4. Cause backfill in the PG. This time, I could occur backfill by setting OSD to `in` from `out`. e.g.) ceph osd in osd.5 5. ceph -s show active+clean status but object is lost on both primary and backfilled OSDs. _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
