Am 10.01.19 um 16:53 schrieb Jason Dillaman: > On Thu, Jan 10, 2019 at 10:50 AM Oliver Freyermuth > <freyermuth@xxxxxxxxxxxxxxxxxx> wrote: >> >> Dear Jason and list, >> >> Am 10.01.19 um 16:28 schrieb Jason Dillaman: >>> On Thu, Jan 10, 2019 at 4:01 AM Oliver Freyermuth >>> <freyermuth@xxxxxxxxxxxxxxxxxx> wrote: >>>> >>>> Dear Cephalopodians, >>>> >>>> I performed several consistency checks now: >>>> - Exporting an RBD snapshot before and after the object map rebuilding. >>>> - Exporting a backup as raw image, all backups (re)created before and after the object map rebuilding. >>>> - md5summing all of that for a snapshot for which the rebuilding was actually needed. >>>> >>>> The good news: I found that all checksums are the same. So the backups are (at least for those I checked) not broken. >>>> >>>> I also checked the source and found: >>>> https://github.com/ceph/ceph/blob/master/src/include/rbd/object_map_types.h >>>> So to my understanding, the object map entries are OBJECT_EXISTS, but should be OBJECT_EXISTS_CLEAN. >>>> Do I understand correctly that OBJECT_EXISTS_CLEAN relates to the object being unchanged ("clean") as compared to another snapshot / the main volume? >>>> >>>> If so, this would explain why the backups, exports etc. are all okay, since the backup tools only got "too many" objects in the fast-diff and >>>> hence extracted too many objects from Ceph-RBD even though that was not needed. Since both Benji and Backy2 deduplicate again in their backends, >>>> this causes only a minor network traffic inefficiency. >>>> >>>> Is my understanding correct? >>>> Then the underlying issue would still be a bug, but (as it seems) a harmless one. >>> >>> Yes, your understanding is correct in that it's harmless from a >>> data-integrity point-of-view. >>> >>> During the creation of the snapshot, the current object map (for the >>> HEAD revision) is copied to a new object map for that snapshot and >>> then all the objects in the HEAD revision snapshot are marked as >>> EXISTS_CLEAN (if they EXIST). Somehow an IO operation is causing the >>> object map to think there is an update, but apparently no object >>> update is actually occurring (or at least the OSD doesn't think a >>> change occurred). >> >> thanks a lot for the clarification! Good to know my understanding is correct. >> >> I re-checked all object maps just now. Again, the most recent snapshots show this issue, but only those. >> The only "special" thing which probably not everybody is doing would likely be us running fstrim in the machines >> running from the RBD regularly, to conserve space. >> >> I am not sure how exactly the DISCARD operation is handled in rbd. But since this was my guess, I just did an fstrim inside one of the VMs, >> and checked the object-maps again. I get: >> 2019-01-10 16:44:25.320 7f06f67fc700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.4f587327b23c6.0000000000000040 marked as 1, but should be 3 >> In this case, I got it for the volume itself and not a snapshot. >> >> So it seems to me that sometimes, DISCARD causes objects to think they have been updated, albeit they have not. >> Sadly due to in-depth code knowledge and lack of a real debug setup I can not track it down further :-(. >> >> Cheers and hope that helps a code expert in tracking it down (at least it's not affecting data integrity), > > Thanks, that definitely provides a good investigation starting point. Should we also put it into a ticket, so it can be tracked? I could do it if you like. On the other hand, maybe you could summarize the issue more concisely than I can. Cheers and all the best, Oliver > >> Oliver >> >>> >>>> I'll let you know if it happens again to some of our snapshots, and if so, if it only happens to newly created ones... >>>> >>>> Cheers, >>>> Oliver >>>> >>>> Am 10.01.19 um 01:18 schrieb Oliver Freyermuth: >>>>> Dear Cephalopodians, >>>>> >>>>> inspired by http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-January/032092.html I did a check of the object-maps of our RBD volumes >>>>> and snapshots. We are running 13.2.1 on the cluster I am talking about, all hosts (OSDs, MONs, RBD client nodes) still on CentOS 7.5. >>>>> >>>>> Sadly, I found that for at least 50 % of the snapshots (only the snapshots, not the volumes themselves), I got something like: >>>>> -------------------------------------------------------------------------------------------------- >>>>> 2019-01-09 23:00:06.481 7f89aeffd700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.519c46b8b4567.0000000000000260 marked as 1, but should be 3 >>>>> 2019-01-09 23:00:06.563 7f89aeffd700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.519c46b8b4567.0000000000000840 marked as 1, but should be 3 >>>>> -------------------------------------------------------------------------------------------------- >>>>> 2019-01-09 23:00:09.166 7fbcff7fe700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.519c46b8b4567.0000000000000480 marked as 1, but should be 3 >>>>> 2019-01-09 23:00:09.228 7fbcff7fe700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.519c46b8b4567.0000000000000840 marked as 1, but should be 3 >>>>> -------------------------------------------------------------------------------------------------- >>>>> It often appears to affect 1-3 entries in the map of a snapshot. The Object Map was *not* marked invalid before I ran the check. >>>>> After rebuilding it, the check is fine again. >>>>> >>>>> The cluster has not yet seen any Ceph update (it was installed as 13.2.1, we plan to upgrade to 13.2.4 soonish). >>>>> There have been no major causes of worries so far. We purged a single OSD disk, balanced PGs with upmap, modified the CRUSH topology slightly etc. >>>>> The cluster never was in a prolonged unhealthy period nor did we have to repair any PG. >>>>> >>>>> Is this a known error? >>>>> Is it harmful, or is this just something like reference counting being off, and objects being in the map which did not really change in the snapshot? >>>>> >>>>> Our usecase, in case that helps to understand or reproduce: >>>>> - RBDs are used as disks for qemu/kvm virtual machines. >>>>> - Every night: >>>>> - We run an fstrim in the VM (which propagates to RBD and purges empty blocks), fsfreeze it, take a snapshot, thaw it again. >>>>> - After that, we run two backups with Benji backup ( https://benji-backup.me/ ) and Backy2 backup ( http://backy2.com/docs/ ) >>>>> which seems to work rather well so far. >>>>> - We purge some old snapshots. >>>>> >>>>> We use the following RBD feature flags: >>>>> layering, exclusive-lock, object-map, fast-diff, deep-flatten >>>>> >>>>> Since Benji and Backy2 are optimized for differential RBD backups to deduplicated storage, they leverage "rbd diff" (and hence make use of fast-diff, I would think). >>>>> If rbd diff produces wrong output due to this issue, it would affect our backups (but it would also affect classic backups of snapshots via "rbd export"...). >>>>> In case the issue is known or understood, can somebody extrapolate whether this means "rbd diff" contains too many blocks or actually misses changed blocks? >>>>> >>>>> >>>>> We are from now on running daily, full object-map checks on all volumes and backups, and automatically rebuild any object-map which was found invalid after the check. >>>>> Hopefully, this will allow to correlate the appearance of these issues with "something" happening on the cluster. >>>>> I did not detect a clean pattern in the affected snapshots, though, it seemed rather random... >>>>> >>>>> Maybe it would also help to understand this issue if somebody else using RBD in a similar manner on Mimic could also check the object-maps. >>>>> Since this issue does not show up until a check is performed, this was below our radar for many months now... >>>>> >>>>> Cheers, >>>>> Oliver >>>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@xxxxxxxxxxxxxx >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> >>> >>
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
