Re: Invalid RBD object maps of snapshots on Mimic

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Am 10.01.19 um 16:53 schrieb Jason Dillaman:
> On Thu, Jan 10, 2019 at 10:50 AM Oliver Freyermuth
> <freyermuth@xxxxxxxxxxxxxxxxxx> wrote:
>>
>> Dear Jason and list,
>>
>> Am 10.01.19 um 16:28 schrieb Jason Dillaman:
>>> On Thu, Jan 10, 2019 at 4:01 AM Oliver Freyermuth
>>> <freyermuth@xxxxxxxxxxxxxxxxxx> wrote:
>>>>
>>>> Dear Cephalopodians,
>>>>
>>>> I performed several consistency checks now:
>>>> - Exporting an RBD snapshot before and after the object map rebuilding.
>>>> - Exporting a backup as raw image, all backups (re)created before and after the object map rebuilding.
>>>> - md5summing all of that for a snapshot for which the rebuilding was actually needed.
>>>>
>>>> The good news: I found that all checksums are the same. So the backups are (at least for those I checked) not broken.
>>>>
>>>> I also checked the source and found:
>>>> https://github.com/ceph/ceph/blob/master/src/include/rbd/object_map_types.h
>>>> So to my understanding, the object map entries are OBJECT_EXISTS, but should be OBJECT_EXISTS_CLEAN.
>>>> Do I understand correctly that OBJECT_EXISTS_CLEAN relates to the object being unchanged ("clean") as compared to another snapshot / the main volume?
>>>>
>>>> If so, this would explain why the backups, exports etc. are all okay, since the backup tools only got "too many" objects in the fast-diff and
>>>> hence extracted too many objects from Ceph-RBD even though that was not needed. Since both Benji and Backy2 deduplicate again in their backends,
>>>> this causes only a minor network traffic inefficiency.
>>>>
>>>> Is my understanding correct?
>>>> Then the underlying issue would still be a bug, but (as it seems) a harmless one.
>>>
>>> Yes, your understanding is correct in that it's harmless from a
>>> data-integrity point-of-view.
>>>
>>> During the creation of the snapshot, the current object map (for the
>>> HEAD revision) is copied to a new object map for that snapshot and
>>> then all the objects in the HEAD revision snapshot are marked as
>>> EXISTS_CLEAN (if they EXIST). Somehow an IO operation is causing the
>>> object map to think there is an update, but apparently no object
>>> update is actually occurring (or at least the OSD doesn't think a
>>> change occurred).
>>
>> thanks a lot for the clarification! Good to know my understanding is correct.
>>
>> I re-checked all object maps just now. Again, the most recent snapshots show this issue, but only those.
>> The only "special" thing which probably not everybody is doing would likely be us running fstrim in the machines
>> running from the RBD regularly, to conserve space.
>>
>> I am not sure how exactly the DISCARD operation is handled in rbd. But since this was my guess, I just did an fstrim inside one of the VMs,
>> and checked the object-maps again. I get:
>> 2019-01-10 16:44:25.320 7f06f67fc700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.4f587327b23c6.0000000000000040 marked as 1, but should be 3
>> In this case, I got it for the volume itself and not a snapshot.
>>
>> So it seems to me that sometimes, DISCARD causes objects to think they have been updated, albeit they have not.
>> Sadly due to in-depth code knowledge and lack of a real debug setup I can not track it down further :-(.
>>
>> Cheers and hope that helps a code expert in tracking it down (at least it's not affecting data integrity),
> 
> Thanks, that definitely provides a good investigation starting point.

Should we also put it into a ticket, so it can be tracked? 
I could do it if you like. On the other hand, maybe you could summarize the issue more concisely than I can. 

Cheers and all the best,
	Oliver

> 
>>         Oliver
>>
>>>
>>>> I'll let you know if it happens again to some of our snapshots, and if so, if it only happens to newly created ones...
>>>>
>>>> Cheers,
>>>>          Oliver
>>>>
>>>> Am 10.01.19 um 01:18 schrieb Oliver Freyermuth:
>>>>> Dear Cephalopodians,
>>>>>
>>>>> inspired by http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-January/032092.html I did a check of the object-maps of our RBD volumes
>>>>> and snapshots. We are running 13.2.1 on the cluster I am talking about, all hosts (OSDs, MONs, RBD client nodes) still on CentOS 7.5.
>>>>>
>>>>> Sadly, I found that for at least 50 % of the snapshots (only the snapshots, not the volumes themselves), I got something like:
>>>>> --------------------------------------------------------------------------------------------------
>>>>> 2019-01-09 23:00:06.481 7f89aeffd700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.519c46b8b4567.0000000000000260 marked as 1, but should be 3
>>>>> 2019-01-09 23:00:06.563 7f89aeffd700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.519c46b8b4567.0000000000000840 marked as 1, but should be 3
>>>>> --------------------------------------------------------------------------------------------------
>>>>> 2019-01-09 23:00:09.166 7fbcff7fe700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.519c46b8b4567.0000000000000480 marked as 1, but should be 3
>>>>> 2019-01-09 23:00:09.228 7fbcff7fe700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.519c46b8b4567.0000000000000840 marked as 1, but should be 3
>>>>> --------------------------------------------------------------------------------------------------
>>>>> It often appears to affect 1-3 entries in the map of a snapshot. The Object Map was *not* marked invalid before I ran the check.
>>>>> After rebuilding it, the check is fine again.
>>>>>
>>>>> The cluster has not yet seen any Ceph update (it was installed as 13.2.1, we plan to upgrade to 13.2.4 soonish).
>>>>> There have been no major causes of worries so far. We purged a single OSD disk, balanced PGs with upmap, modified the CRUSH topology slightly etc.
>>>>> The cluster never was in a prolonged unhealthy period nor did we have to repair any PG.
>>>>>
>>>>> Is this a known error?
>>>>> Is it harmful, or is this just something like reference counting being off, and objects being in the map which did not really change in the snapshot?
>>>>>
>>>>> Our usecase, in case that helps to understand or reproduce:
>>>>> - RBDs are used as disks for qemu/kvm virtual machines.
>>>>> - Every night:
>>>>>     - We run an fstrim in the VM (which propagates to RBD and purges empty blocks), fsfreeze it, take a snapshot, thaw it again.
>>>>>     - After that, we run two backups with Benji backup ( https://benji-backup.me/ ) and Backy2 backup ( http://backy2.com/docs/ )
>>>>>       which seems to work rather well so far.
>>>>>     - We purge some old snapshots.
>>>>>
>>>>> We use the following RBD feature flags:
>>>>> layering, exclusive-lock, object-map, fast-diff, deep-flatten
>>>>>
>>>>> Since Benji and Backy2 are optimized for differential RBD backups to deduplicated storage, they leverage "rbd diff" (and hence make use of fast-diff, I would think).
>>>>> If rbd diff produces wrong output due to this issue, it would affect our backups (but it would also affect classic backups of snapshots via "rbd export"...).
>>>>> In case the issue is known or understood, can somebody extrapolate whether this means "rbd diff" contains too many blocks or actually misses changed blocks?
>>>>>
>>>>>
>>>>> We are from now on running daily, full object-map checks on all volumes and backups, and automatically rebuild any object-map which was found invalid after the check.
>>>>> Hopefully, this will allow to correlate the appearance of these issues with "something" happening on the cluster.
>>>>> I did not detect a clean pattern in the affected snapshots, though, it seemed rather random...
>>>>>
>>>>> Maybe it would also help to understand this issue if somebody else using RBD in a similar manner on Mimic could also check the object-maps.
>>>>> Since this issue does not show up until a check is performed, this was below our radar for many months now...
>>>>>
>>>>> Cheers,
>>>>>        Oliver
>>>>>
>>>>
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users@xxxxxxxxxxxxxx
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>>
>>>
>>

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux