Dear cephers,
We've had a few (dozen or so) rather odd scrub errors in our Mimic
(13.2.6) cephfs:
2019-11-15 07:52:52.614 7fffcc41f700 0 log_channel(cluster) log [DBG] :
2.b5b scrub starts
2019-11-15 07:52:55.190 7fffcc41f700 -1 log_channel(cluster) log [ERR] :
2.b5b shard 599 soid 2:dad01506:::100314224ad.00000160:head : candidate
size 4158512 info size 0 mismatch
2019-11-15 07:52:55.190 7fffcc41f700 -1 log_channel(cluster) log [ERR] :
2.b5b shard 2768 soid 2:dad01506:::100314224ad.00000160:head : candidate
size 4158512 info size 0 mismatch
2019-11-15 07:52:55.190 7fffcc41f700 -1 log_channel(cluster) log [ERR] :
2.b5b shard 3512 soid 2:dad01506:::100314224ad.00000160:head : candidate
size 4158512 info size 0 mismatch
2019-11-15 07:52:55.190 7fffcc41f700 -1 log_channel(cluster) log [ERR] :
2.b5b soid 2:dad01506:::100314224ad.00000160:head : failed to pick
suitable object info
2019-11-15 07:52:55.198 7fffcc41f700 -1 log_channel(cluster) log [ERR] :
scrub 2.b5b 2:dad01506:::100314224ad.00000160:head : on disk size
(4158512) does not match object info size (0) adjusted for ondisk to (0)
2019-11-15 07:53:55.441 7fffcc41f700 -1 log_channel(cluster) log [ERR] :
2.b5b scrub 4 errors
Finding the file - it turns out to be a very small file:
1100338046125 -rw-r----- 1 schen schen 41237 Nov 14 17:18
/mnt/ceph/users/schen/main/Jellium/3u3d_3D/fort.4321169
We use 4MB stripe size - and it looks like the scrub complains about
object 0x160, which is way beyond the end of the file (since the file
should fit in just one object). Retrieving the object gets an empty one
- and it looks like all the objects between object 1 and 0x160 also
exist as empty objects (and object 0 contains the whole, correct file
contents). Any ideas why so many empty objects get created beyond the
end of the file? Would this be the result of the file being
overwritten/truncated? Just for my understanding - if it is truncation,
is that done by the client, or the MDS?
Any ideas how the inconsistencies could have come about? Possibly
something failed during the file truncation?
Thanks,
Andras
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
