Hi Venky, thanks for your reply. I think the first type of messages was a race condition. A user was running rm and find on the same folder at the same time. The second type of message (duplicate inode in stray) might point to an a bit more severe issue. For a while now I observe that ".mds_cache.num_strays" is really large and, on average, constantly increasing: # ssh ceph-08 'ceph daemon mds.$(hostname -s) perf dump | jq .mds_cache.num_strays' 1081531 This is by no means justified by people deleting files. Our snapshots rotate completely every 3 days and the stray buckets should get purged regularly. I have 2 questions: 1) Would a "cephfs-data-scan scan_links" detect and potentially resolve this problem (orphaned inodes in stray bucket)? 2) For a file system of our size, how long would a "cephfs-data-scan scan_links" run approximately (I need to estimate downtime)? I think I can execute up to 35-40 workers. The fs size is: ceph.dir.rbytes="2078289930815425" ceph.dir.rentries="278320382" Thanks for your help! Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Venky Shankar <vshankar@xxxxxxxxxx> Sent: 12 January 2022 12:24 To: Frank Schilder Cc: ceph-users Subject: Re: cephfs: [ERR] loaded dup inode On Tue, Jan 11, 2022 at 6:07 PM Frank Schilder <frans@xxxxxx> wrote: > > Hi all, > > I found a bunch of error messages like below in our ceph log (2 different types). How bad is this and should I do something? > > Ceph version is 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable). > > 2022-01-11 11:49:47.687010 [ERR] loaded dup inode 0x10011bac31c [4f8,head] v1046724308 at ~mds0/stray1/10011bac31c, but inode 0x10011bac31c.head v1046760378 already exists at [...]/miniconda3/envs/ffpy_gwa3/lib/python3.6/site-packages/python_dateutil-2.8.0.dist-info/INSTALLER > > 2022-01-11 11:49:47.682346 [ERR] loaded dup inode 0x10011bac7fc [4f8,head] v1046725418 at ~mds0/stray1/10011bac7fc, but inode 0x10011bac7fc.head v1046760674 already exists at ~mds0/stray2/10011bac7fc I've seen this earlier. Not sure how we end up with an inode in two stray directories, but it doesn't look serious. You could try stopping all MDSs and run `cephfs-data-scan scan_links` (courtesy Zheng) to see if the errors go away. > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > -- Cheers, Venky _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |