On Fri, Jan 14, 2022 at 4:54 PM Frank Schilder <frans@xxxxxx> wrote: > > Hi Venky, > > thanks for your reply. I think the first type of messages was a race condition. A user was running rm and find on the same folder at the same time. The second type of message (duplicate inode in stray) might point to an a bit more severe issue. For a while now I observe that ".mds_cache.num_strays" is really large and, on average, constantly increasing: > > # ssh ceph-08 'ceph daemon mds.$(hostname -s) perf dump | jq .mds_cache.num_strays' > 1081531 > > This is by no means justified by people deleting files. Our snapshots rotate completely every 3 days and the stray buckets should get purged regularly. I have 2 questions: > > 1) Would a "cephfs-data-scan scan_links" detect and potentially resolve this problem (orphaned inodes in stray bucket)? Not exactly. It was for the error you were referring to in the mds log. > 2) For a file system of our size, how long would a "cephfs-data-scan scan_links" run approximately (I need to estimate downtime)? I think I can execute up to 35-40 workers. The fs size is: scan_links runs in the order of magnitude as the number of objects in the metadata pool. > > ceph.dir.rbytes="2078289930815425" > ceph.dir.rentries="278320382" > > Thanks for your help! > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Venky Shankar <vshankar@xxxxxxxxxx> > Sent: 12 January 2022 12:24 > To: Frank Schilder > Cc: ceph-users > Subject: Re: cephfs: [ERR] loaded dup inode > > On Tue, Jan 11, 2022 at 6:07 PM Frank Schilder <frans@xxxxxx> wrote: > > > > Hi all, > > > > I found a bunch of error messages like below in our ceph log (2 different types). How bad is this and should I do something? > > > > Ceph version is 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable). > > > > 2022-01-11 11:49:47.687010 [ERR] loaded dup inode 0x10011bac31c [4f8,head] v1046724308 at ~mds0/stray1/10011bac31c, but inode 0x10011bac31c.head v1046760378 already exists at [...]/miniconda3/envs/ffpy_gwa3/lib/python3.6/site-packages/python_dateutil-2.8.0.dist-info/INSTALLER > > > > 2022-01-11 11:49:47.682346 [ERR] loaded dup inode 0x10011bac7fc [4f8,head] v1046725418 at ~mds0/stray1/10011bac7fc, but inode 0x10011bac7fc.head v1046760674 already exists at ~mds0/stray2/10011bac7fc > > I've seen this earlier. Not sure how we end up with an inode in two > stray directories, but it doesn't look serious. > > You could try stopping all MDSs and run `cephfs-data-scan scan_links` > (courtesy Zheng) to see if the errors go away. > > > > > Best regards, > > ================= > > Frank Schilder > > AIT Risø Campus > > Bygning 109, rum S14 > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > -- > Cheers, > Venky > -- Cheers, Venky _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |