Hi, What is the actual backtrace from the crash? We occasionally had dup inode errors like this in the past but they never escalated to a crash. You can see my old thread here: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-July/036294.html The developer at that time suggested some manipulations to remove the omap entries for the older version of the inode from the stray dirs, and also to eventually run cephfs-data-scan scan_links. Cheers, Dan On Mon, Aug 23, 2021 at 9:01 PM Pickett, Neale T <neale@xxxxxxxx> wrote: > > hello, ceph-users! > > > We have an old cephfs that is ten different kinds of broken, which we are attempting to (slowly) pull files from. The most recent issue we've hit is that the mds will start up, log hundreds of messages like below, then crash. This is happening in a loop; we can never actually get an mds to stay up long enough to try and fix it. > > > The most recent thing we tried was to do a extents scan, followed by an inode scan. No luck. > > > Any suggestions? Could I maybe just manually delete the stray inodes? > > > 2021-08-23T12:59:40.803056-0600 mds.dellingr [ERR] loaded dup inode 0x100028e324a [2,head] v265523356 at ~mds0/stray3/100028e324a, but inode 0x100028e324a.head v265540146 already exists at ~mds0/stray1/100028e324a > 2021-08-23T12:59:40.803082-0600 mds.dellingr [ERR] loaded dup inode 0x100028e3249 [2,head] v265523350 at ~mds0/stray3/100028e3249, but inode 0x100028e3249.head v265540138 already exists at ~mds0/stray1/100028e3249 > 2021-08-23T12:59:40.803109-0600 mds.dellingr [ERR] loaded dup inode 0x100028e3247 [2,head] v265523354 at ~mds0/stray3/100028e3247, but inode 0x100028e3247.head v265540140 already exists at ~mds0/stray1/100028e3247 > 2021-08-23T12:59:40.803136-0600 mds.dellingr [ERR] loaded dup inode 0x100028e3246 [2,head] v265523344 at ~mds0/stray3/100028e3246, but inode 0x100028e3246.head v265540130 already exists at ~mds0/stray1/100028e3246 > 2021-08-23T12:59:40.803162-0600 mds.dellingr [ERR] loaded dup inode 0x100028e3245 [2,head] v265523342 at ~mds0/stray3/100028e3245, but inode 0x100028e3245.head v265540128 already exists at ~mds0/stray1/100028e3245 > 2021-08-23T12:59:40.803188-0600 mds.dellingr [ERR] loaded dup inode 0x100028e3244 [2,head] v265523348 at ~mds0/stray3/100028e3244, but inode 0x100028e3244.head v265540126 already exists at ~mds0/stray1/100028e3244 > 2021-08-23T12:59:40.803214-0600 mds.dellingr [ERR] loaded dup inode 0x100028e3243 [2,head] v265523346 at ~mds0/stray3/100028e3243, but inode 0x100028e3243.head v265540136 already exists at ~mds0/stray1/100028e3243 > 2021-08-23T12:59:40.803241-0600 mds.dellingr [ERR] loaded dup inode 0x100028e323f [2,head] v265523338 at ~mds0/stray3/100028e323f, but inode 0x100028e323f.head v265540124 already exists at ~mds0/stray1/100028e323f > > > > Neale Pickett <neale@xxxxxxxx> > A-4: Advanced Research in Cyber Systems > Los Alamos National Laboratory > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |