Chris,
After you run "scrub repair" followed by a "scrub" without any issues, and
if the "damage ls" still shows you an error, try running "damage rm" and
re-run "scrub" to see if the system still reports a damage.
Please update the upstream tracker with your findings if possible.
--
Milind
On Sun, Mar 13, 2022 at 2:41 AM Chris Palmer <chris.palmer@xxxxxxxxx> wrote:
> Ok, restarting mds.0 cleared it. I then restarted the others until this
> one was again active, and repeated the scrub ~mdsdir which was then clean.
>
> I don't know what caused it, or why restarting the MDS was necessary but
> it has done the trick.
>
> On 12/03/2022 19:14, Chris Palmer wrote:
> > Hi Miland (or anyone else who can help...)
> >
> > Reading this thread made me realise I had overlooked cephfs scrubbing,
> > so i tried it on a small 16.2.7 cluster. The normal forward scrub
> > showed nothing. However "ceph tell mds.0 scrub start ~mdsdir
> > recursive" did find one backtrace error (putting the cluster into
> > HEALTH_ERR). I then did a repair which according to the log did
> > rewrite the inode, and subsequent scrubs have not found it.
> >
> > However the cluster health is still ERR, and the MDS still shows the
> > damage:
> >
> > ceph@xxxx1:~$ ceph tell mds.0 damage ls
> > 2022-03-12T18:42:01.609+0000 7f1b817fa700 0 client.173985213
> > ms_handle_reset on v2:192.168.80.121:6824/939134894
> > 2022-03-12T18:42:01.625+0000 7f1b817fa700 0 client.173985219
> > ms_handle_reset on v2:192.168.80.121:6824/939134894
> > [
> > {
> > "damage_type": "backtrace",
> > "id": 3308827822,
> > "ino": 256,
> > "path": "~mds0"
> > }
> > ]
> >
> > What are the right steps from here? Has the error actually been
> > corrected but just needs clearing or is it still there?
> >
> > In case it is relevant: there is one active and two standby MDS. The
> > log is from the node currently hosting rank 0.
> >
> > From the mds log:
> >
> > 2022-03-12T18:13:41.593+0000 7f61d30c1700 1 mds.xxxx1 asok_command:
> > scrub start {path=~mdsdir,prefix=scrub start,scrubops=[recursive]}
> > (starting...)
> > 2022-03-12T18:13:41.593+0000 7f61cb0b1700 0 log_channel(cluster) log
> > [INF] : scrub queued for path: ~mds0
> > 2022-03-12T18:13:41.593+0000 7f61cb0b1700 0 log_channel(cluster) log
> > [INF] : scrub summary: idle+waiting paths [~mds0]
> > 2022-03-12T18:13:41.593+0000 7f61cb0b1700 0 log_channel(cluster) log
> > [INF] : scrub summary: active paths [~mds0]
> > 2022-03-12T18:13:41.601+0000 7f61cb0b1700 0 log_channel(cluster) log
> > [WRN] : Scrub error on inode 0x100 (~mds0) see mds.xxxx1 log and
> > `damage ls` output for details
> > 2022-03-12T18:13:41.601+0000 7f61cb0b1700 -1 mds.0.scrubstack
> > _validate_inode_done scrub error on inode [inode 0x100 [...2,head]
> > ~mds0/ auth v6798 ap=1 snaprealm=0x55d59548
> > 4800 f(v0 10=0+10) n(v1815 rc2022-03-12T16:01:44.218294+0000
> > b1017620718 375=364+11)/n(v0 rc2019-10-29T10:52:34.302967+0000
> > 11=0+11) (inest lock) (iversion lock) | dirtysca
> > ttered=0 lock=0 dirfrag=1 openingsnapparents=0 dirty=1 authpin=1
> > scrubqueue=0 0x55d595486000]:
> >
> {"performed_validation":true,"passed_validation":false,"backtrace":{"checked"
> >
> :true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(11)0x100:[]//[]","error_str":"failed
>
> > to read off disk; see retval"},"raw_stats":{"ch
> > ecked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0
> > 10=0+10)","ondisk_value.rstat":"n(v0 rc2022-03-12T16:01:44.218294+0000
> > b1017620718 375=364+11)","mem
> > ory_value.dirstat":"f(v0 10=0+10)","memory_value.rstat":"n(v1815
> > rc2022-03-12T16:01:44.218294+0000 b1017620718
> > 375=364+11)","error_str":""},"return_code":-61}
> > 2022-03-12T18:13:41.601+0000 7f61cb0b1700 0 log_channel(cluster) log
> > [INF] : scrub summary: idle+waiting paths [~mds0]
> > 2022-03-12T18:13:45.317+0000 7f61cf8ba700 0 log_channel(cluster) log
> > [INF] : scrub summary: idle
> >
> > 2022-03-12T18:13:52.881+0000 7f61d30c1700 1 mds.xxxx1 asok_command:
> > scrub start {path=~mdsdir,prefix=scrub
> > start,scrubops=[recursive,repair]} (starting...)
> > 2022-03-12T18:13:52.881+0000 7f61cb0b1700 0 log_channel(cluster) log
> > [INF] : scrub queued for path: ~mds0
> > 2022-03-12T18:13:52.881+0000 7f61cb0b1700 0 log_channel(cluster) log
> > [INF] : scrub summary: idle+waiting paths [~mds0]
> > 2022-03-12T18:13:52.881+0000 7f61cb0b1700 0 log_channel(cluster) log
> > [INF] : scrub summary: active paths [~mds0]
> > 2022-03-12T18:13:52.881+0000 7f61cb0b1700 0 log_channel(cluster) log
> > [WRN] : bad backtrace on inode 0x100(~mds0), rewriting it
> > 2022-03-12T18:13:52.881+0000 7f61cb0b1700 0 log_channel(cluster) log
> > [INF] : Scrub repaired inode 0x100 (~mds0)
> > 2022-03-12T18:13:52.881+0000 7f61cb0b1700 -1 mds.0.scrubstack
> > _validate_inode_done scrub error on inode [inode 0x100 [...2,head]
> > ~mds0/ auth v6798 ap=1 snaprealm=0x55d595484800 DIRTYPARENT f(v0
> > 10=0+10) n(v1815 rc2022-03-12T16:01:44.218294+0000 b1017620718
> > 375=364+11)/n(v0 rc2019-10-29T10:52:34.302967+0000 11=0+11) (inest
> > lock) (iversion lock) | dirtyscattered=0 lock=0 dirfrag=1
> > openingsnapparents=0 dirtyparent=1 dirty=1 authpin=1 scrubqueue=0
> > 0x55d595486000]:
> >
> {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(11)0x100:[]//[]","error_str":"failed
>
> > to read off disk; see
> >
> retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0
>
> > 10=0+10)","ondisk_value.rstat":"n(v0 rc2022-03-12T16:01:44.218294+0000
> > b1017620718 375=364+11)","memory_value.dirstat":"f(v0
> > 10=0+10)","memory_value.rstat":"n(v1815
> > rc2022-03-12T16:01:44.218294+0000 b1017620718
> > 375=364+11)","error_str":""},"return_code":-61}
> > 2022-03-12T18:13:52.881+0000 7f61cb0b1700 0 log_channel(cluster) log
> > [INF] : scrub summary: idle+waiting paths [~mds0]
> > 2022-03-12T18:13:55.317+0000 7f61cf8ba700 0 log_channel(cluster) log
> > [INF] : scrub summary: idle
> >
> > 2022-03-12T18:14:12.608+0000 7f61d30c1700 1 mds.xxxx1 asok_command:
> > scrub start {path=~mdsdir,prefix=scrub
> > start,scrubops=[recursive,repair]} (starting...)
> > 2022-03-12T18:14:12.608+0000 7f61cb0b1700 0 log_channel(cluster) log
> > [INF] : scrub queued for path: ~mds0
> > 2022-03-12T18:14:12.608+0000 7f61cb0b1700 0 log_channel(cluster) log
> > [INF] : scrub summary: idle+waiting paths [~mds0]
> > 2022-03-12T18:14:12.608+0000 7f61cb0b1700 0 log_channel(cluster) log
> > [INF] : scrub summary: active paths [~mds0]
> > 2022-03-12T18:14:12.608+0000 7f61cb0b1700 0 log_channel(cluster) log
> > [INF] : scrub summary: idle+waiting paths [~mds0]
> > 2022-03-12T18:14:15.316+0000 7f61cf8ba700 0 log_channel(cluster) log
> > [INF] : scrub summary: idle
> >
> >
> > Thanks, Chris
> >
> >
> > On 11/03/2022 12:24, Milind Changire wrote:
> >> Here's some answers to your questions:
> >>
> >> On Sun, Mar 6, 2022 at 3:57 AM Arnaud M<arnaud.meauzoone@xxxxxxxxx>
> >> wrote:
> >>
> >>> Hello to everyone :)
> >>>
> >>> Just some question about filesystem scrubbing
> >>>
> >>> In this documentation it is said that scrub will help admin check
> >>> consistency of filesystem:
> >>>
> >>> https://docs.ceph.com/en/latest/cephfs/scrub/
> >>>
> >>> So my questions are:
> >>>
> >>> Is filesystem scrubbing mandatory ?
> >>> How often should I scrub the whole filesystem (ie start at /)
> >>> How often should I scrub ~mdsdir
> >>> Should I set up a cronjob ?
> >>> Is filesystem scrubbing considerated armless ? Even with recursive
> >>> force
> >>> repair ?
> >>> Is there any chance for scrubbing to overload mds on a big file
> >>> system (ie
> >>> like find . -ls) ?
> >>> What is the difference between "recursive repair" and "recursive force
> >>> repair" ? Is "force" armless ?
> >>> Is there any way to see at which file/folder is the scrub operation
> >>> ? In
> >>> fact any better way to see srub progress than "scrub status" which
> >>> doesn't
> >>> say much
> >>>
> >>> Sorry for all the questions, but there is not that much
> >>> documentation about
> >>> filesystem scrubbing. And I do think the answers will help a lot of
> >>> cephfs
> >>> administrators :)
> >>>
> >>> Thanks to all
> >>>
> >>> All the best
> >>>
> >>> Arnaud
> >>> _______________________________________________
> >>> ceph-users mailing list --ceph-users@xxxxxxx
> >>> To unsubscribe send an email toceph-users-leave@xxxxxxx
> >>>
> >>>
> >> 1.
> >>
> >> Is filesystem scrubbing mandatory ?
> >> As a routine system administration practice, it is good to ensure
> >> that
> >> your file-system is always in a good state. To avoid getting the
> >> file-system into a bottleneck state during work hours, it's
> >> always a good
> >> idea to reserve some time to run a recursive forward scrub and
> >> use the
> >> in-built scrub automation to fix such issues. Although you can
> >> run the
> >> scrub at any directory of your choice, it's always a good
> >> practice to start
> >> the scrub at the file-system root once in a while.
> >>
> >> So file-system scrubbing is not mandatory but highly recommended.
> >>
> >> Filesystem scrubbing is designed to read CephFS’ metadata and detect
> >> inconsistencies or issues that are generated by bitrot or bugs, just as
> >> RADOS’ pg scrubbing is. In a perfect world without bugs or bit flips it
> >> would be unnecessary, but we don’t live in that world — so a scrub can
> >> detect small issues before they turn into big ones, and the mere act of
> >> reading data can keep it fresh and give storage devices a chance to
> >> correct
> >> any media errors while that’s still possible.
> >>
> >> We don’t have a specific recommended schedule and scrub takes up
> >> cluster IO
> >> and compute resources so its frequency should be tailored to your
> >> workload.
> >>
> >>
> >> 1.
> >>
> >> How often should I scrub the whole filesystem (ie start at /)
> >> Since you'd always want to have a consistent file-system, it
> >> would good
> >> to run scrubbing:
> >> 1.
> >>
> >> before taking a snapshot of the entire file-system OR
> >> 2.
> >>
> >> before taking a backup of the entire file-system OR
> >> 3.
> >>
> >> after significant metadata activity eg. after creating files,
> >> renaming files, deleting files, changing file attributes, etc.
> >>
> >>
> >> There's no one-rule-fixes-all scenario. So, you'll need to follow a
> >> heuristic approach. The type of devices (HDD or SSD), the amount of
> >> activity wearing the device are the typical factors involved when
> >> deciding
> >> to scrub a file-system. If you have some window dedicated for backup
> >> activity, then you’d want to run a recursive forward scrub with
> >> repair on
> >> the entire file-system before it is snapshotted and used for backup.
> >> Although you can run a scrub along with active use of the
> >> file-system, it
> >> is always recommended that you run the scrub on a quiet file-system
> >> so that
> >> neither of the activities get in each other’s way. This also helps in
> >> completing the scrub task quicker.
> >>
> >>
> >> 1.
> >>
> >> How often should I scrub ~mdsdir ?
> >> ~mdsdir is used to collect deleted (stray) entries. So, the
> >> number of
> >> file/dir unlinks in a typical workload should be used to come up
> >> with a
> >> heuristic to scrub the file-system. This activity can be taken up
> >> separately from scrubbing the file-system root.
> >>
> >>
> >>
> >> 1.
> >>
> >> Should I set up a cron job ?
> >>
> >> Yes, you could.
> >>
> >>
> >> 1.
> >>
> >> Is filesystem scrubbing considered harmless ? Even with recursive
> >> force
> >> repair ?
> >>
> >> Yes, scrubbing even with repair is harmless.
> >>
> >> Scrubbing with repair does the following things:
> >>
> >> 1.
> >>
> >> Repair backtrace
> >> If on-disk and in-memory backtraces don't match, then the
> >> DIRTYPARENT
> >> flag is set so that the journal logger thread picks the inode for
> >> writing
> >> the backtrace to the disk.
> >> 2.
> >>
> >> Repair inode
> >> If on-disk and in-memory inode versions don't match, then the
> >> inode is
> >> left untouched. Otherwise, if the inode is marked as "free", the
> >> inode
> >> number is removed from active use.
> >> 3.
> >>
> >> Repair recursive-stats
> >> If on-disk and in-memory raw-stats don't match, then all the
> >> stats for
> >> the leaves in the directory tree are marked dirty and a
> >> scatter-gather
> >> operation is forced to coalesce raw-stats info.
> >>
> >>
> >>
> >> 1.
> >>
> >> Is there any chance for scrubbing to overload mds on a big file
> >> system
> >> ie. like find . -ls ?
> >> Scrubbing on its own should not be able to overload an MDS, but
> >> it is an
> >> additional load on top of whatever client activity the MDS is
> >> serving,
> >> which could exceed the server’s capacity. To put it in short,
> >> yes, it might
> >> overload the mds when done in sustained high I/O scenarios.
> >> The mds config option mds_max_scrub_ops_in_progress, which
> >> defaults to
> >> 5, decides the number of scrubs running at any given time. So,
> >> there is a
> >> small effort at throttling.
> >>
> >>
> >>
> >> 1.
> >>
> >> What is the difference between "recursive repair" and "recursive
> >> force
> >> repair" ? Is "force" harmless ?
> >> If “force” argument is specified, then a dirfrag is scrubbed only if
> >> 1.
> >>
> >> The dentry version is greater than last scrub version AND
> >> 2.
> >>
> >> The dentry type is a DIR
> >>
> >> If “force” is not specified, then dirfrag scrubbing is skipped. You
> >> will be
> >> able to see an mds log saying that the scrubbing is skipped for the
> >> dentry.
> >>
> >> The rest of the scrubbing is done as described in Q5 above.
> >>
> >>
> >> 1.
> >>
> >> Is there any way to see at which file/folder is the scrub
> >> operation ? In
> >> fact any better way to see scrub progress than "scrub status"
> >> which doesn't
> >> say much.
> >> Currently there's no way to see which file/folder is being
> >> scrubbed. At
> >> most we could log a line in the mds logs about it, but it could
> >> soon cause
> >> logs to bloat if the number of entries are large.
> >>
> >>
> >>
> >>
> >>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
>
--
Milind
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
