Hi Stefan,
On Mon, Jul 1, 2024 at 2:30 PM Stefan Kooman <stefan@xxxxxx> wrote:
>
> Hi Dietmar,
>
> On 29-06-2024 10:50, Dietmar Rieder wrote:
> > Hi all,
> >
> > finally we were able to repair the filesystem and it seems that we did
> > not lose any data. Thanks for all suggestions and comments.
> >
> > Here is a short summary of our journey:
>
> Thanks for writing this up. This might be useful for someone in the future.
>
> --- snip ---
>
> > X. Conclusion:
> >
> > If we would have be aware of the bug and its mitigation we would have
> > saved a lot of downtime and some nerves.
> >
> > Is there an obvious place that I missed were such known issues are
> > prominently made public? (The bug tracker maybe, but I think it is easy
> > to miss the important among all others)
>
>
> Not that I know of. But changes in behavior of Ceph (daemons) and or
> Ceph kernels would be good to know about indeed. I follow the
> ceph-kernel mailing list to see what is going on with the development of
> kernel CephFS. And there is a thread about reverting the PR that Enrico
> linked to [1], here the last mail in that thread from Venky to Ilya [2]:
>
> "Hi Ilya,
>
> After some digging and talking to Jeff, I figured that it's possible
> to disable async dirops from the mds side by setting
> `mds_client_delegate_inos_pct` config to 0:
>
> - name: mds_client_delegate_inos_pct
> type: uint
> level: advanced
> desc: percentage of preallocated inos to delegate to client
> default: 50
> services:
> - mds
>
> So, I guess this patch is really not required. We can suggest this
> config update to users and document it for now. We lack tests with
> this config disabled, so I'll be adding the same before recommending
> it out. Will keep you posted."
>
> However, I have not seen any update after this. So apparently it is
> possible to disable this preallocate behavior globally by disabling it
> on the MDS. But there are (were) no MDS tests with this option disabled
> (I guess a percentage of "0" would disable it). So I'm not sure it's
> safe to disable it, and what would happen if you disable this on the MDS
> when there are clients actually using preallocated inodes. I have added
> Venky in the CC so I hope he can give us an update about the recommended
> way(s) of disabling preallocated inodes
It's safe to disable preallocation by setting
`mds_client_delegate_inos_pct = 0'. Once disabled, the MDS will not
delegate (preallocated) inode ranges to clients, effectively disabling
async dirops. We have seen users running with this config (and using
the `wsync' mount option in the kernel driver - although setting both
isn't really required IMO, Xiubo?) and reporting stabe file system
operations.
As far as tests are concerned, the way forward is to have a shot
reproducing this in our test lab. We have a tracker for this:
https://tracker.ceph.com/issues/66250
It's likely that a combination of having the MDS preallocate inodes
and delegate a percentage of those frequently to clients is causing
this bug. Furthermore, an enhancement is proposed (for the shorter
term) to not crash the MDS, but to blocklist the client that's holding
the "problematic" preallocated inode range. That way, the file system
isn't totally unavailable when such a problem occurs (the client would
have to be remounted though, but that's a lesser pain than going
through the disaster recovery steps).
HTH.
>
> Gr. Stefan
>
> [1]:
> https://github.com/gregkh/linux/commit/f7a67b463fb83a4b9b11ceaa8ec4950b8fb7f902
>
> [2]:
> https://lore.kernel.org/all/20231003110556.140317-1-vshankar@xxxxxxxxxx/T/
>
>
>
--
Cheers,
Venky
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
