Hi Stefan, On 7/1/24 10:34, Stefan Kooman wrote:
Hi Dietmar, On 29-06-2024 10:50, Dietmar Rieder wrote:Hi all,finally we were able to repair the filesystem and it seems that we did not lose any data. Thanks for all suggestions and comments.Here is a short summary of our journey:Thanks for writing this up. This might be useful for someone in the future.
Yeah, your welcome, I thought so too
--- snip ---X. Conclusion:If we would have be aware of the bug and its mitigation we would have saved a lot of downtime and some nerves.Is there an obvious place that I missed were such known issues are prominently made public? (The bug tracker maybe, but I think it is easy to miss the important among all others)Not that I know of. But changes in behavior of Ceph (daemons) and or Ceph kernels would be good to know about indeed. I follow the ceph-kernel mailing list to see what is going on with the development of kernel CephFS. And there is a thread about reverting the PR that Enrico linked to [1], here the last mail in that thread from Venky to Ilya [2]:"Hi Ilya, After some digging and talking to Jeff, I figured that it's possible to disable async dirops from the mds side by setting `mds_client_delegate_inos_pct` config to 0: - name: mds_client_delegate_inos_pct type: uint level: advanced desc: percentage of preallocated inos to delegate to client default: 50 services: - mds So, I guess this patch is really not required. We can suggest this config update to users and document it for now. We lack tests with this config disabled, so I'll be adding the same before recommending it out. Will keep you posted."However, I have not seen any update after this. So apparently it is possible to disable this preallocate behavior globally by disabling it on the MDS. But there are (were) no MDS tests with this option disabled (I guess a percentage of "0" would disable it). So I'm not sure it's safe to disable it, and what would happen if you disable this on the MDS when there are clients actually using preallocated inodes. I have added Venky in the CC so I hope he can give us an update about the recommended way(s) of disabling preallocated inodesGr. Stefan[1]: https://github.com/gregkh/linux/commit/f7a67b463fb83a4b9b11ceaa8ec4950b8fb7f902[2]: https://lore.kernel.org/all/20231003110556.140317-1-vshankar@xxxxxxxxxx/T/
I'm curious about any updates as well. I hope that not too many cephfs users will end up in this situation until the bug is fixed furthermore I hope that with posting our experiences here some will get alerted and make the proposed mitigation settings....
Best Dietmar
Attachment:
OpenPGP_signature.asc
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
