Hello Eugen and all, Thanks for the reply. We’ve checked the SuSE doc before raising it twice. From 100k to 125k, then to 150k. We are a bit worried about the continuous growth of strays at 50K a day and would like to find an effective to reduce the strays. Last night another 30K increase in the strays. Thanks Jinfeng From: Eugen Block <eblock@xxxxxx> Date: Sunday, 16 February 2025 at 7:32 PM To: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Subject: Re: How to reduce CephFS num_strays effectively? ⚠ EXTERNAL EMAIL: Do not click links or open any attachments unless you trust the sender and know the content is safe. ⚠ Hi, this SUSE article [0] covers that, it helped us with a customer a few years ago. The recommendation was to double the mds_bal_fragment_size_max (default 100k) to 200k, which worked nicely for them. Also note the mentioned correlation between mds_bal_fragment_size_max and mds_cache_memory_limit. Regards, Eugen [0] https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.suse.com%2Fde-de%2Fsupport%2Fkb%2Fdoc%2F%3Fid%3D000020569&data=05%7C02%7Cjinfeng.biao%40cba.com.au%7Cd2558d1e23384b02c9f308dd4e6cc82a%7Cdddffba06c174f3497483fa5e08cc366%7C0%7C0%7C638752951318780733%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GXn4vSVpmchKLA3IWFZx0HTTRlo%2FHHimg82inRUnHy4%3D&reserved=0<https://www.suse.com/de-de/support/kb/doc/?id=000020569> Zitat von jinfeng.biao@xxxxxxxxxx: > Hello folks, > > We had an issue with the num_strays hit 1 million recently. As a > workaround, max bal was increased to 125,000. > > The stray_num keeps growing at 25k per day. After a recent > observation of 10TiB file deletion, the relevant application was > stopped. > > Then we increased purging options to below values > > mds advanced filer_max_purge_ops 40 > mds advanced mds_max_purge_files 1024 > mds advanced mds_max_purge_ops 32768 > mds advanced mds_max_purge_ops_per_pg 3 > > And run "du -hsx" to the top level directory mounted to the app that > does massive deletion. > > Despite all above, strays still growing at 60K per day. > > There are a lot more applications using this CephFS filesystem and > only this app is observed perform deletion at this scale. > > I'm wondering what would be the effective way to cleanup the strays > in this situation while making the least impact to production. > > Note: We are on 14.2.6 > > thanks > James Biao > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx ************** IMPORTANT MESSAGE ************** This e-mail message is intended only for the addressee(s) and contains information which may be confidential. If you are not the intended recipient please advise the sender by return email, do not use or disclose the contents, and delete the message and any attachments from your system. Unless specifically indicated, this email does not constitute formal advice or commitment by the sender or the Commonwealth Bank of Australia (ABN 48 123 123 124 AFSL and Australian credit licence 234945) or its subsidiaries. We can be contacted through our web site: commbank.com.au. If you no longer wish to receive commercial electronic messages from us, please reply to this e-mail by typing Opt Out in the subject line. ************************************************** _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |