Ilya, Thank you for providing these discussion threads on the Kernel fixes for where there was a change and details on this affects the clients. What is the expected behavior in CephFS client when there are multiple data pools in the CephFS? Does having 'nearfull' in any data pool in the CephFS then trigger the synchronous writes for clients even if they would be writing to a CephFS location mapped to a non-nearfull data pool? I.e. is 'nearfull' / sync behavior global across the same CephFS filesystem? Thanks, Matt On Thu, Nov 16, 2023 at 8:39 AM Ilya Dryomov <idryomov@xxxxxxxxx> wrote: > On Thu, Nov 16, 2023 at 3:21 AM Xiubo Li <xiubli@xxxxxxxxxx> wrote: > > > > Hi Matt, > > > > On 11/15/23 02:40, Matt Larson wrote: > > > On CentOS 7 systems with the CephFS kernel client, if the data pool > has a > > > `nearfull` status there is a slight reduction in write speeds (possibly > > > 20-50% fewer IOPS). > > > > > > On a similar Rocky 8 system with the CephFS kernel client, if the data > pool > > > has `nearfull` status, a similar test shows write speeds at different > block > > > sizes shows the IOPS < 150 bottlenecked vs the typical write > > > performance that might be with 20000-30000 IOPS at a particular block > size. > > > > > > Is there any way to avoid the extremely bottlenecked IOPS seen on the > Rocky > > > 8 system CephFS kernel clients during the `nearfull` condition or to > have > > > behavior more similar to the CentOS 7 CephFS clients? > > > > > > Do different OS or Linux kernels have greatly different ways they > respond > > > or limit on the IOPS? Are there any options to adjust how they limit on > > > IOPS? > > > > Just to be clear that the kernel on CentOS 7 is lower than the kernel on > > Rocky 8, they may behave differently someway. BTW, are the ceph versions > > the same for your test between CentOS 7 and Rocky 8 ? > > > > I saw in libceph.ko there has some code will handle the OSD FULL case, > > but I didn't find the near full case, let's get help from Ilya about > this. > > > > @Ilya, > > > > Do you know will the osdc will behave differently when it detects the > > pool is near full ? > > Hi Xiubo, > > It's not libceph or osdc, but CephFS itself. I think Matt is running > against this fix: > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7614209736fbc4927584d4387faade4f31444fce > > It was previously discussed in detail here: > > > https://lore.kernel.org/ceph-devel/CAOi1vP_K2YBX9+JffMUHCuXsynGfTqJYh+FRUsYY4UrePrK9fQ@xxxxxxxxxxxxxx/ > > The solution is to add additional capacity or bump the nearfull > threshold: > > > https://lore.kernel.org/ceph-devel/23f46ca6dd1f45a78beede92fc91d588@xxxxxxxxxxxxx/ > > Thanks, > > Ilya > -- Matt Larson, PhD Madison, WI 53705 U.S.A. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |