Hi Eugen,
Thanks for the reply, sorry for the delayed response.
The IOPS issues observed in my original email seemed to subside after I did a compaction of the OSDs, we had some other issues with pubsub that we had to delete a very large data pool of pubsub objects to recover from that I understand gave rise to the need for the database compaction. We have 3 RGWs serving our client load and 3 for pubsub acks/notifs (hence the 6 in my original email). Unfortunately for us scaling out isn't an option just now.
We found some other issues on the cluster that we think arose due to misconfiguration - I've redeployed and it seems to be performing much better, virtually never seeing slow ops now. I suspect that it was either caused by:
- The large OMAP alarm "52 large omap objects"
- The level of fragmentation on the OSDs. (osd.0 = 0.83, osd.1=0.80, osd.2=0.77, osd.3 = 0.79)
- A badly chosen block size. We reduced this to 4KB.
Thanks,
Alex
-----Original Message-----
From: Eugen Block <eblock@xxxxxx>
Sent: 03 November 2021 14:56
To: ceph-users@xxxxxxx
Subject: [EXTERNAL] Re: Slow S3 Requests
Hi,
it's possible that the log messages are a consequence of the slow requests. Having only 4 OSDs is kind of a corner case, ceph is designed as a scalable solution, so the real benefits start with a large number of OSDs so you can parallelize many client requests to many OSDs. And with only 4 OSDs the one with the most IO is probably the primary OSD. You have more RGWs than OSDs, are all your RGWs serving client IO? Could you provide more details about the load the clients are producing? I assume the OSDs are also virtual disks?
I would recommend to scale out if you can.
Regards,
Eugen
Zitat von Alex Hussein-Kershaw <alexhus@xxxxxxxxxxxxx>:
> Hi ceph-users,
>
> Having an issue on our test cluster. The problem I have is that S3
> requests are slow. Up to 30s on occasion, but usually taking a few
> seconds.
>
> This is a multisite cluster of 4 VMs running Ceph in containers. We
> have 4 OSDs, 3 MDS, 3 MONs and 3 RGWs. We're running ceph version
> 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable).
> The cluster is backed by SSDs.
>
> One of our 4 OSDs is doing much more read IOPs than the other,
> bouncing between 0 and 300 MB/s. The other 3 OSDs appear mostly idle,
> from an IO perspective. The OSD that is having high IOPS is spamming
> logs with:
>
> Oct 28 14:13:25 albans_sc0 container_name/ceph-osd-0[1002]:
> 2021-10-28 14:13:25.731 7fa06229f700 0 <cls>
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_A
> RCH/x86_64/AVAILABLE_DIST/centos7/
> DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-1
> 4.2.9/src/cls/rgw/cls_rgw.cc:2090: ERROR: rgw_obj_remove():
> cls_cxx_remove returned
> -2
> Oct 28 14:13:50 albans_sc0 container_name/ceph-osd-0[1002]:
> 2021-10-28 14:13:50.095 7fa062aa0700 0 <cls>
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_A
> RCH/x86_64/AVAILABLE_DIST/centos7/
> DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-1
> 4.2.9/src/cls/rgw/cls_rgw.cc:2090: ERROR: rgw_obj_remove():
> cls_cxx_remove returned
> -2
>
> Occasionally I'll have the following health warnings pop up then clear
> after a few seconds before clearing:
>
> 2021-10-28 14:28:46.686909 mon.albans_sc0 [WRN] Health check failed:
> 0 slow ops, oldest one blocked for 32 sec, osd.2 has slow ops
> (SLOW_OPS)
> 2021-10-28 14:28:47.880352 mon.albans_sc0 [WRN] Health check failed:
> 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)
> 2021-10-28 14:28:52.669720 mon.albans_sc0 [WRN] Health check update:
> 4 slow ops, oldest one blocked for 36 sec, daemons [osd.2,osd.3] have
> slow ops. (SLOW_OPS)
>
> I also noticed that the MDS cache appears to be tiny (315MB just now,
> when compared to our other similar test system which uses 4GB - they
> also have the same MDS cache config, although the other system is
> using Octopus):
>
> $ ceph daemon mds.albans_sc1 cache status {
> "pool": {
> "items": 7045301,
> "bytes": 315787403
> }
> }
>
> Otherwise the cluster reports:
>
> cluster:
> id: 29924e01-c131-4457-b252-e7a48200b925
> health: HEALTH_WARN
> 52 large omap objects
>
> services:
> mon: 3 daemons, quorum albans_sc0,albans_sc1,albans_sc2 (age 44h)
> mgr: albans_sc2(active, since 44h), standbys: albans_sc0, albans_sc1
> mds: cephfs:1 {0=albans_sc2=up:active} 2 up:standby
> osd: 4 osds: 4 up (since 3h), 4 in (since 3w)
> rgw: 6 daemons active (albans_sc0.pubsub, albans_sc0.rgw0,
> albans_sc1.pubsub, albans_sc1.rgw0, albans_sc2.pubsub,
> albans_sc2.rgw0)
>
> data:
> pools: 14 pools, 140 pgs
> objects: 4.90M objects, 129 GiB
> usage: 724 GiB used, 676 GiB / 1.4 TiB avail
> pgs: 139 active+clean
> 1 active+clean+scrubbing+deep
>
> io:
> client: 226 KiB/s rd, 9.2 KiB/s wr, 31 op/s rd, 18 op/s wr
>
> My client is reporting the slow S3 requests, and in the RGW logs I can
> also see things like:
>
> Oct 28 14:12:03 albans_sc2
> container_name/ceph-rgw-albans_sc2-rgw0[1001]: 2021-10-28
> 14:12:03.234 7fcd16962700 2 req 132980 22.823s s3:get_obj completing
> Oct 28 14:12:03 albans_sc2
> container_name/ceph-rgw-albans_sc2-rgw0[1001]: 2021-10-28
> 14:12:03.234 7fcd16962700 2 req 132980 22.823s s3:get_obj op status=0
> Oct 28 14:12:03 albans_sc2
> container_name/ceph-rgw-albans_sc2-rgw0[1001]: 2021-10-28
> 14:12:03.234 7fcd16962700 2 req 132980 22.823s s3:get_obj http
> status=200
> Oct 28 14:12:03 albans_sc2
> container_name/ceph-rgw-albans_sc2-rgw0[1001]: 2021-10-28
> 14:12:03.234 7fcd16962700 1 ====== req done req=0x55bca6b625f0 op
> status=0 http_status=200 latency=22.823s ======
>
> I've turned the RGW logs up all the way, but failing to identify what
> is causing such a long delay.
>
> What is the cause of the "ERROR: rgw_obj_remove(): cls_cxx_remove
> returned -2" message? How can I investigate further into these slow
> S3 requests?
>
> Any advice/guidance on how to debug this further is much appreciated.
>
> Thanks,
> Alex
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an
> email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
