Hi,
I have a Ceph cluster with 6 hosts and 24 OSDs (mon and rgw colocated on these hosts) that is used just for RGW. Since upgrading to from Kraken (11.2.0) to Luminous (12.2.0) a week ago I can see lots of IOPS in the default.rgw.gc pool.
The cluster was created with Kraken two months ago and has the following status (yeah I know size=2 but data loss in this cluster is not problem):
root@ceph01:~# ceph status
cluster:
id: 14ff6ba0-6c70-4a01-ac80-5059c4956f2e
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph04,ceph05,ceph06
mgr: ceph04(active), standbys: ceph05, ceph06
osd: 24 osds: 24 up, 24 in
rgw: 4 daemons active
data:
pools: 13 pools, 1280 pgs
objects: 16200k objects, 29355 GB
usage: 59078 GB used, 44005 GB / 100 TB avail
pgs: 1280 active+clean
io:
client: 177 MB/s rd, 24044 kB/s wr, 3260 op/s rd, 148 op/s wr
Normal load (before upgrade) is around 150 IOPS reading + 150 IOPS writing. Now the reading IOPS are between 2000 and 6000 all the time. All this excessive load is done in the default.rgw.gc pool. So my guess is there is some kind of garbage collection done. This started right after restarting the radosgw processes during upgrade. Otherwise the cluster is healthy and normal rgw work does not seem to be affected. The number of objects in the cluster went down by 1000k objects since the upgrade. The data size increased slightly which is expected by our workload.
When looking at the OSDs I can see a pattern of high CPU on a single OSD process in the cluster for about 60 to 90 minutes with lots of messages in the log and then after a break the load switches to another OSD.
2017-09-20 06:05:02.560587 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.560586174
2017-09-20 06:05:02.562030 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.562028463
2017-09-20 06:05:02.563489 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.563487625
2017-09-20 06:05:02.564869 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.564867424
2017-09-20 06:05:02.566284 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.566282739
2017-09-20 06:05:02.567652 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.567650712
2017-09-20 06:05:02.569123 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.569122362
2017-09-20 06:05:02.570587 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.570585626
2017-09-20 06:05:02.572017 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.572015899
2017-09-20 06:05:02.573471 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.573469906
2017-09-20 06:05:02.574912 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.574910343
2017-09-20 06:05:02.576363 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.576362151
2017-09-20 06:05:02.577807 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.577806002
2017-09-20 06:05:02.579190 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.579189306
2017-09-20 06:05:02.580642 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.580641131
2017-09-20 06:05:02.582086 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.582084531
2017-09-20 06:05:02.583506 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.583505141
2017-09-20 06:05:02.584996 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.584994743
2017-09-20 06:05:02.586429 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.586427627
2017-09-20 06:05:02.587885 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.587884152
2017-09-20 06:05:02.589272 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.589271242
2017-09-20 06:05:02.590766 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.590764473
2017-09-20 06:05:02.592184 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.592182375
2017-09-20 06:05:02.593651 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.593649812
2017-09-20 06:05:02.595083 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.595081484
2017-09-20 06:05:02.596534 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.596532864
2017-09-20 06:05:02.597924 7fd1bf114700 0 <cls> /build/ceph-12.2.0/src/cls/rgw/cls_rgw.cc:3251: gc_iterate_entries end_key=1_01505880302.597923234
As you can see this is a lot of stuff logged per second and these OSDs generate several GB of logs per day.
So my question is: Anybody else seeing this high read load since the upgrade? Is there some kind of data migration or intensiv initial GC done in Luminous when the cluster was upgraded from Kraken? Can we perhaps reduce the amount of logged messages in the next update? I remember reading about a fix in Luminous that was about leaked objects when doing multipart uploads. Maybe this is related?
Any more infos I can provide you with?
Thanks!
Uwe
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
