Hi Mahnoosh!
Unfortunately I'm not an expert in RGW hence nothing to recommend from
this side.
Apparently your issues are caused by bulk data removal - it appears that
RocksDB can hardly sustain such things and its performance degrades.
We've seen that plenty of times before.
So far there are two known workarounds - manual DB compaction with using
ceph-kvstore-tool and setting bluefs_buffer_io to true. The latter makes
sense for some Ceph releases which got that parameter set to false by
default, v15.2.12 is one of them. And indeed that setting
On 7/26/2021 1:28 AM, mahnoosh shahidi wrote:
Hi Igor,
Thanks for your response.This problem happens on my osds with hdd
disks. I set the bluefs_buffered_io to true just for these osds but it
caused my bucket index disks (which are ssd) to produce slow ops. I
also tried to set bluefs_buffered_io to true in bucket index osds but
they filled the entire memory (256G) so I had to set the
bluefs_buffered_io back to false in all osds. Is that the only way to
handle the garbage collector problem? Do you have any ideas for the
bucket index problem?
On Thu, Jul 22, 2021 at 3:37 AM Igor Fedotov <ifedotov@xxxxxxx
<mailto:ifedotov@xxxxxxx>> wrote:
Hi Mahnoosh,
you might want to set bluefs_buffered_io to true for every OSD.
It looks it's false by default in v15.2.12
Thanks,
Igor
On 7/18/2021 11:19 PM, mahnoosh shahidi wrote:
> We have a ceph cluster with 408 osds, 3 mons and 3 rgws. We
updated our
> cluster from nautilus 14.2.14 to octopus 15.2.12 a few days ago.
After
> upgrading, the garbage collector process which is run after the
lifecycle
> process, causes slow ops and makes some osds to be restarted. In
each
> process the garbage collector deletes about 1 million objects.
Below are
> the one of the osd's logs before it restarts.
>
> ```
> 2021-07-18T00:44:38.807+0430 7fd1cda76700 1 osd.60 1092400
is_healthy
> false -- internal heartbeat failed
> 2021-07-18T00:44:38.807+0430 7fd1cda76700 1 osd.60 1092400 not
> healthy; waiting to boot
> 2021-07-18T00:44:39.847+0430 7fd1cda76700 1 heartbeat_map
is_healthy
> 'OSD::osd_op_tp thread 0x7fd1b4243700' had timed out after 15
> 2021-07-18T00:44:39.847+0430 7fd1cda76700 1 osd.60 1092400
is_healthy
> false -- internal heartbeat failed
> 2021-07-18T00:44:39.847+0430 7fd1cda76700 1 osd.60 1092400 not
> healthy; waiting to boot
> 2021-07-18T00:44:40.895+0430 7fd1cda76700 1 heartbeat_map
is_healthy
> 'OSD::osd_op_tp thread 0x7fd1b4243700' had timed out after 15
> 2021-07-18T00:44:40.895+0430 7fd1cda76700 1 osd.60 1092400
is_healthy
> false -- internal heartbeat failed
> 2021-07-18T00:44:40.895+0430 7fd1cda76700 1 osd.60 1092400 not
> healthy; waiting to boot
> 2021-07-18T00:44:41.859+0430 7fd1cda76700 1 heartbeat_map
is_healthy
> 'OSD::osd_op_tp thread 0x7fd1b4243700' had timed out after 15
> 2021-07-18T00:44:41.859+0430 7fd1cda76700 1 osd.60 1092400
is_healthy
> false -- internal heartbeat failed
> 2021-07-18T00:44:41.859+0430 7fd1cda76700 1 osd.60 1092400 not
> healthy; waiting to boot
> 2021-07-18T00:44:42.811+0430 7fd1cda76700 1 heartbeat_map
is_healthy
> 'OSD::osd_op_tp thread 0x7fd1b4243700' had timed out after 15
> 2021-07-18T00:44:42.811+0430 7fd1cda76700 1 osd.60 1092400
is_healthy
> false -- internal heartbeat failed
>
> ```
> what is the suitable configuration for gc in such a heavy delete
process so
> it doesn't make slow ops? We had the same delete load in
nautilus but we
> didn't have any problem with that.
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
<mailto:ceph-users@xxxxxxx>
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
<mailto:ceph-users-leave@xxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
<mailto:ceph-users@xxxxxxx>
To unsubscribe send an email to ceph-users-leave@xxxxxxx
<mailto:ceph-users-leave@xxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
