Can you please open a tracker issue and attach detailed rgw logs for
objects that you know aren't getting removed from the gc queue, like the
one that you had pasted above:
2021-11-23T14:54:00.061+0100 7f6afa7fc700 20 garbage collection:
RGWGC::process iterating over entry
tag='23d143e2-d02d-4481-ba81-e783696ec99f.93072205.26537934^@',
time=2021-11-21T12:01:08.225897+0100, chain.objs.size()=3
2021-11-23T14:54:00.061+0100 7f6afa7fc700 5 garbage collection:
RGWGC::process removing
default.rgw.buckets.data:23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_1
2021-11-23T14:54:00.753+0100 7f6afa7fc700 5 garbage collection:
RGWGC::process removing
default.rgw.buckets.data:23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_2
2021-11-23T14:54:00.753+0100 7f6afa7fc700 5 garbage collection:
RGWGC::process removing
default.rgw.buckets.data:23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_3
If you have corresponding osd logs, please attach them as well. Also
please add other details like - the settings that you have used. Did
you change these settings after upgrading from nautilus to
octopus?(Seems like you didn't). And also how many days after
upgrading did you start seeing this problem?
I will take a look at it asap.
Thanks,
Pritha
On Wed, Nov 24, 2021 at 4:54 PM Jaka Močnik <jaka@xxxxxxxxx> wrote:
> hi, pritha,
>
> On Wed, 2021-11-24 at 16:41 +0530, Pritha Srivastava wrote:
>
> On Wed, Nov 24, 2021 at 4:11 PM Jaka Močnik <jaka@xxxxxxxxx> wrote:
>
> [...]
>
> after a bit of investigation it turned out that many of the objects in
> the gc queue were already garbage collected. i.e. rgw has deleted them
> from the rados rgw data pool, but has failed to remove them from the gc
> queue.
>
> How did you diagnose this?
>
> by means dumping the gc queue via radosgw-admin gc list --include all,
> checking the logs that objects still in that list were being deleted before
> the dump, and looking at the rgw logs.
>
> here is an example for one such rados object:
>
> ----
>
> rados object
>
>
> 23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_1
>
>
> has been deleted at least once. logs:
>
>
> 2021-11-23T14:54:00.061+0100 7f6afa7fc700 20 garbage collection: RGWGC::process iterating over entry tag='23d143e2-d02d-4481-ba81-e783696ec99f.93072205.26537934^@', time=2021-11-21T12:01:08.225897+0100, chain.objs.size()=3
>
> 2021-11-23T14:54:00.061+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_1
>
> 2021-11-23T14:54:00.753+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_2
>
> 2021-11-23T14:54:00.753+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_3
>
>
> object indeed does not exist in the data pool anymore:
>
>
> # rados -p default.rgw.buckets.data get 23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_1 out.bin
>
> error getting default.rgw.buckets.data/23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_1: (2) No such file or directory
>
>
> however, it is still present in gc queue list made after time of deletion
>
>
> {
>
> "tag": "23d143e2-d02d-4481-ba81-e783696ec99f.93072205.26537934\u0000",
>
> "time": "2021-11-21T12:01:08.225897+0100",
>
> "objs": [
>
> {
>
> "pool": "default.rgw.buckets.data",
>
> "oid": "23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_1",
>
> "key": "",
>
> "instance": ""
>
> },
>
> {
>
> "pool": "default.rgw.buckets.data",
>
> "oid": "23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_2",
>
> "key": "",
>
> "instance": ""
>
> },
>
> {
>
> "pool": "default.rgw.buckets.data",
>
> "oid": "23d143e2-d02d-4481-ba81-e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_3",
>
> "key": "",
>
> "instance": ""
>
> }
>
> ]
>
> },
>
> ----
>
> [...]
>
> Have you tried running radosgw-admin gc list command? Are some entries
> always there, past their expiration time? There is a flag --include-all
> which can also be used to list all expired and unexpired entries.
>
> yes. there are objects there for each day since 10. 11. 2021. some of them
> are getting deleted over and over again, and remain in the gc queue. the
> number of objects remaining from each day does not ever change, so I'm
> thinking it's a problem only with some of the gc "shards."
>
> Also in the logs - do you see this "RGWGC::process removing entries,
> marker: "? Are the markers getting repeated?
>
> no, the markers are not repeating right after each other. can't really
> tell in the long run. here is a grep for "marker" from today's logs of one
> of the rgws.
>
> ----
>
> 2021-11-24T07:06:50.647+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker=''
>
> 2021-11-24T07:06:51.515+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T07:06:52.115+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker=''
>
> 2021-11-24T07:06:52.411+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T07:06:54.687+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=100, truncated=1, next_marker='0/21854656'
>
> 2021-11-24T07:24:33.287+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker=''
>
> 2021-11-24T07:24:33.479+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T07:24:34.135+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker=''
>
> 2021-11-24T07:24:34.483+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T07:24:34.831+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker=''
>
> 2021-11-24T07:24:35.279+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T07:24:35.591+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker=''
>
> 2021-11-24T07:24:35.775+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T07:24:36.267+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker=''
>
> 2021-11-24T07:24:40.803+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=100, truncated=1, next_marker='0/23111053'
>
> 2021-11-24T07:45:17.850+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker=''
>
> 2021-11-24T07:45:18.366+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T07:45:19.086+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker=''
>
> 2021-11-24T07:45:19.474+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T07:45:20.018+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker=''
>
> 2021-11-24T07:45:20.426+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker=''
>
> 2021-11-24T07:45:22.082+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T07:45:22.554+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker=''
>
> 2021-11-24T07:45:23.310+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T07:45:27.902+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=100, truncated=1, next_marker='0/22157454'
>
> 2021-11-24T08:05:11.768+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=8, truncated=0, next_marker=''
>
> 2021-11-24T08:05:12.896+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T08:05:13.056+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker=''
>
> 2021-11-24T08:05:13.380+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T08:05:13.624+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=8, truncated=0, next_marker=''
>
> 2021-11-24T08:05:14.236+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T08:05:21.392+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=100, truncated=1, next_marker='0/29926681'
>
> 2021-11-24T08:47:37.722+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker=''
>
> 2021-11-24T08:47:45.474+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=49, truncated=0, next_marker=''
>
> 2021-11-24T09:32:49.386+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=11, truncated=0, next_marker=''
>
> 2021-11-24T09:33:03.414+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T09:33:03.662+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=10, truncated=0, next_marker=''
>
> 2021-11-24T09:33:05.442+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T09:33:11.874+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=100, truncated=1, next_marker='0/30714984'
>
> 2021-11-24T10:17:46.603+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker=''
>
> 2021-11-24T10:17:53.991+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:17:54.279+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker=''
>
> 2021-11-24T10:17:58.087+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:17:58.827+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker=''
>
> 2021-11-24T10:18:00.771+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:18:03.363+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=100, truncated=1, next_marker='0/20026975'
>
> 2021-11-24T10:24:25.192+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker=''
>
> 2021-11-24T10:24:25.500+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:24:31.121+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=54, truncated=0, next_marker=''
>
> 2021-11-24T10:49:31.582+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker=''
>
> 2021-11-24T10:49:33.070+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:49:33.318+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker=''
>
> 2021-11-24T10:49:36.294+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:49:36.694+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=7, truncated=0, next_marker=''
>
> 2021-11-24T10:49:42.014+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:49:42.690+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker=''
>
> 2021-11-24T10:49:43.050+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:49:43.718+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker=''
>
> 2021-11-24T10:49:45.994+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:49:46.262+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker=''
>
> 2021-11-24T10:49:47.154+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:49:47.634+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker=''
>
> 2021-11-24T10:49:49.766+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:49:50.318+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker=''
>
> 2021-11-24T10:49:50.614+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker=''
>
> 2021-11-24T10:49:52.006+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:49:52.422+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker=''
>
> 2021-11-24T10:49:54.371+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:49:54.647+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker=''
>
> 2021-11-24T10:49:55.795+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:49:56.227+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker=''
>
> 2021-11-24T10:49:56.731+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:49:57.035+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker=''
>
> 2021-11-24T10:49:58.491+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:49:58.699+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker=''
>
> 2021-11-24T10:50:00.395+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:50:00.711+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker=''
>
> 2021-11-24T10:50:03.099+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T10:50:07.855+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=100, truncated=1, next_marker='0/26269608'
>
> 2021-11-24T11:21:04.061+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker=''
>
> 2021-11-24T11:21:12.849+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T11:21:13.385+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker=''
>
> 2021-11-24T11:21:16.573+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T11:21:17.261+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker=''
>
> 2021-11-24T11:21:18.981+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T11:21:23.617+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=100, truncated=1, next_marker='0/21488979'
>
> 2021-11-24T11:50:19.050+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker=''
>
> 2021-11-24T11:50:23.438+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T11:50:24.554+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker=''
>
> 2021-11-24T11:50:25.798+0100 7f6afa7fc700 5 garbage collection: RGWGC::process removing entries, marker:
>
> 2021-11-24T11:50:27.731+0100 7f6afa7fc700 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=18, truncated=0, next_marker=''
>
> ----
>
> with regard to remedy in case we cannot diagnose the cause and fix it
> soon enough, I was thinking about:
> - stopping deletes to rgws for a short while,
> - dumping the gc queue contents,
> - stopping rgws,
> - clearing or recreating the rgw gc queue structures on rados pools,
> - restarting rgws and deletes,
> - manually deleting the rados objects in the old gc queue dump.
>
> is that a sound plan?
>
> if so, what exactly does the "clearing or recreating the rgw gc queue
> structures on rados pools" entail?
>
> I am under the impression that the gc queue is stored in gc.<number>
> objects in the GC namespace in the default.rgw.log pool.
>
> would just deleting these and starting rgw do the trick? or do I need
> to somehow recreate empty objects in their stead?
>
>
> Have you tried using the command: radosgw-admin gc process, to clear the
> expired entries and with --include-all to clear all entries?
>
> yes. it finishes (takes ~12h). the deletes get run according to logs. the
> problematic objects still remain in the gc queue.
>
> I should perhaps note that not all objects exhibit this problem. some (in
> my estimation ~80%) get removed just fine. but the ones that don't are
> getting deleted over and over again but are never removed.
>
> fwiw, these are the rgw gc settings that we explicitly set (others should
> be default). note that there was no problem with these settings for a long
> time, on either nautilus or octopus:
> ----
> rgw_gc_max_objs = 128
> rgw_gc_obj_min_wait = 3600
> rgw_gc_processor_max_time = 300
> rgw_gc_processor_period = 300
> ----
>
> regards,
> Jaka
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
