Hello,
I've been testing rgw sync policies with bucket sync policies on reef
18.2.2 and discovered something that looks like a bug.
My setup is really simple. I have 2 ceph clusters with RGW. Main cluster
A and backup cluster B.
Sync policy allows user to enable replication from A to B.
I've enabled replication on few buckets via `aws s3api
put-bucket-replication`.
Data seems to replicate between cluster without major issues. But after
sync completes and there is no data waiting for replication cluster's B
rgw starts to spam cluster's A RWG with thousands of requests per second
generating unnecessary load on cluster's A rgw.log pool and huge
ammounts of system logs.
It looks like RGW on cluster B seems to constantly check metadata shard
updates. Is there a way to throttle this behavior?
Log example from RGW B:
2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10
http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]payload
request hash =
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10
http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]canonical
request = GET
/admin/log
id=60&max-entries=100&period=579115d4-b9df-435a-99f1-40c7eb03da28&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55&type=metadata
date:Sun, 01 Sep 2024 13:45:00 +0000
host:10.0.6.10:80
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240901T134500Z
date;host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10
http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]canonical
request hash =
a89e3cdb353e45bbaa5356d1b35cd1bbfd5cb4118da8a42677b8456931b16d13
2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10
http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]string
to sign = AWS4-HMAC-SHA256
20240901T134500Z
20240901/home/s3/aws4_request
a89e3cdb353e45bbaa5356d1b35cd1bbfd5cb4118da8a42677b8456931b16d13
2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10
http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]date_k
= 665f3152284a19f7eb31f7feb891577b045da2ebe65a12675d7b9b549bd7aa55
2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10
http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]region_k
= 81abe23f881a8ff25c0352f5f61a873bcdac805fba60c0001dedba940aac60a7
2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10
http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]service_k
= 49e4c3a8bce4feed131eccf6df5280dacd75d0a7fec6405a1d41e31804c4d742
2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10
http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]signing_k
= 2578ef828af150f3b303e7f02b87663ed9b6f783bec697048a760cc589491d86
2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10
http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=60&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]generated
signature = 1291a73a183e4fabac1494e1124bde56647e5f63618d1cd91c1ab894757e8da1
2024-09-01T15:45:00.757+0200 7fd0a280b6c0 10
http_client[GET/http://10.0.6.10:80/admin/log?type=metadata&id=47&period=579115d4-b9df-435a-99f1-40c7eb03da28&max-entries=100&&rgwx-zonegroup=3e543ae3-24ed-4568-be71-fa50791ecf55]canonical
headers format = date:Sun, 01 Sep 2024 13:45:00 +0000
host:10.0.6.10:80
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240901T134500Z
Best regards
Adam Prycki
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
