Adding a bit more context to this thread.
I added an additional radosgw to each cluser. Radosgw 1-3 are customer facing. Radosgw #4 is dedicated to syncing
Radosgw 1-3 now have an additional lines:
rgw_enable_lc_threads = False
rgw_enable_gc_threads = False
Radosgw4 has the additional line:
rgw_sync_obj_etag_verify = True
The logs on any of the radosgw’s appear to be identical, but here is an example log that is reflect on any of the servers. Notice the IP addresses are 1-4. Where I expect the traffic to be from only 4.
Is this to be expected?
My understanding of this thread is that this traffic would be regulated to radosgw 04.
Example Ceph Conf for a single node, this is RadosGw 01
[client.rgw.west01.rgw0]
host = west01
keyring = /var/lib/ceph/radosgw/west-rgw.west01.rgw0/keyring
log file = /var/log/ceph/west-rgw-west01.rgw0.log
rgw frontends = beast port=8080 num_threads=500
rgw_dns_name = west01.example.com
rgw_max_chunk_size = 67108864
rgw_obj_stripe_size = 67108864
rgw_put_obj_min_window_size = 67108864
rgw_zone = rgw-west
rgw_enable_lc_threads = False
rgw_enable_gc_threads = False
------------
Example Logs:
2023-05-18T19:06:48.295+0000 7fb295f83700 1 beast: 0x7fb3e82b26f0: 10.10.10.1 - synchronization-user [18/May/2023:19:06:48.107 +0000] "GET /admin/log/?type=data&id=69&marker=1_xxxx&extra-info=true&rgwx-zonegroup=xxxxx HTTP/1.1" 200 44 - - - latency=0.188007131s
2023-05-18T19:06:48.371+0000 7fb1dd612700 1 ====== starting new request req=0x7fb3e80ae6f0 =====
2023-05-18T19:06:48.567+0000 7fb1dd612700 1 ====== req done req=0x7fb3e80ae6f0 op status=0 http_status=200 latency=0.196007445s ======
2023-05-18T19:06:48.567+0000 7fb1dd612700 1 beast: 0x7fb3e80ae6f0: 10.10.10.3 - synchronization-user [18/May/2023:19:06:48.371 +0000] "GET /admin/log/?type=data&id=107&marker=1_xxxx&extra-info=true&rgwx-zonegroup=xxxx HTTP/1.1" 200 44 - - - latency=0.196007445s
2023-05-18T19:06:49.023+0000 7fb290f79700 1 ====== starting new request req=0x7fb3e81b06f0 =====
2023-05-18T19:06:49.023+0000 7fb28bf6f700 1 ====== req done req=0x7fb3e81b06f0 op status=0 http_status=200 latency=0.000000000s ======
2023-05-18T19:06:49.023+0000 7fb28bf6f700 1 beast: 0x7fb3e81b06f0: 10.10.10.2 - synchronization-user [18/May/2023:19:06:49.023 +0000] "GET /admin/log?bucket-instance=ceph-bucketxxx%3A81&format=json&marker=00000020447.3609723.6&type=bucket-index&rgwx-zonegroup=xxx HTTP/1.1" 200 2 - - - latency=0.000000000s
2023-05-18T19:06:49.147+0000 7fb27af4d700 1 ====== starting new request req=0x7fb3e81b06f0 =====
2023-05-18T19:06:49.151+0000 7fb27af4d700 1 ====== req done req=0x7fb3e81b06f0 op status=0 http_status=200 latency=0.004000151s ======
2023-05-18T19:06:49.475+0000 7fb280f59700 1 beast: 0x7fb3e82316f0: 10.10.10.4 - synchronization-user [18/May/2023:19:06:49.279 +0000] "GET /admin/log/?type=data&id=58&marker=1_xxxx.1&extra-info=true&rgwx-zonegroup=xxxx HTTP/1.1" 200 312 - - - latency=0.196007445s
2023-05-18T19:06:49.987+0000 7fb27c750700 1 ====== starting new request req=0x7fb3e81b06f0 =====
radosgw-admin zonegroup get
{
"id": "x",
"name": "eastWestceph",
"api_name": "EastWestCeph",
"is_master": "true",
"endpoints": [
"http://east01.noam.lnrm.net:8080";,
"http://east02.noam.lnrm.net:8080";,
"http://east03.noam.lnrm.net:8080";,
"http://east04.noam.lnrm.net:8080";, << ---- sync node
"http://west01.noam.lnrm.net:8080";,
"http://west02.noam.lnrm.net:8080";,
"http://west03.noam.lnrm.net:8080";,
"http://west04.noam.lnrm.net:8080"; << ---- sync node
],
.......
],
"hostnames_s3website": [],
"master_zone": "x",
"zones": [
{
"id": "x",
"name": "rgw-west",
"endpoints": [
"http://west01.noam.lnrm.net:8080";,
"http://west02.noam.lnrm.net:8080";,
"http://west03.noam.lnrm.net:8080";,
"http://west04.noam.lnrm.net:8080"; << -- sync node
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": [],
"redirect_zone": ""
},
{
"id": "x",
"name": "rgw-east",
"endpoints": [
"http://east01.noam.lnrm.net:8080";,
"http://east02.noam.lnrm.net:8080";,
"http://east03.noam.lnrm.net:8080";,
"http://east04.noam.lnrm.net:8080"; << -- sync node
....
From: Ulrich Klein <Ulrich.Klein@xxxxxxxxxxxxxx>
Date: Tuesday, May 16, 2023 at 10:22 AM
To: Konstantin Shalygin <k0ste@xxxxxxxx>
Cc: Michal Strnad <michal.strnad@xxxxxxxxx>, ceph-users <ceph-users@xxxxxxx>
Subject: Re: Dedicated radosgw gateways
[You don't often get email from ulrich.klein@xxxxxxxxxxxxxx. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
*** External email: use caution ***
Hi,
Might be a dumb question …
I'm wondering how I can set those config variables in some but not all RGW processes?
I'm on a cephadm 17.2.6. On 3 nodes I have RGWs. The ones on 8080 are behind haproxy for users. the ones one 8081 I'd like for sync only.
# ceph orch ps | grep rgw
rgw.max.maxvm4.lmjaef maxvm4 *:8080 running (51m) 4s ago 2h 262M - 17.2.6 d007367d0f3c 315f47a4f164
rgw.max.maxvm4.lwzxpf maxvm4 *:8081 running (51m) 4s ago 2h 199M - 17.2.6 d007367d0f3c 7ae82e5f6ef2
rgw.max.maxvm5.syxpnb maxvm5 *:8081 running (51m) 4s ago 2h 137M - 17.2.6 d007367d0f3c c0635c09ba8f
rgw.max.maxvm5.wtpyfk maxvm5 *:8080 running (51m) 4s ago 2h 267M - 17.2.6 d007367d0f3c b4ad91718094
rgw.max.maxvm6.ostneb maxvm6 *:8081 running (51m) 4s ago 2h 150M - 17.2.6 d007367d0f3c 83b2af8f787a
rgw.max.maxvm6.qfulra maxvm6 *:8080 running (51m) 4s ago 2h 262M - 17.2.6 d007367d0f3c 81d01bf9e21d
# ceph config show rgw.max.maxvm4.lwzxpf
Error ENOENT: no config state for daemon rgw.max.maxvm4.lwzxpf
# ceph config set rgw.max.maxvm4.lwzxpf rgw_enable_lc_threads false
Error EINVAL: unrecognized config target 'rgw.max.maxvm4.lwzxpf'
(Not surprised)
# ceph tell rgw.max.maxvm4.lmjaef get rgw_enable_lc_threads
error handling command target: local variable 'poolid' referenced before assignment
# ceph tell rgw.max.maxvm4.lmjaef set rgw_enable_lc_threads false
error handling command target: local variable 'poolid' referenced before assignment
Is there any way to set the config for specific RGWs in a containerized env?
(ceph.conf doesn't work. Doesn't do anything and gets overwritten with a minimall version at "unpredictable" intervalls)
Thanks for any ideas.
Ciao, Uli
> On 15. May 2023, at 14:15, Konstantin Shalygin <k0ste@xxxxxxxx> wrote:
>
> Hi,
>
>> On 15 May 2023, at 14:58, Michal Strnad <michal.strnad@xxxxxxxxx> wrote:
>>
>> at Cephalocon 2023, it was mentioned several times that for service tasks such as data deletion via garbage collection or data replication in S3 via zoning, it is good to do them on dedicated radosgw gateways and not mix them with gateways used by users. How can this be achieved? How can we isolate these tasks? Will using dedicated keyrings instead of admin keys be sufficient? How do you operate this in your environment?
>
> Just:
>
> # don't put client traffic to "dedicated radosgw gateways"
> # disable lc/gc on "gateways used by users" via `rgw_enable_lc_threads = false` & `rgw_enable_gc_threads = false`
>
>
> k
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
________________________________
The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
