Thanks for the suggestion. It may be that it does fit my use case but I wasn't sure from the docs. I have a self-signed CA that I use for my deployment, can pass a root CA cert and key in to Cephadm to have it auto-generate a server certificate as a child of that? I noticed in the docs there is no mention of those and I had a very brief look at the code and didn't see the ability to do that. It looked like perhaps Cephadm would generate its own root cert and sign with that? I've got a bit of baggage around this area (clients, cert rotation processes) that probably complicate using this also. I'm probably going to shy away from this for now. I still think a sensible fix here is to not add the defaulted ssl_certificate option if it's passed as an extra_frontend_arg, which allows my use case without a mainline change to behaviour. Thanks, Alex ________________________________ From: Redouane Kachach Sent: Friday, January 17, 2025 11:14 AM To: Alex Hussein-Kershaw (HE/HIM) Cc: ceph-users Subject: Re: [EXTERNAL] Re: Cephadm: Specifying RGW Certs & Keys By Filepath Thank you Alex, Reading the description of the tracker https://tracker.ceph.com/issues/69567 sounds like the outcome is very similar to when using the self-signed auto-generated certificates by cephadm. Wouldn't that be a viable option? I mean is there a big difference between self-signing the certificates by yourself and having cephadm doing that for you (unless your Org has strict rules around that!)? On Fri, Jan 17, 2025 at 10:19 AM Redouane Kachach <rkachach@xxxxxxxxxx<mailto:rkachach@xxxxxxxxxx>> wrote: I see, unfortunately I can't see an easy way to avoid that. With the current code you will get either port= or ssl_port= depending on whether you enable ssl or not. At the end of the day the result get written in the configuration variable rgw_frontends (ceph config ls | grep rgw_frontends). I'll recommend opening a ticket with a feature request explaining your use case and why the current behavior doesn't meet your needs. Best, Redo. On Fri, Jan 17, 2025 at 9:51 AM Alex Hussein-Kershaw (HE/HIM) <alexhus@xxxxxxxxxxxxx<mailto:alexhus@xxxxxxxxxxxxx>> wrote: Moring Redo, I had dropped the "rgw_frontend_port" as I was providing it via "ssl_port". However I tested with it back in also: service_type: rgw service_id: rgw service_name: rgw.rgw placement: count_per_host: 1 host_pattern: '*' extra_container_args: - -v - /etc/pki:/etc/pki:ro - -v - /etc/ssl/:/etc/ssl:ro spec: rgw_frontend_port: 7480 rgw_frontend_extra_args: - ssl_port=7480 - ssl_certificate=/etc/ssl/certs/server.crt - ssl_private_key=/etc/ssl/private/server.key rgw_realm: geored_realm rgw_zone: siteA rgw_zonegroup: geored_zg Results in config: beast port=7480 ssl_port=7480 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key ...and I then get errors in logs because it's trying to bind to the same port twice. failed to bind address 0.0.0.0:7480<http://0.0.0.0:7480/>: Address already in use Dropping the "ssl_port" just brings it up as a HTTP endpoint. beast port=7480 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key So I don't have a combination that allows me to do just HTTPS with my cert/key provided as a file path. Thanks, Alex ________________________________ From: Redouane Kachach Sent: Thursday, January 16, 2025 8:00 PM To: Alex Hussein-Kershaw (HE/HIM) Cc: ceph-users Subject: Re: [EXTERNAL] Re: Cephadm: Specifying RGW Certs & Keys By Filepath That's strange... in the code I can see that when rgw_frontend_port it's used so I can't see why you get port=80 ... can you plz post your spec? On Thu, Jan 16, 2025 at 7:09 PM Alex Hussein-Kershaw (HE/HIM) <alexhus@xxxxxxxxxxxxx<mailto:alexhus@xxxxxxxxxxxxx>> wrote: Oh actually I have spoke to soon. That does work, but it also exposes port HTTP over port 80. 🙁 beast port=80 ssl_port=7480 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key ________________________________ From: Alex Hussein-Kershaw (HE/HIM) Sent: Thursday, January 16, 2025 5:59 PM To: Redouane Kachach Cc: ceph-users Subject: Re: [EXTERNAL] Re: Cephadm: Specifying RGW Certs & Keys By Filepath Amazing. How did I miss that. Dropping "ssl: true" and adding "ssl_port=1234" to the rgw_frontend_extra_args values has me sorted. Many thanks! ________________________________ From: Redouane Kachach Sent: Thursday, January 16, 2025 4:39 PM To: Alex Hussein-Kershaw (HE/HIM) Cc: ceph-users Subject: [EXTERNAL] Re: Cephadm: Specifying RGW Certs & Keys By Filepath You are getting the double option because "ssl: true" ... try to disable ssl since you are passing the arguments and certificates by hand! Another option is to have cephadm generate the certificates for you by setting the `generate_cert` field in the spec to true. But I'm not sure if that works for your environment or not ... Best, Redo. On Thu, Jan 16, 2025 at 4:37 PM Alex Hussein-Kershaw (HE/HIM) <alexhus@xxxxxxxxxxxxx<mailto:alexhus@xxxxxxxxxxxxx>> wrote: Hi Folks, Looking for some advice on RGW service specs and Cephadm. I've read the docs here: RGW Service — Ceph Documentation<https://docs.ceph.com/en/reef/cephadm/services/rgw/> Using a service spec I can deploy a RGW: service_type: rgw service_id: '25069123' service_name: rgw.25069123 placement: hosts: - raynor-sc-1 extra_container_args: - -v - /etc/pki:/etc/pki:ro - -v - /etc/ssl/:/etc/ssl:ro spec: rgw_frontend_port: 7480 rgw_realm: geored_realm rgw_zone: siteA rgw_zonegroup: geored_zg ssl: true rgw_frontend_extra_args: - "ssl_certificate=/etc/ssl/certs/server.crt" - "ssl_private_key=/etc/ssl/private/server.key" But it fails to start because my "rgw_frontends" config ends up as this. Notice the two entries of "ssl_certificate". beast ssl_port=7480 ssl_certificate=config://rgw/cert/rgw.25069123 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key That causes it to fail to start: debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 deferred set uid:gid to 167:167 (ceph:ceph) debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable), process radosgw, pid 7 debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework: beast debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_port, val: 7480 debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_certificate, val: config://rgw/cert/rgw.25069123 debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_certificate, val: /etc/ssl/certs/server.crt debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_private_key, val: /etc/ssl/private/server.key debug 2025-01-16T15:26:54.520+0000 7f65bcfdc840 -1 LDAP not started since no server URIs were provided in the configuration. debug 2025-01-16T15:26:54.531+0000 7f65bcfdc840 0 framework: beast debug 2025-01-16T15:26:54.531+0000 7f65bcfdc840 0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt debug 2025-01-16T15:26:54.531+0000 7f65bcfdc840 0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key debug 2025-01-16T15:26:54.597+0000 7f65bcfdc840 0 starting handler: beast debug 2025-01-16T15:26:54.601+0000 7f65bcfdc840 -1 ssl_certificate was not found: rgw/cert/rgw.25069123 debug 2025-01-16T15:26:54.602+0000 7f65bcfdc840 -1 no ssl_certificate configured for ssl_port debug 2025-01-16T15:26:54.602+0000 7f65bcfdc840 -1 ERROR: failed initializing frontend debug 2025-01-16T15:26:54.602+0000 7f65bcfdc840 -1 ERROR: initialize frontend fail, r = 22 Correcting that config manually: $ ceph config set client.rgw.25069123.raynor-sc-1.qiatgr rgw_frontends "beast ssl_port=7480 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key" Then allows the RGW to start: debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 deferred set uid:gid to 167:167 (ceph:ceph) debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable), process radosgw, pid 7 debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework: beast debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework conf key: ssl_port, val: 7480 debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework conf key: ssl_certificate, val: /etc/ssl/certs/server.crt debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework conf key: ssl_private_key, val: /etc/ssl/private/server.key debug 2025-01-16T15:28:17.767+0000 7fba6e85a840 -1 LDAP not started since no server URIs were provided in the configuration. debug 2025-01-16T15:28:17.781+0000 7fba6e85a840 0 framework: beast debug 2025-01-16T15:28:17.781+0000 7fba6e85a840 0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt debug 2025-01-16T15:28:17.781+0000 7fba6e85a840 0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key debug 2025-01-16T15:28:17.860+0000 7fba6e85a840 0 starting handler: beast debug 2025-01-16T15:28:17.908+0000 7fba6e85a840 0 set uid:gid to 167:167 (ceph:ceph) debug 2025-01-16T15:28:17.912+0000 7fba6e85a840 1 mgrc service_daemon_register rgw.643295 metadata {arch=x86_64,ceph_release=squid,ceph_version=ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable),ceph_version_short=19.2.0,container_hostname=raynor-sc-1,container_image=ceph/squid:v19.2.0,cpu=Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz,distro=centos,distro_description=CentOS Stream 9,distro_version=9,frontend_config#0=beast ssl_port=7480 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key,frontend_type#0=beast,hostname=raynor-sc-1,id=25069123.raynor-sc-1.qiatgr,kernel_description=#1 SMP Thu Dec 19 20:58:14 EST 2024,kernel_version=5.4.288-1.el8.elrepo.x86_64,mem_swap_kb=0,mem_total_kb=12251748,num_handles=1,os=Linux,pid=7,realm_id=c99973d7-27cb-4abd-8391-bfbe4fa928be,realm_name=geored_realm,zone_id=30816d1d-e315-471b-bc84-6cdc4f6f1408,zone_name=siteA,zonegroup_id=c109a213-6638-4028-b21b-8bc9fcb49cd3,zonegroup_name=geored_zg} Specifying the certificate by file path I think is the right choice for me as: * I don't have a domain, I have a per host certificate covering the IP of the host. Each is different, signed by a self-signed trusted CA. * I don't want to put the hardcoded key into the service spec, as suggested in the docs, as it appears in the mgr logs. My org policy (sensibly) doesn't allow for secrets to appear in logs. * It's just generally more hassle to manage keys on a per VM basis, rather than a static file path. * It will also allow me to not manage RGW services on a per host basis, but use the same spec for all and a count=1 placement, which would be lovely. So my question - is there a way around this such that I can specify the key and certificate by file path and not have to manually change the config to make it work? Thanks, Alex _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |