Turns out it was all actually working fine with the addition of the extra_container_args and the /etc/pki mount. I was running the "radosgw-admin sync status " in a cephadm shell which did not have the certificates in, and it seems the check was getting blocked. Switching into a radosgw-admin container and running "radosgw-admin sync status" shows looks much happier. I presume I could also mount the /etc/pki directory into my cephadm shell container if I desired also. Best wishes, Alex ________________________________ From: Alex Hussein-Kershaw (HE/HIM) <alexhus@xxxxxxxxxxxxx> Sent: Tuesday, July 16, 2024 12:48 PM To: Eugen Block <eblock@xxxxxx>; ceph-users@xxxxxxx <ceph-users@xxxxxxx> Subject: Re: [EXTERNAL] Re: RGW Multisite with a Self-Signed CA Hi Eugen, Thanks for the advice - I've tried something similiar but no luck (my base OS is RHEL 9, so the paths don't quite line up with yours, I have no /var/lib/ca-certificates directory). I can curl inside the RGW container to the remote site and it is happy with the certificate as per below. $ curl https://10.235.22.23:7480 <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/";><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult> But the sync still fails in one direction. Triggering that manually lets me get a packet capture. $ radosgw-admin data sync init --source-zone=siteB ERROR: sync.init() returned ret=-5 2024-07-16T11:40:20.051+0000 7f21e3c30a80 0 ERROR: failed to fetch datalog info It's is still showing the "Unknown CA" error in the TLS handshake. I saw an issue over at Rook which looked similar: Ceph multisite replication with ssl untrusted certs · Issue #9584 · rook/rook (github.com)<https://github.com/rook/rook/issues/9584>, which had a suggestion that it's not just the RGW containers that require the trust. That seems surprising to me, not sure if I believe it. As I mentioned in my initial question, this does seem to work fine for us on Octopus with the Centos8 container base so not sure if something has changed in either Ceph or in the move to the Centos stream 8 base image. I'm going to keep digging to see if I can work it out, but welcome any other suggestions. Kind regards, Alex ________________________________ From: Eugen Block <eblock@xxxxxx> Sent: Monday, July 15, 2024 1:56 PM To: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Subject: [EXTERNAL] Re: RGW Multisite with a Self-Signed CA Hi, I'm not sure if it's a sufficient answer, but we use our own CA and have it integrated with a ca-bundle mapping as extra_container_args for the rgw daemons: extra_container_args: - -v=/var/lib/ca-certificates/:/var/lib/ca-certificates/:ro - -v=/var/lib/ca-certificates/ca-bundle.pem:/etc/pki/tls/certs/ca-bundle.crt:ro The ceph container checks /etc/pki/tls/certs/ca-bundle.crt per default (/etc/krb5.conf), and sometimes we need the /var/lib/ca-certificates/ path as well, so I configured both. That seems to work well, but I don't have multisite enabled. To verify I did a curl test from within the container to the rgw endpoint and got a valid response. This is from a test cluster on Reef. This is not yet in our production cluster, maybe I'll add that as well during a maintenance window, but for now it's not required. Regards, Eugen Zitat von "Alex Hussein-Kershaw (HE/HIM)" <alexhus@xxxxxxxxxxxxx>: > Hi ceph-users! > > I'm going through the process of migrating to use cephadm for my > clusters. Previously I used ceph-ansible. My question is essentially > "How can I configure RGW multisite with self-signed certificates > with cephadm?". I have prototyped the migration and redeployed RGWs. > Everything on my adopted site is running the latest version of Reef > (18.2.2). The remote site is using Octopus as historically deployed > with ceph-ansible. > > My RGWs on both sites are up and I can make requests to them, but > they are failing to sync, and "radosgw-admin sync status" shows a > generic input/output error. Taking some network capture I can see > that the TLS handshake is failing with "Unknown CA", so it looks > like the RGWs don't trust my self-signed certificate, I suppose > that's not a surprise. > > However, I can't work out how to establish the trust. I've tried > mounting in the /etc/pki directory from the machine it's running on > into the RGW containers, which does contain the self-signed CA, but > I still see errors in my multi-site sync. I did notice that I can > curl from within the containers successfully to the remote HTTPS > RGWs after this though, so it did do something. Where do the RGWs > infer which CAs to trust from? > > I should also mention that when stepping all the RGWs down to HTTP > the sync works with no issues (I was nervous about the Reef to > Octopus pairing, but it seems fine). > > Kind regards, > Alex > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |