Nice, great that it works for you. And thanks for the update.
Zitat von "Alex Hussein-Kershaw (HE/HIM)" <alexhus@xxxxxxxxxxxxx>:
Turns out it was all actually working fine with the addition of the
extra_container_args and the /etc/pki mount. I was running the
"radosgw-admin sync status " in a cephadm shell which did not have
the certificates in, and it seems the check was getting blocked.
Switching into a radosgw-admin container and running "radosgw-admin
sync status" shows looks much happier. I presume I could also mount
the /etc/pki directory into my cephadm shell container if I desired
also.
Best wishes,
Alex
________________________________
From: Alex Hussein-Kershaw (HE/HIM) <alexhus@xxxxxxxxxxxxx>
Sent: Tuesday, July 16, 2024 12:48 PM
To: Eugen Block <eblock@xxxxxx>; ceph-users@xxxxxxx <ceph-users@xxxxxxx>
Subject: Re: [EXTERNAL] Re: RGW Multisite with a Self-Signed CA
Hi Eugen,
Thanks for the advice - I've tried something similiar but no luck
(my base OS is RHEL 9, so the paths don't quite line up with yours,
I have no /var/lib/ca-certificates directory). I can curl inside the
RGW container to the remote site and it is happy with the
certificate as per below.
$ curl https://10.235.22.23:7480
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult
xmlns="http://s3.amazonaws.com/doc/2006-03-01/";><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
But the sync still fails in one direction. Triggering that manually
lets me get a packet capture.
$ radosgw-admin data sync init --source-zone=siteB
ERROR: sync.init() returned ret=-5
2024-07-16T11:40:20.051+0000 7f21e3c30a80 0 ERROR: failed to fetch
datalog info
It's is still showing the "Unknown CA" error in the TLS handshake. I
saw an issue over at Rook which looked similar: Ceph multisite
replication with ssl untrusted certs · Issue #9584 · rook/rook
(github.com)<https://github.com/rook/rook/issues/9584>, which had a
suggestion that it's not just the RGW containers that require the
trust. That seems surprising to me, not sure if I believe it.
As I mentioned in my initial question, this does seem to work fine
for us on Octopus with the Centos8 container base so not sure if
something has changed in either Ceph or in the move to the Centos
stream 8 base image. I'm going to keep digging to see if I can work
it out, but welcome any other suggestions.
Kind regards,
Alex
________________________________
From: Eugen Block <eblock@xxxxxx>
Sent: Monday, July 15, 2024 1:56 PM
To: ceph-users@xxxxxxx <ceph-users@xxxxxxx>
Subject: [EXTERNAL] Re: RGW Multisite with a Self-Signed CA
Hi,
I'm not sure if it's a sufficient answer, but we use our own CA and
have it integrated with a ca-bundle mapping as extra_container_args
for the rgw daemons:
extra_container_args:
- -v=/var/lib/ca-certificates/:/var/lib/ca-certificates/:ro
-
-v=/var/lib/ca-certificates/ca-bundle.pem:/etc/pki/tls/certs/ca-bundle.crt:ro
The ceph container checks /etc/pki/tls/certs/ca-bundle.crt per default
(/etc/krb5.conf), and sometimes we need the /var/lib/ca-certificates/
path as well, so I configured both. That seems to work well, but I
don't have multisite enabled. To verify I did a curl test from within
the container to the rgw endpoint and got a valid response.
This is from a test cluster on Reef. This is not yet in our production
cluster, maybe I'll add that as well during a maintenance window, but
for now it's not required.
Regards,
Eugen
Zitat von "Alex Hussein-Kershaw (HE/HIM)" <alexhus@xxxxxxxxxxxxx>:
Hi ceph-users!
I'm going through the process of migrating to use cephadm for my
clusters. Previously I used ceph-ansible. My question is essentially
"How can I configure RGW multisite with self-signed certificates
with cephadm?". I have prototyped the migration and redeployed RGWs.
Everything on my adopted site is running the latest version of Reef
(18.2.2). The remote site is using Octopus as historically deployed
with ceph-ansible.
My RGWs on both sites are up and I can make requests to them, but
they are failing to sync, and "radosgw-admin sync status" shows a
generic input/output error. Taking some network capture I can see
that the TLS handshake is failing with "Unknown CA", so it looks
like the RGWs don't trust my self-signed certificate, I suppose
that's not a surprise.
However, I can't work out how to establish the trust. I've tried
mounting in the /etc/pki directory from the machine it's running on
into the RGW containers, which does contain the self-signed CA, but
I still see errors in my multi-site sync. I did notice that I can
curl from within the containers successfully to the remote HTTPS
RGWs after this though, so it did do something. Where do the RGWs
infer which CAs to trust from?
I should also mention that when stepping all the RGWs down to HTTP
the sync works with no issues (I was nervous about the Reef to
Octopus pairing, but it seems fine).
Kind regards,
Alex
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
