Re: [EXTERNAL] Re: RGW Multisite with a Self-Signed CA

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Nice, great that it works for you. And thanks for the update.

Zitat von "Alex Hussein-Kershaw (HE/HIM)" <alexhus@xxxxxxxxxxxxx>:

Turns out it was all actually working fine with the addition of the extra_container_args and the /etc/pki mount. I was running the "radosgw-admin sync status " in a cephadm shell which did not have the certificates in, and it seems the check was getting blocked.

Switching into a radosgw-admin container and running "radosgw-admin sync status" shows looks much happier. I presume I could also mount the /etc/pki directory into my cephadm shell container if I desired also.

Best wishes,
Alex

________________________________
From: Alex Hussein-Kershaw (HE/HIM) <alexhus@xxxxxxxxxxxxx>
Sent: Tuesday, July 16, 2024 12:48 PM
To: Eugen Block <eblock@xxxxxx>; ceph-users@xxxxxxx <ceph-users@xxxxxxx>
Subject: Re: [EXTERNAL]  Re: RGW Multisite with a Self-Signed CA

Hi Eugen,

Thanks for the advice - I've tried something similiar but no luck (my base OS is RHEL 9, so the paths don't quite line up with yours, I have no /var/lib/ca-certificates directory). I can curl inside the RGW container to the remote site and it is happy with the certificate as per below.

$ curl https://10.235.22.23:7480
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/";><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

But the sync still fails in one direction. Triggering that manually lets me get a packet capture.
$ radosgw-admin data  sync init    --source-zone=siteB
ERROR: sync.init() returned ret=-5
2024-07-16T11:40:20.051+0000 7f21e3c30a80 0 ERROR: failed to fetch datalog info

It's is still showing the "Unknown CA" error in the TLS handshake. I saw an issue over at Rook which looked similar: Ceph multisite replication with ssl untrusted certs · Issue #9584 · rook/rook (github.com)<https://github.com/rook/rook/issues/9584>, which had a suggestion that it's not just the RGW containers that require the trust. That seems surprising to me, not sure if I believe it.

As I mentioned in my initial question, this does seem to work fine for us on Octopus with the Centos8 container base so not sure if something has changed in either Ceph or in the move to the Centos stream 8 base image. I'm going to keep digging to see if I can work it out, but welcome any other suggestions.

Kind regards,
Alex


________________________________
From: Eugen Block <eblock@xxxxxx>
Sent: Monday, July 15, 2024 1:56 PM
To: ceph-users@xxxxxxx <ceph-users@xxxxxxx>
Subject: [EXTERNAL]  Re: RGW Multisite with a Self-Signed CA

Hi,

I'm not sure if it's a sufficient answer, but we use our own CA and
have it integrated with a ca-bundle mapping as extra_container_args
for the rgw daemons:

extra_container_args:
- -v=/var/lib/ca-certificates/:/var/lib/ca-certificates/:ro
-
-v=/var/lib/ca-certificates/ca-bundle.pem:/etc/pki/tls/certs/ca-bundle.crt:ro

The ceph container checks /etc/pki/tls/certs/ca-bundle.crt per default
(/etc/krb5.conf), and sometimes we need the /var/lib/ca-certificates/
path as well, so I configured both. That seems to work well, but I
don't have multisite enabled. To verify I did a curl test from within
the container to the rgw endpoint and got a valid response.

This is from a test cluster on Reef. This is not yet in our production
cluster, maybe I'll add that as well during a maintenance window, but
for now it's not required.

Regards,
Eugen

Zitat von "Alex Hussein-Kershaw (HE/HIM)" <alexhus@xxxxxxxxxxxxx>:

Hi ceph-users!

I'm going through the process of migrating to use cephadm for my
clusters. Previously I used ceph-ansible. My question is essentially
"How can I configure RGW multisite with self-signed certificates
with cephadm?". I have prototyped the migration and redeployed RGWs.
Everything on my adopted site is running the latest version of Reef
(18.2.2). The remote site is using Octopus as historically deployed
with ceph-ansible.

My RGWs on both sites are up and I can make requests to them, but
they are failing to sync, and "radosgw-admin sync status" shows a
generic input/output error. Taking some network capture I can see
that the TLS handshake is failing with "Unknown CA", so it looks
like the RGWs don't trust my self-signed certificate, I suppose
that's not a surprise.

However, I can't work out how to establish the trust. I've tried
mounting in the /etc/pki directory from the machine it's running on
into the RGW containers, which does contain the self-signed CA, but
I still see errors in my multi-site sync. I did notice that I can
curl from within the containers successfully to the remote HTTPS
RGWs after this though, so it did do something. Where do the RGWs
infer which CAs to trust from?

I should also mention that when stepping all the RGWs down to HTTP
the sync works with no issues (I was nervous about the Reef to
Octopus pairing, but it seems fine).

Kind regards,
Alex
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx


_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx


_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux