Hi Frédéric,
thanks, but no, my mistake. We use Reef.
However, I think my problem is regarding Kafka using mTLS for
authentication.
If I am correct this will be implemented somewhere in the future.
See the Github link I posted.
I got a ca.crt, a user.crt and a user.key file for authentication to the
Kafka broker.
Using a simple Python script this works fine. Also "kafkacat" works
good. And of course openssl if I add these three files.
openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.crt -cert
user.crt -key user.key
I can send notifications.
The RGW just does not like this and always says:
RDKAFKA-3-FAIL: rdkafka#producer-1:
[thrd:ssl://kafka-testbroker.org:9094/bootstrap]:
ssl://kafka-testbroker.org:9094/bootstrap: Receive failed:
ssl/record/rec_layer_s3.c:909:error:0A000412:SSL routines::ssl/tls alert
bad certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY)
It somehow does not use these three files or at least the two files
user.crt and user.key.
Best,
Malte
On 19.03.25 17:22, Frédéric Nass wrote:
Hi Malte,
Based on the documentation link you shared in your first message, you must be using Quincy.
Please be aware that this bug [1] that leads to RGWs crashing when using Kafka notifications will not be fixed in Quincy since Quincy has reached EOL.
It's fixed in v19.2.1 and should be fixed in v18.2.5 when it comes out.
Regards,
Frédéric.
[1] https://tracker.ceph.com/issues/68033
----- Le 12 Mar 25, à 17:44, Malte Stroem malte.stroem@xxxxxxxxx a écrit :
I somehow have the feeling this is related to:
https://github.com/ceph/ceph/pull/61572/files/c4df76cdd075f67867403d960dc581bda8c14876#diff-3d118fb9c89f7fcd1a0b2c04df27367c6b7d8b788e52387a27af49820f28a328
Because we have a three node Kafka cluster.
On 12.03.25 17:32, Malte Stroem wrote:
Perhaps this helps with readability:
req 4130898986969377541 0.024000837s s3:complete_multipart name: Version
val: 2010-03-31
req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka-
ack-level val: broker
req 4130898986969377541 0.024000837s s3:complete_multipart name:
persistent val: false
req 4130898986969377541 0.024000837s s3:complete_multipart name: push-
endpoint val: kafka://kafka-testbroker.org:9094
req 4130898986969377541 0.024000837s s3:complete_multipart name:
security.protocol val: ssl
req 4130898986969377541 0.024000837s s3:complete_multipart name:
ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem
req 4130898986969377541 0.024000837s s3:complete_multipart name:
ssl.certificate.location val: /etc/pki/ca-trust/source/anchors/kafka-
rgw-user.pem
req 4130898986969377541 0.024000837s s3:complete_multipart name:
ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key
req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl
val: true
req 4130898986969377541 0.024000837s s3:complete_multipart name: verify-
ssl val: true
Kafka connect: new connection is created. Total connections: 1
Kafka connect: successfully configured SSL security
Kafka connect: using default CA location
Kafka connect: successfully configured security
Kafka connect: successfully created new producer
req 4130898986969377541 0.040001396s INFO: push endpoint created:
kafka://kafka-testbroker.org:9094
Kafka publish: successfully created topic: ktest
Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks
RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-
testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/
bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909:
error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert
number 42 (after 0ms in state APIVERSION_QUERY)
Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap:
Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL
routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms
in state APIVERSION_QUERY)
Kafka run: poll error(-187): 1/1 brokers are down
RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka-
testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/
bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909:
error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert
number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s)
suppressed)
afka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap:
Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL
routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms
in state APIVERSION_QUERY, 1 identical error(s) suppressed)
Kafka run: poll error(-187): 1/1 brokers are down
rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start
Kafka run: poll error(-187): 1/1 brokers are down
BucketsSyncThread: sync user=kuser
bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1])
lua background: cache get: name=default.rgw.log++script.background. :
hit (negative entry)
Kafka run: poll error(-187): 1/1 brokers are down
On 12.03.25 17:26, Malte Stroem wrote:
Hello,
configuring bucket notification for Kafka.
I have the ca.pem, the user.pem and the user.key file.
Running
openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem -
cert user.pem -key user.key
from inside the RGW container everything is fine.
I added the three files via
- mount_path: ...
to the spec file for the RGW service.
Inside the container I ran
update-ca-trust
kafkacat works with the three files as producer.
However when uploading objects to the test bucket the RGW shows the
following (debug is set to 20):
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: Version val: 2010-03-31
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: kafka-ack-level val: broker
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: persistent val: false
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: push-endpoint val: kafka://
kafka-testbroker.org:9094
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: security.protocol val: ssl
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/
pki/ ca-trust/source/anchors/kafka-rgw-ca.pem
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: ssl.certificate.location
val: / etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/
pki/ ca-trust/source/anchors/kafka-rgw-user.key
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: use-ssl val: true
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: verify-ssl val: true
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new
connection is created. Total connections: 1
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect:
successfully configured SSL security
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using
default CA location
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect:
successfully configured security
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect:
successfully created new producer
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541
0.040001396s INFO: push endpoint created: kafka://kafka-
testbroker.org:9094
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish:
successfully created topic: ktest
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with
callback, tag=1): OK. Queue has: 1 callbacks
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL:
rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]:
ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/
rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad
certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY)
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll
error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive
failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL
routines::ssl/tls alert bad certificate: SSL alert number 42 (after
0ms in state APIVERSION_QUERY)
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll
error(-187): 1/1 brokers are down
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL:
rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]:
ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/
rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad
certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY,
1 identical error(s) suppressed)
ar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll
error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive
failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL
routines::ssl/tls alert bad certificate: SSL alert number 42 (after
0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed)
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll
error(-187): 1/1 brokers are down
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start
Mar 12 16:52:31 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll
error(-187): 1/1 brokers are down
Mar 12 16:52:34 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync
user=kuser
bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1])
Mar 12 16:52:34 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache
get: name=default.rgw.log++script.background. : hit (negative entry)
Mar 12 16:52:35 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll
error(-187): 1/1 brokers are down
I can only add ca-location to the topic described here:
https://docs.ceph.com/en/quincy/radosgw/notifications/
But the notification service needs all infos from all three files.
A certificate chain did not work.
I browsed rgw_kafka.cc and did not find anything like
ca-cert or ca-key.
How to add the full chain or all three files to the topic so the RGW
can connect to the Kafka broker?
Best
Malte
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
