Perhaps this helps with readability:
req 4130898986969377541 0.024000837s s3:complete_multipart name: Version
val: 2010-03-31
req 4130898986969377541 0.024000837s s3:complete_multipart name:
kafka-ack-level val: broker
req 4130898986969377541 0.024000837s s3:complete_multipart name:
persistent val: false
req 4130898986969377541 0.024000837s s3:complete_multipart name:
push-endpoint val: kafka://kafka-testbroker.org:9094
req 4130898986969377541 0.024000837s s3:complete_multipart name:
security.protocol val: ssl
req 4130898986969377541 0.024000837s s3:complete_multipart name:
ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem
req 4130898986969377541 0.024000837s s3:complete_multipart name:
ssl.certificate.location val:
/etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem
req 4130898986969377541 0.024000837s s3:complete_multipart name:
ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key
req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl
val: true
req 4130898986969377541 0.024000837s s3:complete_multipart name:
verify-ssl val: true
Kafka connect: new connection is created. Total connections: 1
Kafka connect: successfully configured SSL security
Kafka connect: using default CA location
Kafka connect: successfully configured security
Kafka connect: successfully created new producer
req 4130898986969377541 0.040001396s INFO: push endpoint created:
kafka://kafka-testbroker.org:9094
Kafka publish: successfully created topic: ktest
Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks
RDKAFKA-3-FAIL: rdkafka#producer-1:
[thrd:ssl://kafka-testbroker.org:9094/bootstrap]:
ssl://kafka-testbroker.org:9094/bootstrap: Receive failed:
ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls
alert bad certificate: SSL alert number 42 (after 0ms in state
APIVERSION_QUERY)
Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap:
Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL
routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms
in state APIVERSION_QUERY)
Kafka run: poll error(-187): 1/1 brokers are down
RDKAFKA-3-FAIL: rdkafka#producer-1:
[thrd:ssl://kafka-testbroker.org:9094/bootstrap]:
ssl://kafka-testbroker.org:9094/bootstrap: Receive failed:
ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls
alert bad certificate: SSL alert number 42 (after 0ms in state
APIVERSION_QUERY, 1 identical error(s) suppressed)
afka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap:
Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL
routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms
in state APIVERSION_QUERY, 1 identical error(s) suppressed)
Kafka run: poll error(-187): 1/1 brokers are down
rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start
Kafka run: poll error(-187): 1/1 brokers are down
BucketsSyncThread: sync user=kuser
bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1])
lua background: cache get: name=default.rgw.log++script.background. :
hit (negative entry)
Kafka run: poll error(-187): 1/1 brokers are down
On 12.03.25 17:26, Malte Stroem wrote:
Hello,
configuring bucket notification for Kafka.
I have the ca.pem, the user.pem and the user.key file.
Running
openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem -cert
user.pem -key user.key
from inside the RGW container everything is fine.
I added the three files via
- mount_path: ...
to the spec file for the RGW service.
Inside the container I ran
update-ca-trust
kafkacat works with the three files as producer.
However when uploading objects to the test bucket the RGW shows the
following (debug is set to 20):
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: Version val: 2010-03-31
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: kafka-ack-level val: broker
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: persistent val: false
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: push-endpoint val: kafka://
kafka-testbroker.org:9094
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: security.protocol val: ssl
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/pki/
ca-trust/source/anchors/kafka-rgw-ca.pem
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: ssl.certificate.location val: /
etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/pki/
ca-trust/source/anchors/kafka-rgw-user.key
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: use-ssl val: true
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541
0.024000837s s3:complete_multipart name: verify-ssl val: true
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new
connection is created. Total connections: 1
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully
configured SSL security
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using
default CA location
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: successfully
configured security
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect: successfully
created new producer
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541
0.040001396s INFO: push endpoint created: kafka://kafka-testbroker.org:9094
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish: successfully
created topic: ktest
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with
callback, tag=1): OK. Queue has: 1 callbacks
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL:
rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]:
ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/
rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad
certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY)
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll
error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed:
ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls
alert bad certificate: SSL alert number 42 (after 0ms in state
APIVERSION_QUERY)
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll
error(-187): 1/1 brokers are down
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL:
rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]:
ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/
rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad
certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, 1
identical error(s) suppressed)
ar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll
error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive failed:
ssl/record/rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls
alert bad certificate: SSL alert number 42 (after 0ms in state
APIVERSION_QUERY, 1 identical error(s) suppressed)
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll
error(-187): 1/1 brokers are down
Mar 12 16:52:30 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log:
RGWDataChangesLog::ChangesRenewThread: start
Mar 12 16:52:31 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll
error(-187): 1/1 brokers are down
Mar 12 16:52:34 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync
user=kuser bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1])
Mar 12 16:52:34 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache get:
name=default.rgw.log++script.background. : hit (negative entry)
Mar 12 16:52:35 rgwnode1000 bash[811097]: debug
2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll
error(-187): 1/1 brokers are down
I can only add ca-location to the topic described here:
https://docs.ceph.com/en/quincy/radosgw/notifications/
But the notification service needs all infos from all three files.
A certificate chain did not work.
I browsed rgw_kafka.cc and did not find anything like
ca-cert or ca-key.
How to add the full chain or all three files to the topic so the RGW can
connect to the Kafka broker?
Best
Malte
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
