Hi Malte, Based on the documentation link you shared in your first message, you must be using Quincy. Please be aware that this bug [1] that leads to RGWs crashing when using Kafka notifications will not be fixed in Quincy since Quincy has reached EOL. It's fixed in v19.2.1 and should be fixed in v18.2.5 when it comes out. Regards, Frédéric. [1] https://tracker.ceph.com/issues/68033 ----- Le 12 Mar 25, à 17:44, Malte Stroem malte.stroem@xxxxxxxxx a écrit : > I somehow have the feeling this is related to: > > https://github.com/ceph/ceph/pull/61572/files/c4df76cdd075f67867403d960dc581bda8c14876#diff-3d118fb9c89f7fcd1a0b2c04df27367c6b7d8b788e52387a27af49820f28a328 > > Because we have a three node Kafka cluster. > > On 12.03.25 17:32, Malte Stroem wrote: >> Perhaps this helps with readability: >> >> req 4130898986969377541 0.024000837s s3:complete_multipart name: Version >> val: 2010-03-31 >> req 4130898986969377541 0.024000837s s3:complete_multipart name: kafka- >> ack-level val: broker >> req 4130898986969377541 0.024000837s s3:complete_multipart name: >> persistent val: false >> req 4130898986969377541 0.024000837s s3:complete_multipart name: push- >> endpoint val: kafka://kafka-testbroker.org:9094 >> req 4130898986969377541 0.024000837s s3:complete_multipart name: >> security.protocol val: ssl >> req 4130898986969377541 0.024000837s s3:complete_multipart name: >> ssl.ca.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-ca.pem >> req 4130898986969377541 0.024000837s s3:complete_multipart name: >> ssl.certificate.location val: /etc/pki/ca-trust/source/anchors/kafka- >> rgw-user.pem >> req 4130898986969377541 0.024000837s s3:complete_multipart name: >> ssl.key.location val: /etc/pki/ca-trust/source/anchors/kafka-rgw-user.key >> req 4130898986969377541 0.024000837s s3:complete_multipart name: use-ssl >> val: true >> req 4130898986969377541 0.024000837s s3:complete_multipart name: verify- >> ssl val: true >> Kafka connect: new connection is created. Total connections: 1 >> Kafka connect: successfully configured SSL security >> Kafka connect: using default CA location >> Kafka connect: successfully configured security >> Kafka connect: successfully created new producer >> req 4130898986969377541 0.040001396s INFO: push endpoint created: >> kafka://kafka-testbroker.org:9094 >> Kafka publish: successfully created topic: ktest >> Kafka publish (with callback, tag=1): OK. Queue has: 1 callbacks >> RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- >> testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ >> bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: >> error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert >> number 42 (after 0ms in state APIVERSION_QUERY) >> Kafka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: >> Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL >> routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms >> in state APIVERSION_QUERY) >> Kafka run: poll error(-187): 1/1 brokers are down >> RDKAFKA-3-FAIL: rdkafka#producer-1: [thrd:ssl://kafka- >> testbroker.org:9094/bootstrap]: ssl://kafka-testbroker.org:9094/ >> bootstrap: Receive failed: ssl/record/rec_layer_s3.c:909: >> error:0A000412:SSL routines::ssl/tls alert bad certificate: SSL alert >> number 42 (after 0ms in state APIVERSION_QUERY, 1 identical error(s) >> suppressed) >> afka run: poll error(-195): ssl://kafka-testbroker.org:9094/bootstrap: >> Receive failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL >> routines::ssl/tls alert bad certificate: SSL alert number 42 (after 0ms >> in state APIVERSION_QUERY, 1 identical error(s) suppressed) >> Kafka run: poll error(-187): 1/1 brokers are down >> rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start >> Kafka run: poll error(-187): 1/1 brokers are down >> BucketsSyncThread: sync user=kuser >> bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) >> lua background: cache get: name=default.rgw.log++script.background. : >> hit (negative entry) >> Kafka run: poll error(-187): 1/1 brokers are down >> >> On 12.03.25 17:26, Malte Stroem wrote: >>> Hello, >>> >>> configuring bucket notification for Kafka. >>> >>> I have the ca.pem, the user.pem and the user.key file. >>> >>> Running >>> >>> openssl s_client -connect kafka-testbroker.org:9094 -CAfile ca.pem - >>> cert user.pem -key user.key >>> >>> from inside the RGW container everything is fine. >>> >>> I added the three files via >>> >>> - mount_path: ... >>> >>> to the spec file for the RGW service. >>> >>> Inside the container I ran >>> >>> update-ca-trust >>> >>> kafkacat works with the three files as producer. >>> >>> However when uploading objects to the test bucket the RGW shows the >>> following (debug is set to 20): >>> >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 >>> 0.024000837s s3:complete_multipart name: Version val: 2010-03-31 >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 >>> 0.024000837s s3:complete_multipart name: kafka-ack-level val: broker >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 >>> 0.024000837s s3:complete_multipart name: persistent val: false >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 >>> 0.024000837s s3:complete_multipart name: push-endpoint val: kafka:// >>> kafka-testbroker.org:9094 >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 >>> 0.024000837s s3:complete_multipart name: security.protocol val: ssl >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 >>> 0.024000837s s3:complete_multipart name: ssl.ca.location val: /etc/ >>> pki/ ca-trust/source/anchors/kafka-rgw-ca.pem >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 >>> 0.024000837s s3:complete_multipart name: ssl.certificate.location >>> val: / etc/pki/ca-trust/source/anchors/kafka-rgw-user.pem >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 >>> 0.024000837s s3:complete_multipart name: ssl.key.location val: /etc/ >>> pki/ ca-trust/source/anchors/kafka-rgw-user.key >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 >>> 0.024000837s s3:complete_multipart name: use-ssl val: true >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 req 4130898986969377541 >>> 0.024000837s s3:complete_multipart name: verify-ssl val: true >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.226+0000 7f74e9dc5640 10 Kafka connect: new >>> connection is created. Total connections: 1 >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: >>> successfully configured SSL security >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: using >>> default CA location >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.226+0000 7f74e9dc5640 20 Kafka connect: >>> successfully configured security >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 Kafka connect: >>> successfully created new producer >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.242+0000 7f74e9dc5640 20 req 4130898986969377541 >>> 0.040001396s INFO: push endpoint created: kafka://kafka- >>> testbroker.org:9094 >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish: >>> successfully created topic: ktest >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.246+0000 7f74c8582640 20 Kafka publish (with >>> callback, tag=1): OK. Queue has: 1 callbacks >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.246+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: >>> rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: >>> ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ >>> rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad >>> certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY) >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll >>> error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive >>> failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL >>> routines::ssl/tls alert bad certificate: SSL alert number 42 (after >>> 0ms in state APIVERSION_QUERY) >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.246+0000 7f74c8582640 10 Kafka run: poll >>> error(-187): 1/1 brokers are down >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.498+0000 7f74c6b7b640 1 RDKAFKA-3-FAIL: >>> rdkafka#producer-1: [thrd:ssl://kafka-testbroker.org:9094/bootstrap]: >>> ssl://kafka-testbroker.org:9094/bootstrap: Receive failed: ssl/record/ >>> rec_layer_s3.c:909: error:0A000412:SSL routines::ssl/tls alert bad >>> certificate: SSL alert number 42 (after 0ms in state APIVERSION_QUERY, >>> 1 identical error(s) suppressed) >>> ar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll >>> error(-195): ssl://kafka-testbroker.org:9094/bootstrap: Receive >>> failed: ssl/record/rec_layer_s3.c:909: error:0A000412:SSL >>> routines::ssl/tls alert bad certificate: SSL alert number 42 (after >>> 0ms in state APIVERSION_QUERY, 1 identical error(s) suppressed) >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.498+0000 7f74c8582640 10 Kafka run: poll >>> error(-187): 1/1 brokers are down >>> Mar 12 16:52:30 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:30.586+0000 7f75db013640 2 rgw data changes log: >>> RGWDataChangesLog::ChangesRenewThread: start >>> Mar 12 16:52:31 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:31.118+0000 7f74c8582640 10 Kafka run: poll >>> error(-187): 1/1 brokers are down >>> Mar 12 16:52:34 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:34.758+0000 7f75cdff9640 20 BucketsSyncThread: sync >>> user=kuser >>> bucket=:ktest[29ec749a-9a62-4899-8181-90f82603862f.80959106.1]) >>> Mar 12 16:52:34 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:34.770+0000 7f75ca786640 10 lua background: cache >>> get: name=default.rgw.log++script.background. : hit (negative entry) >>> Mar 12 16:52:35 rgwnode1000 bash[811097]: debug >>> 2025-03-12T15:52:35.318+0000 7f74c8582640 10 Kafka run: poll >>> error(-187): 1/1 brokers are down >>> >>> I can only add ca-location to the topic described here: >>> >>> https://docs.ceph.com/en/quincy/radosgw/notifications/ >>> >>> But the notification service needs all infos from all three files. >>> >>> A certificate chain did not work. >>> >>> I browsed rgw_kafka.cc and did not find anything like >>> >>> ca-cert or ca-key. >>> >>> How to add the full chain or all three files to the topic so the RGW >>> can connect to the Kafka broker? >>> >>> Best >>> Malte >> > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |