Hi Yuval, I stuck on the first step where I am trying to create sns topic but not able to create it. I am not able to figure out the issue. AMQP server is ready, up and running with AMQP 0.9.1. [root@ceprgw01 ~]# aws --endpoint-url http://localhost:8000 sns create-topic --name=mytopic --attributes='{"push-endpoint": "amqp://10.xx.xx.xx:5672", "amqp-exchange": "ex1", "amqp-ack-level": "broker"}' getting below error: Unknown options: {"push-endpoint": "amqp://10.xx.xx.xx:15672", "amqp-ack-level": "broker", "persistent": "true", "amqp-exchange": "topic_logs"}, set-topic-attributes I am using Ceph 4.2. [root@ceprgw01 ~]# aws --version aws-cli/1.14.28 Python/2.7.5 Linux/3.10.0-1160.25.1.el7.x86_64 botocore/1.8.35 Best regards, Sanjeev ________________________________ From: Yuval Lifshitz <ylifshit@xxxxxxxxxx> Sent: Friday, August 20, 2021 3:40 PM To: Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx> Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Subject: Re: S3 Bucket Notification requirement On Thu, Aug 19, 2021 at 6:30 PM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>> wrote: Hi Yuval, Thanks very much for your reply. I am using AMQP 0.9.1. Can I use aws sns create-topic command to create a topic in Ceph's RadosGW ? yes. note that you define the topic on the RGW, not the rabbitmq broker. e.g. aws --endpoint-url http://<rgw host>:<rgw port> sns create-topic --name=mytopic --attributes='{"push-endpoint": "amqp://<amqp host>:5672", "amqp-exchange": "ex1"}' (see also: https://github.com/ceph/ceph/tree/master/examples/boto3#aws-cli) ,, If Yes then How and where to define notifications to associate with created topics? Basically, I want to understand the tasks needs to be defined in Ceph and RabbitMQ Broker. you can use the aws cli to define the topic. a minimal setup to get all notifications for object creation would be: aws --region=default --endpoint-url http://<rgw host>:<rgw posrt> s3api put-bucket-notification-configuration --bucket=mybucket --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:default::mytopic", "Events": ["s3:ObjectCreated:*"]}]}' I have found few formats like below in the developer documentation but wondering how/where to define it: -- Where and how to define these below codes, In ceph or AMQP broker? in ceph (RGW). these are the formats of the raw HTTP messages used to create the topic. when you use AWS cli, they crafty these messages and send them to the RGW. unless you want to craft these messages yourself (e.g. using CURL) you should not really care about them. For Example: POST Action=CreateTopic &Name=<topic-name> &push-endpoint=<endpoint> [&Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=<exchange>] ... .... Java is being used in RabbitMQ broker with AMQP 0.9.1 protocol.. creating topics and notifications is also possible using the AWS Java SDK: * topics: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/examples-simple-notification-service.html * notifications: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/BucketNotificationConfiguration.html so, if your application wants to create them, the application developers will have to write code that does that (sadly, we don't have Java examples like the CLI and python ones, but you can use the example from the AWS docs). if the topics and notifications are more static, you can create them up front using the AWS cli, and then you don't need to do anything in your Java code. Best regards, Sanjeev ________________________________ From: Yuval Lifshitz <ylifshit@xxxxxxxxxx<mailto:ylifshit@xxxxxxxxxx>> Sent: Thursday, August 19, 2021 8:38 PM To: Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>> Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> <ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>> Subject: Re: S3 Bucket Notification requirement Hi Sanjeev, Welcome to the Ceph community! Which protocol do you intend to use in ActiveMQ? If you want to use AMQP1.0, you will have to wait, as this is still not officially supported [1]. Currently, we support AMQP0.9.1, Kafka, and HTTP. As for the more general question. To make bucket notifications work you first need to define a "topic", with an endpoint configured to point your message broker. Next, you need to define a "notification" that associates the topic with a bucket on which you want to get notifications. In the "notification" you can define a filter and the events on which you want to be notified. You can find examples of how to do that using the AWS CLI tool (or in python, using the boto3 library) here [2]. If you are looking for a different client or client SDK let me know. More details on the bucket notification feature (without any client specific examples) could be found here [3]. Yuval [1] https://github.com/ceph/ceph/pull/42548 [2] https://github.com/ceph/ceph/tree/master/examples/boto3 [3] https://docs.ceph.com/en/latest/radosgw/notifications/ On Thu, Aug 19, 2021 at 5:10 PM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>> wrote: Dear Sir, I would like to inform you that I am new to Ceph, specially new to the S3 bucket notification topic and not able to understand how exactly to configure it in RadaosGW. We are using an ActiveMQ broker. There is no easy technical documentation for Ceph available and I am not sure how and where to configure the coding. I am not the developer, rather just an administrator who manages Ceph. I would appreciate If you could let me know the steps if possible so that I could achieve this goal. Best regards, Sanjeev _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |