Hi Yuval, Thanks, after using aws-cli v1, no signature related issue occurs. However, I can still see the issue related to sns topic creation. [ansibleuser@ceprgw02 z_ejbg]$ aws --endpoint-url http://objects.dev.xx.xx.xx:80 sns create-topic --name=mytopic --attributes='{"push-endpoint": "amqp://10.33.xx.xx:15672", "amqp-exchange": "ex1", "amqp-ack-level": "broker"}' An error occurred (Unknown) when calling the CreateTopic operation: Unknown [ansibleuser@ceprgw02 z_ejbg]$ aws --version aws-cli/1.16.312 Python/2.7.5 Linux/3.10.0-1160.25.1.el7.x86_64 botocore/1.13.48 [ansibleuser@ceprgw02 z_ejbg]$ I have tried running topic_with_endpoint.py from RadosGW node and gets below response. I am not sure whether it is ok or not. [ansibleuser@ceprgw01 z_ejbg]$ python3 topic_with_endpoint.py Usage: topic_with_endpoint.py <topic name> [region name] [ansibleuser@ceprgw01 z_ejbg]$ python3 topic_with_endpoint.py --debug {'TopicArn': 'arn:aws:sns:poc:app1:--debug', 'ResponseMetadata': {'RequestId': '8b7fe743-ecc5-42b4-8e55-a3feb9dc0501.98885264.50', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-request-id': 'tx000000000000000000032-006156c3dd-5e4de90-moi', 'content-type': 'application/xml', 'content-length': '289', 'date': 'Fri, 01 Oct 2021 08:16:29 GMT'}, 'RetryAttempts': 0}} Best regards, Sanjeev ________________________________ From: Yuval Lifshitz <ylifshit@xxxxxxxxxx> Sent: Wednesday, September 29, 2021 2:24 PM To: Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx> Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Subject: Re: S3 Bucket Notification requirement aws-cli v2 do not support the old signature types. can you please install aws-cli v1 [1] and try with it? [1] https://docs.aws.amazon.com/cli/latest/userguide/install-linux.html On Mon, Sep 27, 2021 at 6:45 PM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>> wrote: Hi Yuval, I have changed the sns signature version as suggested.. I tried creating simple sns topic even without any attributes, but it does not seem to be working and getting below error: [ansibleuser@ceprgw01 ~]$ aws --profile test sns create-topic --region=poc --name=mytopic --endpoint-url http://objects.dev.esoc.esa.int An error occurred (Unknown) when calling the CreateTopic operation: Unknown When I try to list objects using aws command, below message appears: [ansibleuser@ceprgw01 ~]$ aws --profile test --endpoint-url http://objects.dev.esoc.esa.int s3api list-objects --bucket=alicia Unknown Signature Version: s3. The same command is working fine when ruuning from root. Still I have problem creating sns topic and listing object which was working earlier from non root account. versions installed: [ansibleuser@ceprgw01 ~]$ ceph --version ceph version 14.2.11-181.el7cp (68fea1005601531fe60d2979c56ea63bc073c84f) nautilus (stable) [ansibleuser@ceprgw01 ~]$ aws --version aws-cli/2.0.30 Python/3.7.3 Linux/3.10.0-1160.25.1.el7.x86_64 botocore/2.0.0dev34 [ansibleuser@ceprgw01 ~]$ Best regards, Sanjeev ________________________________ From: Yuval Lifshitz <ylifshit@xxxxxxxxxx<mailto:ylifshit@xxxxxxxxxx>> Sent: Sunday, September 26, 2021 2:40 PM To: Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>> Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> <ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>> Subject: Re: S3 Bucket Notification requirement Hi Sanjeev, You need to install our aws cli extensions. Please see the instructions here: https://github.com/ceph/ceph/blob/master/examples/boto3/README.md If you are using a version of ceph which is older than "pacific", you would also need to change the default SNS signer to use v2 instead of v4: aws configure set default.sns.signature_version s3 Please let me know if it helped, Yuval On Tue, Sep 21, 2021 at 6:12 PM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>> wrote: Hi Yuval, I stuck on the first step where I am trying to create sns topic but not able to create it. I am not able to figure out the issue. AMQP server is ready, up and running with AMQP 0.9.1. [root@ceprgw01 ~]# aws --endpoint-url http://localhost:8000 sns create-topic --name=mytopic --attributes='{"push-endpoint": "amqp://10.xx.xx.xx:5672", "amqp-exchange": "ex1", "amqp-ack-level": "broker"}' getting below error: Unknown options: {"push-endpoint": "amqp://10.xx.xx.xx:15672", "amqp-ack-level": "broker", "persistent": "true", "amqp-exchange": "topic_logs"}, set-topic-attributes I am using Ceph 4.2. [root@ceprgw01 ~]# aws --version aws-cli/1.14.28 Python/2.7.5 Linux/3.10.0-1160.25.1.el7.x86_64 botocore/1.8.35 Best regards, Sanjeev ________________________________ From: Yuval Lifshitz <ylifshit@xxxxxxxxxx<mailto:ylifshit@xxxxxxxxxx>> Sent: Friday, August 20, 2021 3:40 PM To: Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>> Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> <ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>> Subject: Re: S3 Bucket Notification requirement On Thu, Aug 19, 2021 at 6:30 PM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>> wrote: Hi Yuval, Thanks very much for your reply. I am using AMQP 0.9.1. Can I use aws sns create-topic command to create a topic in Ceph's RadosGW ? yes. note that you define the topic on the RGW, not the rabbitmq broker. e.g. aws --endpoint-url http://<rgw host>:<rgw port> sns create-topic --name=mytopic --attributes='{"push-endpoint": "amqp://<amqp host>:5672", "amqp-exchange": "ex1"}' (see also: https://github.com/ceph/ceph/tree/master/examples/boto3#aws-cli) ,, If Yes then How and where to define notifications to associate with created topics? Basically, I want to understand the tasks needs to be defined in Ceph and RabbitMQ Broker. you can use the aws cli to define the topic. a minimal setup to get all notifications for object creation would be: aws --region=default --endpoint-url http://<rgw host>:<rgw posrt> s3api put-bucket-notification-configuration --bucket=mybucket --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:default::mytopic", "Events": ["s3:ObjectCreated:*"]}]}' I have found few formats like below in the developer documentation but wondering how/where to define it: -- Where and how to define these below codes, In ceph or AMQP broker? in ceph (RGW). these are the formats of the raw HTTP messages used to create the topic. when you use AWS cli, they crafty these messages and send them to the RGW. unless you want to craft these messages yourself (e.g. using CURL) you should not really care about them. For Example: POST Action=CreateTopic &Name=<topic-name> &push-endpoint=<endpoint> [&Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=<exchange>] ... .... Java is being used in RabbitMQ broker with AMQP 0.9.1 protocol.. creating topics and notifications is also possible using the AWS Java SDK: * topics: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/examples-simple-notification-service.html * notifications: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/BucketNotificationConfiguration.html so, if your application wants to create them, the application developers will have to write code that does that (sadly, we don't have Java examples like the CLI and python ones, but you can use the example from the AWS docs). if the topics and notifications are more static, you can create them up front using the AWS cli, and then you don't need to do anything in your Java code. Best regards, Sanjeev ________________________________ From: Yuval Lifshitz <ylifshit@xxxxxxxxxx<mailto:ylifshit@xxxxxxxxxx>> Sent: Thursday, August 19, 2021 8:38 PM To: Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>> Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> <ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>> Subject: Re: S3 Bucket Notification requirement Hi Sanjeev, Welcome to the Ceph community! Which protocol do you intend to use in ActiveMQ? If you want to use AMQP1.0, you will have to wait, as this is still not officially supported [1]. Currently, we support AMQP0.9.1, Kafka, and HTTP. As for the more general question. To make bucket notifications work you first need to define a "topic", with an endpoint configured to point your message broker. Next, you need to define a "notification" that associates the topic with a bucket on which you want to get notifications. In the "notification" you can define a filter and the events on which you want to be notified. You can find examples of how to do that using the AWS CLI tool (or in python, using the boto3 library) here [2]. If you are looking for a different client or client SDK let me know. More details on the bucket notification feature (without any client specific examples) could be found here [3]. Yuval [1] https://github.com/ceph/ceph/pull/42548 [2] https://github.com/ceph/ceph/tree/master/examples/boto3 [3] https://docs.ceph.com/en/latest/radosgw/notifications/ On Thu, Aug 19, 2021 at 5:10 PM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>> wrote: Dear Sir, I would like to inform you that I am new to Ceph, specially new to the S3 bucket notification topic and not able to understand how exactly to configure it in RadaosGW. We are using an ActiveMQ broker. There is no easy technical documentation for Ceph available and I am not sure how and where to configure the coding. I am not the developer, rather just an administrator who manages Ceph. I would appreciate If you could let me know the steps if possible so that I could achieve this goal. Best regards, Sanjeev _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |