On Fri, Oct 1, 2021 at 11:20 AM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx> wrote: > Hi Yuval, > > Thanks, after using aws-cli v1, no signature related issue occurs. > However, I can still see the issue related to sns topic creation. > > [ansibleuser@ceprgw02 z_ejbg]$ aws --endpoint-url > http://objects.dev.xx.xx.xx:80 sns create-topic --name=mytopic > --attributes='{"push-endpoint": "amqp://10.33.xx.xx:15672", > "amqp-exchange": "ex1", "amqp-ack-level": "broker"}' > > An error occurred (Unknown) when calling the CreateTopic operation: Unknown > did you set the signature to v2 on the v1 aws-cli? [ansibleuser@ceprgw02 z_ejbg]$ aws --version > aws-cli/1.16.312 Python/2.7.5 Linux/3.10.0-1160.25.1.el7.x86_64 > botocore/1.13.48 > [ansibleuser@ceprgw02 z_ejbg]$ > > I have tried running topic_with_endpoint.py from RadosGW node and gets > below response. I am not sure whether it is ok or not. > > [ansibleuser@ceprgw01 z_ejbg]$ python3 topic_with_endpoint.py > Usage: topic_with_endpoint.py <topic name> [region name] > > [ansibleuser@ceprgw01 z_ejbg]$ python3 topic_with_endpoint.py --debug > there is no debug flag for the script... it assumed that the topic name is "debug" :-) > {'TopicArn': 'arn:aws:sns:poc:app1:--debug', 'ResponseMetadata': > {'RequestId': '8b7fe743-ecc5-42b4-8e55-a3feb9dc0501.98885264.50', > 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-request-id': > 'tx000000000000000000032-006156c3dd-5e4de90-moi', 'content-type': > 'application/xml', 'content-length': '289', 'date': 'Fri, 01 Oct 2021 > 08:16:29 GMT'}, 'RetryAttempts': 0}} > > seems to work. you got 200OK as the response, and more important, you got the topic ARN back (with topic name as "--debug"): "arn:aws:sns:poc:app1:--debug" later you use this ARN when you set up the notification. > Best regards, > Sanjeev > > ------------------------------ > *From:* Yuval Lifshitz <ylifshit@xxxxxxxxxx> > *Sent:* Wednesday, September 29, 2021 2:24 PM > *To:* Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx> > *Cc:* ceph-users@xxxxxxx <ceph-users@xxxxxxx> > *Subject:* Re: S3 Bucket Notification requirement > > aws-cli v2 do not support the old signature types. > can you please install aws-cli v1 [1] and try with it? > > [1] https://docs.aws.amazon.com/cli/latest/userguide/install-linux.html > > On Mon, Sep 27, 2021 at 6:45 PM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx> > wrote: > > Hi Yuval, > > I have changed the sns signature version as suggested.. > > I tried creating simple sns topic even without any attributes, but it does > not seem to be working and getting below error: > [ansibleuser@ceprgw01 ~]$ aws --profile test sns create-topic > --region=poc --name=mytopic --endpoint-url http://objects.dev.esoc.esa.int > > An error occurred (Unknown) when calling the CreateTopic operation: Unknown > > When I try to list objects using aws command, below message appears: > [ansibleuser@ceprgw01 ~]$ aws --profile test --endpoint-url > http://objects.dev.esoc.esa.int s3api list-objects --bucket=alicia > Unknown Signature Version: s3. > > The same command is working fine when ruuning from root. > > Still I have problem creating sns topic and listing object which was > working earlier from non root account. > > versions installed: > > [ansibleuser@ceprgw01 ~]$ ceph --version > ceph version 14.2.11-181.el7cp (68fea1005601531fe60d2979c56ea63bc073c84f) > nautilus (stable) > [ansibleuser@ceprgw01 ~]$ aws --version > aws-cli/2.0.30 Python/3.7.3 Linux/3.10.0-1160.25.1.el7.x86_64 > botocore/2.0.0dev34 > [ansibleuser@ceprgw01 ~]$ > > Best regards, > Sanjeev > > ------------------------------ > *From:* Yuval Lifshitz <ylifshit@xxxxxxxxxx> > *Sent:* Sunday, September 26, 2021 2:40 PM > *To:* Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx> > *Cc:* ceph-users@xxxxxxx <ceph-users@xxxxxxx> > *Subject:* Re: S3 Bucket Notification requirement > > Hi Sanjeev, > You need to install our aws cli extensions. Please see the instructions > here: > https://github.com/ceph/ceph/blob/master/examples/boto3/README.md > > If you are using a version of ceph which is older than "pacific", you > would also need to change the default SNS signer to use v2 instead of v4: > aws configure set default.sns.signature_version s3 > > Please let me know if it helped, > > Yuval > > > > On Tue, Sep 21, 2021 at 6:12 PM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx> > wrote: > > Hi Yuval, > > I stuck on the first step where I am trying to create sns topic but not > able to create it. I am not able to figure out the issue. AMQP server is > ready, up and running with AMQP 0.9.1. > > [root@ceprgw01 ~]# aws --endpoint-url http://localhost:8000 sns create-topic --name=mytopic --attributes='{"push-endpoint": "amqp://10.xx.xx.xx:5672", "amqp-exchange": "ex1", "amqp-ack-level": "broker"}' > > getting below error: > Unknown options: {"push-endpoint": "amqp://10.xx.xx.xx:15672", "amqp-ack-level": "broker", "persistent": "true", "amqp-exchange": "topic_logs"}, set-topic-attributes > > I am using Ceph 4.2. > > [root@ceprgw01 ~]# aws --version > aws-cli/1.14.28 Python/2.7.5 Linux/3.10.0-1160.25.1.el7.x86_64 botocore/1.8.35 > > > > Best regards, > Sanjeev > > ------------------------------ > *From:* Yuval Lifshitz <ylifshit@xxxxxxxxxx> > *Sent:* Friday, August 20, 2021 3:40 PM > *To:* Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx> > *Cc:* ceph-users@xxxxxxx <ceph-users@xxxxxxx> > *Subject:* Re: S3 Bucket Notification requirement > > > > On Thu, Aug 19, 2021 at 6:30 PM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx> > wrote: > > Hi Yuval, > > Thanks very much for your reply. > > I am using AMQP 0.9.1. > > Can I use aws sns create-topic command to create a topic in Ceph's RadosGW > ? > > yes. note that you define the topic on the RGW, not the rabbitmq broker. > e.g. > > aws --endpoint-url http://<rgw host>:<rgw port> sns create-topic > --name=mytopic --attributes='{"push-endpoint": "amqp://<amqp host>:5672", > "amqp-exchange": "ex1"}' > > (see also: https://github.com/ceph/ceph/tree/master/examples/boto3#aws-cli > ) > > ,, If Yes then How and where to define notifications to associate with > created topics? Basically, I want to understand the tasks needs to be > defined in Ceph and RabbitMQ Broker. > > you can use the aws cli to define the topic. a minimal setup to get all > notifications for object creation would be: > > aws --region=default --endpoint-url http://<rgw host>:<rgw posrt> s3api > put-bucket-notification-configuration --bucket=mybucket > --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", > "TopicArn": "arn:aws:sns:default::mytopic", "Events": > ["s3:ObjectCreated:*"]}]}' > > > I have found few formats like below in the developer documentation but > wondering how/where to define it: -- Where and how to define these below > codes, In ceph or AMQP broker? > > > in ceph (RGW). > these are the formats of the raw HTTP messages used to create the topic. > when you use AWS cli, they crafty these messages and send them to the RGW. > unless you want to craft these messages yourself (e.g. using CURL) you > should not really care about them. > > > For Example: > POST > Action=CreateTopic > &Name=<topic-name> > &push-endpoint=<endpoint> > [&Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=<exchange>] > ... > .... > > > > Java is being used in RabbitMQ broker with AMQP 0.9.1 protocol.. > > creating topics and notifications is also possible using the AWS Java SDK: > * topics: > https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/examples-simple-notification-service.html > * notifications: > > https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/BucketNotificationConfiguration.html > > so, if your application wants to create them, the application developers > will have to write code that does that (sadly, we don't have Java examples > like the CLI and python ones, but you can use the example from the AWS > docs). > > if the topics and notifications are more static, you can create them up > front using the AWS cli, and then you don't need to do anything in your > Java code. > > > Best regards, > Sanjeev > > > ------------------------------ > *From:* Yuval Lifshitz <ylifshit@xxxxxxxxxx> > *Sent:* Thursday, August 19, 2021 8:38 PM > *To:* Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx> > *Cc:* ceph-users@xxxxxxx <ceph-users@xxxxxxx> > *Subject:* Re: S3 Bucket Notification requirement > > Hi Sanjeev, > Welcome to the Ceph community! > > Which protocol do you intend to use in ActiveMQ? > If you want to use AMQP1.0, you will have to wait, as this is still not > officially supported [1]. > Currently, we support AMQP0.9.1, Kafka, and HTTP. > > As for the more general question. > To make bucket notifications work you first need to define a "topic", with > an endpoint configured to point your message broker. > Next, you need to define a "notification" that associates the topic with a > bucket on which you want to get notifications. > In the "notification" you can define a filter and the events on which you > want to be notified. > > You can find examples of how to do that using the AWS CLI tool (or in > python, using the boto3 library) here [2]. > If you are looking for a different client or client SDK let me know. > > More details on the bucket notification feature (without any client > specific examples) could be found here [3]. > > Yuval > > [1] https://github.com/ceph/ceph/pull/42548 > [2] https://github.com/ceph/ceph/tree/master/examples/boto3 > [3] https://docs.ceph.com/en/latest/radosgw/notifications/ > > > On Thu, Aug 19, 2021 at 5:10 PM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx> > wrote: > > Dear Sir, > > I would like to inform you that I am new to Ceph, specially new to the S3 > bucket notification topic and not able to understand how exactly to > configure it in RadaosGW. We are using an ActiveMQ broker. > There is no easy technical documentation for Ceph available and I am not > sure how and where to configure the coding. > I am not the developer, rather just an administrator who manages Ceph. > I would appreciate If you could let me know the steps if possible so that > I could achieve this goal. > > Best regards, > Sanjeev > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |