Thanks very much Yuval for your confirmation.
I have set the signature to v2 on the v1 aws-cli?
[ansibleuser@ceprgw01 z_ejbg]$ python3 topic_with_endpoint.py mytopic
{'TopicArn': 'arn:aws:sns:poc:app1:mytopic', 'ResponseMetadata': {'RequestId': '8b7fe743-ecc5-42b4-8e55-a3feb9dc0501.98885252.51', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-request-id': 'tx000000000000000000033-006156d5da-5e4de84-moi', 'content-type': 'application/xml', 'content-length': '289', 'date': 'Fri, 01 Oct 2021 09:33:14 GMT'}, 'RetryAttempts': 0}}
I will let you know once notification is set with the ARN number.
Best regards,
Sanjeev
________________________________
From: Yuval Lifshitz <ylifshit@xxxxxxxxxx>
Sent: Friday, October 1, 2021 2:23 PM
To: Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx>
Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx>
Subject: Re: S3 Bucket Notification requirement
On Fri, Oct 1, 2021 at 11:20 AM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>> wrote:
Hi Yuval,
Thanks, after using aws-cli v1, no signature related issue occurs. However, I can still see the issue related to sns topic creation.
[ansibleuser@ceprgw02 z_ejbg]$ aws --endpoint-url http://objects.dev.xx.xx.xx:80 sns create-topic --name=mytopic --attributes='{"push-endpoint": "amqp://10.33.xx.xx:15672", "amqp-exchange": "ex1", "amqp-ack-level": "broker"}'
An error occurred (Unknown) when calling the CreateTopic operation: Unknown
did you set the signature to v2 on the v1 aws-cli?
[ansibleuser@ceprgw02 z_ejbg]$ aws --version
aws-cli/1.16.312 Python/2.7.5 Linux/3.10.0-1160.25.1.el7.x86_64 botocore/1.13.48
[ansibleuser@ceprgw02 z_ejbg]$
I have tried running topic_with_endpoint.py from RadosGW node and gets below response. I am not sure whether it is ok or not.
[ansibleuser@ceprgw01 z_ejbg]$ python3 topic_with_endpoint.py
Usage: topic_with_endpoint.py <topic name> [region name]
[ansibleuser@ceprgw01 z_ejbg]$ python3 topic_with_endpoint.py --debug
there is no debug flag for the script... it assumed that the topic name is "debug" :-)
{'TopicArn': 'arn:aws:sns:poc:app1:--debug', 'ResponseMetadata': {'RequestId': '8b7fe743-ecc5-42b4-8e55-a3feb9dc0501.98885264.50', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-request-id': 'tx000000000000000000032-006156c3dd-5e4de90-moi', 'content-type': 'application/xml', 'content-length': '289', 'date': 'Fri, 01 Oct 2021 08:16:29 GMT'}, 'RetryAttempts': 0}}
seems to work. you got 200OK as the response, and more important, you got the topic ARN back (with topic name as "--debug"):
"arn:aws:sns:poc:app1:--debug"
later you use this ARN when you set up the notification.
Best regards,
Sanjeev
________________________________
From: Yuval Lifshitz <ylifshit@xxxxxxxxxx<mailto:ylifshit@xxxxxxxxxx>>
Sent: Wednesday, September 29, 2021 2:24 PM
To: Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>>
Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> <ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>>
Subject: Re: S3 Bucket Notification requirement
aws-cli v2 do not support the old signature types.
can you please install aws-cli v1 [1] and try with it?
[1] https://docs.aws.amazon.com/cli/latest/userguide/install-linux.html
On Mon, Sep 27, 2021 at 6:45 PM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>> wrote:
Hi Yuval,
I have changed the sns signature version as suggested..
I tried creating simple sns topic even without any attributes, but it does not seem to be working and getting below error:
[ansibleuser@ceprgw01 ~]$ aws --profile test sns create-topic --region=poc --name=mytopic --endpoint-url http://objects.dev.esoc.esa.int
An error occurred (Unknown) when calling the CreateTopic operation: Unknown
When I try to list objects using aws command, below message appears:
[ansibleuser@ceprgw01 ~]$ aws --profile test --endpoint-url http://objects.dev.esoc.esa.int s3api list-objects --bucket=alicia
Unknown Signature Version: s3.
The same command is working fine when ruuning from root.
Still I have problem creating sns topic and listing object which was working earlier from non root account.
versions installed:
[ansibleuser@ceprgw01 ~]$ ceph --version
ceph version 14.2.11-181.el7cp (68fea1005601531fe60d2979c56ea63bc073c84f) nautilus (stable)
[ansibleuser@ceprgw01 ~]$ aws --version
aws-cli/2.0.30 Python/3.7.3 Linux/3.10.0-1160.25.1.el7.x86_64 botocore/2.0.0dev34
[ansibleuser@ceprgw01 ~]$
Best regards,
Sanjeev
________________________________
From: Yuval Lifshitz <ylifshit@xxxxxxxxxx<mailto:ylifshit@xxxxxxxxxx>>
Sent: Sunday, September 26, 2021 2:40 PM
To: Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>>
Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> <ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>>
Subject: Re: S3 Bucket Notification requirement
Hi Sanjeev,
You need to install our aws cli extensions. Please see the instructions here:
https://github.com/ceph/ceph/blob/master/examples/boto3/README.md
If you are using a version of ceph which is older than "pacific", you would also need to change the default SNS signer to use v2 instead of v4:
aws configure set default.sns.signature_version s3
Please let me know if it helped,
Yuval
On Tue, Sep 21, 2021 at 6:12 PM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>> wrote:
Hi Yuval,
I stuck on the first step where I am trying to create sns topic but not able to create it. I am not able to figure out the issue. AMQP server is ready, up and running with AMQP 0.9.1.
[root@ceprgw01 ~]# aws --endpoint-url http://localhost:8000 sns create-topic --name=mytopic --attributes='{"push-endpoint": "amqp://10.xx.xx.xx:5672", "amqp-exchange": "ex1", "amqp-ack-level": "broker"}'
getting below error:
Unknown options: {"push-endpoint": "amqp://10.xx.xx.xx:15672", "amqp-ack-level": "broker", "persistent": "true", "amqp-exchange": "topic_logs"}, set-topic-attributes
I am using Ceph 4.2.
[root@ceprgw01 ~]# aws --version
aws-cli/1.14.28 Python/2.7.5 Linux/3.10.0-1160.25.1.el7.x86_64 botocore/1.8.35
Best regards,
Sanjeev
________________________________
From: Yuval Lifshitz <ylifshit@xxxxxxxxxx<mailto:ylifshit@xxxxxxxxxx>>
Sent: Friday, August 20, 2021 3:40 PM
To: Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>>
Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> <ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>>
Subject: Re: S3 Bucket Notification requirement
On Thu, Aug 19, 2021 at 6:30 PM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>> wrote:
Hi Yuval,
Thanks very much for your reply.
I am using AMQP 0.9.1.
Can I use aws sns create-topic command to create a topic in Ceph's RadosGW ?
yes. note that you define the topic on the RGW, not the rabbitmq broker. e.g.
aws --endpoint-url http://<rgw host>:<rgw port> sns create-topic --name=mytopic --attributes='{"push-endpoint": "amqp://<amqp host>:5672", "amqp-exchange": "ex1"}'
(see also: https://github.com/ceph/ceph/tree/master/examples/boto3#aws-cli)
,, If Yes then How and where to define notifications to associate with created topics? Basically, I want to understand the tasks needs to be defined in Ceph and RabbitMQ Broker.
you can use the aws cli to define the topic. a minimal setup to get all notifications for object creation would be:
aws --region=default --endpoint-url http://<rgw host>:<rgw posrt> s3api put-bucket-notification-configuration --bucket=mybucket --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:default::mytopic", "Events": ["s3:ObjectCreated:*"]}]}'
I have found few formats like below in the developer documentation but wondering how/where to define it: -- Where and how to define these below codes, In ceph or AMQP broker?
in ceph (RGW).
these are the formats of the raw HTTP messages used to create the topic. when you use AWS cli, they crafty these messages and send them to the RGW.
unless you want to craft these messages yourself (e.g. using CURL) you should not really care about them.
For Example:
POST
Action=CreateTopic
&Name=<topic-name>
&push-endpoint=<endpoint>
[&Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=<exchange>]
...
....
Java is being used in RabbitMQ broker with AMQP 0.9.1 protocol..
creating topics and notifications is also possible using the AWS Java SDK:
* topics: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/examples-simple-notification-service.html
* notifications:
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/BucketNotificationConfiguration.html
so, if your application wants to create them, the application developers will have to write code that does that (sadly, we don't have Java examples like the CLI and python ones, but you can use the example from the AWS docs).
if the topics and notifications are more static, you can create them up front using the AWS cli, and then you don't need to do anything in your Java code.
Best regards,
Sanjeev
________________________________
From: Yuval Lifshitz <ylifshit@xxxxxxxxxx<mailto:ylifshit@xxxxxxxxxx>>
Sent: Thursday, August 19, 2021 8:38 PM
To: Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>>
Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> <ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>>
Subject: Re: S3 Bucket Notification requirement
Hi Sanjeev,
Welcome to the Ceph community!
Which protocol do you intend to use in ActiveMQ?
If you want to use AMQP1.0, you will have to wait, as this is still not officially supported [1].
Currently, we support AMQP0.9.1, Kafka, and HTTP.
As for the more general question.
To make bucket notifications work you first need to define a "topic", with an endpoint configured to point your message broker.
Next, you need to define a "notification" that associates the topic with a bucket on which you want to get notifications.
In the "notification" you can define a filter and the events on which you want to be notified.
You can find examples of how to do that using the AWS CLI tool (or in python, using the boto3 library) here [2].
If you are looking for a different client or client SDK let me know.
More details on the bucket notification feature (without any client specific examples) could be found here [3].
Yuval
[1] https://github.com/ceph/ceph/pull/42548
[2] https://github.com/ceph/ceph/tree/master/examples/boto3
[3] https://docs.ceph.com/en/latest/radosgw/notifications/
On Thu, Aug 19, 2021 at 5:10 PM Sanjeev Jha <sanjeev_mac@xxxxxxxxxxx<mailto:sanjeev_mac@xxxxxxxxxxx>> wrote:
Dear Sir,
I would like to inform you that I am new to Ceph, specially new to the S3 bucket notification topic and not able to understand how exactly to configure it in RadaosGW. We are using an ActiveMQ broker.
There is no easy technical documentation for Ceph available and I am not sure how and where to configure the coding.
I am not the developer, rather just an administrator who manages Ceph.
I would appreciate If you could let me know the steps if possible so that I could achieve this goal.
Best regards,
Sanjeev
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
