Re: Sending notification after multiple objects are created in a ceph bucket.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Lokendra and Kushagra,
We don't have such an enhancement on the roadmap. Would think of 2 options:
(1) implement the special logic using lua scripting. We have an example on
how to send notifications to a NATS broker from lua [1]. you can easily
adjust that to kafka. the 2 main drawbacks with this solution are:
* performance and latency, everything done in the lua script is
synchronous with the S3 operation
* lua scripts have the "global" RGW table for aggregations. however, this
is local to an RGW and not shared across RGWs. in addition, the information
there is not persistent, and will be lost if the RGW crash/restart
(2) implement that externally to the RGW. e.g. use bucket notifications to
trigger a serverless function (probably backed with some db) that does the
aggregation and send the notifications according to the logic that
you want. see [2]

also, there is the possibility of developing that feature and contributing
it to ceph. note that it would be a considerably more effort than the
options above.

Yuval

[1] https://github.com/ceph/ceph/blob/main/examples/rgw/lua/nats_adapter.md
[2] https://www.youtube.com/watch?v=1E5Q-ErF2sI


On Thu, Jan 11, 2024 at 11:08 AM Lokendra Rathour <lokendrarathour@xxxxxxxxx>
wrote:

> Facing a similar situation, any support would be helpful.
>
> -Lokendra
>
>
>
> On Tue, Jan 9, 2024 at 10:47 PM Kushagr Gupta <
> kushagrguptasps.mun@xxxxxxxxx>
> wrote:
>
> > Hi Team,
> >
> > Features used: Rados gateway, ceph S3 buckets
> >
> > We are trying to create a data pipeline using the S3 buckets capability
> > and rado gateway in ceph.
> > Our endpoint is a kafka topic.
> >
> > Currently, as soon as an object is created, a notification is sent to
> > kafka topic.
> > Our goal is to create multiple object first and then send the
> notification.
> > Also if there is a way such that we can assign objects to a group and
> > collectively send notification on an event on that group.
> >
> > Could anyone please help me?
> >
> > Thanks and Regards,
> > Kushagra Gupta
> >
>
>
> --
> ~ Lokendra
> skype: lokendrarathour
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux