On 03/04 09:46, Clyso GmbH - Ceph Foundation Member wrote: > Hello Cary, > > take a look at the Deutsche Telekom project. It is also based on librados > and they also needed a solution for managing the metadata of over 30 million > email accounts. > > https://github.com/ceph-dovecot/dovecot-ceph-plugin > > but also think about how such a system will behave during recovery. > > I also listened to an interesting lecture on this subject by nasa at cern. > > https://indico.cern.ch/event/765214/timetable/?print=1&view=standard#23-ceph-at-nasa-atmosphere-sip watching O.O, thanks for sharing! > > Hope this helps you, Joachim > > ___________________________________ > Clyso GmbH - Ceph Foundation Member > support@xxxxxxxxx > https://www.clyso.com > > Am 03.03.2021 um 21:20 schrieb Gregory Farnum: > > Unfortunately, nothing like this exists in RADOS. It can't really -- > > scaling is inimical to the sort of data collation you seem to be > > looking for. If you use librados, you need to maintain all your own > > metadata. RGW has done a lot of work to support these features; > > depending on what you need you may be able to make do by implementing > > your own centralized metadata system in other objects (omap might be > > useful); if not you'll need to set up a more extensive system. > > -Greg > > > > On Tue, Mar 2, 2021 at 1:12 PM Cary FitzHugh <cary.fitzhugh@xxxxxxxxx> wrote: > > > Phooey. :) > > > > > > Do you know of any notification subsystems in libRADOS that might be > > > useful? > > > > > > Will have to think on this... > > > > > > Thanks > > > > > > On Tue, Mar 2, 2021 at 4:05 PM Matt Benjamin <mbenjami@xxxxxxxxxx> wrote: > > > > > > > Right. The elastic search integration--or something custom you could > > > > base on s3 bucket notifications--would both be working with events > > > > generated in RGW. > > > > > > > > Matt > > > > > > > > On Tue, Mar 2, 2021 at 3:55 PM Cary FitzHugh <cary.fitzhugh@xxxxxxxxx> > > > > wrote: > > > > > Understood. > > > > > > > > > > With the RGW architecture comes more load balancing concerns, more > > > > moving parts, more tedious (to me) ACLs, less features (append and some > > > > other things not supported in S3). Was hoping for a solution which didn't > > > > require us to be hamstrung and only read / write to a pool via the gateway. > > > > > If the RGW Metadata search was able to "source" it's data from the OSDs > > > > and sync that way, then I'd be up for setting up a skeleton > > > > implementation, but it sounds like RGW Metadata is only going to record > > > > things which are flowing through the gateway. (Is that correct?) > > > > > > > > > > > > > > > > > > > > On Tue, Mar 2, 2021 at 3:46 PM Matt Benjamin <mbenjami@xxxxxxxxxx> > > > > wrote: > > > > > > Hi Cary, > > > > > > > > > > > > As you've said, these are well-developed features of RGW, I think that > > > > > > would be the way to go, in the Ceph ecosystem. > > > > > > > > > > > > Matt > > > > > > > > > > > > On Tue, Mar 2, 2021 at 3:41 PM Cary FitzHugh <cary.fitzhugh@xxxxxxxxx> > > > > wrote: > > > > > > > Hello - > > > > > > > > > > > > > > We're trying to use native libRADOS and the only challenge we're > > > > running > > > > > > > into is searching metadata. > > > > > > > > > > > > > > Using the rgw metadata sync seems to require all data to be pushed > > > > through > > > > > > > the rgw, which is not something we're interested in setting up at the > > > > > > > moment. > > > > > > > > > > > > > > Are there hooks or features of libRADOS which could be leveraged to > > > > enable > > > > > > > syncing of metadata to an external system (elastic-search / postgres > > > > / etc)? > > > > > > > Is there a way to listen to a stream of updates to a pool in > > > > real-time, > > > > > > > with some guarantees I wouldn't miss things? > > > > > > > > > > > > > > Are there any features like this in libRADOS? > > > > > > > > > > > > > > Thank you > > > > > > > _______________________________________________ > > > > > > > ceph-users mailing list -- ceph-users@xxxxxxx > > > > > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Matt Benjamin > > > > > > Red Hat, Inc. > > > > > > 315 West Huron Street, Suite 140A > > > > > > Ann Arbor, Michigan 48103 > > > > > > > > > > > > http://www.redhat.com/en/technologies/storage > > > > > > > > > > > > tel. 734-821-5101 > > > > > > fax. 734-769-8938 > > > > > > cel. 734-216-5309 > > > > > > > > > > > > > > -- > > > > > > > > Matt Benjamin > > > > Red Hat, Inc. > > > > 315 West Huron Street, Suite 140A > > > > Ann Arbor, Michigan 48103 > > > > > > > > http://www.redhat.com/en/technologies/storage > > > > > > > > tel. 734-821-5101 > > > > fax. 734-769-8938 > > > > cel. 734-216-5309 > > > > > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@xxxxxxx > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
