That makes sense. Just wanted to communicate that you folks are welcome to join some upstream RGW standups, where similar and different solutions to (among other things) bucket index scaling challenges are going on. Matt On Tue, Nov 6, 2018 at 3:28 AM, 吕珊春 <lvshanchun@xxxxxxxxx> wrote: > At present, we are developing and testing based on v12.2.2, we'll > rebase it into master branch and put a PR later on . > > > regards, > Leeshine > Matt Benjamin <mbenjami@xxxxxxxxxx> 于2018年11月6日周二 上午10:37写道： >> >> i.e., there is definitely a community of interest in external index >> strategies, and it would be desirable to share/reuse code and >> invariants among them. >> >> Matt >> >> On Mon, Nov 5, 2018 at 9:35 PM, Matt Benjamin <mbenjami@xxxxxxxxxx> wrote: >> > Hi, >> > >> > I don't know if this is a sustainable direction or now, but we welcome >> > you to develop it upstream. >> > >> > regards, >> > >> > Matt >> > >> > On Mon, Nov 5, 2018 at 9:31 PM, liuchang0812 <liuchang0812@xxxxxxxxx> wrote: >> >> Hi, all, >> >> >> >> We(@liuchang0812 & @Leeshine) are currently working on implementing a >> >> new bucket index in the Ceph RGW (called FDBIndex). The key points of >> >> FDBIndex are as below: >> >> >> >> 1. likes tail objects, we make head objects immutable too. >> >> 2. stores raw_rgw_key into FDB; uses `rgw_obj_key::parse_raw_oid` to >> >> get corresponding head object name >> >> 3. uses FDB's transaction to atomicly update a) object id b) object >> >> meta info c) bucket usage. >> >> >> >> currently we uploads some RGW objects. RGW will create some rados objects as : >> >> >> >> 0c4d3ad9-8ff1-45da-9e58-20b5e3766b6d.4158.1__head_ceph.confEMLK8gcVFNgi3O0YpZi7ldQPV_PtyBC >> >> 0c4d3ad9-8ff1-45da-9e58-20b5e3766b6d.4158.1__head_unittest_workqueuebqvuHn7qym69q2JHNURjOUgc0YHlx4l >> >> 0c4d3ad9-8ff1-45da-9e58-20b5e3766b6d.4158.1__shadow_unittest_workqueuebqvuHn7qym69q2JHNURjOUgc0YHlx4l.1_1 >> >> >> >> We have implemented a prototype already. The new bucket index bucket >> >> supports almost core APIs: >> >> >> >> 1. put object >> >> 2. put multi-part object >> >> 3. get object >> >> 4. delete object >> >> 5. list objects >> >> 6. head object >> >> >> >> ==== testing result >> >> >> >> Test cluster is consists of 36 FDB processes(3machines) and 84 OSDs(7 machines). >> >> >> >> The bucket which used FDBIndex is expected to store unlimited number >> >> of objects, besides this, as long as the FDB cluster is writable, the >> >> bucket is writable. In order to test this feature, I first uploaded >> >> almost 700W objects into a FDBIndex bucket and ran a backgroud task to >> >> upload objects continuously, When I killed a FDB process or rebooted >> >> a FDB machine, thae backgroud task was not blocked(i.e. the bucket is >> >> always writable). >> >> >> >> Upload 10MB file 100 times, FDB index bucket takes 50.814s, but normal >> >> index bucket takes 53.447s. >> >> >> >> I created a bucket with 100 subdirectories and 9999 files under each >> >> subdirectory. It takes 0.8s to list root directory. >> >> >> >> Thanks, comments are appreciated! >> >> >> >> 1. foundationdb: https://github.com/apple/foundationdb >> > >> > >> > >> > -- >> > >> > Matt Benjamin >> > Red Hat, Inc. >> > 315 West Huron Street, Suite 140A >> > Ann Arbor, Michigan 48103 >> > >> > http://www.redhat.com/en/technologies/storage >> > >> > tel. 734-821-5101 >> > fax. 734-769-8938 >> > cel. 734-216-5309 >> >> >> >> -- >> >> Matt Benjamin >> Red Hat, Inc. >> 315 West Huron Street, Suite 140A >> Ann Arbor, Michigan 48103 >> >> http://www.redhat.com/en/technologies/storage >> >> tel. 734-821-5101 >> fax. 734-769-8938 >> cel. 734-216-5309 -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
