Thank you, Matt. It seems that RGW has recently made a very large refactoring. I need more time to rebase our code. see https://github.com/Liuchang0812/ceph/commits/wip-rgw-fdb-index Additionally, I uploaded 100W objects to normal index bucket named `lctest3`, and listed both buckets as below. ``` [root@100 ~]# time s3cmd ls s3://lctest3 1>/dev/null # nornal index bucket real 0m6.290s user 0m0.080s sys 0m0.012s [root@100 ~]# time s3cmd ls s3://lctest2 1>/dev/null # fdb bucket index real 0m0.848s user 0m0.072s sys 0m0.020s ``` Matt Benjamin <mbenjami@xxxxxxxxxx> 于2018年11月7日周三 上午12:15写道： > > That makes sense. Just wanted to communicate that you folks are > welcome to join some upstream RGW standups, where similar and > different solutions to (among other things) bucket index scaling > challenges are going on. > > Matt > > On Tue, Nov 6, 2018 at 3:28 AM, 吕珊春 <lvshanchun@xxxxxxxxx> wrote: > > At present, we are developing and testing based on v12.2.2, we'll > > rebase it into master branch and put a PR later on . > > > > > > regards, > > Leeshine > > Matt Benjamin <mbenjami@xxxxxxxxxx> 于2018年11月6日周二 上午10:37写道： > >> > >> i.e., there is definitely a community of interest in external index > >> strategies, and it would be desirable to share/reuse code and > >> invariants among them. > >> > >> Matt > >> > >> On Mon, Nov 5, 2018 at 9:35 PM, Matt Benjamin <mbenjami@xxxxxxxxxx> wrote: > >> > Hi, > >> > > >> > I don't know if this is a sustainable direction or now, but we welcome > >> > you to develop it upstream. > >> > > >> > regards, > >> > > >> > Matt > >> > > >> > On Mon, Nov 5, 2018 at 9:31 PM, liuchang0812 <liuchang0812@xxxxxxxxx> wrote: > >> >> Hi, all, > >> >> > >> >> We(@liuchang0812 & @Leeshine) are currently working on implementing a > >> >> new bucket index in the Ceph RGW (called FDBIndex). The key points of > >> >> FDBIndex are as below: > >> >> > >> >> 1. likes tail objects, we make head objects immutable too. > >> >> 2. stores raw_rgw_key into FDB; uses `rgw_obj_key::parse_raw_oid` to > >> >> get corresponding head object name > >> >> 3. uses FDB's transaction to atomicly update a) object id b) object > >> >> meta info c) bucket usage. > >> >> > >> >> currently we uploads some RGW objects. RGW will create some rados objects as : > >> >> > >> >> 0c4d3ad9-8ff1-45da-9e58-20b5e3766b6d.4158.1__head_ceph.confEMLK8gcVFNgi3O0YpZi7ldQPV_PtyBC > >> >> 0c4d3ad9-8ff1-45da-9e58-20b5e3766b6d.4158.1__head_unittest_workqueuebqvuHn7qym69q2JHNURjOUgc0YHlx4l > >> >> 0c4d3ad9-8ff1-45da-9e58-20b5e3766b6d.4158.1__shadow_unittest_workqueuebqvuHn7qym69q2JHNURjOUgc0YHlx4l.1_1 > >> >> > >> >> We have implemented a prototype already. The new bucket index bucket > >> >> supports almost core APIs: > >> >> > >> >> 1. put object > >> >> 2. put multi-part object > >> >> 3. get object > >> >> 4. delete object > >> >> 5. list objects > >> >> 6. head object > >> >> > >> >> ==== testing result > >> >> > >> >> Test cluster is consists of 36 FDB processes(3machines) and 84 OSDs(7 machines). > >> >> > >> >> The bucket which used FDBIndex is expected to store unlimited number > >> >> of objects, besides this, as long as the FDB cluster is writable, the > >> >> bucket is writable. In order to test this feature, I first uploaded > >> >> almost 700W objects into a FDBIndex bucket and ran a backgroud task to > >> >> upload objects continuously, When I killed a FDB process or rebooted > >> >> a FDB machine, thae backgroud task was not blocked(i.e. the bucket is > >> >> always writable). > >> >> > >> >> Upload 10MB file 100 times, FDB index bucket takes 50.814s, but normal > >> >> index bucket takes 53.447s. > >> >> > >> >> I created a bucket with 100 subdirectories and 9999 files under each > >> >> subdirectory. It takes 0.8s to list root directory. > >> >> > >> >> Thanks, comments are appreciated! > >> >> > >> >> 1. foundationdb: https://github.com/apple/foundationdb > >> > > >> > > >> > > >> > -- > >> > > >> > Matt Benjamin > >> > Red Hat, Inc. > >> > 315 West Huron Street, Suite 140A > >> > Ann Arbor, Michigan 48103 > >> > > >> > http://www.redhat.com/en/technologies/storage > >> > > >> > tel. 734-821-5101 > >> > fax. 734-769-8938 > >> > cel. 734-216-5309 > >> > >> > >> > >> -- > >> > >> Matt Benjamin > >> Red Hat, Inc. > >> 315 West Huron Street, Suite 140A > >> Ann Arbor, Michigan 48103 > >> > >> http://www.redhat.com/en/technologies/storage > >> > >> tel. 734-821-5101 > >> fax. 734-769-8938 > >> cel. 734-216-5309 > > > > -- > > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://www.redhat.com/en/technologies/storage > > tel. 734-821-5101 > fax. 734-769-8938 > cel. 734-216-5309
