Bonjour, In the past months benchmarks were written[0] and run in the grid5000 cluster[1] to verify there was no blocker with the proposed approach[2]. The results were published today[3]. In the following weeks the Software Heritage codebase will be modified to implement the design[4]. Another round of benchmarks will be run when the implementation is complete and a final report will be posted as a followup in this thread. Cheers [0] https://git.easter-eggs.org/biceps/biceps [1] https://www.grid5000.fr/ [2] https://wiki.softwareheritage.org/wiki/A_practical_approach_to_efficiently_store_100_billions_small_objects_in_Ceph [3] https://wiki.softwareheritage.org/wiki/Premilinary_Object_Storage_Benchmarks [4] https://forge.softwareheritage.org/T3432 On 10/03/2021 15:53, Loïc Dachary wrote: > Bonjour, > > In the past weeks a few mailing list threads[0][1][2] explored the problem of storing billions of small objects in Ceph. There was great feedback (I learned at lot) and it turns out the solution is a rather simple aggregation of the ideas that were suggested during these discussions. It is described in detail here: > > https://wiki.softwareheritage.org/wiki/A_practical_approach_to_efficiently_store_100_billions_small_objects_in_Ceph > > The next step will be to write and run benchmarks[3]. Although I'm convinced it is a good solution, I've been wrong before and confirmation is required :-) Many thanks to all the participants in the discussions on behalf of the Software Heritage project[4]. > > Cheers > > [0] Storing 20 billions of immutable objects in Ceph, 75% <16KB https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/JSG2TXKNXPXEKZOJZGYF2ZPTQHOB4LHJ/ > [1] Small RGW objects and RADOS 64KB minimun size https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/AEMW6O7WVJFMUIX7QGI2KM7HKDSTNIYT/ > [2] Using RBD to pack billions of small files https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/RHQ5ZCHJISXIXOJSH3TU7DLYVYHRGTAT/ > [3] https://forge.softwareheritage.org/T3054 > [4] https://www.softwareheritage.org/ > -- Loïc Dachary, Artisan Logiciel Libre _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |