Hi list, we are thinking of building relatively big CEPH-based object storage for storage of our sample files - we have about 700M files ranging from very small (1-4KiB) files to pretty big ones (several GiB). Median of file size is 64KiB. Since the required space is relatively large (1PiB of usable storage), we are thinking of utilizing erasure coding for this case. On the other hand, we need to achieve at least 1200MiB/s throughput on reads. The working assumption is 4+2 EC (thus 50% overhead). Since the EC is per-object, the small objects will be stripped to even smaller ones. With 4+2 EC, one needs (at least) 4 IOs to read a single object in this scenario -> number of required IOPS when using EC is relatively high. Some vendors (such as Hitachi, but I believe EMC as well) do offline, predefined-chunk size EC instead. The idea is to first write objects with replication factor of 3, wait for enough objects to fill 4x 64MiB chunks and only do EC on that. This not only makes the EC less computationally intensive, and repairs much faster, but it also allows reading majority of the small objects directly by reading just part of one of the chunk from it (assuming non degraded state) - one chunk actually contains the whole object. I wonder if something similar is already possible with CEPH and/or is planned. For our use case of very small objects, it would mean near 3-4x performance boosts in terms of required IOPS performance. Another option how to get out of this situation is to be able to specify different storage pools/policies based on file size - i.e. to do 3x replication of the very small files and only use EC for bigger files, where the performance hit with 4x IOPS won't be that painful. But I I am afraid this is not possible... Any other hint is sincerely welcome. Thank you Jiri Horky _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |