| I think what Gaurav is proposing is only 3000 files *as Ceph sees them*, with a filesystem structure within each. This is very similar to a thread from around a year ago from the Software Heritage folks.
Though in one spot he describes 10 billion files total, and in another it seems like 600 billion.
Since CSI is mentioned, I think RBD would fit.
One issue with RGW here is the object size. The 10 Billion Object project uses 64KB S3 objects and IIRC ran afoul of the min_alloc_size factor. Gaurav's files sizes are described as 1-10KB which would be subject to massive space amplification I suspect that 1KB might be an impractical value for min_alloc_size to compensate.
Gaurav, might you share more details about the nature of the workload?
As I understand it the workload you describe is way more suitable for object (S3) storage than for file. These numbers were also tested successfully with RGW. Hey Cephers, We have a CephFS use case with the following workload requirements with around 10 Billion small sized files each around 1 to 10 kb Workloads might be very write intensive. For example when we need to perform backup recovery that writes hundreds of millions of files or large-scale imports that also happen from time to time. Hierarchy is the following: CephFS CSI Kubernetes volumes that can reach a billion or more files each. We expect up to 100,000 files/directories.
Overall it would consist of 3000 CephFS volumes each of size about 20 to 30 TB based on PVC each having about 200 million files. While having a discussion with Dan he mentioned one fear about one FS with 10B files is that it would be impractical to scrub -- it would take too long. And suggested that we should split this into several small clusters for this particular reason alone. We are looking forward to documentation, case studies and MDS tuning, configuration references as well as examples if anyone has any knowledge or suggestions about such a workload. We also want to understand if there are any limitations as well if anyone has tested such a kind of workload in a Ceph cluster.
Kind regards, Gaurav
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx
_______________________________________________ Dev mailing list -- dev@xxxxxxxTo unsubscribe send an email to dev-leave@xxxxxxx
|
_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx
