On Sun, Feb 2, 2025 at 10:40 PM RIc Wheeler <ricwheeler@xxxxxxxxx> wrote: > > > I have always been super interested in how much we can push the > scalability limits of file systems and for the workloads we need to > support, we need to scale up to supporting absolutely ridiculously large > numbers of files (a few billion files doesn't meet the need of the > largest customers we support). > Hi Ric, Since LSFMM is not about presentations, it would be better if the topic to discuss was trying to address specific technical questions that developers could discuss. If a topic cannot generate a discussion on the list, it is not very likely that it will generate a discussion on-prem. Where does the scaling with the number of files in a filesystem affect existing filesystems? What are the limitations that you need to overcome? > Zach Brown is leading a new project on ngnfs (FOSDEM talk this year gave > a good background on this - > https://www.fosdem.org/2025/schedule/speaker/zach_brown/). We are > looking at taking advantage of modern low latency NVME devices and > today's networks to implement a distributed file system that provides > better concurrency that high object counts need and still have the > bandwidth needed to support the backend archival systems we feed. > I heard this talk and it was very interesting. Here's a direct link to slides from people who may be too lazy to follow 3 clicks: https://www.fosdem.org/2025/events/attachments/fosdem-2025-5471-ngnfs-a-distributed-file-system-using-block-granular-consistency/slides/236150/zach-brow_aqVkVuI.pdf I was both very impressed by the cache coherent rename example and very puzzled - I do not know any filesystem where rename can be synchronized on a single block io, and looking up ancestors is usually done on in-memory dentries, so I may not have understood the example. > ngnfs as a topic would go into the coherence design (and code) that > underpins the increased concurrency it aims to deliver. > > Clear that the project is in early days compared to most of the proposed > content, but it can be useful to spend some of the time on new ideas. > This sounds like an interesting topic to discuss. I would love it if you or Zach could share more details on the list so that more people could participate in the discussion leading to LSFMM. Also, I think it is important to mention, as you told me, that the server implementation of ngnfs is GPL and to provide some pointers, because IMO this is very important when requesting community feedback on a new filesystem. Thanks, Amir.
