On Thu, Jan 25, 2024 at 01:37:02PM -0800, David Rientjes wrote: > > On Thu, 25 Jan 2024, Matthew Wilcox wrote: > > > > That's a huge relief. I was not looking forward to the patches to add > > support for pooling (etc). > > > > Using CXL as cold-data-storage makes a certain amount of sense, although > > I'm not really sure why it offers an advantage over NAND. It's faster > > than NAND, but you still want to bring it back locally before operating > > on it. NAND is denser, and consumes less power while idle. NAND comes > > > > This is **exactly** the type of discussion we're looking to have :) > > There are some things that I've chatted informally with folks about that > I'd like to bring to the forum: > [...snip...] Just going to toss in that tiering from a latency-only perspective is also too narrow a focus. We should also consider bandwidth. Saturating a channel means increased latencies - but the problem can be alleviated by moving hot data *off* of heavily contended devices, even if they are farther away. In a best case scenario, the addition of local CXL devices can bring bandwidth-saturated DRAM back down into the "maximum sustainable" (around ~70% of max), reducing average latencies and increasing CPU utilization. We've been finding there are a non-trivial number of workloads that benefit more from distributing their hot data across their available bandwidth than they do from just jamming it all on the local socket. ~Gregory
