On 24/04/28 01:47PM, Dongsheng Yang wrote: > > > 在 2024/4/27 星期六 上午 12:14, Gregory Price 写道: > > On Fri, Apr 26, 2024 at 10:53:43PM +0800, Dongsheng Yang wrote: > > > > > > > > > 在 2024/4/26 星期五 下午 9:48, Gregory Price 写道: > > > > > > > > > > In (5) of the cover letter, I mentioned that cbd addresses cache coherence > > > at the software level: > > > > > > (5) How do blkdev and backend interact through the channel? > > > a) For reader side, before reading the data, if the data in this channel > > > may be modified by the other party, then I need to flush the cache before > > > reading to ensure that I get the latest data. For example, the blkdev needs > > > to flush the cache before obtaining compr_head because compr_head will be > > > updated by the backend handler. > > > b) For writter side, if the written information will be read by others, > > > then after writing, I need to flush the cache to let the other party see it > > > immediately. For example, after blkdev submits cbd_se, it needs to update > > > cmd_head to let the handler have a new cbd_se. Therefore, after updating > > > cmd_head, I need to flush the cache to let the backend see it. > > > > > > > Flushing the cache is insufficient. All that cache flushing guarantees > > is that the memory has left the writer's CPU cache. There are potentially > > many write buffers between the CPU and the actual backing media that the > > CPU has no visibility of and cannot pierce through to force a full > > guaranteed flush back to the media. > > > > for example: > > > > memcpy(some_cacheline, data, 64); > > mfence(); > > > > Will not guarantee that after mfence() completes that the remote host > > will have visibility of the data. mfence() does not guarantee a full > > flush back down to the device, it only guarantees it has been pushed out > > of the CPU's cache. > > > > similarly: > > > > memcpy(some_cacheline, data, 64); > > mfence(); > > memcpy(some_other_cacheline, data, 64); > > mfence() > > > > Will not guarantee that some_cacheline reaches the backing media prior > > to some_other_cacheline, as there is no guarantee of write-ordering in > > CXL controllers (with the exception of writes to the same cacheline). > > > > So this statement: > > > > > I need to flush the cache to let the other party see it immediately. > > > > Is misleading. They will not see is "immediately", they will see it > > "eventually at some completely unknowable time in the future". > > This is indeed one of the issues I wanted to discuss at the RFC stage. Thank > you for pointing it out. > > In my opinion, using "nvdimm_flush" might be one way to address this issue, > but it seems to flush the entire nd_region, which might be too heavy. > Moreover, it only applies to non-volatile memory. > > This should be a general problem for cxl shared memory. In theory, FAMFS > should also encounter this issue. > > Gregory, John, and Dan, Any suggestion about it? > > Thanx a lot > > > > ~Gregory > > Hi Dongsheng, Gregory is right about the uncertainty around "clflush" operations, but let me drill in a bit further. Say you copy a payload into a "bucket" in a queue and then update an index in a metadata structure; I'm thinking of the standard producer/ consumer queuing model here, with one index mutated by the producer and the other mutated by the consumer. (I have not reviewed your queueing code, but you *must* be using this model - things like linked-lists won't work in shared memory without shared locks/atomics.) Normal logic says that you should clflush the payload before updating the index, then update and clflush the index. But we still observe in non-cache-coherent shared memory that the payload may become valid *after* the clflush of the queue index. The famfs user space has a program called pcq.c, which implements a producer/consumer queue in a pair of famfs files. The only way to currently guarantee a valid read of a payload is to use sequence numbers and checksums on payloads. We do observe mismatches with actual shared memory, and the recovery is to clflush and re-read the payload from the client side. (Aside: These file pairs theoretically might work for CBD queues.) Anoter side note: it would be super-helpful if the CPU gave us an explicit invalidate rather than just clflush, which will write-back before invalidating *if* the cache line is marked as dirty, even when software knows this should not happen. Note that CXL 3.1 provides a way to guarantee that stuff that should not be written back can't be written back: read-only mappings. This one of the features I got into the spec; using this requires CXL 3.1 DCD, and would require two DCD allocations (i.e. two tagged-capacity dax devices - one writable by the server and one by the client). Just to make things slightly gnarlier, the MESI cache coherency protocol allows a CPU to speculatively convert a line from exclusive to modified, meaning it's not clear as of now whether "occasional" clean write-backs can be avoided. Meaning those read-only mappings may be more important than one might think. (Clean write-backs basically make it impossible for software to manage cache coherency.) Keep in mind that I don't think anybody has cxl 3 devices or CPUs yet, and shared memory is not explicitly legal in cxl 2, so there are things a cpu could do (or not do) in a cxl 2 environment that are not illegal because they should not be observable in a no-shared-memory environment. CBD is interesting work, though for some of the reasons above I'm somewhat skeptical of shared memory as an IPC mechanism. Regards, John
