On Fri, Dec 27, 2024 at 02:09:50PM -0500, Gregory Price wrote:
> On Fri, Dec 27, 2024 at 10:40:36AM -0500, Gregory Price wrote:
just adding some follow-up data
test is essentially
membind(1) - node1 is cxl
read() - filecache is initialized on cxl
set_mempolicy(MPOL_DEFAULT) - allow migrations
while true:
start = time()
read()
print(time()-start)
// external events cause migration/drop cache while running
baseline: .93-1s/read()
from cxl: ~1.15-1.2s/read()
So we are seeing anywhere from 20-25% overhead from the filecache living
on CXL right out of the box. At least we have good clear signal, right?
tests:
echo 3 > drop_cache - filecache refills into node 1
result => ~.95-1s/read()
we return back to the baseline, which is expected
enable promotion - numactl shows promotion occurs
result => ~1.15-1.2s/read()
No effect?! Even offlining the dax devices does nothing.
enable promotion, wait for it to complete, drop cache
after promotion => 1.15-1.2s/read
after drop cache => .95-1s/read()
Back to baseline!
This seems to imply that the overhead we're seeing from read() even
when filecache is on the remote node isn't actually related to the
memory speed, but instead likely related to some kind of stale
metadata in the filesystem or filecache layers.
This is going to take me a bit to figure out. I need to isolate the
filesystem influence (we are using btrfs, i want to make sure this
behavior is consistent on other file systems).
~Gregory
