Hello! I’m trying to use overlayfs over Lustre (as lower and upper). I saw the following patch from Miklos to allow distributed fs as lower layer: >From 7c03b5d45b8eebf0111125053d8fe887cc262ba6 Mon Sep 17 00:00:00 2001 From: Miklos Szeredi <mszeredi@xxxxxxx> Date: Mon, 22 Jun 2015 13:53:48 +0200 Subject: [PATCH 2/2] ovl: allow distributed fs as lower layer I know that officially distributed fs are not supported as upper, but I tested myself with Lustre as lower and upper by removing the filesystem checks (in super.c/util.c), and while it seemed to work at first (performance is great), writing from multiple lustre clients (different cluster nodes) to a single shared file led to unexpected results. This is unfortunately a case that we have to support... Example with two clients writing to a shared file using the IOR benchmark, only one client can see the resulting shared file: mounts on both nodes: 10.210.34.201@o2ib1:10.210.34.202@o2ib1:/regal on /regal type lustre (rw,flock,user_xattr,lazystatfs) 10.0.10.51@o2ib7:10.0.10.52@o2ib7:/fir on /fir type lustre (rw,flock,user_xattr,lazystatfs) overlay on /scratch/users type overlay (rw,relatime,lowerdir=/regal/users,upperdir=/fir/users,workdir=/fir/.work) multiple writers to /scratch/users/sthiell/ior/shared node #1 [sthiell@sh-101-60 /scratch/users/sthiell/ior]$ ls -l total 33554440 -rw-r--r-- 1 sthiell ruthm 68719476736 Oct 26 11:17 shared node #2 [sthiell@sh-101-59 /scratch/users/sthiell/ior]$ ls -l ls: cannot access shared: No such file or directory total 0 -????????? ? ? ? ? ? shared [sthiell@sh-101-59 /scratch/users/sthiell/ior]$ stat shared stat: cannot stat ‘shared’: No such file or directory Is anyone aware of any work that has already been done in supporting a distributed filesystem like Lustre as an upper? If not, do you think it’s easily doable? Is it a matter of checking for d_revalidate()? If you wonder why I’m trying to do that, this is because it would be a killer feature for seamlessly migrating cluster scratch filesystems (eg. to new hardware). Thanks! -- Stephane Thiell Stanford Research Computing
