API
===
mshare does not introduce a new API. It instead uses existing APIs
to implement page table sharing. The steps to use this feature are:
1. Mount msharefs on /sys/fs/mshare -
mount -t msharefs msharefs /sys/fs/mshare
2. mshare regions have alignment and size requirements. Start
address for the region must be aligned to an address boundary and
be a multiple of fixed size. This alignment and size requirement
can be obtained by reading the file /sys/fs/mshare/mshare_info
which returns a number in text format. mshare regions must be
aligned to this boundary and be a multiple of this size.
3. For the process creating an mshare region:
a. Create a file on /sys/fs/mshare, for example -
fd = open("/sys/fs/mshare/shareme",
O_RDWR|O_CREAT|O_EXCL, 0600);
b. Establish the starting address and size of the region
struct mshare_info minfo;
minfo.start = TB(2);
minfo.size = BUFFER_SIZE;
ioctl(fd, MSHAREFS_SET_SIZE, &minfo)
We could set the size using ftruncate, just like for any other file. It
would have to be the first thing after creating the file, and before we
allow any other modifications.
Idealy, we'd be able to get rid of the "start", use something resaonable
(e.g., TB(2)) internally, and allow processes to mmap() it at different
(suitably-aligned) addresses.
I recall we discussed that in the past. Did you stumble over real
blockers such that we really must mmap() the file at the same address in
all processes? I recall some things around TLB flushing, but not sure.
So we might have to stick to an mmap address for now.
When using fallocate/stat to set/query the file size, we could end up with:
/*
* Set the address where this file can be mapped into processes. Other
* addresses are not supported for now, and mmap will fail. Changing the
* mmap address after mappings were already created is not supported.
*/
MSHAREFS_SET_MMAP_ADDRESS
MSHAREFS_GET_MMAP_ADDRESS
c. Map some memory in the region
struct mshare_create mcreate;
mcreate.addr = TB(2);
Can we use the offset into the virtual file instead? We should be able
to perform that translation internally fairly easily I assume.
mcreate.size = BUFFER_SIZE;
mcreate.offset = 0;
mcreate.prot = PROT_READ | PROT_WRITE;
mcreate.flags = MAP_ANONYMOUS | MAP_SHARED | MAP_FIXED;
mcreate.fd = -1;
ioctl(fd, MSHAREFS_CREATE_MAPPING, &mcreate)
Would examples with multiple mappings work already in this version?
Did you experiment with other mappings (e.g., ordinary shared file
mappings), and what are the blockers to make that fly?
d. Map the mshare region into the process
mmap((void *)TB(2), BUF_SIZE, PROT_READ | PROT_WRITE,
MAP_SHARED, fd, 0);
e. Write and read to mshared region normally.
4. For processes attaching an mshare region:
a. Open the file on msharefs, for example -
fd = open("/sys/fs/mshare/shareme", O_RDWR);
b. Get information about mshare'd region from the file:
struct mshare_info minfo;
ioctl(fd, MSHAREFS_GET_SIZE, &minfo);
c. Map the mshare'd region into the process
mmap(minfo.start, minfo.size,
PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
5. To delete the mshare region -
unlink("/sys/fs/mshare/shareme");
I recall discussions around cgroup accounting, OOM handling etc. I
thought the conclusion was that we need an "mshare process" where the
memory is accounted to, and once that process is killed (e.g., OOM), it
must tear down all mappings/pages etc.
How does your design currently look like in that regard? E.g., how can
OOM handling make progress, how is cgroup accounting handled?
--
Cheers,
David / dhildenb
