> > On Wed, May 10, 2017 at 09:26:00PM +0530, Pankaj Gupta wrote: > > We are sharing initial project proposal for > > 'KVM "fake DAX" device flushing' project for feedback. > > Got the idea during discussion with 'Rik van Riel'. > > CCing NVDIMM folks. > > > > > Also, request answers to 'Questions' section. > > > > Abstract : > > ---------- > > Project idea is to use fake persistent memory with direct > > access(DAX) in virtual machines. Overall goal of project > > is to increase the number of virtual machines that can be > > run on a physical machine, in order to increase the density > > of customer virtual machines. > > > > The idea is to avoid the guest page cache, and minimize the > > memory footprint of virtual machines. By presenting a disk > > image as a nvdimm direct access (DAX) memory region in a > > virtual machine, the guest OS can avoid using page cache > > memory for most file accesses. > > > > Problem Statement : > > ------------------ > > * Guest uses page cache in memory to process fast requests > > for disk read/write. This results in big memory footprint > > of guests without host knowing much details of the guest > > memory. > > > > * If guests use direct access(DAX) with fake persistent > > storage, the host manages the page cache for guests, > > allowing the host to easily reclaim/evict less frequently > > used page cache pages without requiring guest cooperation, > > like ballooning would. > > > > * Host manages guest cache as ‘mmaped’ disk image area in > > qemu address space. This region is passed to guest as fake > > persistent memory range. We need a new flushing interface > > to flush this cache to secondary storage to persist guest > > writes. > > > > * New asynchronous flushing interface will allow guests to > > cause the host flush the dirty data to backup storage file. > > Systems with pmem storage make use of CLFLUSH instruction > > to flush single cache line to persistent storage and it > > takes care of flushing. With fake persistent storage in > > guest we cannot depend on CLFLUSH instruction to flush entire > > dirty cache to backing storage. Even If we trap and emulate > > CLFLUSH instruction guest vCPU has to wait till we flush all > > the dirty memory. Instead of this we need to implement a new > > asynchronous guest flushing interface, which allows the guest > > to specify a larger range to be flushed at once, and allows > > the vCPU to run something else while the data is being synced > > to disk. > > > > * New flushing interface will consists of a para virt driver to > > new fake nvdimm like device which will process guest flushing > > requests like fsync/msync etc instead of pmem library calls > > like clflush. The corresponding device at host side will be > > responsible for flushing requests for guest dirty pages. > > Guest can put current task in sleep and vCPU can run any other > > task while host side flushing of guests pages is in progress. > > > > Host controlled fake nvdimm DAX to avoid guest page cache : > > ------------------------------------------------------------- > > * Bypass guest page cache by using a fake persistent storage > > like nvdimm & DAX. Guest Read/Write is directly done on > > fake persistent storage without involving guest kernel for > > caching data. > > > > * Fake nvdimm device passed to guest is backed by a regular > > file in host stored in secondary storage. > > > > * Qemu has implementation of fake NVDIMM/DAX device. Use this > > capability of passing regular host file(disk) as nvdimm device > > to guest. > > > > * Nvdimm with DAX works for ext4/xfs filesystem. Supported > > filesystem should be DAX compatible. > > > > * As we are using guest disk as fake DAX/NVDIMM device, we > > need a mechanism for persistence of data backed on regular > > host storage file. > > > > * For live migration use case, if host side backing file is > > shared storage, we need to flush the page cache for the disk > > image at the destination (new fadvise interface, FADV_INVALIDATE_CACHE?) > > before starting execution of the guest on the destination host. > > Good point. QEMU currently only supports live migration with O_DIRECT. > I think the problem was that userspace cannot guarantee consistency in > the general case. If you find a solution to this problem for fake > NVDIMM then maybe the QEMU block layer can also begin supporting live > migration with buffered I/O. > > > > > Design : > > --------- > > * In order to not have page cache inside the guest, qemu would: > > > > 1) mmap the guest's disk image and present that disk image to > > the guest as a persistent memory range. > > > > 2) Present information to the guest telling it that the persistent > > memory range is not physical persistent memory. > > Steps 1 & 2 are already supported by QEMU NVDIMM emulation today. Yes. I have also tested guest 'fake DAX' device using QEMU NVDIMM emulation. > > > 3) Present an additional paravirt device alongside the persistent > > memory range, that can be used to sync (ranges of) data to disk. > > > > * Guest would use the disk image mostly like a persistent memory > > device, with two exceptions: > > > > 1) It would not tell userspace that the files on that device are > > persistent memory. This is done so userspace knows to call > > fsync/msync, instead of the pmem clflush library call. > > Not sure I agree with hiding the nvdimm nature of the device. Instead I > think you need to build this capability into the Linux nvdimm code. > libpmem will detect these types of devices and issue fsync/msync when > the application wants to flush. > > > 2) When userspace calls fsync/msync on files on the fake persistent > > memory device, issue a request through the paravirt device that > > causes the host to flush the device back end. > > > > * Guest uses fake persistent storage data updates can be still in > > qemu memory. We need a way to flush cached data in host to backed > > s/qemu memory/host memory/ > > I guess you mean that host userspace needs a way to reliably flush an > address range to the underlying storage. right. > > > secondary storage. > > > > * Once the guest receives a completion event from the host, it will > > allow userspace programs that were waiting on the fsync/msync to > > continue running. > > > > * Host is responsible for paging in pages in host backing area for > > guest persistent memory as they are accessed by the guest, and > > for evicting pages as host memory fills up. > > > > Questions : > > ----------- > > * What should the flushing interface between guest and host look > > like? > > A simple hack for prototyping is to instantiate an virtio-blk-pci for > the mmapped host file. The guest can send flush commands on the > virtio-blk-pci device but will otherwise use the mapped memory directly. okay. I will check this. > > > * Any suggestions to hook the IO caching code with KVM/Qemu or > > thoughts on how we should do it? > > > > * Thinking of implementing a guest para virt driver which will send > > guest requests to Qemu to flush data to disk. Not sure at this > > point how to tell userspace to work on this device as any regular > > device without considering it as persistent device. Any suggestions > > on this? > > > > * Not thought yet about ballooning impact. But feel this solution > > could be better than ballooning in long term? As we will be > > managing all guests cache from host side. > > > > * Not sure this solution works for ARM and other architectures and > > Windows? >
