On Wed, May 10, 2017 at 09:26:00PM +0530, Pankaj Gupta wrote: > We are sharing initial project proposal for > 'KVM "fake DAX" device flushing' project for feedback. > Got the idea during discussion with 'Rik van Riel'. CCing NVDIMM folks. > > Also, request answers to 'Questions' section. > > Abstract : > ---------- > Project idea is to use fake persistent memory with direct > access(DAX) in virtual machines. Overall goal of project > is to increase the number of virtual machines that can be > run on a physical machine, in order to increase the density > of customer virtual machines. > > The idea is to avoid the guest page cache, and minimize the > memory footprint of virtual machines. By presenting a disk > image as a nvdimm direct access (DAX) memory region in a > virtual machine, the guest OS can avoid using page cache > memory for most file accesses. > > Problem Statement : > ------------------ > * Guest uses page cache in memory to process fast requests > for disk read/write. This results in big memory footprint > of guests without host knowing much details of the guest > memory. > > * If guests use direct access(DAX) with fake persistent > storage, the host manages the page cache for guests, > allowing the host to easily reclaim/evict less frequently > used page cache pages without requiring guest cooperation, > like ballooning would. > > * Host manages guest cache as ‘mmaped’ disk image area in > qemu address space. This region is passed to guest as fake > persistent memory range. We need a new flushing interface > to flush this cache to secondary storage to persist guest > writes. > > * New asynchronous flushing interface will allow guests to > cause the host flush the dirty data to backup storage file. > Systems with pmem storage make use of CLFLUSH instruction > to flush single cache line to persistent storage and it > takes care of flushing. With fake persistent storage in > guest we cannot depend on CLFLUSH instruction to flush entire > dirty cache to backing storage. Even If we trap and emulate > CLFLUSH instruction guest vCPU has to wait till we flush all > the dirty memory. Instead of this we need to implement a new > asynchronous guest flushing interface, which allows the guest > to specify a larger range to be flushed at once, and allows > the vCPU to run something else while the data is being synced > to disk. > > * New flushing interface will consists of a para virt driver to > new fake nvdimm like device which will process guest flushing > requests like fsync/msync etc instead of pmem library calls > like clflush. The corresponding device at host side will be > responsible for flushing requests for guest dirty pages. > Guest can put current task in sleep and vCPU can run any other > task while host side flushing of guests pages is in progress. > > Host controlled fake nvdimm DAX to avoid guest page cache : > ------------------------------------------------------------- > * Bypass guest page cache by using a fake persistent storage > like nvdimm & DAX. Guest Read/Write is directly done on > fake persistent storage without involving guest kernel for > caching data. > > * Fake nvdimm device passed to guest is backed by a regular > file in host stored in secondary storage. > > * Qemu has implementation of fake NVDIMM/DAX device. Use this > capability of passing regular host file(disk) as nvdimm device > to guest. > > * Nvdimm with DAX works for ext4/xfs filesystem. Supported > filesystem should be DAX compatible. > > * As we are using guest disk as fake DAX/NVDIMM device, we > need a mechanism for persistence of data backed on regular > host storage file. > > * For live migration use case, if host side backing file is > shared storage, we need to flush the page cache for the disk > image at the destination (new fadvise interface, FADV_INVALIDATE_CACHE?) > before starting execution of the guest on the destination host. Good point. QEMU currently only supports live migration with O_DIRECT. I think the problem was that userspace cannot guarantee consistency in the general case. If you find a solution to this problem for fake NVDIMM then maybe the QEMU block layer can also begin supporting live migration with buffered I/O. > > Design : > --------- > * In order to not have page cache inside the guest, qemu would: > > 1) mmap the guest's disk image and present that disk image to > the guest as a persistent memory range. > > 2) Present information to the guest telling it that the persistent > memory range is not physical persistent memory. Steps 1 & 2 are already supported by QEMU NVDIMM emulation today. > 3) Present an additional paravirt device alongside the persistent > memory range, that can be used to sync (ranges of) data to disk. > > * Guest would use the disk image mostly like a persistent memory > device, with two exceptions: > > 1) It would not tell userspace that the files on that device are > persistent memory. This is done so userspace knows to call > fsync/msync, instead of the pmem clflush library call. Not sure I agree with hiding the nvdimm nature of the device. Instead I think you need to build this capability into the Linux nvdimm code. libpmem will detect these types of devices and issue fsync/msync when the application wants to flush. > 2) When userspace calls fsync/msync on files on the fake persistent > memory device, issue a request through the paravirt device that > causes the host to flush the device back end. > > * Guest uses fake persistent storage data updates can be still in > qemu memory. We need a way to flush cached data in host to backed s/qemu memory/host memory/ I guess you mean that host userspace needs a way to reliably flush an address range to the underlying storage. > secondary storage. > > * Once the guest receives a completion event from the host, it will > allow userspace programs that were waiting on the fsync/msync to > continue running. > > * Host is responsible for paging in pages in host backing area for > guest persistent memory as they are accessed by the guest, and > for evicting pages as host memory fills up. > > Questions : > ----------- > * What should the flushing interface between guest and host look > like? A simple hack for prototyping is to instantiate an virtio-blk-pci for the mmapped host file. The guest can send flush commands on the virtio-blk-pci device but will otherwise use the mapped memory directly. > * Any suggestions to hook the IO caching code with KVM/Qemu or > thoughts on how we should do it? > > * Thinking of implementing a guest para virt driver which will send > guest requests to Qemu to flush data to disk. Not sure at this > point how to tell userspace to work on this device as any regular > device without considering it as persistent device. Any suggestions > on this? > > * Not thought yet about ballooning impact. But feel this solution > could be better than ballooning in long term? As we will be > managing all guests cache from host side. > > * Not sure this solution works for ARM and other architectures and > Windows?
Attachment:
signature.asc
Description: PGP signature
