RFC: Virtual-Bus (applies to v2.6.30-rc2) ---------------------- Virtual-Bus is a Linux-kernel based virtual IO resource container technology. It allows you to declare virtualized device models directly within a host kernel that can be uniformly accessed from a variety of environments, such as KVM, userspace, lguest, Xen, etc. The goal is to reduce overhead while still preserving proper isolation to achieve maximum throughput and latency from your IO subsystem. It is installed on the host and configured using simple filesystem operations like "mkdir" to create a new container, "ln -s" to associate a device with a container, and "echo" to configure the device thanks to integration with configfs/sysfs. Therefore, managing a virtual-bus does not require special userspace tools. Why would we want this? ---------------------- We believe that a combination of in-kernel models, coupled with optimizations in various areas such as reducing the number of context switches and executing the algorithms in parallel on modern multicore CPUs will result in the best possible IO performance for technologies such as virtualization. This facility aims to provide the framework for building such constructs in in software in the most efficient way we believe is possible. Preliminary Results: ----------------------- As a test, we wrote a virtual-ethernet backend and a Linux driver (called venet) and tested the setup with v2.6.29 on two 8-core x86_64 boxes with Chelsio T3 10GE connected back to back via cross over. We measured bare-metal performance, as well as a kvm guest (running the same kernel) connected to the T3 via a linux-bridge+tap configuration with a 1500 MTU. The results are as follows: Bare metal : tput = 4078Mb/s, round-trip = 25593pps (39us rtt) Virtio-net (PCI) : tput = 4003Mb/s, round-trip = 320pps (3125us rtt) Venet (VBUS): tput = 4050Mb/s, round-trip = 15255 (65us rtt) When we jump to 9000 byte MTU, the situation looks similar Bare metal : tput = 9717Mb/s, round-trip = 30396pps (33us rtt) Virtio-net (PCI) : tput = 4578Mb/s, round-trip = 249pps (4016us rtt) Venet (VBUS): tput = 5802Mb/s, round-trip = 15127 (66us rtt) Note that to be fair, part of the latency response from virtio-pci is due to the way that the driver/backend coalesces packets, and not a true limitation of the userspace path as the results may imply. Anthony Liguori is already looking into improving this based on the results we presented here, and is making good headway. Also note that as of v2, we support a virtio-vbus transport, and therefore will soon be able to compare "Virtio-net (VBUS)" as an additional target in the near future once we have developed a virtio-net backend for the host. This virtio-net(pci) to virtio-net(vbus) comparison is probably more interesting than comparing to Venet as it would be more apples to apples. Also it would be nice to only need one PV driver per IO subsystem in the long term. Changes v2->v3: ------------------ *) VBUS now renders as a PCI/MSI device instead of via cpuid/dynirq *) Dropped dynirq (replaced with MSI facility) *) Dropped "reset" patch in favor of more reliably detecting reset via BUSOPEN *) Incorporated more review feedback from Stephen Hemminger on vbus-enet driver *) Added support for setting the guests MAC during host config (Pat Mullaney) *) Cleaned up documentation per feedback from Ben Hutchings *) Numerous small tweaks to clean up things like warnings, comments, etc. (Alex Tsariounov, Greg Haskins) *) Fixed Kconfig dependency problem with virtio-vbus with feedback from Peter Morreale. Changes v1->v2: *) Incorporated review feedback from Stephen Hemminger on vbus-enet driver *) Added support for connecting to vbus devices from userspace *) Added support for a virtio-vbus transport to allow virtio drivers to work with vbus (needs testing and backend models). Todo: *) Beef up the userspace event channel ABI to support different event types *) Add memory-registration support *) Develop some virtio backend devices. *) Support ethtool_ops for venet. *) Convert from MSI to MSIX to truely support multiple vectors --------------------------------------- For more details, please see our Wiki at http://developer.novell.com/wiki/index.php/Virtual-bus --- Gregory Haskins (16): virtio: add a vbus transport vbus: add a userspace connector kvm: Add guest-side support for VBUS kvm: Add VBUS support to the host venettap: add scatter-gather support venet: add scatter-gather support venet-tap: Adds a "venet" compatible "tap" device to VBUS net: Add vbus_enet driver venet: add the ABI definitions for an 802.x packet interface ioq: add vbus helpers ioq: Add basic definitions for a shared-memory, lockless queue vbus: add a "vbus-proxy" bus model for vbus_driver objects vbus: add bus-registration notifiers vbus: add connection-client helper infrastructure vbus: add virtual-bus definitions shm-signal: shared-memory signals Patrick Mullaney (1): venet-tap: add the ability to set the client's mac address via sysfs Documentation/vbus.txt | 385 +++++++++ arch/x86/Kconfig | 11 arch/x86/include/asm/kvm_para.h | 1 arch/x86/kvm/Kconfig | 9 arch/x86/kvm/Makefile | 3 arch/x86/kvm/x86.c | 6 arch/x86/kvm/x86.h | 12 drivers/Makefile | 2 drivers/net/Kconfig | 13 drivers/net/Makefile | 1 drivers/net/vbus-enet.c | 898 +++++++++++++++++++++ drivers/vbus/devices/Kconfig | 17 drivers/vbus/devices/Makefile | 1 drivers/vbus/devices/venet-tap.c | 1641 ++++++++++++++++++++++++++++++++++++++ drivers/vbus/proxy/Makefile | 2 drivers/vbus/proxy/kvm.c | 751 +++++++++++++++++ drivers/virtio/Kconfig | 15 drivers/virtio/Makefile | 1 drivers/virtio/virtio_vbus.c | 496 +++++++++++ fs/proc/base.c | 96 ++ include/linux/ioq.h | 410 +++++++++ include/linux/kvm.h | 7 include/linux/kvm_host.h | 26 + include/linux/kvm_para.h | 58 + include/linux/sched.h | 4 include/linux/shm_signal.h | 188 ++++ include/linux/vbus.h | 166 ++++ include/linux/vbus_client.h | 115 +++ include/linux/vbus_device.h | 424 ++++++++++ include/linux/vbus_driver.h | 80 ++ include/linux/vbus_userspace.h | 48 + include/linux/venet.h | 82 ++ include/linux/virtio_vbus.h | 163 ++++ kernel/Makefile | 1 kernel/exit.c | 2 kernel/fork.c | 2 kernel/vbus/Kconfig | 56 + kernel/vbus/Makefile | 11 kernel/vbus/attribute.c | 52 + kernel/vbus/client.c | 543 +++++++++++++ kernel/vbus/config.c | 275 ++++++ kernel/vbus/core.c | 626 ++++++++++++++ kernel/vbus/devclass.c | 124 +++ kernel/vbus/map.c | 72 ++ kernel/vbus/map.h | 41 + kernel/vbus/proxy.c | 216 +++++ kernel/vbus/shm-ioq.c | 89 ++ kernel/vbus/userspace-client.c | 485 +++++++++++ kernel/vbus/vbus.h | 117 +++ kernel/vbus/virtio.c | 627 +++++++++++++++ lib/Kconfig | 21 lib/Makefile | 2 lib/ioq.c | 298 +++++++ lib/shm_signal.c | 186 ++++ virt/kvm/kvm_main.c | 10 virt/kvm/vbus.c | 1392 ++++++++++++++++++++++++++++++++ 56 files changed, 11380 insertions(+), 0 deletions(-) create mode 100644 Documentation/vbus.txt create mode 100644 drivers/net/vbus-enet.c create mode 100644 drivers/vbus/devices/Kconfig create mode 100644 drivers/vbus/devices/Makefile create mode 100644 drivers/vbus/devices/venet-tap.c create mode 100644 drivers/vbus/proxy/Makefile create mode 100644 drivers/vbus/proxy/kvm.c create mode 100644 drivers/virtio/virtio_vbus.c create mode 100644 include/linux/ioq.h create mode 100644 include/linux/shm_signal.h create mode 100644 include/linux/vbus.h create mode 100644 include/linux/vbus_client.h create mode 100644 include/linux/vbus_device.h create mode 100644 include/linux/vbus_driver.h create mode 100644 include/linux/vbus_userspace.h create mode 100644 include/linux/venet.h create mode 100644 include/linux/virtio_vbus.h create mode 100644 kernel/vbus/Kconfig create mode 100644 kernel/vbus/Makefile create mode 100644 kernel/vbus/attribute.c create mode 100644 kernel/vbus/client.c create mode 100644 kernel/vbus/config.c create mode 100644 kernel/vbus/core.c create mode 100644 kernel/vbus/devclass.c create mode 100644 kernel/vbus/map.c create mode 100644 kernel/vbus/map.h create mode 100644 kernel/vbus/proxy.c create mode 100644 kernel/vbus/shm-ioq.c create mode 100644 kernel/vbus/userspace-client.c create mode 100644 kernel/vbus/vbus.h create mode 100644 kernel/vbus/virtio.c create mode 100644 lib/ioq.c create mode 100644 lib/shm_signal.c create mode 100644 virt/kvm/vbus.c -- -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
