On Sat, Apr 13, 2013 at 05:23:41PM -0400, Sasha Levin wrote: > On 04/12/2013 07:36 AM, Rusty Russell wrote: > > Sasha Levin <sasha.levin@xxxxxxxxxx> writes: > >> On 04/11/2013 12:36 PM, Will Deacon wrote: > >>> Hello folks, > >>> > >>> Here's the latest round of ARM fixes and updates for kvmtool. Most of > >>> this is confined to the arm/ subdirectory, with the exception of a fix > >>> to the virtio-mmio vq definitions due to the multi-queue work from > >>> Sasha. I'm not terribly happy about that code though, since it seriously > >>> increases the memory footprint of the guest. > >>> > >>> Without multi-queue, we can boot Debian Wheezy to a prompt in 38MB. With > >>> the new changes, that increases to 170MB! Any chance we can try and tackle > >>> this regression please? I keep getting bitten by the OOM killer :( > >> > >> (cc Rusty, MST) > >> > >> The spec defines the operation of a virtio-net device with regards to multiple > >> queues as follows: > >> > >> """ > >> Device Initialization > >> > >> 1. The initialization routine should identify the receive and transmission > >> virtqueues, up to N+1 of each kind. If VIRTIO_NET_F_MQ feature > >> bit is negotiated, N=max_virtqueue_pairs-1, otherwise identify N=0. > >> > >> [...] > >> > >> 5. Only receiveq0, transmitq0 and controlq are used by default. To use more > >> queues driver must negotiate the VIRTIO_NET_F_MQ feature; initialize > >> up to max_virtqueue_pairs of each of transmit and receive queues; execute_ > >> VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command specifying > >> the number of the transmit and receive queues that is going to be > >> used and wait until the device consumes the controlq buffer and acks this > >> command. > >> """ > >> > >> And kvmtool follows that to the letter: It will initialize the maximum amount of > >> queues it can support during initialization and will start using them only when > >> the device tells it it should use them. > >> > >> As Will has stated, this causes a memory issue since all the data structures that hold > >> all possible queues get initialized regardless of whether we actually need them or not, > >> which is quite troublesome for systems with small RAM. > >> > >> > >> Rusty, MST, would you be open to a spec and code change that would initialize the > >> RX/TX vqs on demand instead of on device initialization? Or is there an easier way > >> to work around this issue? > > > > I'm confused. kvmtool is using too much memory, or the guest? If > > kvmtool, the Device Initialization section above applies to the driver, > > not the device. If the guest, well, the language says "UP TO N+1". You > > want a small guest, don't use them all. Or any... > > > > What am I missing? > > It's in the guest - sorry. I was only trying to say that kvmtool doesn't do anything > odd with regards to initializing virtio-net. > > The thing is that there should be a difference between just allowing a larger number > of queues and actually using them (i.e. enabling them with ethtool). Right now I see > the kernel lose 130MB just by having kvmtool offer 8 queue pairs, without actually > using those queues. > > Yes, we can make it configurable in kvmtool (and I will make it so so the arm folks > could continue working with tiny guests) but does it make sense that you have to do > this configuration in *2* places? First in the hypervisor and then inside the guest? > > Memory usage should ideally depend on whether you are actually using multiple queues, > not on whether you just allow using those queues. > > > Thanks, > Sasha 8 queues eat up 130MB? Most of the memory is likely for the buffers? I think we could easily allocate these lazily as queues are enabled, without protocol changes. It's harder to clean them as there's no way to reset a specific queue, but maybe that' good enough for your purposes? -- MST -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
