On 01/09/2013 05:30 PM, Jason Wang wrote:
> On 01/09/2013 04:23 PM, Wanlong Gao wrote:
>> On 01/08/2013 06:14 PM, Jason Wang wrote:
>>> On 01/08/2013 06:00 PM, Wanlong Gao wrote:
>>>> On 01/08/2013 05:51 PM, Jason Wang wrote:
>>>>> On 01/08/2013 05:49 PM, Wanlong Gao wrote:
>>>>>> On 01/08/2013 05:29 PM, Jason Wang wrote:
>>>>>>> On 01/08/2013 05:07 PM, Wanlong Gao wrote:
>>>>>>>> On 12/28/2012 06:32 PM, Jason Wang wrote:
>>>>>>>>> + } else if (nc->peer->info->type != NET_CLIENT_OPTIONS_KIND_TAP) {
>>>>>>>>> + ret = -1;
>>>>>>>>> + } else {
>>>>>>>>> + ret = tap_detach(nc->peer);
>>>>>>>>> + }
>>>>>>>>> +
>>>>>>>>> + return ret;
>>>>>>>>> +}
>>>>>>>>> +
>>>>>>>>> +static void virtio_net_set_queues(VirtIONet *n)
>>>>>>>>> +{
>>>>>>>>> + int i;
>>>>>>>>> +
>>>>>>>>> + for (i = 0; i < n->max_queues; i++) {
>>>>>>>>> + if (i < n->curr_queues) {
>>>>>>>>> + assert(!peer_attach(n, i));
>>>>>>>>> + } else {
>>>>>>>>> + assert(!peer_detach(n, i));
>>>>>>>> I got a assert here,
>>>>>>>> qemu-system-x86_64: /work/git/qemu/hw/virtio-net.c:330: virtio_net_set_queues: Assertion `!peer_detach(n, i)' failed.
>>>>>>>>
>>>>>>>> Any thoughts?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Wanlong Gao
>>>>>>> Thanks for the testing, which steps or cases did you met this assertion,
>>>>>>> migration, reboot or just changing the number of virtqueues?
>>>>>> I use the 3.8-rc2 to test it again, I saw this tag has the multi-tap support.
>>>>>>
>>>>>> I just can't start the QEMU use -netdev tap,id=hostnet0,queues=2,fd=%d,fd=%d -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:ce:7b:29,bus=pci.0,addr=0x3
>>>>>>
>>>>>> I pre-opened two tap fds, did I missing something?
>>>>> Nothing missed :) It should work.
>>>>>
>>>>> Could you please try not use fd=X and let qemu to create the file
>>>>> descriptors by itself? Btw, how did you create the two tap fds?
>>>> Can it create descriptors itself? I get
>>>> qemu-system-x86_64: -netdev tap,id=hostnet0,queues=2: Device 'tap' could not be initialized
>>> You need prepare an ifup script which default at /etc/qemu-ifup (like
>>> following). Or you may try to add a script=no after:
>>>
>>> #!/bin/sh
>>>
>>> switch=kvmbr0
>>>
>>> /sbin/ifconfig $1 0.0.0.0 up
>>> /usr/sbin/brctl addif $switch $1
>>> /usr/sbin/brctl stp $switch off
>>>
>>> This will let qemu create a tap fd itself and make it to be connected to
>>> a port of the bridge caled kvmbr0.
>> But how to support multi-queue in this way?
>
> Qemu will create the necessary multiqueue tap by itself, see patch 0/12.
>> I got guest kernel panic when using this way and set queues=4.
>
> Does it happens w/o or w/ a fd parameter? What's the qemu command line?
> Did you meet it during boot time?
The QEMU command line is
/work/git/qemu/x86_64-softmmu/qemu-system-x86_64 -name f17 -M pc-0.15 -enable-kvm -m 3096 \
-smp 4,sockets=4,cores=1,threads=1 \
-uuid c31a9f3e-4161-c53a-339c-5dc36d0497cb -no-user-config -nodefaults \
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/f17.monitor,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc -no-shutdown \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0xb,num_queues=4,hotplug=on \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 \
-drive file=/vm/f17.img,if=none,id=drive-virtio-disk0,format=qcow2 \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-drive file=/vm2/f17-kernel.img,if=none,id=drive-virtio-disk1,format=qcow2 \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk1,id=virtio-disk1 \
-drive file=/vm/virtio-scsi/scsi3.img,if=none,id=drive-scsi0-0-2-0,format=raw \
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi0-0-2-0,id=scsi0-0-2-0,removable=on \
-drive file=/vm/virtio-scsi/scsi4.img,if=none,id=drive-scsi0-0-3-0,format=raw \
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=3,drive=drive-scsi0-0-3-0,id=scsi0-0-3-0 \
-drive file=/vm/virtio-scsi/scsi1.img,if=none,id=drive-scsi0-0-0-0,format=raw \
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 \
-drive file=/vm/virtio-scsi/scsi2.img,if=none,id=drive-scsi0-0-1-0,format=raw \
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0 \
-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 \
-chardev file,id=charserial1,path=/vm/f17.log \
-device isa-serial,chardev=charserial1,id=serial1 \
-device usb-tablet,id=input0 -vga std \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 \
-netdev tap,id=hostnet0,vhost=on,queues=4 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:ce:7b:29,bus=pci.0,addr=0x3 \
-monitor stdio
I got panic just after booting the system, did nothing, waited for a while, the guest panicked.
[ 28.053004] BUG: soft lockup - CPU#1 stuck for 23s! [ip:592]
[ 28.053004] Modules linked in: ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables uinput joydev microcode virtio_balloon pcspkr virtio_net i2c_piix4 i2c_core virtio_scsi virtio_blk floppy
[ 28.053004] CPU 1
[ 28.053004] Pid: 592, comm: ip Not tainted 3.8.0-rc1-net+ #3 Bochs Bochs
[ 28.053004] RIP: 0010:[<ffffffff8137a9ab>] [<ffffffff8137a9ab>] virtqueue_get_buf+0xb/0x120
[ 28.053004] RSP: 0018:ffff8800bc913550 EFLAGS: 00000246
[ 28.053004] RAX: 0000000000000000 RBX: ffff8800bc49c000 RCX: ffff8800bc49e000
[ 28.053004] RDX: 0000000000000000 RSI: ffff8800bc913584 RDI: ffff8800bcfd4000
[ 28.053004] RBP: ffff8800bc913558 R08: ffff8800bcfd0800 R09: 0000000000000000
[ 28.053004] R10: ffff8800bc49c000 R11: ffff880036cc4de0 R12: ffff8800bcfd4000
[ 28.053004] R13: ffff8800bc913558 R14: ffffffff8137ad73 R15: 00000000000200d0
[ 28.053004] FS: 00007fb27a589740(0000) GS:ffff8800c1480000(0000) knlGS:0000000000000000
[ 28.053004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 28.053004] CR2: 0000000000640530 CR3: 00000000baeff000 CR4: 00000000000006e0
[ 28.053004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 28.053004] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 28.053004] Process ip (pid: 592, threadinfo ffff8800bc912000, task ffff880036da2e20)
[ 28.053004] Stack:
[ 28.053004] ffff8800bcfd0800 ffff8800bc913638 ffffffffa003e9bb ffff8800bc913656
[ 28.053004] 0000000100000002 ffff8800c17ebb08 000000500000ff10 ffffea0002f244c0
[ 28.053004] 0000000200000582 0000000000000000 0000000000000000 ffffea0002f244c0
[ 28.053004] Call Trace:
[ 28.053004] [<ffffffffa003e9bb>] virtnet_send_command.constprop.26+0x24b/0x270 [virtio_net]
[ 28.053004] [<ffffffff812ed963>] ? sg_init_table+0x23/0x50
[ 28.053004] [<ffffffffa0040629>] virtnet_set_rx_mode+0x99/0x300 [virtio_net]
[ 28.053004] [<ffffffff8152306f>] __dev_set_rx_mode+0x5f/0xb0
[ 28.053004] [<ffffffff815230ef>] dev_set_rx_mode+0x2f/0x50
[ 28.053004] [<ffffffff815231b7>] __dev_open+0xa7/0xf0
[ 28.053004] [<ffffffff81523461>] __dev_change_flags+0xa1/0x180
[ 28.053004] [<ffffffff815235f8>] dev_change_flags+0x28/0x70
[ 28.053004] [<ffffffff8152ff20>] do_setlink+0x3b0/0xa50
[ 28.053004] [<ffffffff812fb6b1>] ? nla_parse+0x31/0xe0
[ 28.053004] [<ffffffff815325de>] rtnl_newlink+0x36e/0x580
[ 28.053004] [<ffffffff811355cc>] ? get_page_from_freelist+0x37c/0x730
[ 28.053004] [<ffffffff81531e13>] rtnetlink_rcv_msg+0x113/0x2f0
[ 28.053004] [<ffffffff8117d973>] ? __kmalloc_node_track_caller+0x63/0x1c0
[ 28.053004] [<ffffffff8151526b>] ? __alloc_skb+0x8b/0x2a0
[ 28.053004] [<ffffffff81531d00>] ? __rtnl_unlock+0x20/0x20
[ 28.053004] [<ffffffff8154b571>] netlink_rcv_skb+0xb1/0xc0
[ 28.053004] [<ffffffff8152ea05>] rtnetlink_rcv+0x25/0x40
[ 28.053004] [<ffffffff8154ae91>] netlink_unicast+0x1a1/0x220
[ 28.053004] [<ffffffff8154b211>] netlink_sendmsg+0x301/0x3c0
[ 28.053004] [<ffffffff81508530>] sock_sendmsg+0xb0/0xe0
[ 28.053004] [<ffffffff8113a45b>] ? lru_cache_add_lru+0x3b/0x60
[ 28.053004] [<ffffffff811608b7>] ? page_add_new_anon_rmap+0xc7/0x180
[ 28.053004] [<ffffffff81509efc>] __sys_sendmsg+0x3ac/0x3c0
[ 28.053004] [<ffffffff8162e47c>] ? __do_page_fault+0x23c/0x4d0
[ 28.053004] [<ffffffff8115c9ef>] ? do_brk+0x1ff/0x370
[ 28.053004] [<ffffffff8150bec9>] sys_sendmsg+0x49/0x90
[ 28.053004] [<ffffffff81632d59>] system_call_fastpath+0x16/0x1b
[ 28.053004] Code: 04 0f ae f0 48 8b 47 50 5d 0f b7 50 02 66 39 57 64 0f 94 c0 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 <53> 80 7f 59 00 48 89 fb 0f 85 90 00 00 00 48 8b 47 50 0f b7 50
The QEMU tree I used is git://github.com/jasowang/qemu.git
Thanks,
Wanlong Gao
>
> Thanks
>>
>> Thanks,
>> Wanlong Gao
>>
>>>> I create the tap fd like this, and dup create the second fd, third fd, right?
>>> The second and third fd should be created with TUNSETIFF with the same
>>> tap_name also. Btw, you need to specify a IFF_MULTI_QUEUE flag to tell
>>> the kernel you want to create a multiqueue tap device, otherwise the
>>> second and third calling of TUNSETIFF will fail.
>>>
>>> Thanks
>>>> int tap_fd = open("/dev/net/tun", O_RDWR);
>>>> int vhost_fd = open("/dev/vhost-net", O_RDWR);
>>>> char *tap_name = "tap";
>>>> char cmd[2048];
>>>> char brctl[256];
>>>> char netup[256];
>>>> struct ifreq ifr;
>>>> if (tap_fd < 0) {
>>>> printf("open tun device failed\n");
>>>> return -1;
>>>> }
>>>> if (vhost_fd < 0) {
>>>> printf("open vhost-net device failed\n");
>>>> return -1;
>>>> }
>>>> memset(&ifr, 0, sizeof(ifr));
>>>> memcpy(ifr.ifr_name, tap_name, sizeof(tap_name));
>>>> ifr.ifr_flags = IFF_TAP | IFF_NO_PI;
>>>>
>>>> /*
>>>> * setup tap net device
>>>> */
>>>> if (ioctl(tap_fd, TUNSETIFF, &ifr) < 0) {
>>>> printf("setup tap net device failed\n");
>>>> return -1;
>>>> }
>>>>
>>>> sprintf(brctl, "brctl addif virbr0 %s", tap_name);
>>>> sprintf(netup, "ifconfig %s up", tap_name);
>>>> system(brctl);
>>>> system(netup);
>>>>
>>>> Thanks,
>>>> Wanlong Gao
>>>>
>>>>
>>>>> Thanks
>>>>>> Thanks,
>>>>>> Wanlong Gao
>>>>>>
>>>>>>>>> + }
>>>>>>>>> + }
>>>>>>>>> +}
>>>>>>>>> +
>>>>>>>>> +static void virtio_net_set_multiqueue(VirtIONet *n, int multiqueue, int ctrl);
>>>>>>>>> +
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
