On 01/07/2013 03:48 PM, Wanlong Gao wrote:
> On 01/07/2013 03:28 PM, Jason Wang wrote:
>> On 01/07/2013 03:15 PM, Wanlong Gao wrote:
>>> Add a cpu notifier to virtio-net, so that we can reset the
>>> virtqueue affinity if the cpu hotplug happens. It improve
>>> the performance through enabling or disabling the virtqueue
>>> affinity after doing cpu hotplug.
>>> Adding the notifier block to virtnet_info is suggested by
>>> Jason, thank you.
>>>
>>> Cc: Rusty Russell <rusty@xxxxxxxxxxxxxxx>
>>> Cc: "Michael S. Tsirkin" <mst@xxxxxxxxxx>
>>> Cc: Jason Wang <jasowang@xxxxxxxxxx>
>>> Cc: Eric Dumazet <erdnetdev@xxxxxxxxx>
>>> Cc: virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx
>>> Cc: netdev@xxxxxxxxxxxxxxx
>>> Signed-off-by: Wanlong Gao <gaowanlong@xxxxxxxxxxxxxx>
>>> ---
>>> drivers/net/virtio_net.c | 30 ++++++++++++++++++++++++++++++
>>> 1 file changed, 30 insertions(+)
>>>
>>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
>>> index b483fb5..9547b4c 100644
>>> --- a/drivers/net/virtio_net.c
>>> +++ b/drivers/net/virtio_net.c
>>> @@ -26,6 +26,7 @@
>>> #include <linux/scatterlist.h>
>>> #include <linux/if_vlan.h>
>>> #include <linux/slab.h>
>>> +#include <linux/cpu.h>
>>>
>>> static int napi_weight = 128;
>>> module_param(napi_weight, int, 0444);
>>> @@ -123,6 +124,9 @@ struct virtnet_info {
>>>
>>> /* Does the affinity hint is set for virtqueues? */
>>> bool affinity_hint_set;
>>> +
>>> + /* CPU hot plug notifier */
>>> + struct notifier_block nb;
>>> };
>>>
>>> struct skb_vnet_hdr {
>>> @@ -1051,6 +1055,23 @@ static void virtnet_set_affinity(struct virtnet_info *vi, bool set)
>>> }
>>> }
>>>
>>> +static int virtnet_cpu_callback(struct notifier_block *nfb,
>>> + unsigned long action, void *hcpu)
>>> +{
>>> + struct virtnet_info *vi = container_of(nfb, struct virtnet_info, nb);
>>> + switch(action) {
>>> + case CPU_ONLINE:
>>> + case CPU_ONLINE_FROZEN:
>>> + case CPU_DEAD:
>>> + case CPU_DEAD_FROZEN:
>>> + virtnet_set_affinity(vi, true);
>>> + break;
>>> + default:
>>> + break;
>>> + }
>>> + return NOTIFY_OK;
>>> +}
>>> +
>> I think you'd better fix the .ndo_select_queue() as well (as Michael
>> said in your V1) since it currently uses smp processor id which may not
>> work very well in this case also.
> The bug is we can't get the right txq if the CPU IDs are not consecutive,
> right? Do you have any good idea about fixing this?
>
> Thanks,
> Wanlong Gao
The point is make the virtqueue private to a specific cpu when the
number of queue pairs is equal to the number of cpus. So after you bind
the vq affinity to a specific cpu, you'd better use the reverse mapping
of this affinity to do .ndo_select_queue(). One possible idea, as
Michael suggested, is a per-cpu structure to record the preferable
virtqueue and do both .ndo_select_queue() and affinity hint setting
based on this.
>
>> Thanks
>>> static void virtnet_get_ringparam(struct net_device *dev,
>>> struct ethtool_ringparam *ring)
>>> {
>>> @@ -1509,6 +1530,13 @@ static int virtnet_probe(struct virtio_device *vdev)
>>> }
>>> }
>>>
>>> + vi->nb.notifier_call = &virtnet_cpu_callback;
>>> + err = register_hotcpu_notifier(&vi->nb);
>>> + if (err) {
>>> + pr_debug("virtio_net: registering cpu notifier failed\n");
>>> + goto free_recv_bufs;
>>> + }
>>> +
>>> /* Assume link up if device can't report link status,
>>> otherwise get link status from config. */
>>> if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_STATUS)) {
>>> @@ -1553,6 +1581,8 @@ static void virtnet_remove(struct virtio_device *vdev)
>>> {
>>> struct virtnet_info *vi = vdev->priv;
>>>
>>> + unregister_hotcpu_notifier(&vi->nb);
>>> +
>>> /* Prevent config work handler from accessing the device. */
>>> mutex_lock(&vi->config_lock);
>>> vi->config_enable = false;
>>
_______________________________________________
Virtualization mailing list
Virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
