On 01/07/2013 03:55 PM, Jason Wang wrote:
> On 01/07/2013 03:48 PM, Wanlong Gao wrote:
>> On 01/07/2013 03:28 PM, Jason Wang wrote:
>>> On 01/07/2013 03:15 PM, Wanlong Gao wrote:
>>>> Add a cpu notifier to virtio-net, so that we can reset the
>>>> virtqueue affinity if the cpu hotplug happens. It improve
>>>> the performance through enabling or disabling the virtqueue
>>>> affinity after doing cpu hotplug.
>>>> Adding the notifier block to virtnet_info is suggested by
>>>> Jason, thank you.
>>>>
>>>> Cc: Rusty Russell <rusty@xxxxxxxxxxxxxxx>
>>>> Cc: "Michael S. Tsirkin" <mst@xxxxxxxxxx>
>>>> Cc: Jason Wang <jasowang@xxxxxxxxxx>
>>>> Cc: Eric Dumazet <erdnetdev@xxxxxxxxx>
>>>> Cc: virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx
>>>> Cc: netdev@xxxxxxxxxxxxxxx
>>>> Signed-off-by: Wanlong Gao <gaowanlong@xxxxxxxxxxxxxx>
>>>> ---
>>>> drivers/net/virtio_net.c | 30 ++++++++++++++++++++++++++++++
>>>> 1 file changed, 30 insertions(+)
>>>>
>>>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
>>>> index b483fb5..9547b4c 100644
>>>> --- a/drivers/net/virtio_net.c
>>>> +++ b/drivers/net/virtio_net.c
>>>> @@ -26,6 +26,7 @@
>>>> #include <linux/scatterlist.h>
>>>> #include <linux/if_vlan.h>
>>>> #include <linux/slab.h>
>>>> +#include <linux/cpu.h>
>>>>
>>>> static int napi_weight = 128;
>>>> module_param(napi_weight, int, 0444);
>>>> @@ -123,6 +124,9 @@ struct virtnet_info {
>>>>
>>>> /* Does the affinity hint is set for virtqueues? */
>>>> bool affinity_hint_set;
>>>> +
>>>> + /* CPU hot plug notifier */
>>>> + struct notifier_block nb;
>>>> };
>>>>
>>>> struct skb_vnet_hdr {
>>>> @@ -1051,6 +1055,23 @@ static void virtnet_set_affinity(struct virtnet_info *vi, bool set)
>>>> }
>>>> }
>>>>
>>>> +static int virtnet_cpu_callback(struct notifier_block *nfb,
>>>> + unsigned long action, void *hcpu)
>>>> +{
>>>> + struct virtnet_info *vi = container_of(nfb, struct virtnet_info, nb);
>>>> + switch(action) {
>>>> + case CPU_ONLINE:
>>>> + case CPU_ONLINE_FROZEN:
>>>> + case CPU_DEAD:
>>>> + case CPU_DEAD_FROZEN:
>>>> + virtnet_set_affinity(vi, true);
>>>> + break;
>>>> + default:
>>>> + break;
>>>> + }
>>>> + return NOTIFY_OK;
>>>> +}
>>>> +
>>> I think you'd better fix the .ndo_select_queue() as well (as Michael
>>> said in your V1) since it currently uses smp processor id which may not
>>> work very well in this case also.
>> The bug is we can't get the right txq if the CPU IDs are not consecutive,
>> right? Do you have any good idea about fixing this?
>>
>> Thanks,
>> Wanlong Gao
>
> The point is make the virtqueue private to a specific cpu when the
> number of queue pairs is equal to the number of cpus. So after you bind
> the vq affinity to a specific cpu, you'd better use the reverse mapping
> of this affinity to do .ndo_select_queue(). One possible idea, as
> Michael suggested, is a per-cpu structure to record the preferable
> virtqueue and do both .ndo_select_queue() and affinity hint setting
> based on this.
Yeah, I think I got it now, will address it in V3. thank you. ;)
Regards,
Wanlong Gao
>>
>>> Thanks
>>>> static void virtnet_get_ringparam(struct net_device *dev,
>>>> struct ethtool_ringparam *ring)
>>>> {
>>>> @@ -1509,6 +1530,13 @@ static int virtnet_probe(struct virtio_device *vdev)
>>>> }
>>>> }
>>>>
>>>> + vi->nb.notifier_call = &virtnet_cpu_callback;
>>>> + err = register_hotcpu_notifier(&vi->nb);
>>>> + if (err) {
>>>> + pr_debug("virtio_net: registering cpu notifier failed\n");
>>>> + goto free_recv_bufs;
>>>> + }
>>>> +
>>>> /* Assume link up if device can't report link status,
>>>> otherwise get link status from config. */
>>>> if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_STATUS)) {
>>>> @@ -1553,6 +1581,8 @@ static void virtnet_remove(struct virtio_device *vdev)
>>>> {
>>>> struct virtnet_info *vi = vdev->priv;
>>>>
>>>> + unregister_hotcpu_notifier(&vi->nb);
>>>> +
>>>> /* Prevent config work handler from accessing the device. */
>>>> mutex_lock(&vi->config_lock);
>>>> vi->config_enable = false;
>>>
>
>
_______________________________________________
Virtualization mailing list
Virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
