On 01/08/2013 06:26 PM, Jason Wang wrote:
> On 01/08/2013 06:07 PM, Wanlong Gao wrote:
>> As Michael mentioned, set affinity and select queue will not work very
>> well when CPU IDs are not consecutive, this can happen with hot unplug.
>> Fix this bug by traversal the online CPUs, and create a per cpu variable
>> to find the mapping from CPU to the preferable virtual-queue.
>>
>> Cc: Rusty Russell <rusty@xxxxxxxxxxxxxxx>
>> Cc: "Michael S. Tsirkin" <mst@xxxxxxxxxx>
>> Cc: Jason Wang <jasowang@xxxxxxxxxx>
>> Cc: Eric Dumazet <erdnetdev@xxxxxxxxx>
>> Cc: virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx
>> Cc: netdev@xxxxxxxxxxxxxxx
>> Signed-off-by: Wanlong Gao <gaowanlong@xxxxxxxxxxxxxx>
>> ---
>> drivers/net/virtio_net.c | 39 +++++++++++++++++++++++++++++----------
>> 1 file changed, 29 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
>> index a6fcf15..a77f86c 100644
>> --- a/drivers/net/virtio_net.c
>> +++ b/drivers/net/virtio_net.c
>> @@ -41,6 +41,8 @@ module_param(gso, bool, 0444);
>> #define VIRTNET_SEND_COMMAND_SG_MAX 2
>> #define VIRTNET_DRIVER_VERSION "1.0.0"
>>
>> +DEFINE_PER_CPU(int, vq_index) = -1;
>> +
>
> I think this should not be a global one, consider we may have more than
> one virtio-net cards with different max queues.
Yes, would you move this into virtio_info?
>> struct virtnet_stats {
>> struct u64_stats_sync tx_syncp;
>> struct u64_stats_sync rx_syncp;
>> @@ -1016,6 +1018,7 @@ static int virtnet_vlan_rx_kill_vid(struct net_device *dev, u16 vid)
>> static void virtnet_set_affinity(struct virtnet_info *vi, bool set)
>> {
>> int i;
>> + int cpu;
>>
>> /* In multiqueue mode, when the number of cpu is equal to the number of
>> * queue pairs, we let the queue pairs to be private to one cpu by
>> @@ -1029,16 +1032,29 @@ static void virtnet_set_affinity(struct virtnet_info *vi, bool set)
>> return;
>> }
>>
>> - for (i = 0; i < vi->max_queue_pairs; i++) {
>> - int cpu = set ? i : -1;
>> - virtqueue_set_affinity(vi->rq[i].vq, cpu);
>> - virtqueue_set_affinity(vi->sq[i].vq, cpu);
>> - }
>> + if (set) {
>> + i = 0;
>> + for_each_online_cpu(cpu) {
>> + virtqueue_set_affinity(vi->rq[i].vq, cpu);
>> + virtqueue_set_affinity(vi->sq[i].vq, cpu);
>> + per_cpu(vq_index, cpu) = i;
>> + i++;
>> + if (i >= vi->max_queue_pairs)
>> + break;
>
> Can this happen? we check only set when the number are equal.
will remove.
>> + }
>>
>> - if (set)
>> vi->affinity_hint_set = true;
>> - else
>> + } else {
>> + for(i = 0; i < vi->max_queue_pairs; i++) {
>> + virtqueue_set_affinity(vi->rq[i].vq, -1);
>> + virtqueue_set_affinity(vi->sq[i].vq, -1);
>> + }
>> +
>> + for_each_online_cpu(cpu)
>> + per_cpu(vq_index, cpu) = -1;
>> +
>
> This looks suboptimal since it may leads only txq zero is used.
So, which value is best for txq when we don't set affinity?
just remain to smp_processor_id()?
Thanks,
Wanlong Gao
>> vi->affinity_hint_set = false;
>> + }
>> }
>>
>> static void virtnet_get_ringparam(struct net_device *dev,
>> @@ -1127,12 +1143,15 @@ static int virtnet_change_mtu(struct net_device *dev, int new_mtu)
>>
>> /* To avoid contending a lock hold by a vcpu who would exit to host, select the
>> * txq based on the processor id.
>> - * TODO: handle cpu hotplug.
>> */
>> static u16 virtnet_select_queue(struct net_device *dev, struct sk_buff *skb)
>> {
>> - int txq = skb_rx_queue_recorded(skb) ? skb_get_rx_queue(skb) :
>> - smp_processor_id();
>> + int txq = 0;
>> +
>> + if (skb_rx_queue_recorded(skb))
>> + txq = skb_get_rx_queue(skb);
>> + else if ((txq = per_cpu(vq_index, smp_processor_id())) == -1)
>> + txq = 0;
>>
>> while (unlikely(txq >= dev->real_num_tx_queues))
>> txq -= dev->real_num_tx_queues;
>
>
_______________________________________________
Virtualization mailing list
Virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
