Willem de Bruijn wrote:
> Richard Gobert wrote:
>> Willem de Bruijn wrote:
>>> Richard Gobert wrote:
>>>> {inet,ipv6}_gro_receive functions perform flush checks (ttl, flags,
>>>> iph->id, ...) against all packets in a loop. These flush checks are used
>>>> currently only in tcp flows in GRO.
>>>>
>>>> These checks need to be done only once in tcp_gro_receive and only against
>>>> the found p skb, since they only affect flush and not same_flow.
>>>
>>> I don't quite understand where the performance improvements arise.
>>> As inet_gro_receive will skip any p that does not match:
>>>
>>> if (!NAPI_GRO_CB(p)->same_flow)
>>> continue;
>>>
>>> iph2 = (struct iphdr *)(p->data + off);
>>> /* The above works because, with the exception of the top
>>> * (inner most) layer, we only aggregate pkts with the same
>>> * hdr length so all the hdrs we'll need to verify will start
>>> * at the same offset.
>>> */
>>> if ((iph->protocol ^ iph2->protocol) |
>>> ((__force u32)iph->saddr ^ (__force u32)iph2->saddr) |
>>> ((__force u32)iph->daddr ^ (__force u32)iph2->daddr)) {
>>> NAPI_GRO_CB(p)->same_flow = 0;
>>> continue;
>>> }
>>>
>>> So these checks are already only performed against a p that matches.
>>>
>>
>>
>> Thanks for the review!
>>
>> flush/flush_id is calculated for all other p with same_flow = 1 (which is
>> not always determined to be 0 before inet_gro_receive) and same src/dst
>> addr in the bucket. Moving it to udp_gro_receive_segment/tcp_gro_receive
>> will make it run only once when a matching p is found.
>
> So this optimization is for flows that are the same up to having the
> same saddr/daddr. Aside from stress tests, it seems rare to have many
> concurrent flows between the same pair of machines?
>
Yes exactly, sorry if I wasn't clear enough earlier. Multiple connections
with the same srcaddr+dstaddr are not necessarily rare (e.g. devices behind
a large NAT network all connecting to the same servers, thus sharing the
same IP srcaddr).
>>
>> In addition, UDP flows where skb_gro_receive_list is called -
>> flush/flush_id is not relevant and does not need to be calculated.
>
> That makes sense
>
I ran a UDP forwarding benchmark similar to how I did in the commit
message. GRO's flush/flush_id values were not relevant as all packets
reached skb_gro_receive_list instead of skb_gro_receive.
These numbers show CPU utilization under the same load (64 concurrent
IP/UDP connections):
net-next:
3.03% [kernel] [k] inet_gro_receive
patch applied:
2.78% [kernel] [k] inet_gro_receive
And under encapsulated load, 64 concurrent IP/IP/UDP connections:
net-next:
10.50% [kernel] [k] inet_gro_receive
patch applied:
8.19% [kernel] [k] inet_gro_receive
Total time spent in GRO reduced significantly due to fewer opcodes and
branches in inet_gro_receive :)
>> In these
>> cases total CPU time in GRO should drop. I could post perf numbers for
>> this flow as well.
>>
>>
>>>> Leveraging the previous commit in the series, in which correct network
>>>> header offsets are saved for both outer and inner network headers -
>>>> allowing these checks to be done only once, in tcp_gro_receive. As a
>>>
>>> Comments should be updated to reflect both TCP and L4 UDP. Can
>>> generalize to transport callbacks.
>>>
>>>> result, NAPI_GRO_CB(p)->flush is not used at all. In addition, flush_id
>>>> checks are more declarative and contained in inet_gro_flush, thus removing
>>>> the need for flush_id in napi_gro_cb.
>>>>
>>>> This results in less parsing code for UDP flows and non-loop flush tests
>>>> for TCP flows.
>>>
>>> This moves network layer tests out of the network layer callbacks into
>>> helpers called from the transport layer callback. And then the helper
>>> has to look up the network layer header and demultiplex the protocol
>>> again:
>>>
>>> + if (((struct iphdr *)nh)->version == 6)
>>> + flush |= ipv6_gro_flush(nh, nh2);
>>> + else
>>> + flush |= inet_gro_flush(nh, nh2, p, i != encap_mark);
>>>
>>> That just seems a bit roundabout.
>>
>> IMO this commit could be a part of a larger change, where all
>> loops in gro_list_prepare, inet_gro_receive and ipv6_gro_receive can be
>> removed, and the logic for finding a matching p will be moved to L4. This
>> means that when p is found, the rest of the gro_list would not need to be
>> traversed and thus would not even dirty cache lines at all. I can provide a
>> code snippet which would explain it better.
>
> These loops are exactly the mechanism to find a matching p. Though
> with all the callbacks perhaps not the most efficient model. The
> hashtable should have solved much of that.
>
> Yes, please share a snippet to understand how you would replace this.
>
This is still a rough idea, and I still think the current patch is
significant by itself due to its performance gains and more readable code.
The idea is that an skb from gro_list will be loaded to cache only when GRO
is trying to find out if it matches the current skb, avoiding the current
multiple list traversals design in GRO for IP-TCP.
For a simple IP-TCP packet, gro_list_prapare will not be called from
dev_gro_receive anymore:
@@ -450,8 +489,6 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff
if (netif_elide_gro(skb->dev))
goto normal;
- gro_list_prepare(&gro_list->list, skb);
The matching loop from inet_gro_receive is removed as well:
@@ -1507,27 +1506,6 @@ struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb)
NAPI_GRO_CB(skb)->proto = proto;
flush = (u16)((ntohl(*(__be32 *)iph) ^ skb_gro_len(skb)) | (ntohl(*(__be32 *)&iph->id) & ~IP_DF));
-
- list_for_each_entry(p, head, list) {
- struct iphdr *iph2;
-
- if (!NAPI_GRO_CB(p)->same_flow)
- continue;
-
- iph2 = (struct iphdr *)(p->data + off);
- /* The above works because, with the exception of the top
- * (inner most) layer, we only aggregate pkts with the same
- * hdr length so all the hdrs we'll need to verify will start
- * at the same offset.
- */
- if ((iph->protocol ^ iph2->protocol) |
- ((__force u32)iph->saddr ^ (__force u32)iph2->saddr) |
- ((__force u32)iph->daddr ^ (__force u32)iph2->daddr)) {
- NAPI_GRO_CB(p)->same_flow = 0;
- continue;
- }
- }
-
and change tcp_gro_receive such that p is retrieved from gro_list using a new function:
--- a/net/ipv4/tcp_offload.c
+++ b/net/ipv4/tcp_offload.c
@@ -215,24 +215,12 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb)
len = skb_gro_len(skb);
flags = tcp_flag_word(th);
- list_for_each_entry(p, head, list) {
- if (!NAPI_GRO_CB(p)->same_flow)
- continue;
-
- th2 = tcp_hdr(p);
-
- if (*(u32 *)&th->source ^ *(u32 *)&th2->source) {
- NAPI_GRO_CB(p)->same_flow = 0;
- continue;
- }
+ if (!(p = gro_list_find(head, skb, &flush)))
+ goto out_check_final;
- goto found;
- }
- p = NULL;
- goto out_check_final;
+ th2 = tcp_hdr(p);
-found:
- flush = (__force int)(flags & TCP_FLAG_CWR);
+ flush |= (__force int)(flags & TCP_FLAG_CWR);
flush |= (__force int)((flags ^ tcp_flag_word(th2)) &
~(TCP_FLAG_CWR | TCP_FLAG_FIN | TCP_FLAG_PSH));
flush |= (__force int)(th->ack_seq ^ th2->ack_seq);
The first time gro_list is traversed is in tcp_gro_receive. gro_list_find
is very similar to gro_list_prepare - but it also matches L3 & L4 (UDP and
TCP port offsets are identical), and it returns L3 flush value as well.
The main difference between the current design and what I propose is that
once a matched p is found - there's no need to keep traversing the rest of
gro_list (as opposed to gro_list_prepare and inet_gro_receive for example),
and that's why certain checks are saved especially under load. This way
less data is loaded to cache overall for every skb entering dev_gro_receive.
> In the meantime, I do suggest sending the first two patches to net,
> as they have Fixes tags. And then follow up with this for net-next
> separately.
Just submitted the first two commits to net.
Thanks.
