On 10/14/2023 4:05 AM, Maciej Żenczykowski wrote:
The intent of posting the diff was two fold:
1. The question Greg asked regarding why the max segment size was
limited to 15014 was valid. When I thought about it, I actually wanted
to limit the max MTU to 15000, so the max segment size automatically
needs to be limited to 15014.
Note that this is a *very* abstract value.
I get you want L3 MTU of 10 * 1500, but this value is not actually meaningful.
IPv4/IPv6 fragmentation and IPv4/IPv6 TCP segmentation
do not result in a trivial multiplication of the standard 1500 byte
ethernet L3 MTU.
Indeed aggregating 2 1500 L3 mtu frames results in *different* sized
frames depending on which type of aggregation you do.
(and for tcp it even depends on the number and size of tcp options,
though it is often assumed that those take up 12 bytes, since that's the
normal for Linux-to-Linux tcp connections)
For example if you aggregate N standard Linux ipv6/tcp L3 1500 mtu frames,
this means you have
N frames: ethernet (14) + ipv6 (40) + tcp (20) + tcp options (12) +
payload (1500-12-20-40=1500-72=1428)
post aggregation:
1 frame: ethernet (14) + ipv6 (40) + tcp (20) + tcp options (12) +
payload (N*1428)
so N * 1500 == N * (72 + 1428) --> 1 * (72 + N * 1428)
That value of 72 is instead 52 for 'standard Linux ipv4/tcp),
it's 40/60 if there's no tcp options (which I think happens when
talking to windows)
it's different still with ipv4 fragmentation... and again different
with ipv6 fragmentation...
etc.
ie. 15000 L3 mtu is exactly as meaningless as 14000 L3 mtu.
Either way you don't get full frames.
As such I'd recommend going with whatever is the largest mtu that can
be meaningfully made to fit in 16K with all the NCM header overhead.
That's likely closer to 15500-16000 (though I have *not* checked).
But my commit text didn't mention this
properly which was a mistake on my behalf. But when I looked at the
code, limiting the max segment size 15014 would force the practical
max_mtu to not cross 15000 although theoretical max_mtu was set to:
(GETHER_MAX_MTU_SIZE - 15412) during registration of net device.
So my assumption of limiting it to 15000 was wrong. It must be limited
to 15412 as mentioned in u_ether.c This inturn means we must limit
max_segment_size to:
GETHER_MAX_ETH_FRAME_LEN (GETHER_MAX_MTU_SIZE + ETH_HLEN)
as mentioned in u_ether.c.
I wanted to confirm that setting MAX_DATAGRAM_SIZE to
GETHER_MAX_ETH_FRAME_LEN was correct.
2. I am not actually able to test with MTU beyond 15000. When my host
device is a linux machine, the cdc_ncm.c limits max_segment_size to:
CDC_NCM_MAX_DATAGRAM_SIZE 8192 /* bytes */
In practice you get 50% of the benefits of infinitely large mtu by
going from 1500 to ~2980.
you get 75% of the benefits by going to ~6K
you get 87.5% of the benefits by going to ~12K
the benefits of going even higher are smaller and smaller...
> If the host side is limited to 8192, maybe we should match that here too?
Hi Maciej,
Thanks for the detailed explanation. I agree with you on setting
device side also to 8192 instead of what max_mtu is present in u_ether
or practical max segment size possible.
But the host side limitation of 8192 doesn't seem particularly sane either...
Maybe we should relax that instead?
I really didn't understand why it was set to 8192 in first place.
(especially since for things like tcp zero copy you want an mtu which
is slighly more then N * 4096,
ie. around 4.5KB, 8.5KB, 12.5KB or something like that)
I am not sure about host mode completely. If we want to increase though,
just increasing the MAX_DATAGRAM_SIZE to some bigger value help ? (I
don't know the entire code of cdc_ncm, so I might be wrong).
Regards,
Krishna,
Hmm, I'm not sure. I know I've experimented with high mtu ncm in the past
(around 2.5 years ago). I got it working between my Linux desktop (host)
and a Pixel 6 (device/gadget) with absolutely no problems.
I'm pretty sure I didn't change my desktop kernel, so I was probably
limited to 8192 there
(and I do more or less remember that).
From what I vaguely remember, it wasn't difficult (at all) to hit
upwards of 7gbps for iperf tests.
I don't remember how close to the theoretical USB 10gbps maximum of
9.7gbps I could get...
[this was never the real bottleneck / issue, so I didn't ever dig
particularly deep]
I'm pretty sure my gadget side changes were non-configurable...
Probably just bumped one or two constants...
Could you share what parameters you changed to get this high value of
iperf throughput.
I do *very* *vaguely* recall there being some funkiness though, where 8192 was
*less* efficient than some slightly smaller value.
If I recall correctly the issue is that 8192 + ethernet overhead + NCM
overhead only fits *once* into 16384, which leaves a lot of space
wasted.
While ~7.5 kb + overhead fits twice and is thus a fair bit better.
Right, same goes for using 5K vs 5.5K MTU. If MTU is 5K, 3 packets can
conveniently fit into an NTB but if its 5.5, at max only two (5.5k)
packets can fit in (essentially filling ~11k of the 16384 bytes and
wasting the rest)
And whether its Ipv4/Ipv6 like you mentioned on [1], the MTU is what NCM
layer receives and we append the Ethernet header and add NCM headers and
send it out after aggregation. Why can't we set the MAX_DATAGRAM_SIZE to
~8050 or ~8100 ? The reason I say this value is, obviously setting it to
8192 would not efficiently use the NTB buffer. We need to fill as much
space in buffer as possible and assuming that each packet received on
ncm layer is of MTU size set (not less that that), we can assume that
even if only 2 packets are aggregated (minimum aggregation possible), we
would be filling (2 * (8050 + ETH_HLEN) + (room for NCM headers)) would
almost be close to 16384 ep max packet size. I already check 8050 MTU
and it works. We can add a comment in code detailing the above
explanation and why we chose to use 8050 or 8100 as MAX_DATAGRAM_SIZE.
Hope my reasoning of why we can chose 8.1K or 8.05K makes sense. Let me
know your thoughts on this.
Regards,
Krishna,
