I am beginning to suspect there is a problem with Scatter/Gather I/O.
If I disable this feature on the NIC (e1000), and use the version with
two calls to ip_append_data, the wmem_alloc limit is not exceeded
(though values are still higher than with a single call to
ip_append_data). Of course, I cannot use ip_append_page in that case,
since it depends on S/G.
Elad
Elad Lahav wrote:
I wrote a function that is equivalent to udp_sendmsg, but uses
ip_append_page to attach data to an skb. The function is implemented as
follows:
1. Allocate a page and copy the given data to that page
2. Set up routing and cork the socket
3. Call ip_append_data to create an initial skb (with data length set to 0)
4. Call ip_append_page with the allocated page
5. Call udp_push_pending_frames to send the packet
The function works correctly. Packets are generated and sent as
expected: this was verified by looking at the packet contents on the
receiving machine.
However, under load, there is a significant difference between
udp_sendmsg and my function. The problem is that the socket send buffer
(wmem_alloc) quickly grows beyond its upper limit (which is 131071 by
default). This results in numerous failures of ip_append_data with
EAGAIN, degrading performance considerably.
udp_sendmsg, on the other hand, keeps wmem_alloc in a much smaller range
under the same load.
Two notes:
1. Modifying the upper limit to 524287 solved the problem completely
(regardless of the load)
2. The same thing happens with multiple calls to ip_append_data (e.g.,
if I want to copy the data in two sections), so it is not a problem with
ip_append_page.
Any thoughts?
Elad
--
Kernelnewbies: Help each other learn about the Linux kernel.
Archive: http://mail.nl.linux.org/kernelnewbies/
FAQ: http://kernelnewbies.org/faq/
--
Kernelnewbies: Help each other learn about the Linux kernel.
Archive: http://mail.nl.linux.org/kernelnewbies/
FAQ: http://kernelnewbies.org/faq/
