I would like to hear comments about this documentation I made.
Dave: maybe this is useful to be included in kernel documentation or at
least should be mentioned in CONFIG_PACKET_MMAP help?
Ulisses
Debian GNU/Linux: a dream come true
-----------------------------------------------------------------------------
"Computers are useless. They can only give answers." Pablo Picasso
Humans are slow, innaccurate, and brilliant.
Computers are fast, acurrate, and dumb.
Together they are unbeatable
---> Visita http://www.valux.org/ para saber acerca de la <---
---> Asociación Valenciana de Usuarios de Linux <---
--------------------------------------------------------------------------------
+ ABSTRACT
--------------------------------------------------------------------------------
This file documents the following function
static int packet_set_ring(struct sock *sk, struct tpacket_req *req, int closing);
This is the low level function used for setting up the PACKET_MMAP feature
of the PACKET socket interface. This type of sockets are used for packet capture
in Linux 2.4/2.6. The setup of PACKET_MMAP is done by user level code
with a call to the conventional setsockopt function.
Maybe this file is too much descriptive, please send me your comments to
Ulises Alonso Camaró <uaca@i.hate.spam.alumni.uv.es>
--------------------------------------------------------------------------------
+ PACKET_MMAP settings
--------------------------------------------------------------------------------
To setup PACKET_MMAP from user level code is done with a call like
setsockopt(fd, SOL_PACKET, PACKET_RX_RING, (void *) &req, sizeof(req))
The most significative in the previous call is the req parameter, this parameter has the
following structure:
struct tpacket_req
{
unsigned int tp_block_size; /* Minimal size of contiguous block */
unsigned int tp_block_nr; /* Number of blocks */
unsigned int tp_frame_size; /* Size of frame */
unsigned int tp_frame_nr; /* Total number of frames */
};
This structure is defined in include/linux/if_packet.h and establishes a circular buffer (ring) of
unswapable memory mapped in the capture process. Being mapped in the capture process allows reading
the captured frames and related meta-information like timestamps without requiring a system call.
Capured frames are grouped in blocks. Each block is a phisically contiguous region of memory
and holds tp_block_size/tp_frame_size frames. The total number of blocks is tp_block_nr. Note that
tp_frame_nr is a redundant parameter because
frames_per_block = tp_block_size/tp_frame_size
indeed, packet_set_ring checks that the following condition is true
frames_per_block * tp_block_nr == tp_frame_nr
Lets see an example, with the following values:
tp_block_size= 4096
tp_frame_size= 2048
tp_block_nr = 4
tp_frame_nr = 8
we will get the following buffer structure:
block #1 block #2 block #3 block #4
--------------------- --------------------- --------------------- ---------------------
| frame 1 | frame 2 | | frame 3 | frame 4 | | frame 5 | frame 6 | | frame 7 | frame 8 |
--------------------- --------------------- --------------------- ---------------------
--------------------------------------------------------------------------------
+ PACKET_MMAP setting constraints
--------------------------------------------------------------------------------
Block size limit
------------------
As stated earlier, each block is a contiguous phisical region of memory. These memory regions
are allocated with calls to the __get_free_pages() function. As the name indicates, this
function allocates pages of memory, it allocates a power of two number of pages, that is
4096, 8192, 16384, etc. The maximum size of a region allocated by __get_free_pages is determined
by the MAX_ORDER macro. More precisely the limit can be calculated as:
PAGE_SIZE << MAX_ORDER
In a i386 architecture PAGE_SIZE is 4096 bytes
In a 2.4/i386 kernel MAX_ORDER is 10
In a 2.6/i386 kernel MAX_ORDER is 11
So get_free_pages can allocate as much as 4MB or 8MB in a 2.4/2.6 kernel respectively, with a
i386 architecture.
Block number limit
--------------------
To understand the constraints of PACKET_MMAP settings we have to see two aditional
data structures used to support the ring. One of this structures limits the
number of blocks as we will see next, the other structure limits the total
number of frames.
There is a vector that mantains a pointer to each block, this vector is called pg_vec
wich stands for page vector. The following figure represents the pg_vec that is used
with the buffer shown before.
-----------------
| x | x | x | x |
-----------------
| | | |
| | | v
| | v block #4
| v block #3
v block #2
block #1
The number of blocks that can be allocated is determined by the size of pg_vec. This vector
is allocated with a call to the kmalloc function.
kmalloc allocates any number of bytes of phisically contiguous memory
from a pool of pre-determined sizes. This pool of memory is mantained
by the slab allocator wich is at the end the responsible of doing
the allocation and hence wich imposes the maximum memory
that kmalloc can allocate.
In a 2.4/2.6 kernel and the i386 architecture, the limit is 131072 bytes. This
limit can be checked in the "size-<bytes>" entries of /proc/slabinfo
In a i386 pointers are 4 bytes long, so the total number of pointers to blocks
(and hence blocks) is
131072/4 = 32768 blocks
Total Frame number limit
--------------------------
There is another vector of pointers, wich hold references to each frame
in the buffer, this vector is called io_vec. This vector is also allocated with kmalloc,
so we the maximum number of frames is the same as for the block number. Indeed, the limit to the
size of the buffer is impossed by the io_vec vector because we have at least the same number
of frames than blocks.
If we continue with the previous example the resulting io_vec is:
---------------------------------
| y | y | y | y | y | y | y | y |
---------------------------------
| | | | | | | |
| | | | | | | v
| | | | | | v frame #8 --- in block #4
| | | | | v frame #7 ------- in block #4
| | | | v frame #6 ----------- in block #3
| | | v frame #5 --------------- in block #3
| | v frame #4 ------------------- in block #2
| v frame #3 ----------------------- in block #2
v frame #2 --------------------------- in block #1
frame #1 ------------------------------- in block #1
If you check the source code you will see that what I draw here as a frame
is not only the link level frame. At the begining of each frame there is a
header called struct tpacket_hdr used in PACKET_MMAP to hold link level's frame
meta information like timestamp. So what we draw here a frame it's really
the following (from include/linux/if_packet.h):
/*
Frame structure:
- Start. Frame must be aligned to TPACKET_ALIGNMENT=16
- struct tpacket_hdr
- pad to TPACKET_ALIGNMENT=16
- struct sockaddr_ll
- Gap, chosen so that packet data (Start+tp_net) alignes to TPACKET_ALIGNMENT=16
- Start+tp_mac: [ Optional MAC header ]
- Start+tp_net: Packet data, aligned to TPACKET_ALIGNMENT=16.
- Pad to align to TPACKET_ALIGNMENT=16
*/
Other constraints
-------------------
The following are conditions that are checked in packet_set_ring
tp_block_size must be a multiple of PAGE_SIZE (1)
tp_frame_size must be greater than TPACKET_HDRLEN (obvious)
tp_frame_size must be a multiple of TPACKET_ALIGNMENT
tp_frame_nr must be exactly frames_per_block*tp_block_nr
I believe the check check (1) should be changed to check if
tp_block_size is also a power of two.
I suposse that alignment to 16 bytes boundaries is to fit better
in cache lines.
--------------------------------------------------------------------------------
+ Details and discusion
--------------------------------------------------------------------------------
All memory allocations done in packet_set_ring are not freed until the socket
is closed. The memory allocations are done with GFP_KERNEL priority, this basically
means that the allocation can wait and swap other process' memory in order
to allocate the nececessary memory, so normally limits can be reached.
While reading packet_set_ring I asked myself some questions:
+ Why pointers for both blocks and frames?
io_vec and pg_vec pointers are asigned to a struct packet_op
wich is held in the packet socket, not freed until socket close. In
struct packet_opt io_vec renames to iovec.
By having frame pointers there is a fast access to each frame when
needed, and this is fine because it will be very often. Block pointers
are used only in the setup/shutdown of the PACKET_MMAP infraestructure,
mostly in packet_set_ring and packet_mmap. It is possible to infer block
position by taking into acount the number of frames each block has,
PACKET_MMAP designers seems that tought it is worth having pg_vec
to make code more readable.
+ The maximum number of frames, is really a limitating factor?
Next I will consider the following scenario:
In the Internet, packet average size, including the link layer,
is around 575 bytes.
In a i386 architecture PACKET_MMAP can hold up to 32768 frames.
If we want to monitor a link at a rate of 1 Gb/s, PACKET_MMAP
will only buffer as much as 0.150 seconds ((575*8*32768)/10^9).
With 10GE Wan interfaces going to be mainstream, this limit will
have to go away.
kmalloc limits (128 KiB by default) are defined at include/linux/kmalloc_sizes.h, and
is raised in case the CPU doesn't have an MMU (CONFIG_MMU undefined) and can be raised
further with CONFIG_LARGE_ALLOCS.
It's straight forward do modify kmalloc_sizes.h to increase the limits, but you also
have to modify MAX_OBJ_ORDER and MAX_GFP_ORDER in slab.c.
Another possibility would be to change the allocation of io_vec and pg_vec
to use vmalloc instead of kmalloc.
--------------------------------------------------------------------------------
+ MISC
--------------------------------------------------------------------------------
You can find the lastest version of this document at
http://pusa.uv.es/~ulisses/packet_mmap/packet_set_ring.txt
You can download a libpcap library with PACKET_MMAP from
http://pusa.uv.es/~ulisses/packet_mmap/libpcap-0.6.2-packet_mmap.tar.gz
-
: send the line "unsubscribe linux-net" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
