Patrick McHardy wrote:
Michael Tokarev wrote:
Hello.
Right now I'm evaluating various ways to perform traffic accounting
in Linux. Nowadays solutions are all based on netfilter and packet
queuing over netlink interface. And most tools used for that sort
of stuff are based on Cisco "flows". So far so good.
There are several tools available to perform logging of various
statistics which are being logging over netlink, including the
successor of ulogd. But all the tools I've seen so far perform
"flow management" in userspace in addition to all the housekeeping
the kernel already does - namely, mixing-n-matching various
conntrack entries together, keeping/managing hash tables of
various (src_IP, dst_IP, src_port, dst_port etc) stuff.
For flow-based accounting (compatible with cisco netflows I mean)
this all is only to get one field of the flow structure: it's
the start_time, i.e., in terms of conntrack, it's the time when
the conntrack entry were created.
But isn't it a bit excessive to duplicate all the (quite heavy
and fast-changing) data structures in both kernel- and user-space?
I mean, at least for the "industry standard" format, the one
missing field can be kept by kernel, especially since there,
it costs only extra 4 bytes, and zero CPU, while in duplicating
the whole thing in user space is something entirely else?
I think its done for aggregation, but not sure. And we used
to have 32bit counters for a while, so userspace needed to
track overflows.
Aggregation is nice when needed. Most tools that deals with
netflows does their own aggregation later, probably after some
filtering and probably on another hardware.
As for counters, they're 64bits now, so should be fine.
[]
So, are there any objections/comments/suggestions about just
adding this one thing (conditionally using CONFIG_foo or not)?
Or maybe it is already there and there's nothing to do?
How are you going to log the data of a connection thats removed
from the conntrack tables without using netlink? If you're using
netlink, you can generate a time stamp when you receive a NEW
event.
In case either "new" or "remove" event is missing now, that connection
can't be logged properly, because we don't see some part of the
info in either of the two cases.
For nowadays tools, in case we've seen "new" but not "remove" event,
that connection will be in our hash table sorta forever (ok, till
restart). Which is, I think, worse than not logging it at all (it
will not be logged anyway).
Basically, I don't see such a case (lost "remove" event) as valid.
But what I do see is: if our accounting tool were started long
time after system startup and when some connections are already
established, that tool will not know the actual start time --
the tool can force all current entries to be sent over netlink
again to catch up, but all that will be with current timestamp
obviously, since kernel does not keep that info now.
And there's one more interesting thing, too.
Suppose our tool gets restarted somehow. Currently, it's forced
to write down all its current connections as if they were all
suddenly finished, and after restart there will be a bunch of
"new" connections -- all the same as just logged as "finished".
With the suggested approach, we don't need to care about that,
since we only interested in really finished connections, kernel
keeps track of the start time for us. Sure we still can do our
own timekeeping as before -- for example, to force "daily" stats
at midnight even for connections which are still established.
Thanks.
/mjt
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
