On Wed, 4 May 2005, Taylor, Grant wrote:
How about using a fifo (man mkfifo and man syslog) and let syslog pipe to that fifo. Some program can then read from the fifo, parse data, and maybe use a database for storing the parsed, now more limited, data. Might be a good ide to have the database on another system :-
Using a FIFO to a program that parses and transmits the data to another system to network might be a possibility. Keep in mind that any processing that you do on the packets has to be able to be done at least as fast if not faster than the rate the packets come in. If you ever end up getting behind on the processing things will snowball on you VERY quickly and more than likely end up in a very nasty mess. This is why I think it would be better to use something like TCPDump or Snort to sniff the network and then post process the dumps.
Why where a FIFO and a program which parses and transmit the data to another system any faster than syslog/syslog-ng/ulogd/etc? (Why reinvent the wheel?)
It might not.. AFAIK, the FIFO is implemented not as disk I/O, but is a memory thingy.
It *appears* as file I/O, but the filesystem is used only to create that named pipe.
Hence, my thought was that since logging with iptables has to go throu syslog, this might offload faster through a FIFO.
The app reading the FIFO would preprocess the datastream and turn it into chunks, exactly as Taylor put it.
Depending on what one really wants to look at / dig out of logging, the FIFO-reading app could also reduce data.
-- Kind regards, Mogens Valentin
