10 000 PPS is, at worst, around 15 000 000 bytes/s. Even today's lamest SATA drives should be able to handle that data rate; USB thumb drives can be a different story. Turn off O_SYNC. Let Linux do its job; it handles disk I/O very well. Write to the file cache and let Linux handle flushing the data to disk. Be sure you have plenty of RAM to cache plenty of data. And if you expect to read the data simultaneously, you'll want even *more* RAM to give the system half a chance of keeping some of the data cached. If you're worried about power inerruptions, use a UPS. If you're worried about FS corruption, use a journalled FS. N On Friday, November 21, 2014 07:43:09 AM Joel Gerber wrote: > What type of storage are you writing to? The slow-down might be disk > access. > > If you really want to log traffic in the 10,000 pkt/sec range, I would > start by writing to memory storage, and then have a process in the > background sync to disk. Hopefully the sync will be able to keep up > relatively well. > > You'll also want a writing system with very little overhead. Writing > directly to a file with no abstraction layer, using binary output, might > be the fastest thing possible, short of writing to a raw partition table. > > The other thing to consider, is even with all of the data written, how will > you read it? Having a file constantly open for writing, precludes being > able to reliably read from it. > > Joel Gerber > Network Specialist > Network Operations > Eastlink > E: Joel.Gerber@xxxxxxxxxxxxxxxx T: 519.786.1241 > > > -----Original Message----- > From: netfilter-owner@xxxxxxxxxxxxxxx > [mailto:netfilter-owner@xxxxxxxxxxxxxxx] On Behalf Of Akshat Kakkar Sent: > November-21-14 7:36 AM > To: netfilter@xxxxxxxxxxxxxxx > Subject: iptables logging using ulog : which can handle high traffic, > writing in db or json or xml? > > i want to do logging of traffic iptables rules. > Traffic can go upto 10000pkts/sec. > > While using ulog and writing in mysql db, I could not get good results. Max > around 300 pkts/sec were able to handle. Further traffic was delayed in > db. > > Then I used json, it was able to handle around 1700 pkts/sec, beyond that > there is traffic loss in json file. > > Is there any other mode which can be faster than json without losing any > traffic? -- > To unsubscribe from this list: send the line "unsubscribe netfilter" in the > body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at > http://vger.kernel.org/majordomo-info.html > N�����r��y����b�X��ǧv�^�){.n�+���z����{ay� ʇڙ�,j��f���h���z� �w��� ���j > :+v���w�j�m��������zZ+�����ݢj"��!�i -- To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
