Justin, Is you're recv error(-1) print's : bufferspace unavailable? If so, the problem is not with the tcp sequence, and the multiple tcp sessions, it is again with the face that you're process take's alot of time.. I don't recommend using more threads than you're cpu count, and ofcourse try using a thread pool.. (If someone has better knowledge of schedulers in userland, please input here ). even if you are dispatching the packet to a different thread to work / queue .( as in thread pool ), maybe the throughput you are getting is way over you're processing capability. (Also, if anyone was successful in doing deep processing on packets with libnetfilter_queue, and was successful in outstanding big throughputs, you're input will be appreciated here ) If you want, you can increase the buffer space size, and i said in my previous mail, but it will not solve the issue, since over time / more tcp sessions, even that buffer will fill up. Increasing the buffer can be done via: "nfnl_rcvbufsiz(nfq_nfnlh(my_nfq_handle), NFQ_NF_BUFSIZE);" .. NOTE: if you write to hdd while processing, don't / use an advanced flushing method's in order to avoid writing to disk so much... for example try using syslog... ) Kind regards Yechiel Levi. On Wed, Nov 3, 2010 at 3:53 AM, Justin Yaple <yaplej@xxxxxxxxx> wrote: > Yechiel, > > Thank you so much I have been trying to figure this exact issue out > for the past few day and was getting nowhere. Your suggestion to > ignore the error and keep going worked fine but not sure if its the > best solution. > > On Tue, Nov 2, 2010 at 10:51 AM, Mistick Levi <gmistick@xxxxxxxxx> wrote: >> Hi, >> >> This error is kind of showing up alot in this mailing list.. ( I'd >> love to hear a response about my thought on how to solve those >> re-occurring mail's, in the last paragraph ). >> >> What's causing this error is that you do not handle packet's fast enough... >> meaning that you're callback takes time to finish, therefor it delay >> the recv functions. >> The bufferspace that is filling up is actually the socket buffer.. the >> fd you work the recv function on... >> You can tune the socket buffer size, though it won't help because with >> time you're buffer will fill up. >> and as such you must handle you're packets asap, maybe in a different >> thread( if you have multiple cpu's otherwise its kind of a waste). > > My application already using multiple threads one to recv() packets > from the queue and then one or several others to do more advanced > processing (ie compression/optimizations on TCP segments). When any > more than 1 TCP session was being processed I would get rv = -1. > > My project allows packets that don't need advanced processing to > bypass this in the queue handler function so it can causes packets to > be returned to the queue out of order when there is more than 1 > session involved. These could be SYN packets for new sessions that > dont carry any data so bypass the internal processing queues > altogether. > > Is this the problem here that packets being nfq_set_verdict() out of > queue order causing the rv = -1? I am handling packets in a manner > than ensures that is processes TCP sessions in sequence but not > necessarily in the order received by the queue. > > Perhaps there is a nfq_set_verdict() that would inform the queue that > the process is holding this packet so go ahead and move on to the next > packet in the queue? > > Again thanks, > Justin. > >> >> I hope this mail will be available as an answer to everyone searching >> this error on the web.. i know that when i looked for it, i found very >> little information. >> >> Maybe this information should be added to the doc's or maybe we could >> create a Wiki for netfilter that will help newcomers and solve most of >> those problems before they get to the mailing list, thus leaving the >> mailing list for new issue's as they arise. >> >> >> Kind Regards >> Yechiel Levi >> >> On Tue, Nov 2, 2010 at 7:30 PM, Rajkumar S <rajkumars@xxxxxxxxx> wrote: >>> Hi, >>> >>> Thanks for the reply, you were spot on. I removed && rv >= 0 and now >>> it's working fine. >>> >>> btw, what could have caused buffer space unavailable error? >>> >>> Thanks and regards, >>> >>> raj >>> >>> On Tue, Nov 2, 2010 at 10:30 PM, Mistick Levi <gmistick@xxxxxxxxx> wrote: >>>> Hi, >>>> >>>> Well, if you didn't change the nfqnl_test program at all, what i think >>>> happend is that you got : buffer space unavailable error... >>>> >>>> meaning that in you're loop ( " while ((rv = recv(fd, buf, >>>> sizeof(buf), 0)) && rv >= 0) " >>>> you get rv < 0, and then you exit properly. >>>> You could ignore this "recv error" and just continue on packeting. >>>> >>>> Try removing the "( && rv >= 0 ) ,and let us know if it helped. >>>> >>>> Kind Regards, >>>> Yechiel Levi >>>> >>>> On Tue, Nov 2, 2010 at 5:46 PM, Rajkumar S <rajkumars@xxxxxxxxx> wrote: >>>>> Hi all, >>>>> >>>>> I am using latest git checkout of libnetfilter_queue and libnfnetlink >>>>> on debian etch with kernel 2.6.26-2-686. The iptables rules used while >>>>> testing are: >>>>> >>>>> -A INPUT -s 192.168.3.22/32 -m state --state NEW,ESTABLISHED -j >>>>> NFQUEUE --queue-num 0 >>>>> -A OUTPUT -d 192.168.3.22/32 -m state --state NEW,ESTABLISHED -j >>>>> NFQUEUE --queue-num 0 >>>>> >>>>> I am using utils/nfqnl_test.c as my test program and using wget to get >>>>> a file from 192.168.3.22 for testing. The program runs okay when >>>>> getting smaller files but if number of packets go above say 200 >>>>> nfqnl_test exits with following message: >>>>> >>>>> hw_protocol=0x0800 hook=1 id=389 hw_src_addr=00:14:2a:c9:e1:5d indev=2 >>>>> payload_len=1500 >>>>> entering callback >>>>> hw_protocol=0x0800 hook=1 id=390 hw_src_addr=00:14:2a:c9:e1:5d indev=2 >>>>> payload_len=1500 >>>>> entering callback >>>>> closing library handle >>>>> >>>>> The number of packets to trigger this condition varies from say 200 to >>>>> about 1000 and changes with each run. >>>>> >>>>> dmesg does not show any error, the last lines of dmesg are: >>>>> [76465.470246] ip_tables: (C) 2000-2006 Netfilter Core Team >>>>> [92735.818567] Netfilter messages via NETLINK v0.30. >>>>> [92793.863824] nf_conntrack version 0.5.0 (6144 buckets, 24576 max) >>>>> >>>>> Before testing with compiled git version I was trying with ubuntu >>>>> (lucid) and nfqueue-bindings for python and got the same error. >>>>> >>>>> I am not sure what goes wrong here, I can help with any debug steps to >>>>> find out the exact error if required. Any help to locate and fix this >>>>> issue is much appreciated. >>>>> >>>>> with regards, >>>>> >>>>> raj >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe netfilter" in >>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>> >>> >> -- >> To unsubscribe from this list: send the line "unsubscribe netfilter" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
