On Tue, Jan 10, 2023 at 3:27 PM Toke Høiland-Jørgensen <toke@xxxxxxxxxx> wrote: > > Vincent Li <vincent.mc.li@xxxxxxxxx> writes: > > > On Tue, Jan 10, 2023 at 7:23 AM Toke Høiland-Jørgensen <toke@xxxxxxxxxx> wrote: > >> > >> Vincent Li <vincent.mc.li@xxxxxxxxx> writes: > >> > >> > On Mon, Jan 2, 2023 at 3:34 AM Toke Høiland-Jørgensen <toke@xxxxxxxxxx> wrote: > >> >> > >> >> Vincent Li <vincent.mc.li@xxxxxxxxx> writes: > >> >> > >> >> > On Wed, Dec 14, 2022 at 2:53 PM Toke Høiland-Jørgensen <toke@xxxxxxxxxx> wrote: > >> >> >> > >> >> >> Vincent Li <vincent.mc.li@xxxxxxxxx> writes: > >> >> >> > >> >> >> > Hi, > >> >> >> > > >> >> >> > If I have an user space stack like mTCP works on top of AF_XDP as tcp > >> >> >> > stateful packet filter to drop tcp packet like tcp syn/rst/ack flood > >> >> >> > or other tcp attack, and redirect good tcp packet back to linux host > >> >> >> > stack after mTCP filtering, is that possible? > >> >> >> > >> >> >> Not really, no. You can inject it using regular userspace methods (say, > >> >> >> a TUN device), or using AF_XDP on a veth device. But in both cases the > >> >> >> packet will come in on a different interface, so it's not really > >> >> >> transparent. And performance is not great either. > >> >> > > >> >> > I have thought about it more :) what about this scenario > >> >> > > >> >> > > >> >> > good tcp rst/ack or bad flooding rst/ack -> NIC1 -> mTCP+AF_XDP ->NIC2 > >> >> > > >> >> > NIC1 and NIC2 on the same host, drop flooding rst/ack by mTCP, > >> >> > redirect good tcp rst/ack to NIC2, is that possible? > >> >> > >> >> You can do this if NIC2 is a veth device: you inject packets into the > >> >> veth on the TX side, they come out on the other side and from the kernel > >> >> PoV it looks like all packets come in on the peer veth. You'll need to > >> >> redirect packets the other way as well. > >> >> > >> >> > any performance impact? > >> >> > >> >> Yes, obviously :) > >> >> > >> >> >> In general, if you want to filter traffic before passing it on to the > >> >> >> kernel, the best bet is to implement your filtering in BPF and run it as > >> >> >> an XDP program. > >> >> > > >> >> > I am thinking for scenario like tcp rst/ack flood DDOS attack to NIC1 > >> >> > above, I can't simply drop every rst/ack because there could be > >> >> > legitimate rst/ack, in this case since mTCP can validate legitimate > >> >> > stateful tcp connection, drop flooding rst/ack packet, redirect good > >> >> > rst/ack to NIC2. I am not sure a BPF XDP program attached to NIC1 is > >> >> > able to do stateful TCP packet filtering, does that make sense to you? > >> >> > >> >> It makes sense in the "it can probably be made to work" sense. Not in > >> >> the "why would anyone want to do this" sense. If you're trying to > >> >> protect against SYN flooding using XDP there are better solutions than > >> >> proxying things through a user space TCP stack. See for instance Maxim's > >> >> synproxy patches: > >> >> > >> > > >> > SYN flooding is just one of the example, what I have in mind is an > >> > user space TCP/IP stack runs on top of AF_XDP as middle box/proxy for > >> > packet filtering or load balancing, like F5 BIG-IP runs an user space > >> > TCP/IP stack on top of AF_XDP. I thought open source mTCP + AF_XDP > >> > could be a similar use case as middle box. user space TCP/IP stack + > >> > AF_XDP as middle box/proxy, the performance is not going to be good? > >> > >> Well, you can certainly build a proxy using AF_XDP by intercepting all > >> the traffic and bridging it onto a veth device, say. I've certainly > >> heard of people doing that. It'll have some non-trivial overhead, > >> though; even if AF_XDP is fairly high performance, you're still making > >> all traffic take an extra hop through userspace, and you'll lose > >> features like hardware TSO, etc. Whether it can be done with "good" > >> performance depends on your use case, I guess (i.e., how do you define > >> "good performance"?). > >> > >> I guess I don't really see the utility in having a user-space TCP stack > >> be a middlebox? If you're doing packet-level filtering, you could just > >> do that in regular XDP (and the same for load balancing, see e.g., > >> Katran), and if you want to do application-level filtering (say, a WAF), > >> you could just use the kernel TCP stack? > >> > > > > the reason I mention user-space TCP stack is user space stack appears > > performs better than kernel TCP stack, and we see user-space stack + > > DPDK for high speed packet processing applications out there, since > > XDP/AF_XDP seems to be competing with DPDK, so I thought why not user > > space stack + AF_XDP :) > > Well, there's a difference between running a user-level stack directly > in the end application, or using it as a middlebox. The latter just adds > overhead, and again, I really don't see why you'd want to do that? The middle box indeed adds overhead in the packet path, but commercial products like BIG-IP is a middle box with a well tuned user-space stack that runs on DPDK successfully, and could potentially run on AF_XDP if AF_XDP performs well as DPDK. I think there's no reason open source user-space stack can't do this :). Suricata also added AF_XDP support, no TX support though https://github.com/OISF/suricata/pull/8210/commits/2cf0b3616c668c0d3fda4de47a6b93630dcc366d, Suricata could work as inline middle box in IPS mode, but seems not supported so far for AF_XDP. so I am thinking of an user-space stack + AF_XDP that works as a stealth inline middle box to do the advanced packet processing/filtering/proxying to protect application servers, maybe AF_XDP is not designed for such use cases? > > Also, the mTCP web site cites tests against a 3.10 kernel, and the code > doesn't look like it's been touched for years. So I'd suggest running > some up-to-date tests against a modern kernel (and trying things like > io_uring if your concern is syscall overhead for small flows) before > drawing any conclusions about performance :) > I cloned https://github.com/mcabranches/mtcp that added AF_XDP support, I could run the example epserver app fine on Ubuntu 20.04 5.4 kernel, and yes, it appears performance worse from CPU usage wise than kernel stack when under hping3 RST flood. perf top example when under hping3 RST flood: Samples: 120K of event 'cycles', 4000 Hz, Event count (approx.): 6522491740 lost: 0/0 drop: 0/0 Overhead Shared Object Symbol 21.42% epserver [.] ProcessPacket 10.89% epserver [.] StreamHTSearch 7.07% [kernel] [k] native_write_msr 5.75% epserver [.] HashFlow 5.69% [kernel] [k] native_read_msr 3.62% epserver [.] xsk_ring_prod__fill_addr 2.20% [kernel] [k] xsk_poll 2.05% epserver [.] afxdp_get_rptr 1.87% [kernel] [k] native_queued_spin_lock_slowpath 1.57% epserver [.] ProcessIPv4Packet 1.55% [kernel] [k] _raw_spin_lock_irqsave 1.38% [kernel] [k] do_sys_poll 1.34% epserver [.] TCPCalcChecksum 1.22% [kernel] [k] vmxnet3_poll_rx_only 1.21% libpthread-2.31.so [.] pthread_mutex_unlock 1.21% epserver [.] xsk_alloc_umem_frame 1.04% [kernel] [k] psi_task_change 0.99% [kernel] [k] cpuacct_charge 0.91% epserver [.] ProcessTCPPacket 0.90% [kernel] [k] vmware_sched_clock 0.80% [kernel] [k] poll_schedule_timeout.constprop.0 0.72% epserver [.] MTCPRunThread 0.69% epserver [.] xsk_prod_nb_free 0.64% epserver [.] afxdp_recv_pkts 0.64% [kernel] [k] read_tsc 0.56% [kernel] [k] xsk_generic_rcv 0.55% [kernel] [k] memcpy 0.53% [kernel] [k] x86_pmu_disable_all 0.51% [kernel] [k] native_apic_msr_eoi_write 0.50% [kernel] [k] fput_many 0.49% [kernel] [k] __perf_event_task_sched_in 0.48% [kernel] [k] __fget_light 0.48% [kernel] [k] __schedule 0.47% epserver [.] xsk_cons_nb_avail 0.46% [kernel] [k] memset 0.41% [kernel] [k] irq_entries_start if I run hping3 RST flood a ngnix web server runs on Linux kernel, no visible CPU usage overhead > That being said, it's certainly *possible* to do what you're suggesting; > there's even a PMD driver in DPDK for AF_XDP, so in that sense it's > pluggable. So, like, feel free to try it out? I'm just cautioning > against thinking it some kind of magic bullet; packet processing at high > speeds in software is *hard*, so the details matter a lot, and it's > really easy to throw away any performance gains by inefficiencies > elsewhere in the stack (which goes for both the kernel stack, XDP, and > AF_XDP). I agree there is no magic bullet, I am just thinking if there is potential for AF_XDP as a middle box senario. Thank you for your patience, feel free to drop the discussion :) > > -Toke >
|