On 07.11.2018 11:48, Nikolay Shirokovskiy wrote: > Hi, all! > > There is performance issue with network filters and broadcast ethernet traffic. > If L2 segment is large enough (several thousands of VMs) then there is a lot of > broadcast ARP traffic (about frames 100/s). As aresult on host with several hundreds > VMs (say 300) we have kernel thread eating 100% of CPUs just for checking this traffic > against firewall rules. The problem is if there are rules in ebtables POSTROUTING chain > (clean-traffic is example of such filter) then when every single broadcast frame turns into > 300, one for every distinct bridge port and then each one of these 300 is checked against > 300 / 2 rules average to find chain for that port. As a result we have 100 * 300 * 300 / 2 > = 4.5 * 10^6 rules checks per second. Kernel does not spread this workload onto > different CPUs and anyway this is wasting CPUs! > > The simple solution is to put rules that ACCEPT ARP traffic into POSTROUTING > itself before any port specific chains. But this will affect non-VM ports too > and host itself. So can we instead make a distinct network namespace for every > VM and put tap there, next add the bridge into the namespace too so we can apply > ebtables rules there and insert tap into the bridge. Finally connect the bridges > in root namespace and VM namespace by veth pair. As result in the situation > described above each cloned frame will be cheched only againt rules for this > very VM. The regular TCP traffic will have same benefits. On the other hand we > need a bridge and veth pair for every VM and some CPU power to process this extra > traffic path. > > The proposed approach also fixes the problem of slow libvirtd restarting with > network filters ([1], [2]) as it is rather difficult to mess network rules in > different network namespace, at least restarting/reloading firewalld won't > hurt such rules so we just don't need to reinstantiate rules at all. Not true. I forgot sometimes libvirt updates can bring updated filters definitions and/or daemon code that instantiates them so we still need reinstantiate on restart generally. In this case we can use approach of [2] or [3] which one is prefearable. [3] [PATCH] nwfilter: intantiate filters on loading driver https://www.redhat.com/archives/libvir-list/2018-October/msg00787.html Nikolay > > > > [1] [RFC] Faster libvirtd restart with nwfilter rules > https://www.redhat.com/archives/libvir-list/2018-September/msg01206.html > which continues in > https://www.redhat.com/archives/libvir-list/2018-October/msg00657.html > > [2] [PATCH v2 0/2] nwfilter: don't reinstantiate rules if there is no need to > https://www.redhat.com/archives/libvir-list/2018-October/msg01317.html > > > Nikolay > > -- > libvir-list mailing list > libvir-list@xxxxxxxxxx > https://www.redhat.com/mailman/listinfo/libvir-list > -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list
