Problem: A problem has been identified in a cluster environment using IPVS with Direct Routing where multiple appliances can end up in the "active forwarder/distributor" state simultaneously. As an "active distributor" the appliance balances workload by forwarding packets to the group members. Because "active distributors" also consider each other as group members available to receive forwarded packets (i.e. the load balancers also front as real servers and are working in a HA mode with active/backup roles), the distributors may forward the same packet to each other forming a routing loop. While the immediate trigger in the aforesaid scenario is CPU starvation caused by lock contention leading to an active/active scenario (i.e. two instances both acting as "active" virtualservers), similar route loops in an ip_vs installation is possible through other means as well (e.g. http://marc.info/?l=linux-virtual-server&m=136008320907330&w=2). As it stands now, there is no mitigation/damping mechanism available in ip_vs to limit the impact of the routing loop as described above. When the scenario occurs it leads to starvation and requires administrative network action on the cluster controller to terminate the routing loop and recover. Although the situation described above was observed in a Virtual Server with Direct Routing, it is just as applicable in Virtual Servers via NAT and IP Tunneling. ip_vs does not decrement ip_ttl as standard routers do and as a result does not have anything to protect itself from re-forwarding the same packet an unbounded number of times. Standard IP routers always decrement the IP TTL as required by rfc791, but ip_vs does not even though ip_vs is acting as a specialized kind of IP router. In a scenario where two ip_vs instances are forwarding to each other (which admittedly should not happen but is not impossible, as illustrated above), there is no way for the system to recover due to the persistence of the route loop. The two hosts will forward the same packet between each other at speed. Test Case: It is possible to configure two ip_vs instances to forward to each other and cause it to starve the network. The starvation itself makes it impossible to recover from this situation since the communication channel is blocked by the forwarding loop. Proposed fix: Sample fix for Linux v4.7 which decrements the TTL when forwarding, is for the Direct Routing Transmitter. ============================================================================ diff -Naur linux_4.7/net/netfilter/ipvs/ip_vs_xmit.c linux_ipvs_patch/net/netfilter/ipvs/ip_vs_xmit.c --- linux_4.7/net/netfilter/ipvs/ip_vs_xmit.c 2016-07-28 00:01:10.040974435 -0500 +++ linux_ipvs_patch/net/netfilter/ipvs/ip_vs_xmit.c 2016-07-28 00:01:42.900977155 -0500 @@ -1156,10 +1156,18 @@ struct ip_vs_protocol *pp, struct ip_vs_iphdr *ipvsh) { int local; + struct iphdr *iph = ip_hdr(skb); EnterFunction(10); rcu_read_lock(); + if (iph->ttl <= 1) { + /* Tell the sender its packet died... */ + __IP_INC_STATS(dev_net(skb_dst(skb)->dev), IPSTATS_MIB_INHDRERRORS); + icmp_send(skb, ICMP_TIME_EXCEEDED, ICMP_EXC_TTL, 0); + goto tx_error; + } + local = __ip_vs_get_out_rt(cp->ipvs, cp->af, skb, cp->dest, cp->daddr.ip, IP_VS_RT_MODE_LOCAL | IP_VS_RT_MODE_NON_LOCAL | @@ -1171,7 +1179,10 @@ return ip_vs_send_or_cont(NFPROTO_IPV4, skb, cp, 1); } - ip_send_check(ip_hdr(skb)); + /* Decrease ttl */ + ip_decrease_ttl(iph); + + ip_send_check(iph); /* Another hack: avoid icmp_send in ip_fragment */ skb->ignore_df = 1; ================================================================================== p.s. A similar fix may be made to the other modes too ( NAT, IP Tunneling, ICMP Package transmitter). -- To unsubscribe from this list: send the line "unsubscribe lvs-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
