2013/8/26 Julian Anastasov <ja@xxxxxx>: > On Mon, 26 Aug 2013, Drunkard Zhang wrote: > >> Good news, I finally found the crap source, it's keepalived. I tested >> several times without keepalived in runlevel 3, after kernel boots I >> add the ipvs service by hand: > > OK, I was worried that my recent RCU changes broke > something in the WRR scheduler and the configuration process. > >> ./ipvsadm -C >> # Clear previous log >> > /var/log/kern.log >> sleep 1 >> # Start debug >> echo 20 > /proc/sys/net/ipv4/vs/debug_level >> ./ipvsadm -R < /etc/keepalived/rules-with-ops >> usleep 30000 >> # Stop debug >> echo 0 > /proc/sys/net/ipv4/vs/debug_level >> >> Then add VIP manually, then do ARP announce manually: >> vs3 ~/pkgs # ip a add 150.164.100.120/32 dev eno1 >> vs3 ~/pkgs # arp-sk -i eno1 -S 150.164.100.120:90:b1:1c:1a:59:46 -d >> 150.164.100.126 >> >> After these actions, traffic starts come in. and all ipvsadm checks >> are fine, OPS is fine too. So I figured that maybe outdated libipvs in >> keepalived broke the ipvs in kernel. I'll try to report this to >> upstream. > > OK, I have no more doubts. To summarize, > here is what I think happened: > > - packet is scheduled while there is virtual service without > the --ops flag. The result is that an UDP connection is > created that expires after 5mins by default, if there are > no more packets. > > - traffic is not stopped, it hits the connection and > restarts its timer. As result, this connection stays > forever and forwards traffic to single server. This explains why expire time from "ipvsadm -lcn" keeps at 5.00min. > - as single connection is used we see that the stats for > Conns and CPS rate do not move because we do not create > connections anymore, all traffic comes from single client > address and the scheduler is not called. > > - there is one variation here: ipvsadm -C is called, > dests are moved to the trash list, new rules are > added but before the RCU grace period is expired. > In such case IP_VS_DEST_STATE_REMOVING is still set and > prevents the same dest to be reused when adding the > same dest parameters. In this case the connection will point > to unavailable dest for 5mins and the traffic that hits it > will not restart its timer. After 5mins the connection > will be removed and the first packet that comes > will use the --ops flag. There is a chance everything > to work. So, if new rules are added we have 2 > situations: > > 1. rules reuse old dests and traffic goes to single server. > This happens if the new rules are added after at least > 10ms (the RCU grace period, in fact), eg. with > usleep 10000 after ipvsadm -C. We have CPS=0 and > InPPS above 0 for single server. > > 2. rules allocate new dest and traffic is stopped > for 5mins. This will happen if rules are added > immediately after ipvsadm -C (while in RCU grace period). > After 5mins everything works. > > - CPS 0 means we are reusing existing connection > > - even if you replace the service or set --ops, the > existing connection is still used, even ipvsadm -C > can not remove it. There is only one chance: to set > expire_nodest_conn=1, to call ipvsadm -C and to wait > next packet to remove the connection. Then to add > all rules again but not before the connection is removed. > >> On the other hand, ipvs didn't recovery from ipvsadm -C, rmmod ip_vs >> && ./ipvsadm -R < rules-with-ops is needed (I tested, reload ip_vs >> module could make OPS work). So robustness of IPVS needs improvement. > > Some problem? May be you refer to the fact that > connections survive ipvsadm -C and that is what prevented > your traffic to be scheduled. > > So, I see two problems here: > > - tools do not set --ops, connection is created and is > reused from all packets from same client. The trick > to add --ops later can not work. Idea: drop traffic > before reaching IPVS (-j DROP) until --ops is applied, > by this way no connections should be created. > > - no way to flush connections in IPVS without removing the > module because expire_nodest_conn works only when traffic is > received. I think, your above remark points here. Again, thanks for your explanation, now I understand all these "weird" things, it's all because of not supporting --ops by keepalived. -- To unsubscribe from this list: send the line "unsubscribe lvs-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
