Hello, On Sat, 24 Aug 2013, Drunkard Zhang wrote: > I'm running x86_64 kernel. I compared kernel config of my two servers, > a big difference between them is CONFIG_PREEMPT. While CONFIG_PREEMPT > is disabled, trying plenty times of "ipvsadm -C && ipvsadm -R < > rules-with-ops" will finally succeed, but with CONFIG_PREEMPT enabled There is no "./" in above ipvsadm commands, I hope you put everything in scripts to make sure the new ipvsadm binary is used. > it's too hard to get --ops work. I will test again on my "good" server > another day to prove my guessing. My tests are on 32-bit UP, may be that is why I can not reproduce it. > Is there any good debug method for this? Tuning > /proc/sys/net/ipv4/vs/debug_level didn't gave me much. echo 20 > /proc/sys/net/ipv4/vs/debug_level should show something but don't do it for 60K packets/sec > I use keepalived to manage the ipvs configuration, but as vrrp > heartbeat going on and no realserver up/down, it won't interact with > ipvs, right? So I can temporarily modify ipvs rule via ipvsadm after > keepalived started, and the modified rules didn't changed as time fly, > so do the --ops setting. Yes, just make sure ops is present after the tests, in case some daemon removes the flag. > > More things to check: > > > > - if traffic stops check if some real server is hijacking the > > traffic from director due to ARP problem in the real server. > > Or explain how exactly OPS stops to work, do you see other > > traffic for the VIP coming to director during such problem? > > > No possibility, I configured VIP on lo of realserver. > for IP in $VIP; do > ip addr add $IP/32 dev $VIP_NIC brd $IP > done Setting these flags on "lo" is useless but "all" values should do the job, so ARP problem is solved. > sysctl -q -w net.ipv4.conf.lo.arp_ignore=1 > sysctl -q -w net.ipv4.conf.lo.arp_announce=2 > sysctl -q -w net.ipv4.conf.all.arp_ignore=1 > sysctl -q -w net.ipv4.conf.all.arp_announce=2 > > > - Build ipvsadm with 'make HAVE_NL=0' to check if Conns=0 problem > > in --stats output is netlink related. This builds ipvsadm without > > netlink support but use this binary only to see stats, not > > for configuration. > > > > - show output from 'cat /proc/net/ip_vs_stats_percpu' to see > > the kernel's stats and rates. Note that these stats are not > > zeroed while stats in /proc/net/ip_vs_stats are zeroed. > > Always changing. Even when OPS does not work? > vs3 ~ # cat /proc/net/ip_vs_stats_percpu > Total Incoming Outgoing Incoming Outgoing > CPU Conns Packets Packets Bytes Bytes > 0 8F11751F 70455AB5 0 10AA672610D 0 > 1 1A780554 1A780554 0 E2AB71BCA 0 > 2 0 0 0 0 0 > 3 BF0E0B BF0E0B 0 4B7E409C 0 > 4 244BAF54 244BAF54 0 2224071265 0 > 5 2360B25C 2360B25B 0 1715A45DB3 0 > 6 0 0 0 0 0 > 7 E88FEF E88FEF 0 6ECC3067 0 > 8 1E2477AE 1E2477AE 0 12726CDE2E 0 > 9 10BD4D97 10BD4D97 0 A35650024 0 > A BE81916 BE81914 0 6D9FD6CEF 0 > B 4474D837 4474D836 0 3FCEC43B56 0 > C 0 0 0 0 0 > D 0 0 0 0 0 > E 0 0 0 0 0 > F 0 0 0 0 0 > ~ 721BAF1B 534F94AD 0 1B61556B50B 0 > > Conns/s Pkts/s Pkts/s Bytes/s Bytes/s > 1120F 1120F 0 C1FEB1 0 So, to summarize for the both cases when OPS works and when OPS does not work: - you check after every rule restoring that the ops is present in kernel rules: cat /proc/net/ip_vs - in both cases traffic is received on director (no ARP problem): tcpdump -lnnn -i $INPUT_DEVICE -c 10 $VIP - cat /proc/net/ip_vs_stats_percpu in both cases shows that Conns for CPU "~" (Totals) are increasing and "Conns/s" rate is above 0. Help me to understand the Conns=0 and CPS=0 values in ipvsadm, they are showing 0 in both cases, right? - where do you see that OPS is not working? In ipvsadm -ln --stats/--rate ? Or packets do not reach real servers? Do you see that rates or stats for the real servers stop in ipvsadm output? May be we can enable debug for short time when OPS is not working: # Start debug for 10ms echo 20 > /proc/sys/net/ipv4/vs/debug_level usleep 10000 # Stop debug echo 0 > /proc/sys/net/ipv4/vs/debug_level You can show me such debug. The main thing to understand is where in IPVS the traffic is lost, the debug will be helpful, it should be no more than one page per packet. I need debug for one packet, something that you see is repeated in logs. May be due to the destination trash mechanism something is not set properly after the ipvsadm -C && ipvsadm -R sequence. Regards -- Julian Anastasov <ja@xxxxxx> -- To unsubscribe from this list: send the line "unsubscribe lvs-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
