Hello, On Fri, 11 Apr 2008, Jason Stubbs wrote: > Greetings, > > Ok, things are mostly working now. The patch is a little messy as in there's > old comments remaining and function names are left as is, but hopefully > reviewable. If it's not, I'll split it up and add appropriate comments... Your changes will break existing setups. I'll recommend you to start by reading http://www.ssi.bg/~ja/LVS.txt. I just updated it with some 2.6 info as it was too old document. There you can see some requirements and motivation why IPVS uses specific hooks and priority. I think, for such changes there are many things to be considered and carefully tested: - all forwarding methods can be tested on LAN, even LVS-TUN - forwarding of related ICMP traffic (ICMP errors) in both directions, for all methods - ICMP generation to both sides (client and real server): when there is no real server, when skb is longer than PMTU. - scheduling by nfmark - firewall: at least basic packet fields matching - ip_vs_ftp testing (LVS-NAT) when netfilter ftp module is in effect: test if double NAT happens resulting in broken packets (TCP sequence numbers or payload) when payload is changed if IP:PORT strings in FTP commands have different length (VIP and RIP). Note that there are many new changes in Netfilter and Networking after IPVS was included in early 2.6. Even I already don't know what happens in latest kernels for POST_ROUTING, with fragmentation, etc. May be some things work by luck because IPVS tries to work closely with Netfilter without breaking things. That is why a careful testing is needed for any new changes if such changes are planned for inclusion in kernel. > With local node, 127.0.0.1 doesn't work but an IP address on a local interface > does. When the address is 127.0.0.1, the SYN makes it all the way through > INPUT, but the SYN/ACK doesn't come into OUTPUT. Something to investigate > further... Also, null_xmit doesn't work as ipvs_in is being done in > POSTROUTING, so I've simple aliased LOCAL to MASQ for the time being. LOCAL replaced with MASQ? Such changes can not be accepted for inclusion, they break existing setups just because something does not work in your new way to handle things. You should always remember that there must be a reason some code to exist. If you really want to modify IPVS I'll recommend you to create some short document that explains: - how do you plan out->in (ip_vs_in) and in->out (ip_vs_out) packets to traverse netfilter hooks, when addresses, ports and payload are modified (ip_vs_ftp) - what setups you are going to break because you consider them as not used anymore - use defines/configuration options to preserve old handling for existing setups. If your changes are not planned for inclusion you can do whatever you want, of course. > What I haven't tested: > * LVS-TUN > * ICMP for LVS-NAT You can test if related ICMP errors are forwarded by adding REJECT-with-ICMP rules in client and real server. > * IP_VS_CONN_F_BYPASS - what is this? IP_VS_CONN_F_BYPASS is used for transparent proxy setups when real server (cache server) is not present and we should forward the traffic to original destination. The idea is request still to be served. In such case IPVS traffic uses the original destination instead of real server. > I realized I haven't explained at all why I chose POST/PRE as the hook points. > Firstly the cropped output from a LOG target in every mangle table for the > SYN SYN/ACK of a LVS-NAT connection: > > PREROUTING IN=eth0 OUT= SRC=192.168.0.104 DST=192.168.0.SYN > FORWARD IN=eth0 OUT=eth0 SRC=192.168.0.104 DST=192.168.0.7 SYN > POSTROUTING IN= OUT=eth0 SRC=192.168.0.104 DST=192.168.0.7 SYN > POSTROUTING IN= OUT=eth1 SRC=192.168.0.104 DST=192.168.1.3 SYN > > PREROUTING IN=eth1 OUT= SRC=192.168.0.7 DST=192.168.0.104 ACK SYN > FORWARD IN=eth1 OUT=eth0 SRC=192.168.0.7 DST=192.168.0.104 ACK SYN > POSTROUTING IN= OUT=eth0 SRC=192.168.0.7 DST=192.168.0.104 ACK SYN > > 192.168.0.104 is the client, 192.168.0.7 is the VIP and 192.168.1.3 is the > real server. Other than the second POSTROUTING entry on the SYN side, > netfilter isn't dealing with the real server's IP at all. This will > theoretically make writing firewall rules much easier and also limits what > netfilter's conntracking has to deal with. > > Actually, I don't know why the second POSTROUTING entry is there at all. It > seems that after the packet is injected into the end of POSTROUTING, a > routing decision is being made again and POSTROUTING is rerun. Preferable the > packet would go straight out the appropriate interface after ipvs_in is run. Not sure what happens, it is a good idea to put some printk()s in netfilter (eg. hooks) when testing IPVS changes. > Similar behaviour happens with a local node: > > PREROUTING IN=eth0 OUT= SRC=192.168.0.104 DST=192.168.0.7 SYN > FORWARD IN=eth0 OUT=eth0 SRC=192.168.0.104 DST=192.168.0.7 SYN > POSTROUTING IN= OUT=eth0 SRC=192.168.0.104 DST=192.168.0.7 SYN > POSTROUTING IN= OUT=lo SRC=192.168.0.104 DST=192.168.0.5 SYN > PREROUTING IN=lo OUT= SRC=192.168.0.104 DST=192.168.0.5 SYN > INPUT IN=lo OUT= SRC=192.168.0.104 DST=192.168.0.5 SYN > > OUTPUT IN= OUT=eth0 SRC=192.168.0.7 DST=192.168.0.104 ACK SYN > POSTROUTING IN= OUT=eth0 SRC=192.168.0.7 DST=192.168.0.104 ACK SYN > > 192.168.0.5 is an IP local to the director. I had to add the ipvs_out hooks to > the beginning of OUTPUT as the local reply never hits PREROUTING. Again with > the above, I'd prefer the POST/PRE/INPUT disappear. Why? I don't think it is possible without changes in Netfilter. There are some issues that prevent IPVS to benefit from Netfilter connection tracking: - Netfilter's NAT and routing are not in single place (hook), difficult to handle LVS-DR - Netfilter can re-route sometimes (eg. after mangle), it can cause properly routed LVS-DR traffic to fail. - Double NAT for ip_vs_ftp > Anyway, that's pretty much my intention. Is there any problem with essentially > hiding the real servers from netfilter? Is there a way to get the packet out > of the netfilter loop earlier? IPVS traffic should not be NAT-ed by Netfilter. This double-NAT leads to broken packets as I already mentioned above. What I do not understand is what is the end goal for your changes? Speed or IPVS traffic to fully benefit from Netfilter features? Or some setup does not work? Regards -- Julian Anastasov <ja@xxxxxx> -- To unsubscribe from this list: send the line "unsubscribe lvs-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
