Hi Anton, thanks for the hints. I learned few bits in the process
which is always nice.
It would seem default QubesOS routing is fine and that I don't need to
add anything routing wise.
The issue is therefore still unresolved.
Here are the checks done:
<new-dst> 10.137.0.25 (my sys-firewall:PREROUTING DNAT --to-destination IP)
<src> 10.137.0.5 (my sys-net:vif3.0 interface)
<input-iface> eth0 (my sys-firewall:eth0 interface connected to sys-net)
sudo ip route get 10.137.0.25 from 10.137.0.5 iif eth0
10.137.0.25 from 10.137.0.5 dev vif4.0
cache iif eth0
vif4.0 is the correct interface traffic should go to. Note that there
is no via (as local traffic) which is correct.
for the return packet:
sudo ip route get 10.137.0.5 from 10.137.0.25 iif vif4.0
10.137.0.5 from 10.137.0.25 dev eth0
cache iif vif4.0
eth0 is the correct interface return traffic should go to.
sudo nstat -az | grep TcpExtIPReverse
TcpExtIPReversePathFilter 0 0.0
Which if I understood correctly shows a counter of zero for rp_filter
(so no packet dropped by rp_filter).
On 25 February 2018 at 11:13, Anton Danilov
<littlesmilingcloud@xxxxxxxxx> wrote:
> Hi.
> For future: use the 'iptables-save -c' to list the rule set.
> After nat/PREROUTING should be the route lookup stage. Maybe the packets are
> being dropped by the rp_filter or the suitable route not found.
> Check the routes (ip route get <new-dst> from <src> iif <input-iface>), and,
> especially the value of the reverse path filter (you can use the 'nstat -az'
> command and check the TcpExtIPReversePathFilter line).
>
>
> On 25 February 2018 at 12:03, Alex Dubois <bowabos@xxxxxxxxx> wrote:
>>
>> Hi,
>>
>> I have had iptables rules working for some time using QubesOS 3.2
>> (Fedora set of interconnected VMs on top of Xen). After upgrading to
>> QubesOS 4rc4 (and Fedora 26), I am having issues.
>>
>> Here is a subpart of the set-up
>>
>> Connectivity: sys-server-test <--> sys-firewall <--> sys-net
>>
>> sys-net VM:
>> - Interface: ens4 192.168.0.2/24 (not relevant)
>> - Interface: vif3.0 10.137.0.5/32
>> - Routing: 0.0.0.0/0 -> 192.168.0.254 via ens4 (not relevant)
>> - Routing: 10.137.0.6/32 -> 0.0.0.0 via vif3.0
>> - Routing: 192.168.0.0/24 -> 0.0.0.0 via ens4 (not relevant)
>> sys-firewall VM:
>> - Interface: eth0 10.137.0.6/32 (connected to sys-net:vif3.0)
>> - Interface: vif4.0 10.137.0.6/32
>> - Routing: 0.0.0.0/0 -> 10.137.0.5 via eth0
>> - Routing: 10.137.0.25/32 -> 0.0.0.0 via vif4.0
>> sys-server-web VM:
>> - Interface: eth0 10.137.0.25/32 (connected to sys-firewall:vif4.0)
>> - Routing: 0.0.0.0/0 -> 10.137.0.6 via eth0
>> - Routing: 10.137.0.6/32 -> 0.0.0.0 via eth0
>>
>> All 3 VMs have:
>> Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
>> num pkts bytes target prot opt in out source
>> destination
>> 1 75670 5188K ACCEPT all -- * vif+ 0.0.0.0/0
>> 0.0.0.0/0
>> 2 4 208 ACCEPT all -- * lo
>> 0.0.0.0/0 0.0.0.0/0
>> 3 892K 49M MASQUERADE all -- * * 0.0.0.0/0
>> 0.0.0.0/0
>>
>> sys-server-web can connect outbound without any issue (i.e. browse
>> internet).
>>
>> I want to server a website on 443 in sys-server-web. Testing access
>> will be from sys-net in this thread.
>>
>> I have configured
>> sys-firewall VM:
>> In nat tables:
>> Chain PREROUTING (policy ACCEPT 964K packets, 54M bytes)
>> pkts bytes target prot opt in out source destination
>> 0 0 Web-Server tcp -- eth0 * 0.0.0.0/0 10.137.0.6
>> tcp spts:1024:65535 dpt:443
>> Chain Web-Server (1 references)
>> pkts bytes target prot opt in out source
>> destination
>> 0 0 DNAT all -- * * 0.0.0.0/0
>> 0.0.0.0/0 to:10.137.0.25
>> In default tables
>> Chain FORWARD (policy DROP 0 packets, 0 bytes)
>> pkts bytes target prot opt in out source
>> destination
>> 118M 122G ACCEPT all -- * * 0.0.0.0/0
>> 0.0.0.0/0 ctstate RELATED,ESTABLISHED
>> 0 0 Web-Server tcp -- * * 0.0.0.0/0
>> 10.137.0.25 tcp spts:1024:65535 dpt:443
>> Chain Web-Server (1 references)
>> pkts bytes target prot opt in out source
>> destination
>> 0 0 ACCEPT tcp -- * * 0.0.0.0/0
>> 0.0.0.0/0 tcp spts:1024:65535 dpt:443 ctstate NEW
>>
>> If from sys-net I issue
>> wget https://10.137.0.6
>> The counter for sys-firewall iptables -t nat Web-Server rule with
>> target DNAT increments
>> Chain Web-Server (1 references)
>> pkts bytes target prot opt in out source
>> destination
>> 2 104 DNAT all -- * * 0.0.0.0/0
>> 0.0.0.0/0 to:10.137.0.25
>> However the counter for sys-firewall iptables FORWARD rule does not
>> Chain FORWARD (policy DROP 0 packets, 0 bytes)
>> pkts bytes target prot opt in out source
>> destination
>> 118M 122G ACCEPT all -- * * 0.0.0.0/0
>> 0.0.0.0/0 ctstate RELATED,ESTABLISHED
>> 0 0 Web-Server tcp -- * * 0.0.0.0/0
>> 10.137.0.25 tcp spts:1024:65535 dpt:443
>>
>> Trying to trace the packet:
>> sudo iptables -t raw -A PREROUTING -i eth0 -p tcp --dport 443 -j TRACE
>>
>> sudo journalctl | grep TRACE
>> Feb 25 08:37:18 sys-firewall kernel: TRACE: raw:PREROUTING:policy:8
>> IN=eth0 OUT= MAC=00:16:3e:5e:6c:00:fe:ff:ff:ff:ff:ff:08:00
>> SRC=10.137.0.5 DST=10.137.0.6 LEN=52 TOS=0x00 PREC=0x00 TTL=64
>> ID=13711 DF PROTO=TCP SPT=55684 DPT=443 SEQ=4023764561 ACK=0
>> WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40101040201030306)
>> Feb 25 08:37:18 sys-firewall kernel: TRACE: nat:PREROUTING:rule:1
>> IN=eth0 OUT= MAC=00:16:3e:5e:6c:00:fe:ff:ff:ff:ff:ff:08:00
>> SRC=10.137.0.5 DST=10.137.0.6 LEN=52 TOS=0x00 PREC=0x00 TTL=64
>> ID=13711 DF PROTO=TCP SPT=55684 DPT=443 SEQ=4023764561 ACK=0
>> WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40101040201030306)
>> Feb 25 08:37:18 sys-firewall kernel: TRACE: nat:Web-Server:rule:1
>> IN=eth0 OUT= MAC=00:16:3e:5e:6c:00:fe:ff:ff:ff:ff:ff:08:00
>> SRC=10.137.0.5 DST=10.137.0.6 LEN=52 TOS=0x00 PREC=0x00 TTL=64
>> ID=13711 DF PROTO=TCP SPT=55684 DPT=443 SEQ=4023764561 ACK=0
>> WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40101040201030306)
>>
>> It hits the TRACE rule in raw:PREROUTING
>> it hits the nat:PREROUTING
>> it hits the nat:Web-Server which has a target of DNAT to 10.137.0.25
>> I would have expected a hit on filter:FORWARD
>>
>> So I believe the next it looks into the routing table which would
>> instruct it to go out of vif4.0
>> It then go to Mangle rules (I have nothing)
>> It then go to filter:FORWARD
>> The counter does not increment for my rule.
>>
>> At this stage I would ideally like:
>> - some info on how to attach a debugger when the 3rd TRACE gets triggered
>> or
>> - some info on how to increase the logs generated by iptables
>>
>> Thanks in advance for your help.
>>
>> Regards,
>> Alex
>> --
>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
>
> --
> Anton.
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
