Hi there, I recently had trouble deploying my Linux based BGP router, ip rule cannot selecting route table based on source ipv4 address Our network structure is like this. GRE Server(IN) >>>>> Server(SG)>>> upstream. SG(root@yxvm-2252644:~#) just means the Singapore node from another provider that supports BGP. IN(root@ip007-20230227) refers to my vultr dedicated server in India, I did not configure BGP with vultr here, because vultr refused to accept my prefix. The following is the routing information to 8.8.8.8 obtained on my SG server root@ip007-20230227:~# ip route 46.23.100.0 8.8.8.8 Command "46.23.100.0" is unknown, try "ip route help". root@ip007-20230227:~# ip route from 46.23.100.0 8.8.8.8 Command "from" is unknown, try "ip route help". root@ip007-20230227:~# ip route get 8.8.8.8 8.8.8.8 via 139.84.140.1 dev enp1s0f0 src 139.84.140.60 uid 0 cache root@ip007-20230227:~# ip route get 8.8.8.8 table 147 Error: inet prefix is expected rather than "table". root@ip007-20230227:~# ip route show table 147 | grep 8.8.8 8.8.8.0/24 via 10.0.5.1 dev gre_sg proto bird metric 32 98.8.8.0/23 via 10.0.5.1 dev gre_sg proto bird metric 32 198.8.80.0/24 via 10.0.5.1 dev gre_sg proto bird metric 32 198.8.81.0/24 via 10.0.5.1 dev gre_sg proto bird metric 32 198.8.82.0/24 via 10.0.5.1 dev gre_sg proto bird metric 32 198.8.83.0/24 via 10.0.5.1 dev gre_sg proto bird metric 32 198.8.84.0/24 via 10.0.5.1 dev gre_sg proto bird metric 32 198.8.85.0/24 via 10.0.5.1 dev gre_sg proto bird metric 32 198.8.86.0/24 via 10.0.5.1 dev gre_sg proto bird metric 32 198.8.87.0/24 via 10.0.5.1 dev gre_sg proto bird metric 32 198.8.88.0/24 via 10.0.5.1 dev gre_sg proto bird metric 32 198.8.89.0/24 via 10.0.5.1 dev gre_sg proto bird metric 32 root@ip007-20230227:~# When I use ping, I did not got any output from tcpdump for interface gre_sg. I think it would be better to paste the full output so we can communicate, let's call it TEST 1. --------------------------------------------- TEST 1 --------------------------------------------- root@ip007-20230227:~# ping -I dummy2 8.8.8.8 PING 8.8.8.8 (8.8.8.8) from 46.23.100.0 dummy2: 56(84) bytes of data. root@ip007-20230227:~# tcpdump -i gre_sg host 8.8.8.8 -v -v tcpdump: listening on gre_sg, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes root@ip007-20230227:~# tcpdump -i gre_sg host 8.8.8.8 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on gre_sg, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel root@ip007-20230227:~# tcpdump -i gre_sg host 8.8.8.8 -v tcpdump: listening on gre_sg, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel root@ip007-20230227:~# tcpdump -i gre_sg host 8.8.8.8 -v -v tcpdump: listening on gre_sg, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel root@ip007-20230227:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 3c:ec:ef:ba:6c:80 brd ff:ff:ff:ff:ff:ff inet 139.84.140.60/23 brd 139.84.141.255 scope global dynamic enp1s0f0 valid_lft 75843sec preferred_lft 75843sec inet6 2401:c080:3000:2a5c:3eec:efff:feba:6c80/64 scope global dynamic mngtmpaddr valid_lft 2591985sec preferred_lft 604785sec inet6 fe80::3eec:efff:feba:6c80/64 scope link valid_lft forever preferred_lft forever 3: enp1s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 3c:ec:ef:ba:6c:81 brd ff:ff:ff:ff:ff:ff 6: dummy2: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 2e:45:0b:65:83:40 brd ff:ff:ff:ff:ff:ff inet 46.23.100.0/32 scope global dummy2 valid_lft forever preferred_lft forever inet6 fe80::2c45:bff:fe65:8340/64 scope link valid_lft forever preferred_lft forever 7: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/sit 0.0.0.0 brd 0.0.0.0 10: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1000 link/gre 0.0.0.0 brd 0.0.0.0 11: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 12: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 14: sit2@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000 link/sit 139.84.140.60 peer 178.173.233.141 inet 10.0.2.2/24 brd 10.0.2.255 scope global sit2 valid_lft forever preferred_lft forever inet6 2407:d340:7500:2::2/64 scope global valid_lft forever preferred_lft forever inet6 fe80::8b54:8c3c/64 scope link valid_lft forever preferred_lft forever 15: gre_sg@enp1s0f0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1476 qdisc noqueue state UNKNOWN group default qlen 1000 link/gre 139.84.140.60 peer 178.173.233.141 inet 10.0.5.2/24 brd 10.0.5.255 scope global gre_sg valid_lft forever preferred_lft forever inet6 fe80::200:5efe:8b54:8c3c/64 scope link valid_lft forever preferred_lft forever 16: twix: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UNKNOWN group default qlen 1000 link/ether 76:a0:c4:30:96:38 brd ff:ff:ff:ff:ff:ff inet 10.0.4.2/24 scope global twix valid_lft forever preferred_lft forever inet6 fe80::c44f:2eff:fe15:8d02/64 scope link valid_lft forever preferred_lft forever root@ip007-20230227:~# tcpdump -i enp1s0f0 host 8.8.8.8 -v -v tcpdump: listening on enp1s0f0, link-type EN10MB (Ethernet), snapshot length 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel root@ip007-20230227:~# tcpdump -i any host 8.8.8.8 -v -v tcpdump: data link type LINUX_SLL2 tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 11:57:26.104394 dummy2 Out IP (tos 0x0, ttl 64, id 34501, offset 0, flags [DF], proto ICMP (1), length 84) 46.23.100.0 > dns.google: ICMP echo request, id 9579, seq 221, length 64 11:57:27.128409 dummy2 Out IP (tos 0x0, ttl 64, id 34623, offset 0, flags [DF], proto ICMP (1), length 84) 46.23.100.0 > dns.google: ICMP echo request, id 9579, seq 222, length 64 11:57:28.152409 dummy2 Out IP (tos 0x0, ttl 64, id 34771, offset 0, flags [DF], proto ICMP (1), length 84) 46.23.100.0 > dns.google: ICMP echo request, id 9579, seq 223, length 64 11:57:29.176411 dummy2 Out IP (tos 0x0, ttl 64, id 34918, offset 0, flags [DF], proto ICMP (1), length 84) 46.23.100.0 > dns.google: ICMP echo request, id 9579, seq 224, length 64 11:57:30.200413 dummy2 Out IP (tos 0x0, ttl 64, id 35005, offset 0, flags [DF], proto ICMP (1), length 84) 46.23.100.0 > dns.google: ICMP echo request, id 9579, seq 225, length 64 11:57:31.224391 dummy2 Out IP (tos 0x0, ttl 64, id 35173, offset 0, flags [DF], proto ICMP (1), length 84) 46.23.100.0 > dns.google: ICMP echo request, id 9579, seq 226, length 64 11:57:32.248404 dummy2 Out IP (tos 0x0, ttl 64, id 35175, offset 0, flags [DF], proto ICMP (1), length 84) 46.23.100.0 > dns.google: ICMP echo request, id 9579, seq 227, length 64 ^C 7 packets captured 8 packets received by filter 0 packets dropped by kernel root@ip007-20230227:~# tcpdump -i any host 8.8.8.8 -v tcpdump: data link type LINUX_SLL2 tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 11:57:37.368392 dummy2 Out IP (tos 0x0, ttl 64, id 36107, offset 0, flags [DF], proto ICMP (1), length 84) 46.23.100.0 > dns.google: ICMP echo request, id 9579, seq 232, length 64 11:57:38.392401 dummy2 Out IP (tos 0x0, ttl 64, id 36316, offset 0, flags [DF], proto ICMP (1), length 84) 46.23.100.0 > dns.google: ICMP echo request, id 9579, seq 233, length 64 11:57:39.416433 dummy2 Out IP (tos 0x0, ttl 64, id 36335, offset 0, flags [DF], proto ICMP (1), length 84) 46.23.100.0 > dns.google: ICMP echo request, id 9579, seq 234, length 64 11:57:40.440394 dummy2 Out IP (tos 0x0, ttl 64, id 36550, offset 0, flags [DF], proto ICMP (1), length 84) 46.23.100.0 > dns.google: ICMP echo request, id 9579, seq 235, length 64 11:57:41.464401 dummy2 Out IP (tos 0x0, ttl 64, id 36651, offset 0, flags [DF], proto ICMP (1), length 84) 46.23.100.0 > dns.google: ICMP echo request, id 9579, seq 236, length 64 11:57:42.488411 dummy2 Out IP (tos 0x0, ttl 64, id 36825, offset 0, flags [DF], proto ICMP (1), length 84) 46.23.100.0 > dns.google: ICMP echo request, id 9579, seq 237, length 64 ^C 6 packets captured 7 packets received by filter 0 packets dropped by kernel ---------------------------------------------------------------------END OF TEST 1----------------------------------------------------- For the second test, I switched to MTR. Let's call it TEST 2 ------------------------------------------------------------------------------------------TEST 2------------------------------ root@ip007-20230227:~# mtr -a 46.23.100.0 8.8.8.8 root@ip007-20230227:~# tcpdump -i any host 8.8.8.8 -v tcpdump: data link type LINUX_SLL2 tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 12:02:37.138196 enp1s0f0 Out IP (tos 0x0, ttl 1, id 6453, offset 0, flags [none], proto ICMP (1), length 64) 46.23.100.0 > dns.google: ICMP echo request, id 54563, seq 33000, length 44 12:02:37.238495 enp1s0f0 Out IP (tos 0x0, ttl 2, id 6475, offset 0, flags [none], proto ICMP (1), length 64) 46.23.100.0 > dns.google: ICMP echo request, id 54563, seq 33001, length 44 12:02:37.338799 enp1s0f0 Out IP (tos 0x0, ttl 3, id 6498, offset 0, flags [none], proto ICMP (1), length 64) 46.23.100.0 > dns.google: ICMP echo request, id 54563, seq 33002, length 44 12:02:37.439118 enp1s0f0 Out IP (tos 0x0, ttl 4, id 6510, offset 0, flags [none], proto ICMP (1), length 64) 46.23.100.0 > dns.google: ICMP echo request, id 54563, seq 33003, length 44 12:02:37.539427 enp1s0f0 Out IP (tos 0x0, ttl 5, id 6528, offset 0, flags [none], proto ICMP (1), length 64) 46.23.100.0 > dns.google: ICMP echo request, id 54563, seq 33004, length 44 12:02:37.639748 enp1s0f0 Out IP (tos 0x0, ttl 6, id 6536, offset 0, flags [none], proto ICMP (1), length 64) 46.23.100.0 > dns.google: ICMP echo request, id 54563, seq 33005, length 44 12:02:37.740067 enp1s0f0 Out IP (tos 0x0, ttl 7, id 6561, offset 0, flags [none], proto ICMP (1), length 64) 46.23.100.0 > dns.google: ICMP echo request, id 54563, seq 33006, length 44 root@ip007-20230227:~# tcpdump -i gre_sg host 8.8.8.8 -v -v tcpdump: listening on gre_sg, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes ^C 0 packets captured 0 packets received by filter root@ip007-20230227:~# tcpdump -i enp1s0f0 host 8.8.8.8 -v -v tcpdump: listening on enp1s0f0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 12:03:46.936558 IP (tos 0x0, ttl 6, id 15388, offset 0, flags [none], proto ICMP (1), length 64) 46.23.100.0 > dns.google: ICMP echo request, id 54563, seq 33971, length 44 12:03:47.008335 IP (tos 0x0, ttl 7, id 15405, offset 0, flags [none], proto ICMP (1), length 64) 46.23.100.0 > dns.google: ICMP echo request, id 54563, seq 33972, length 44 12:03:47.079988 IP (tos 0x0, ttl 8, id 15412, offset 0, flags [none], proto ICMP (1), length 64) 46.23.100.0 > dns.google: ICMP echo request, id 54563, seq 33973, length 44 12:03:47.151624 IP (tos 0x0, ttl 9, id 15426, offset 0, flags [none], proto ICMP (1), length 64) 46.23.100.0 > dns.google: ICMP echo request, id 54563, seq 33974, length 44 12:03:47.223336 IP (tos 0x0, ttl 10, id 15439, offset 0, flags [none], proto ICMP (1), length 64) ------------------------------------------------END OF TEST 2----------------------------------------------------- For netcat, let's call it TEST 3 For the method used by different tcpdumps, I send "hello" every time ---------------------------------------------------------------------------------------TEST 3------------------------------------------------------------------- root@ip007-20230227:~# nc -s 46.23.100.0 -u 8.8.8.8 53 hello hello hello root@ip007-20230227:~# tcpdump -i enp1s0f0 host 8.8.8.8 -v -v tcpdump: listening on enp1s0f0, link-type EN10MB (Ethernet), snapshot length 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel root@ip007-20230227:~# tcpdump -i gre_sg host 8.8.8.8 -v -v tcpdump: listening on gre_sg, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes 12:09:16.198374 IP (tos 0x0, ttl 64, id 58243, offset 0, flags [DF], proto UDP (17), length 34) 46.23.100.0.24017 > dns.google.domain: [bad udp cksum 0xa246 -> 0xbbc8!] domain [length 6 < 12] (invalid) ^C 1 packet captured 1 packet received by filter 0 packets dropped by kernel root@ip007-20230227:~# tcpdump -i any host 8.8.8.8 -v tcpdump: data link type LINUX_SLL2 tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 12:09:27.702784 gre_sg Out IP (tos 0x0, ttl 64, id 58244, offset 0, flags [DF], proto UDP (17), length 34) 46.23.100.0.24017 > dns.google.domain: domain [length 6 < 12] (invalid) Last login: Tue Mar 21 11:37:12 2023 from 140.138.182.239 root@yxvm-2252644:~# tcpdump -i any host 8.8.8.8 -v tcpdump: data link type LINUX_SLL2 tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 12:10:58.237149 vultr_IN In IP (tos 0x0, ttl 64, id 58245, offset 0, flags [DF], proto UDP (17), length 61) 46.23.100.0.24017 > dns.google.domain: 18031 zoneInit [b2&3=0x7220] [29795a] [28261q] [24948n] [11296au] [|domain] 12:10:58.237168 eth0 Out IP (tos 0x0, ttl 63, id 58245, offset 0, flags [DF], proto UDP (17), length 61) 46.23.100.0.24017 > dns.google.domain: 18031 zoneInit [b2&3=0x7220] [29795a] [28261q] [24948n] [11296au] [|domain] ^C 2 packets captured 4 packets received by filter 0 packets dropped by kernel root@yxvm-2252644:~# tcpdump -i eth0 host 8.8.8.8 -v tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 12:11:22.977896 IP (tos 0x0, ttl 63, id 58246, offset 0, flags [DF], proto UDP (17), length 34) 46.23.100.0.24017 > dns.google.domain: domain [length 6 < 12] (invalid) ^C 1 packet captured 1 packet received by filter 0 packets dropped by kernel root@yxvm-2252644:~# tcpdump -i vultr_IN host 8.8.8.8 -v tcpdump: listening on vultr_IN, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes 12:12:02.239933 IP (tos 0x0, ttl 64, id 58247, offset 0, flags [DF], proto UDP (17), length 29) 46.23.100.0.24017 > dns.google.domain: domain [length 1 < 12] (invalid) root@yxvm-2252644:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,L OWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether ba:5e:16:eb:32:7d brd ff:ff:ff:ff:ff:ff altname enp0s18 altname ens18 inet 178.173.233.141/25 brd 178.173.233.255 scope global eth0 valid_lft forever preferred_lft forever inet6 2404:92c0:233:2271:b85e:16ff:feeb:327d/64 scope global dynamic mngtmpaddr valid_lft 2591779sec preferred_lft 604579sec inet6 fe80::b85e:16ff:feeb:327d/64 scope link valid_lft forever preferred_lft forever 3: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1000 link/gre 0.0.0.0 brd 0.0.0.0 4: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 5: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 6: gretap1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1462 qdisc fq state UNKNOWN group default qlen 1000 link/ether c2:42:c3:89:8a:3d brd ff:ff:ff:ff:ff:ff inet 10.0.0.1/24 brd 10.0.0.255 scope global gretap1 valid_lft forever preferred_lft forever inet6 2407:d340:7500:1::1/64 scope global valid_lft forever preferred_lft forever inet6 fe80::c042:c3ff:fe89:8a3d/64 scope link valid_lft forever preferred_lft forever 7: dummy3: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 2e:7c:bc:6d:7d:f2 brd ff:ff:ff:ff:ff:ff inet6 2407:d340:7100::1/128 scope global valid_lft forever preferred_lft forever inet6 2407:d340:7100::/128 scope global valid_lft forever preferred_lft forever inet6 fe80::2c7c:bcff:fe6d:7df2/64 scope link valid_lft forever preferred_lft forever 8: dummy1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 0a:29:d5:31:67:84 brd ff:ff:ff:ff:ff:ff inet6 2407:d340:7500::1/128 scope global valid_lft forever preferred_lft forever inet6 2407:d340:7500::/128 scope global valid_lft forever preferred_lft forever inet6 fe80::829:d5ff:fe31:6784/64 scope link valid_lft forever preferred_lft forever 9: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/sit 0.0.0.0 brd 0.0.0.0 10: sit2@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000 link/sit 178.173.233.141 peer 139.84.140.60 inet6 2407:d340:7500:2::1/64 scope global valid_lft forever preferred_lft forever inet6 fe80::b2ad:e98d/64 scope link valid_lft forever preferred_lft forever 11: vultr_IN@eth0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1476 qdisc noqueue state UNKNOWN group default qlen 1000 link/gre 178.173.233.141 peer 139.84.140.60 inet 10.0.5.1/24 brd 10.0.5.255 scope global vultr_IN valid_lft forever preferred_lft forever inet6 fe80::200:5efe:b2ad:e98d/64 scope link valid_lft forever preferred_lft forever root@yxvm-2252644:~#- -----------------------------------------------------------END OF TEST 3-------------------------------------------------- This is the full mash BGP routing table. Please check the email attachment. https://drive.google.com/file/d/1HWuVecElPgHt2VKvFxq88-YFOiP-ecrT/view?usp=drive_web I think it might have something to do with the software calling network-related modules(netcat follows IP rule but mtr and ping don't), but if some software doesn't obey the ip rule, then it will be a problem (the ip -6 rule doesn't have this problem) Best,
Attachment:
publickey - Huiyuze_Zhi@protonmail.com - 0x3063182D.asc
Description: application/pgp-keys
Attachment:
signature.asc
Description: OpenPGP digital signature
