We are experiencing a very odd issue with our QFX5100 switching and the routing tables.
Issue:
Ubuntu Test Server (96.126.81.60) making any TCP or UDP connection to off-net services is unable to connect.
After spending a few weeks trying to identify the issue, we found the following to be happening:
1) When no default route is populated on the QFX5100 switch, and only the full BGP table, the Loopback0.0 filter is being applied to all transit traffic.
2) When a default route is populated on the QFX5100 switch, and a full BGP table, only the default route is utilized, the BGP forwarding table appears to be ignored, and the Loopback0.0 filter is not applied.
Below is a simplified network topology showing how all of the Juniper devices are interconnected, as well as how the routing tables are being populated.
I say simplified, because there are actually two MX80 routers, with full BGP tables from two different carriers.
The QFX5100 switches are two physical switches in a VC, using LACP bonding connectivity ( ae[0-5] ) setup for "flexible-vlan-tagging".
Can anyone throw out some ideas as to why the the lo0.0 input filter is being processed on transit traffic?
To my understanding, and all of our research and training, transit traffic should never touch the RE unless exceptions in the packets are experienced.
However, when digging into this issue, I ran across this KB article, but I don't think it applies to my setup (or is even relevant) as I do not run any firewalls on this equipment other than the lo0.0 firewall policy to protect the RE (management, etc).
https://kb.juniper.net/InfoCenter/index?page=content&id=KB32041&cat=QFX_SERIES&actp=LIST
https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1080758
Let me know what you would like to see for configurations or routing table output and I will gladly show it.
Ping Google DNS from Ubuntu Test Server
# ping 8.8.8.8 -c 5 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=120 time=13.0 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=120 time=16.6 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=120 time=38.2 ms 64 bytes from 8.8.8.8: icmp_seq=4 ttl=120 time=19.1 ms 64 bytes from 8.8.8.8: icmp_seq=5 ttl=120 time=11.9 ms --- 8.8.8.8 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4005ms rtt min/avg/max/mdev = 11.988/19.828/38.263/9.566 ms
Traceroute to Google DNS (default UDP mode)
# traceroute 8.8.8.8 -n traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets 1 96.126.81.57 0.786 ms 0.806 ms 0.847 ms 2 96.126.81.37 19.928 ms 19.916 ms 19.897 ms 3 96.126.81.25 20.177 ms 20.207 ms 20.187 ms 4 62.115.170.158 20.059 ms 20.054 ms 20.106 ms 5 62.115.137.106 20.341 ms 20.463 ms 20.488 ms 6 62.115.113.84 20.330 ms 19.770 ms * 7 * * * 8 * * * 9 * * * 10 * * *
Traceroute to Google DNS (ICMP Mode)
# traceroute 8.8.8.8 -n --icmp traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets 1 96.126.81.57 0.872 ms 0.906 ms 0.941 ms 2 96.126.81.37 18.166 ms 18.171 ms 18.169 ms 3 96.126.81.25 18.384 ms 18.404 ms 18.420 ms 4 62.115.170.158 18.383 ms 18.383 ms 18.406 ms 5 62.115.137.106 19.947 ms 20.009 ms 20.105 ms 6 62.115.113.84 19.020 ms 18.184 ms 18.194 ms 7 72.14.243.44 17.788 ms 22.440 ms 22.462 ms 8 * * * 9 72.14.236.240 22.136 ms 22.148 ms 22.115 ms 10 209.85.248.109 23.025 ms 22.848 ms 22.542 ms 11 8.8.8.8 21.378 ms 21.378 ms 20.519 ms
DIG to Google DNS
# dig google.com @8.8.8.8 ; <<>> DiG 9.10.3-P4-Ubuntu <<>> google.com @8.8.8.8 ;; global options: +cmd ;; connection timed out; no servers could be reached
QFX5100 lo0.0 Firewall Filter
> show log RE_FIREWALL | match 96.126.81.60
Mar 27 20:30:24 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 49279 53 (2 packets)
Mar 27 20:30:29 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 49279 53 (2 packets)
Mar 27 20:30:34 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 56371 53 (2 packets)
Mar 27 20:30:39 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 56371 53 (2 packets)
Mar 27 21:04:52 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 52793 53 (2 packets)
Mar 27 21:04:57 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 52793 53 (2 packets)
Mar 27 21:05:02 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 45378 53 (2 packets)
Mar 27 21:05:08 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 45378 53 (2 packets)
Mar 27 21:39:20 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 39696 53 (2 packets)
Mar 27 21:39:25 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 39696 53 (2 packets)
Mar 27 21:39:30 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 58978 53 (2 packets)
Mar 27 21:39:35 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 58978 53 (2 packets)
Mar 27 22:01:48 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 37783 53 (1 packets)
Mar 27 22:01:53 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 37783 53 (1 packets)
Mar 27 22:01:58 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 37783 53 (1 packets)
QFX Configurations - Loopback 0
# show interfaces lo0.0
family inet {
filter {
input PROTECT_RE_v4;
}
address 96.126.81.3/32;
}
family inet6 {
filter {
input PROTECT_RE_v6;
}
address 2607:f170:110:0::3/128;
}QFX Configurations - Facing MX80 Router
# show interfaces ae1
flexible-vlan-tagging;
aggregated-ether-options {
minimum-links 1;
link-speed 10g;
lacp {
active;
periodic fast;
}
}
unit 0 {
family ethernet-switching {
interface-mode trunk;
vlan {
members 1013;
}
}
}
# show interfaces irb.1013
family inet {
address 96.126.81.25/30;
}
family inet6 {
address 2607:f170:110:1014::25/116;
}
QFX Configurations - Facing EX4200 Switch
# show interfaces ae4
flexible-vlan-tagging;
aggregated-ether-options {
minimum-links 1;
link-speed 10g;
lacp {
active;
periodic fast;
}
}
unit 0 {
family ethernet-switching {
interface-mode trunk;
vlan {
members [ 19-20 220-225 1016 2010 ];
}
}
}
# show interfaces irb.1016
family inet {
address 96.126.81.37/30;
}
family inet6 {
address 2607:f170:110:1016::37/116;
}