Hello,
I have a pair of QFX5100's configured as MC-LAG peers. I have an MC-LAG going up to an MX5 router - ge-0/0/41 on both QFX5100-A and QFX5100-B connect to MX5.
The problem is I'm seeing drops in my best-effort queue even though I'm pushing less than 500Mbps through the interface. I'm using iperf to generate UDP traffic. When I push more than 100Mbps, I'm seeing thousands of drops on my ge-0/0/41 interfaces (the mc-ae interface is ae8). Here is my QoS configuration:
admin@QFX5100-B# show class-of-service
classifiers {
dscp CLASSIFIER-DSCP {
forwarding-class FC-BE {
loss-priority low code-points [ 000000 000001 000010 000011 000100 000101 000110 000111 001000 001001 001010 001011 001100 001101 001110 001111 010000 010001 010010 010100 010011 010101 010110 010111 011000 011100 011001 011010 011011 011101 011110 011111 100000 100001 100011 100100 100101 100110 100111 101000 101001 101010 101011 101100 101101 101111 110001 110010 110011 110100 110101 110110 110111 111000 111001 111010 111011 111100 111101 111110 111111 ];
}
forwarding-class FC-AF {
loss-priority low code-points 100010;
}
forwarding-class FC-NC {
loss-priority low code-points 110000;
}
forwarding-class FC-EF {
loss-priority low code-points 101110;
}
}
}
drop-profiles {
DP-STANDARD {
interpolate {
fill-level [ 95 99 ];
drop-probability [ 0 50 ];
}
}
}
shared-buffer {
ingress {
percent 100;
buffer-partition lossless {
percent 5;
}
buffer-partition lossless-headroom {
percent 0;
}
buffer-partition lossy {
percent 95;
}
}
egress {
percent 100;
buffer-partition lossless {
percent 5;
}
buffer-partition lossy {
percent 75;
}
buffer-partition multicast {
percent 20;
}
}
}
forwarding-classes {
class FC-BE queue-num 0;
class FC-AF queue-num 1;
class FC-NC queue-num 2;
class FC-EF queue-num 3;
}
traffic-control-profiles {
TC-PROFILE-STANDARD {
scheduler-map SCHEDMAP-STANDARD;
}
TC-PROFILE-STRICT {
scheduler-map SCHEDMAP-STRICT;
}
}
forwarding-class-sets {
FC-SET-STANDARD {
class FC-BE;
class FC-AF;
class FC-NC;
}
FC-SET-STRICT {
class FC-EF;
}
}
interfaces {
ge-* {
forwarding-class-set {
FC-SET-STANDARD {
output-traffic-control-profile TC-PROFILE-STANDARD;
}
FC-SET-STRICT {
output-traffic-control-profile TC-PROFILE-STRICT;
}
}
unit 0 {
classifiers {
dscp CLASSIFIER-DSCP;
}
}
}
xe-* {
forwarding-class-set {
FC-SET-STANDARD {
output-traffic-control-profile TC-PROFILE-STANDARD;
}
FC-SET-STRICT {
output-traffic-control-profile TC-PROFILE-STRICT;
}
}
unit 0 {
classifiers {
dscp CLASSIFIER-DSCP;
}
}
}
ae* {
forwarding-class-set {
FC-SET-STANDARD {
output-traffic-control-profile TC-PROFILE-STANDARD;
}
FC-SET-STRICT {
output-traffic-control-profile TC-PROFILE-STRICT;
}
}
unit 0 {
classifiers {
dscp CLASSIFIER-DSCP;
}
}
}
}
scheduler-maps {
SCHEDMAP-STANDARD {
forwarding-class FC-BE scheduler SCHED-BE;
forwarding-class FC-AF scheduler SCHED-AF;
forwarding-class FC-NC scheduler SCHED-NC;
}
SCHEDMAP-STRICT {
forwarding-class FC-EF scheduler SCHED-EF;
}
}
schedulers {
SCHED-BE {
transmit-rate percent 80;
buffer-size {
remainder;
}
priority low;
drop-profile-map loss-priority low protocol any drop-profile DP-STANDARD;
}
SCHED-AF {
transmit-rate percent 5;
buffer-size percent 5;
priority low;
drop-profile-map loss-priority low protocol any drop-profile DP-STANDARD;
}
SCHED-NC {
transmit-rate percent 5;
buffer-size percent 5;
priority low;
drop-profile-map loss-priority low protocol any drop-profile DP-STANDARD;
}
SCHED-EF {
shaping-rate percent 10;
buffer-size percent 0;
priority strict-high;
}
}Here is the interface stats:
admin@QFX5100-B# run show interfaces ge-0/0/41 extensive
Physical interface: ge-0/0/41, Enabled, Physical link is Up
Interface index: 670, SNMP ifIndex: 651, Generation: 161
Description: L3 Link to MX5-B - NeutralPath
Link-level type: Ethernet, MTU: 1514, MRU: 0, Speed: Auto, Duplex: Auto, BPDU Error: None,
MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled,
Auto-negotiation: Enabled, Remote fault: Online, Media type: Copper
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x4000
Link flags : None
CoS queues : 12 supported, 12 maximum usable queues
Hold-times : Up 0 ms, Down 0 ms
Current address: 28:a2:4b:42:33:d8, Hardware address: 28:a2:4b:42:33:0c
Last flapped : 2018-02-14 22:47:30 CST (4w6d 11:44 ago)
Statistics last cleared: 2018-03-21 09:35:39 CDT (01:55:53 ago)
Traffic statistics:
Input bytes : 1584906758 14984 bps
Output bytes : 943837004 101200 bps
Input packets: 1214425 19 pps
Output packets: 1471527 96 pps
IPv6 transit statistics:
Input bytes : 0
Output bytes : 0
Input packets: 0
Output packets: 0
Input errors:
Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Bucket drops: 0, Policed discards: 0,
L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0,
Resource errors: 0
Output errors:
Carrier transitions: 0, Errors: 0, Drops: 12360, Collisions: 0, Aged packets: 0, FIFO errors: 0,
HS link CRC errors: 0, MTU errors: 0, Resource errors: 0, Bucket drops: 0
Egress queues: 12 supported, 7 in use
Queue counters: Queued packets Transmitted packets Dropped packets
0 0 1307498 12360
1 0 0 0
2 0 14 0
3 0 39 0
4 0 0 0
7 0 160280 0
8 0 0 0
Queue number: Mapped forwarding classes
0 FC-BE
1 FC-AF
2 FC-NC
3 FC-EF
4 no-loss
7 network-control
8 mcast
Active alarms : None
Active defects : None
MAC statistics: Receive Transmit
Total octets 1584906758 943837004
Total packets 1214425 1471527
Unicast packets 1207472 1460998
Broadcast packets 0 23
Multicast packets 6953 10506
CRC/Align errors 0 0
FIFO errors 0 0
MAC control frames 0 0
MAC pause frames 0 0
Oversized frames 0
Jabber frames 0
Fragment frames 0
VLAN tagged frames 1207472
Code violations 0
MAC Priority Flow Control Statistics:
Priority : 0 0 0
Priority : 1 0 0
Priority : 2 0 0
Priority : 3 0 0
Priority : 4 0 0
Priority : 5 0 0
Priority : 6 0 0
Priority : 7 0 0
Filter statistics:
Input packet count 0
Input packet rejects 0
Input DA rejects 0
Input SA rejects 0
Output packet count 0
Output packet pad count 0
Output packet error count 0
CAM destination filters: 1, CAM source filters: 0
Autonegotiation information:
Negotiation status: Complete
Link partner:
Link mode: Full-duplex, Flow control: None, Remote fault: OK, Link partner Speed: 1000 Mbps
Local resolution:
Flow control: None, Flow control tx: None, Flow control rx: None, Remote fault: Link OK
Packet Forwarding Engine configuration:
Destination slot: 0 (0x00)
Interface transmit statistics: Disabled
Logical interface ge-0/0/41.0 (Index 552) (SNMP ifIndex 662) (HW Token 2147483649) (Generation 184)
Flags: Up SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
Traffic statistics:
Input bytes : 0
Output bytes : 389618
Input packets: 0
Output packets: 2861
Local statistics:
Input bytes : 0
Output bytes : 389618
Input packets: 0
Output packets: 2861
Transit statistics:
Input bytes : 0 0 bps
Output bytes : 0 0 bps
Input packets: 0 0 pps
Output packets: 0 0 pps
Protocol aenet, AE bundle: ae8.0, Generation: 208, Route table: 0And here are some QoS output commnads:
admin@QFX5100-B# run show interfaces queue ge-0/0/41
Physical interface: ge-0/0/41, Enabled, Physical link is Up
Interface index: 670, SNMP ifIndex: 651
Description: L3 Link to DAL-MX5-B - NeutralPath
Forwarding classes: 16 supported, 7 in use
Egress queues: 12 supported, 7 in use
Queue: 0, Forwarding classes: FC-BE
Queued:
Packets : 0 0 pps
Bytes : 0 0 bps
Transmitted:
Packets : 1317060 36 pps
Bytes : 933203718 45760 bps
Tail-dropped packets : Not Available
RL-dropped packets : 0 0 pps
RL-dropped bytes : 0 0 bps
Total-dropped packets: 12360 0 pps
Total-dropped bytes : 17389225 0 bps
Queue: 1, Forwarding classes: FC-AF
Queued:
Packets : 0 0 pps
Bytes : 0 0 bps
Transmitted:
Packets : 0 0 pps
Bytes : 0 0 bps
Tail-dropped packets : Not Available
RL-dropped packets : 0 0 pps
RL-dropped bytes : 0 0 bps
Total-dropped packets: 0 0 pps
Total-dropped bytes : 0 0 bps
Queue: 2, Forwarding classes: FC-NC
Queued:
Packets : 0 0 pps
Bytes : 0 0 bps
Transmitted:
Packets : 14 0 pps
Bytes : 1372 0 bps
Tail-dropped packets : Not Available
RL-dropped packets : 0 0 pps
RL-dropped bytes : 0 0 bps
Total-dropped packets: 0 0 pps
Total-dropped bytes : 0 0 bps
Queue: 3, Forwarding classes: FC-EF
Queued:
Packets : 0 0 pps
Bytes : 0 0 bps
Transmitted:
Packets : 39 0 pps
Bytes : 3822 0 bps
Tail-dropped packets : Not Available
RL-dropped packets : 0 0 pps
RL-dropped bytes : 0 0 bps
Total-dropped packets: 0 0 pps
Total-dropped bytes : 0 0 bps
Queue: 4, Forwarding classes: no-loss
Queued:
Packets : 0 0 pps
Bytes : 0 0 bps
Transmitted:
Packets : 0 0 pps
Bytes : 0 0 bps
Tail-dropped packets : Not Available
RL-dropped packets : 0 0 pps
RL-dropped bytes : 0 0 bps
Total-dropped packets: 0 0 pps
Total-dropped bytes : 0 0 bps
Queue: 7, Forwarding classes: network-control
Queued:
Packets : 0 0 pps
Bytes : 0 0 bps
Transmitted:
Packets : 162251 22 pps
Bytes : 12489518 13936 bps
Tail-dropped packets : Not Available
RL-dropped packets : 0 0 pps
RL-dropped bytes : 0 0 bps
Total-dropped packets: 0 0 pps
Total-dropped bytes : 0 0 bps
Queue: 8, Forwarding classes: mcast
Queued:
Packets : 0 0 pps
Bytes : 0 0 bps
Transmitted:
Packets : 0 0 pps
Bytes : 0 0 bps
Tail-dropped packets : Not Available
RL-dropped packets : 0 0 pps
RL-dropped bytes : 0 0 bps
Total-dropped packets: 0 0 pps
Total-dropped bytes : 0 0 bpsadmin@QFX5100-B# run show class-of-service interface ge-0/0/41
Physical interface: ge-0/0/41, Index: 670
Queues supported: 12, Queues in use: 7
Congestion-notification: Disabled
Forwarding class set: FC-SET-STANDARD, Index: 52944
Output traffic control profile: TC-PROFILE-STANDARD, Index: 36518
Forwarding class set: FC-SET-STRICT, Index: 43469
Output traffic control profile: TC-PROFILE-STRICT, Index: 46173
Logical interface: ge-0/0/41.0, Index: 552
Object Name Type Index
Classifier CLASSIFIER-DSCP dscp 49277
Classifier ieee8021p-default ieee8021p 11admin@QFX5100-B# run show class-of-service shared-buffer
Ingress:
Total Buffer : 12480.00 KB
Dedicated Buffer : 2912.81 KB
Shared Buffer : 9567.19 KB
Lossless : 478.36 KB
Lossless Headroom : 0.00 KB
Lossy : 9088.83 KB
Lossless Headroom Utilization:
Node Device Total Used Free
0 0.00 KB 0.00 KB 0.00 KB
Egress:
Total Buffer : 12480.00 KB
Dedicated Buffer : 3744.00 KB
Shared Buffer : 8736.00 KB
Lossless : 436.80 KB
Multicast : 1747.20 KB
Lossy : 6552.00 KB
Finally, the interface config:
admin@QFX5100-B# show interfaces ae8
description "L3 Link to DAL-MX5-B - NeutralPath";
aggregated-ether-options {
lacp {
active;
system-id 00:00:00:00:00:08;
admin-key 1;
}
mc-ae {
mc-ae-id 8;
chassis-id 1;
mode active-active;
status-control standby;
init-delay-time 420;
}
}
unit 0 {
family ethernet-switching {
interface-mode trunk;
vlan {
members [ VLAN-100 VLAN-200 VLAN-300 VLAN-400 ];
}
}
}
{master:0}[edit]
admin@QFX5100-B# show interfaces ge-0/0/41
description "L3 Link to DAL-MX5-B - NeutralPath";
ether-options {
802.3ad ae8;
}
In my opinion, there should be NO reason for the QFX to drop anything on this interface. Like I said, I'm pushing between 200Mbps and 400Mbps on a 1Gbps interface. Why are there drops?
I'm running 14.1X53-D40.8. Any help would be appreciated. Thanks!
R - experiencing dropped packets on an interface, suspect buffer overflows, how to fix?
