Hi Juniper gurus!
I have 2x2 leaf-spine topology made of vqfx 18.1R1 VMs running on KVM:
me@kvm-server $ virsh list | grep junos-mcae
1791 junos-mcae.aos-aos-vm-aos_server running
1792 junos-mcae.aeon_ztp-vm1 running
1793 junos-mcae.vqfx-leaf21 running
1794 junos-mcae.vqfx-leaf22 running
1795 junos-mcae.vqfx-leaf11 running
1796 junos-mcae.vqfx-leaf12 running
1797 junos-mcae.vqfx-leaf11_pfe running
1798 junos-mcae.vqfx-leaf21_pfe running
1799 junos-mcae.ubuntu-server1 running
1800 junos-mcae.ubuntu-server2 running
1801 junos-mcae.vqfx-spine2_pfe running
1802 junos-mcae.vqfx-spine1 running
1803 junos-mcae.vqfx-leaf12_pfe running
1804 junos-mcae.vqfx-spine2 running
1805 junos-mcae.vqfx-spine1_pfe running
1806 junos-mcae.vqfx-leaf22_pfe running
1807 junos-mcae.veos-router running
After some time (~24 hours) `show version` started to take really long time to run, approximately 2-3 minutes. Other commands, like `show interfaces` or `show route` or `show bgp neighbor` execute immediately.
The configuration on vqfx boxes seem to be fine. KVM servers run at 0.2% cpu utilization, and have tons of unused RAM.
How do you troubleshoot? Are there any logs to look at? What may be causing such a huge delay?
me@kvm-server $ kvm --version QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.40), Copyright (c) 2003-2008 Fabrice Bellard me@kvm-server $ libvirtd --version libvirtd (libvirt) 1.2.2 me@kvm-server $ uname -a Linux bs32 4.13.0-041300-generic #201709031731 SMP Sun Sep 3 21:33:09 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux me@kvm-server $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"
Junos boxes:
admin@l2-virtual-mlag-2-leaf1> show version fpc0: -------------------------------------------------------------------------- Hostname: l2-virtual-mlag-2-leaf1 Model: vqfx-10000 Junos: 18.1R1.9 limited JUNOS Base OS boot [18.1R1.9] JUNOS Base OS Software Suite [18.1R1.9] JUNOS Crypto Software Suite [18.1R1.9] JUNOS Online Documentation [18.1R1.9] JUNOS Kernel Software Suite [18.1R1.9] JUNOS Packet Forwarding Engine Support (qfx-10-f) [18.1R1.9] JUNOS Routing Software Suite [18.1R1.9] JUNOS jsd [i386-18.1R1.9-jet-1] JUNOS SDN Software Suite [18.1R1.9] JUNOS Enterprise Software Suite [18.1R1.9] JUNOS Web Management [18.1R1.9] JUNOS py-base-i386 [18.1R1.9] JUNOS py-extensions-i386 [18.1R1.9]