vMX and vEOS RFC 5549 interop, EVPN Layer-2 inter-op, etc.

Posted on by user

I began this lab with the hopes of just showing some simple inter-op use cases but unfortunately so much time lapsed between my last blog post I kept having new ideas before I could publish. So this blog post has these subjects packed in for vMX and vEOS:

  • Explore a way to make interface creation less of a burden on KVM:
    • sub-ints (dot1q tagging) as a method to get around KVM interface creation
    • vlans as a method to get around KVM interface creation
  • An eBGP unnumbered setup for an underlay using RFC 5549 (interop b/t vEOS and vMX)
  • vMX and vEOS EVPN Layer-2 interop (type-2 routes)
  • vMX stitch from EVPN to Layer-3 VPN (vpnv4)

Let’s dive in.

Lab Topology

JunOS version: 21.3R2

vEOS version: 4.27.1

Device configs on my git-hub page

Here is a picture of the lab topology in it’s end-state. I’ll peel back the layers of this onion below in each individual section

Making interface creation less of burden on KVM

The reason interface creation is a burden on KVM is that without scripting it is error prone. You must create bridges, attach interfaces to those bridges and you are limited to the amount of interfaces on vMX (my person experience is that it’s around 10). And if your KVM instance is already running you must destroy and recreate it.

So I had a few different approaches I wanted to try:

1.) Sub-Interfaces (dot1q tagging) – This is an approach I’ve used on real physical devices some time ago. It’s easy to follow from a CLI perspective and while I hadn’t tried it on EOS/vEOS lately I thought maybe this would be a good approach.

2.) VLAN Interfaces (w/ a VLAN trunk) – This is very similar to the above. I find it a bit harder to flow on the CLI but we shouldn’t be depending on the CLI as much these days anyways.

3.) KVM virtual interface – This is necessary for a baseline and the hosts on the right side of the diagram (LEAF2, SPINE2, R2) are setup this way. This provides me a sanity check that when code versions get upgraded or i’m trying some new knob that the problem isn’t the interface configuration.

And here’s what I found out:

1.) Sub-interfaces on vEOS are great if you aren’t doing any more encapsulation (VxLAN or MPLS). If you are, it won’t work on vEOS. I do not know why but I suspect it has to do with how vEOS treats a dot1q tagged interface w/ an IP and it doesn’t give it a chance to do a pass for encapsulation w/ VxLAN or MPLS.

2.) VLAN interfaces (w/ a VLAN trunk) – This worked. This technique is something I found in certification lab books which allowed the authors of those books to make up different lab scenarios while leaving the physical cabling intact. So for example all device should physically (or in my case virtually) cable to a common switch or bridge. This allows me to use that one port on all devices as a multiplexer and the individual VLAN interfaces as their P2P link between devices.

In the below picture I can make all of those links listed as trunks and then create a ‘vlan interface’ on each vEOS to create ad-hoc P2P links:

3.) KVM virtual interface – Nothing more to say here, but if the majority of my lab was using the VLAN interface technique I still want a few links between some devices that are ‘real’. It allows me to get a cleaner tcpdump and rule out the variable of using a VLAN interface for anything new I’m trying.

RFC 5549 interop

I set out using a VLAN trunk and VLAN interfaces (for P2P) on the left side of this diagram as my easy way to create more P2P’s. And on the right side I’m using ‘real’ (real to vMX/vEOS) physical links between devices as a tried and true setup.

I don’t enjoy coming up with IP schemas for a lab. However it’s so common to me these days it doesn’t take much thought. Still though, it’d be nice to have an ‘unnumbered’ setup. But I am using BGP in my lab so the closest thing I could find was RFC 5549.

The idea RFC 5549 is to have IPv6 enabled on both router’s interfaces, allow IPv6 LLA neighbor discovery to happen, dynamically have a BGP session over these addresses setup, and also advertise IPv4 NRLI via that IPv6 BGP session. Easy right?

First issue I had was Juniper states 21.1 is the supported release for this. That is wrong. Juniper’s Feature Explorer states it is 21.3 for the PTX (it worked on vMX in 21.3R2). My testing in 21.1 supported this as the session would tear itself down upon receiving the v4 NRLI.

vMX Configuration

set protocols bgp group autodisc type external
set protocols bgp group autodisc family inet unicast extended-nexthop
set protocols bgp group autodisc export REDIS-CONNECTED
set protocols bgp group autodisc multipath multiple-as
set protocols bgp group autodisc dynamic-neighbor NDP peer-auto-discovery family inet6 ipv6-nd
set protocols bgp group autodisc dynamic-neighbor NDP peer-auto-discovery interface ge-0/0/0.101
set protocols bgp group autodisc peer-as-list AS-LIST

set protocols router-advertisement interface ge-0/0/0.101

set policy-options as-list AS-LIST members 65002

set policy-options policy-statement REDIS-CONNECTED term 1 from protocol direct
set policy-options policy-statement REDIS-CONNECTED term 1 from route-filter 10.0.0.3/32 exact
set policy-options policy-statement REDIS-CONNECTED term 1 then accept
set policy-options policy-statement REDIS-CONNECTED term 2 from protocol direct
set policy-options policy-statement REDIS-CONNECTED term 2 then reject

set interfaces ge-0/0/0 flexible-vlan-tagging
set interfaces ge-0/0/0 encapsulation flexible-ethernet-services
set interfaces ge-0/0/0 unit 101 family inet6

vMX side peering session up and receiving IPv4 routes:

vEOS Configuration

interface Ethernet1
   switchport trunk allowed vlan 98,100
   switchport mode trunk

interface Vlan100
   ipv6 enable

ip routing ipv6 interfaces
ipv6 unicast-routing

router bgp 65001
   router-id 10.0.0.1
   bgp default ipv4-unicast transport ipv6
   maximum-paths 4 ecmp 4
<snip>
   neighbor underlay peer group
   neighbor underlay allowas-in 2
   neighbor underlay send-community extended
   neighbor underlay maximum-routes 12000
   redistribute connected
   neighbor interface Vl100 peer-group underlay remote-as 65002
   !
<snip>
   address-family ipv4
      no neighbor overlay activate
      neighbor underlay activate
      neighbor underlay next-hop address-family ipv6 originate

vEOS side peering session up (other session is to leaf) and receiving routes from vMX

EVPN Layer-2 Inter-Op (type-2 routes)

The BGP-EVPN session between Leaf1 and R1 came up with no issues once the above underlay was online. However I ran into an issue with my linux host connected to only a MAC-VRF on the vEOS side pinging the IRB at the vMX. I found that the arista side was broadcasting ARP for “who has 10.0.0.1” and the Juniper vMX was never responding. I began searching the web and found this blog post.

The issue from that blog post is that Juniper is actively suppressing ARP. Comments within that blog post indicate that there’s a hidden JunOS command to re-enable it. I was having no luck getting the hidden knob to work (I did find it though).

I found another knob on the arista side:

router l2-vpn
   arp learning bridged

This knob allows the arista side to create Type-2 MAC and MAC-IP routes from ARP packets. Once implemented R1 gets Leaf1’s advertisements and the pings are able to pass between Linux Host 1 and R1.

NOTE: My preference for a real world deployment is IRB/VLAN Interface at the TOR and to pass only Type-5 routes. As type-5 routes seems to leave less interpretation to the vendors to mess up inter-op. But I thought this was an interesting exercise in inter-op. If I were forced in a scenario of layer-2 I’d prefer single vendor.

In this case I am doing a VLAN-AWARE bundle type.

R1 (vMX):

set routing-instances VLAWARE instance-type virtual-switch
set routing-instances VLAWARE protocols evpn encapsulation vxlan
set routing-instances VLAWARE protocols evpn default-gateway no-gateway-community
set routing-instances VLAWARE protocols evpn extended-vni-list 1098
set routing-instances VLAWARE protocols evpn multicast-mode ingress-replication
set routing-instances VLAWARE vtep-source-interface lo0.0
set routing-instances VLAWARE bridge-domains bd98 vlan-id 98
set routing-instances VLAWARE bridge-domains bd98 routing-interface irb.98
set routing-instances VLAWARE bridge-domains bd98 vxlan vni 1098
set routing-instances VLAWARE bridge-domains bd98 vxlan ingress-node-replication
set routing-instances VLAWARE route-distinguisher 10.0.0.3:1
set routing-instances VLAWARE vrf-target target:1:2

set protocols bgp group EVPN multihop
set protocols bgp group EVPN local-address 10.0.0.3
set protocols bgp group EVPN family evpn signaling
set protocols bgp group EVPN peer-as 65001
set protocols bgp group EVPN neighbor 10.0.0.1

set interfaces irb unit 98 family inet address 10.0.0.1/24

set routing-instances IRBVL98 instance-type vrf
set routing-instances IRBVL98 protocols evpn ip-prefix-routes advertise direct-nexthop
set routing-instances IRBVL98 protocols evpn ip-prefix-routes encapsulation vxlan
set routing-instances IRBVL98 protocols evpn ip-prefix-routes vni 2098
set routing-instances IRBVL98 interface irb.98
set routing-instances IRBVL98 route-distinguisher 10.0.0.3:2
set routing-instances IRBVL98 vrf-target target:1:1
set routing-instances IRBVL98 vrf-table-label

Leaf1 (vEOS):

router bgp 65001
   router-id 10.0.0.1
   bgp default ipv4-unicast transport ipv6
   maximum-paths 4 ecmp 4
   neighbor overlay peer group
   neighbor overlay remote-as 65003
   neighbor overlay update-source Loopback0
   neighbor overlay ebgp-multihop
   neighbor overlay send-community standard extended
   neighbor overlay maximum-routes 12000
   neighbor underlay peer group
   neighbor underlay allowas-in 2
   neighbor underlay send-community extended
   neighbor underlay maximum-routes 12000
   neighbor 10.0.0.3 peer group overlay
   redistribute connected
   neighbor interface Vl100 peer-group underlay remote-as 65002
   !
   vlan-aware-bundle VLAN_BUNDLE
      rd 10.0.0.1:2
      route-target both 1:2
      redistribute learned
      no redistribute host-route
      vlan 98
   !
   address-family evpn
      neighbor overlay activate
      no neighbor underlay activate
   !
   address-family ipv4
      no neighbor overlay activate
      neighbor underlay activate
      neighbor underlay next-hop address-family ipv6 originate
!

interface Vxlan1
   vxlan source-interface Loopback0
   vxlan udp-port 4789
   vxlan vlan 98 vni 1098
   vxlan vrf DC-HOST vni 2098

interface Ethernet1
   switchport trunk allowed vlan 98,100
   switchport mode trunk

vMX stitch from EVPN to Layer-3 VPN (vpnv4)

It would be boring if I just peered EVPN to R1 and R2 and passed EVPN Type-5 routes. Instead to spice it up I stitched the routes into vpnv4 and use MPLS as my dataplane between these two DC’s. I wouldn’t recommend this in a greenfield network but if you have an existing MPLS network with vpnv4 routes that must communicate it’s a nice option Juniper is providing.

R1 (vMX) config:

set routing-instances IRBVL98 instance-type vrf
set routing-instances IRBVL98 protocols evpn ip-prefix-routes advertise direct-nexthop
set routing-instances IRBVL98 protocols evpn ip-prefix-routes encapsulation vxlan
set routing-instances IRBVL98 protocols evpn ip-prefix-routes vni 2098
set routing-instances IRBVL98 interface irb.98
set routing-instances IRBVL98 route-distinguisher 10.0.0.3:2
set routing-instances IRBVL98 vrf-target target:1:1
set routing-instances IRBVL98 vrf-table-label

set protocols bgp group VPNV4 traceoptions file BGP-TRACE
set protocols bgp group VPNV4 traceoptions flag normal
set protocols bgp group VPNV4 traceoptions flag update
set protocols bgp group VPNV4 local-address 10.0.0.3
set protocols bgp group VPNV4 family inet-vpn unicast
set protocols bgp group VPNV4 peer-as 65003
set protocols bgp group VPNV4 neighbor 10.0.0.6

R2 (vMX) config:

set routing-instances IRBVL99 instance-type vrf
set routing-instances IRBVL99 protocols evpn ip-prefix-routes advertise direct-nexthop
set routing-instances IRBVL99 protocols evpn ip-prefix-routes encapsulation vxlan
set routing-instances IRBVL99 protocols evpn ip-prefix-routes vni 2099
set routing-instances IRBVL99 interface irb.99
set routing-instances IRBVL99 route-distinguisher 10.0.0.6:2
set routing-instances IRBVL99 vrf-target target:1:1
set routing-instances IRBVL99 vrf-table-label

set protocols bgp group VPNV4 traceoptions file BGP-TRACE
set protocols bgp group VPNV4 traceoptions flag normal
set protocols bgp group VPNV4 traceoptions flag update
set protocols bgp group VPNV4 local-address 10.0.0.6
set protocols bgp group VPNV4 family inet-vpn unicast
set protocols bgp group VPNV4 peer-as 65003
set protocols bgp group VPNV4 neighbor 10.0.0.3

What’s interesting is the same VRF type is used for either EVPN or L3VPN on MX. If I were to enable evpn signaling between R1 and R2 and disable inet-vpn, routes would go from R1<-> R2 as EVPN and VxLAN would be the dataplane encapsulation.

You can see that here in a packet capture I did on the pure P router BB01. And all I did was disable inet-vpn address family and enable evpn AF on both PE’s (R1 and R2)

BGP-vpnv4 control-plane

MPLS Encap Dataplane

BGP-EVPN Control-Plane (disabled LSP as well)

VxLAN Encap

The obligatory pings pass moment:

Closing Thoughts

1.) Vlan interfaces acting as my ‘layer one switch’ isn’t something new or inventive but just an idea to throw out there incase anyone else builds virtual labs like I do. Automation is the next step but I reserve automation for repeated tasks and my labs are usually one off’s so I end up just saving the config for a particular scenario and moving on.

2.) RFC 5549 seems like a valid way for not only lab but potentially for production to go for an ‘unnumbered bgp underlay’. Alternatives would be OSPF or even IS-IS if the vendors support unnumbered.

3.) Layer-2 interop on EVPN is something I would try to avoid if possible. I’m unqualified to pass judgement on which vendor is right/wrong but from just a pure operator view I wouldn’t want to support this kind of setup without a lot more testing to see where things can go wrong.

4.) MPLS networks exist so it’s nice Juniper provided a way to stitch BGP-EVPN and BGP-vpnv4. There is more to the story here however when you stitch protocols like this together. In a redundant router setup, a community needs to be used (or route attribute) to prevent the route from potentially coming back and resulting in a loop (ie. OSPF as CE <-> PE and domain-id/domain-vpn-tag ..). But a good first iteration from Juniper.

5.) I shouldn’t allow so many days between blog posts because I keep thinking of new ideas to show different implementations 🙂

Leave a Reply