Configuring ACI Border Gateways

These procedures show how to use the APIC REST API to perform the GUI procedures of the same name in the Cisco APIC Layer 3 Networking Configuration Guide.

Creating Border Gateway Sets and VXLAN Remote Fabrics

  • These are a set of border gateway nodes that are used to connect to the remote VXLAN EVPN fabrics. These BGW nodes could either be part of an ACI pod or be deployed across different pods when the ACI fabric is a multi-pod fabric. All BGWs within a POD as are assigned the same TEP to attract traffic for endpoints within this POD from the remote fabric.

  • Cisco APIC assigns a unique internal anycast TEP for a border gateway set, which is common across all the pods for a border gateway set. In Cisco APIC Release 6.1(1), only one border gateway set can be configured.

Before you begin

This policy assigns a data plane TEP for border gateways in each POD, which is used to communicate with remote non-ACI fabrics. This is the external anycast TEP for the POD. Cisco APIC also allocates one internal anycast TEP for all the border gateways within the fabric.

Procedure

Post with information similar to the following:

POST https://{{IP}}/api/node/mo/uni.xml

<polUni>
    <fvTenant name="infra">
        <vxlanBgwSet name="bgwSetOne" status="deleted" />

        <vxlanBgwSet name="bgwSet">
            <vxlanExtAnycastIP addr="122.1.1.1" podId="1" />
            <vxlanExtAnycastIP addr="122.1.1.2" podId="2" />
        </vxlanBgwSet>
        <vxlanRemoteFabric name="nxosSite1">
            <bgpInfraPeerP addr="10.11.0.3" peerT="vxlan-bgw" ttl="2">
                <bgpAsP asn="110" />
            </bgpInfraPeerP>
            <bgpInfraPeerP addr="10.11.0.5" peerT="vxlan-bgw" ttl="2">
                <bgpAsP asn="110" />
            </bgpInfraPeerP>
            <vxlanRsRemoteFabricToBgwSet tDn="uni/tn-infra/vxlanbgwset-bgwSet" status="" />
        </vxlanRemoteFabric>
        <vxlanRemoteFabric name="nxosSite2">
            <bgpInfraPeerP addr="10.12.0.1" peerT="vxlan-bgw" ttl="2">
                <bgpAsP asn="111" />
            </bgpInfraPeerP>
            <vxlanRsRemoteFabricToBgwSet tDn="uni/tn-infra/vxlanbgwset-bgwSet" status="" />
        </vxlanRemoteFabric>
    </fvTenant>
</polUni>

Configuring a VXLAN Infra L3Out

The VXLAN infra L3Out configuration allows you to select the ACI Border Gateway nodes and interfaces to establish EBGP underlay adjacencies with the external network devices. This is required to exchange underlay reachability information with the remote NX-OS Border Gateways and establish the overlay EVPN adjacencies with them

You will configure the following pieces when configuring the VXLAN infra L3Out:

  • Configure the ACI Border Gateway Set.

  • Configure the remote VXLAN fabric.

  • Nodes

  • Only border gateways are allowed to be configured as nodes in the VXLAN infra L3Out.

  • Each VXLAN infra L3Out can have border gateways from multiple pods that are part of the same ACI multi-pod fabric.

  • The border gateway can either be configured in a single VXLAN infra L3Out or multiple VXLAN infra L3Outs.

  • When you configure a node profile, you can configure the Router ID and the loopback interface underneath the node. The loopback interface is the control plane TEP on a BGW, which is used for the BGP EVPN peering with the VXLAN gateway on the remote fabric.

  • Interfaces

    • Supported types of interfaces are:
      • Routed interface or sub-interface

  • You will also configure the underlay BGP peer policy in the interfaces tab in the VXLAN infra L3Out. This is the basic underlay configuration that is needed to bring the BGP underlay to exchange the loopback address to a connected device.

  • QoS rules

    • You can configure the VXLAN ingress rule and VXLAN egress rule through the VXLAN QoS

policy in the VXLAN Infra L3Out.

  • If you do not create a VXLAN QoS policy, any ingressing VXLAN traffic is assigned the default QoS level.

You will also configure the underlay and overlay through the VXLAN Infra L3Out:

  • Underlay: BGP peer IP configuration as part of the interface configuration.
  • Overlay: BGP EVPN remote configuration is part of the remote fabric configuration.

Before you begin

  • Ensure that you have registered the leaf node as a new node type border-gateway for it to be displayed as a VXLAN EVPN border gateway.
  • Complete setting up the Border Gateway Set.
  • Complete setting up the Remote VXLAN Fabrics.
  • Configure the VXLAN Infra L3Out custom QoS policy using the procedures.

Procedure

Post with information similar to the following:

POST https://{{IP}}/api/node/mo/uni.xml

<polUni>
    <fvTenant name="infra">
        <l3extOut name="epvnsite" >
            <vxlanExtP/>
            <bgpExtP/>
            <l3extRsProvBgwSet tDn="uni/tn-infra/vxlanbgwset-bgwSet"/>
            <l3extRsEctx tnFvCtxName="overlay-1"/>
            <l3extLNodeP name="bgwLeaf">
                <l3extRsNodeL3OutAtt rtrId="110.0.1.1" rtrIdLoopBack="yes" tDn="topology/pod-1/node-1091">
                    <l3extLoopBackIfP addr="110.0.1.1"/>
                </l3extRsNodeL3OutAtt>
                <l3extRsNodeL3OutAtt rtrId="110.0.1.2" rtrIdLoopBack="yes" tDn="topology/pod-1/node-1092">
                    <l3extLoopBackIfP addr="110.0.1.2"/>
                </l3extRsNodeL3OutAtt>

                <l3extRsNodeL3OutAtt rtrId="110.0.2.1" rtrIdLoopBack="yes" tDn="topology/pod-2/node-1093">
                    <l3extLoopBackIfP addr="110.0.2.1"/>
                </l3extRsNodeL3OutAtt>

                <l3extLIfP name="portIf">
                    <l3extRsPathL3OutAtt addr="102.1.1.1/24" encapScope="local" ifInstT="l3-port" mac="00:22:BD:F8:19:FF" mode="regular" mtu="9000" tDn="topology/pod-1/paths-1091/pathep-[eth1/11]">
                        <bgpPeerP addr="102.1.1.2" addrTCtrl="af-ucast" adminSt="enabled" allowedSelfAsCnt="3" connectivityType="tenant" ctrl="allow-self-as" ttl="1" peerCtrl="bfd">
                            <bgpAsP asn="500"/>
                        </bgpPeerP>
                    </l3extRsPathL3OutAtt>                
                    <l3extRsPathL3OutAtt addr="102.1.2.1/24" encapScope="local" ifInstT="l3-port" mac="00:22:BD:F8:19:FF" mode="regular" mtu="9000" tDn="topology/pod-1/paths-1091/pathep-[eth1/12]" status="">
                        <bgpPeerP addr="102.1.2.2" addrTCtrl="af-ucast" adminSt="enabled" allowedSelfAsCnt="3" connectivityType="tenant" ctrl="allow-self-as" ttl="1" peerCtrl="bfd">
                            <bgpAsP asn="500"/>
                        </bgpPeerP>
                    </l3extRsPathL3OutAtt>

                    <l3extRsPathL3OutAtt addr="102.1.3.1/24" encapScope="local" ifInstT="l3-port" mac="00:22:BD:F8:19:FF" mode="regular" mtu="9000" tDn="topology/pod-1/paths-1092/pathep-[eth1/13]">
                        <bgpPeerP addr="102.1.3.2" addrTCtrl="af-ucast" adminSt="enabled" allowedSelfAsCnt="3" connectivityType="tenant" ctrl="allow-self-as" ttl="1" peerCtrl="bfd">
                            <bgpAsP asn="500"/>
                        </bgpPeerP>
                    </l3extRsPathL3OutAtt>
                    
                    <l3extRsPathL3OutAtt addr="102.2.1.1/24" encapScope="local" ifInstT="l3-port" mac="00:22:BD:F8:19:FF" mode="regular" mtu="9000" tDn="topology/pod-2/paths-1093/pathep-[eth1/11]">
                        <bgpPeerP addr="102.2.1.2" addrTCtrl="af-ucast" adminSt="enabled" allowedSelfAsCnt="3" connectivityType="tenant" ctrl="allow-self-as" ttl="1" peerCtrl="bfd">
                            <bgpAsP asn="500"/>
                        </bgpPeerP>
                    </l3extRsPathL3OutAtt>                
                </l3extLIfP>
            </l3extLNodeP>
            <l3extInstP name="vxlanInstP"/>
        </l3extOut>
    </fvTenant>
</polUni>

Configuring a VXLAN VRF Stretch

Using the procedure in this section, you can stretch tenant VRF(s) between the ACI and VXLAN EVPN domains. This ensures that routed communications for the tenants between those domains can happen by leveraging the VXLAN data-plane encapsulation. Some specific deployment considerations when stretching a tenant VRF are.

  • User tenant VRFs that are stretched are associated to a BGW set, which are associated to the VXLAN infra L3Out.

  • Only one VXLAN VRF L3Out is supported on each VRF. This is used to stretch the VRF towards a BGW.

Before you begin

  • Review the Guidelines and Limitations for ACI Border Gateways.
  • Configure the VXLAN Gateway infra L3Out using the procedures.

Procedure

Post with information similar to the following:

POST https://{{IP}}/api/node/mo/uni.xml

<polUni>
    <fvTenant name="vxlan0">
        <l3extOut name="vxlan_vrf_vxlan0_vxlan0ctx0">
            <l3extVxGwFabrics>
                <l3extConsBgwSet name="bgwSet"/>
                <l3extVxGwRemoteFabric name="nxosSite1"/>
                <l3extVxGwRemoteFabric name="nxosSite2"/>
            </l3extVxGwFabrics>
            <l3extRsEctx tnFvCtxName="vxlan0ctx0"/>
            <l3extInstP name="vxlan_vrf_vxlan0_vxlan0ctx0_vxlanInstP"/>
        </l3extOut>
    </fvTenant>
</polUni>

Configuring a VXLAN Bridge Domain Stretch

Using the procedure in this section, you can stretch tenant bridge domain (s) between the ACI and VXLAN EVPN domains. This ensures that bridged communications for the tenants between those domains can happen by leveraging VXLAN data-plane encapsulation.

Before you begin

  • Review the Guidelines and Limitations for ACI Border Gateways.
  • Configure the VXLAN Gateway infra L3Out using the procedures.

Procedure

Post with information similar to the following:

POST to 
https://{{IP}}/api/node/mo/uni.xml

<polUni>
    <fvTenant name="vxlan0">
        <fvBD name="vxlan0ctx0BD0">
            <fvVxGwFabrics>
                <fvConsBgwSet name="bgwSet"/>
                <fvVxGwRemoteFabric name="nxosSite1"/>
                <fvVxGwRemoteFabric name="nxosSite2"/>
            </fvVxGwFabrics>
        </fvBD>
    </fvTenant>
</polUni>

Configuring a VXLAN VRF Stretch with A Route-Map Configuration

Starting from Cisco APIC 6.1(2), the ACI Border Gateway feature also supports VRF level route-maps that can be configured on the stretched VRFs. These Route-maps are applicable for all the remote fabrics that are associated to the border gateway set. The route-map set rules are configured with the route control profile policies and the action rule profiles.

Using the procedure in this section, you can configure a VXLAN VRF Stretch by using the rest API to specify the outbound and inbound route-maps.

NOTE This is an optional configuration. If you do not configure import route-maps, all the routes received from remote VXLAN EVPN fabrics are accepted. If you do not configure export route-maps, all the local bridge domain subnets are advertised to the remote VXLAN EVPN fabrics that are associated to the border gateway set.

Following are the list of match and set clauses that are supported by both the inbound route-map and the outbound route-map:

Supported Match Clauses • IP Prefix List • AS-Path • Community • Extended Community (match on color extended community is not supported) • Regex Community • Regex Extended Community

Supported Set Clauses • Community • Extended Community • Weight • Preference • Metric

Before you begin

  • Review the Guidelines and Limitations for ACI Border Gateways.
  • Configure the VXLAN Gateway infra L3Out using the procedures.

Procedure

Post with information similar to the following:

POST to 
https://{{IP}}/api/node/mo/uni.xml

<polUni>
    <fvTenant  name="t1" >
        <l3extOut enforceRtctrl="export" mplsEnabled="no" name="userl3out_1" >
            <rtctrlProfile autoContinue="no" name="RT2" type="combinable">
                <rtctrlCtxP action="permit" name="CtxP2" order="0" >
                    <rtctrlRsCtxPToSubjP tnRtctrlSubjPName="Rule2" />
                </rtctrlCtxP>
            </rtctrlProfile>
            <rtctrlProfile autoContinue="no" name="RT1" type="combinable">
                <rtctrlCtxP action="permit" name="CtxP1" nameAlias="" order="0" >
                    <rtctrlRsCtxPToSubjP tnRtctrlSubjPName="Rule1" />
                </rtctrlCtxP>
            </rtctrlProfile>
            <l3extRsL3DomAtt tDn="uni/l3dom-L3Dom" />
            <l3extRsEctx tnFvCtxName="vrf1" />
            <l3extVxGwFabrics>
                <l3extRsVxGwToRtProfile annotation="" direction="import" tDn="uni/tn-t1/prof-RT2" userdom=":all:"/>
                <l3extRsVxGwToRtProfile annotation="" direction="export" tDn="uni/tn-t1/prof-RT1" userdom=":all:"/>
                <l3extVxGwRemoteFabric name="remote_fabric1">
                </l3extVxGwRemoteFabric>
                <l3extVxGwRemoteFabric name="remote_fabric2">
                </l3extVxGwRemoteFabric>
                <l3extConsBgwSet name="bgwSet2" nameAlias="" />
            </l3extVxGwFabrics>
            <l3extInstP floodOnEncap="disabled" matchT="AtleastOne" name="instp3" pcEnfPref="unenforced" prefGrMemb="exclude">
                <fvRsCustQosPol annotation="" tnQosCustomPolName="" userdom="all"/>
            </l3extInstP>
            <bgpExtP/>
            <vxlanExtP/>
        </l3extOut>
    </fvTenant>
</polUni>

Configuring a VXLAN Infra L3Out With VXLAN Site ID

Starting from Cisco APIC 6.1(2), you must configure a site ID. You will not be able to configure the border gateway set policy if you do not have this site ID.

NOTE If you have already configured the ACI Border Gateway feature for Cisco APIC 6.1(1), and upgrade to Cisco APIC 6.1(2) without creating a VXLAN site ID a fault is generated for all the stretched VRFs and bridge domains.

Procedure

Post with information similar to the following:

POST to 
https://{{IP}}/api/node/mo/uni.xml

<polUni>
   <fvTenant name="infra">
      <vxlanSite name="ACI-SJC" descr="San Jose ACI Data Center" id="100" status=""/>
      <vxlanBgwSet name="bgwSet2" status="">
         <vxlanExtAnycastIP podId="1" addr="192.168.12.1/32" />
         <vxlanExtAnycastIP podId="2" addr="192.168.12.2/32" />
      </vxlanBgwSet>
      <vxlanRemoteFabric name="remote_fabric1" status="">
         <bgpInfraPeerP addr="1.1.1.1" ctrl="send-com,send-ext-com" peerT="vxlan-bgw" ttl="5" status="">
            <bgpAsP annotation="" asn="200"/>
         </bgpInfraPeerP>
         <bgpInfraPeerP addr="2.2.2.2" ctrl="send-com,send-ext-com" peerT="vxlan-bgw" ttl="5" status="">
            <bgpAsP annotation="" asn="200"/>
         </bgpInfraPeerP>
         <vxlanRsRemoteFabricToBgwSet tDn="uni/tn-infra/vxlanbgwset-bgwSet2" status="" />
      </vxlanRemoteFabric>
      <vxlanRemoteFabric name="remote_fabric2" status="">
         <bgpInfraPeerP addr="3.3.3.3" ctrl="send-com,send-ext-com" peerT="vxlan-bgw" ttl="5" status="">
            <bgpAsP annotation="" asn="300"/>
         </bgpInfraPeerP>
         <bgpInfraPeerP addr="4.4.4.4" ctrl="send-com,send-ext-com" peerT="vxlan-bgw" ttl="5" status="">
            <bgpAsP annotation="" asn="300"/>
         </bgpInfraPeerP>
         <vxlanRsRemoteFabricToBgwSet tDn="uni/tn-infra/vxlanbgwset-bgwSet2" status="" />
      </vxlanRemoteFabric>
      <l3extOut name="vxlanL3out" status="">
         <vxlanExtP />
         <l3extRsEctx tnFvCtxName="overlay-1" />
         <l3extRsProvBgwSet tDn="uni/tn-infra/vxlanbgwset-bgwSet2" status="" />
         <l3extLNodeP name="Node102" status="">
            <l3extRsNodeL3OutAtt rtrId="102.102.102.2" rtrIdLoopBack="no" tDn="topology/pod-1/node-102" status="">
               <l3extLoopBackIfP addr="102.102.102.3" status="" />
            </l3extRsNodeL3OutAtt>
            <l3extLIfP name="LIfP11">
               <l3extRsPathL3OutAtt addr="102.102.11.11/24" ifInstT="sub-interface" tDn="topology/pod-1/paths-102/pathep-[eth1/11]" encap="vlan-13" />
            </l3extLIfP>
         </l3extLNodeP>
         <l3extLNodeP name="Node101" status="">
            <l3extRsNodeL3OutAtt rtrId="101.101.101.1" rtrIdLoopBack="no" tDn="topology/pod-1/node-101" status="">
               <l3extLoopBackIfP addr="101.101.101.2" status="" />
            </l3extRsNodeL3OutAtt>
            <l3extLIfP name="LIfP9">
               <l3extRsPathL3OutAtt addr="101.101.9.9/24" ifInstT="sub-interface" tDn="topology/pod-1/paths-101/pathep-[eth1/9]" encap="vlan-13">
                  <bgpPeerP addr="101.101.11.1">
                     <bgpAsP asn="111" />
                  </bgpPeerP>
               </l3extRsPathL3OutAtt>
            </l3extLIfP>
         </l3extLNodeP>
         <l3extInstP name="InstP" />
         <bgpExtP />
      </l3extOut>
   </fvTenant>
</polUni>

Configuring the EVPN VXLAN Selectors

Selectors are configured under each ESG with a variety of matching criteria to classify endpoints to the ESG. Starting with Cisco APIC 6.1(2), two new selectors have been added to classify endpoints and external destinations learned from remote VXLAN EVPN fabrics.

VXLAN Stretched Bridge Domain Selectors

Use this selector to classify all the L2 MAC addresses learned from the remote VXLAN fabrics into a corresponding ESG. This selector can be configured only for bridge domains that are VXLAN stretched. The endpoints from all the remote fabrics belonging to this bridge domain are classified as part of the same ESG.

VXLAN External Subnet Selector

Use this selector to classify EVPN Type-5 prefixes received from a remote VXLAN fabric into a corresponding ESG. You cannot have the same prefix configured under an external subnet selector and an external EPG selector under a local L3Out. If you have an overlap, the longest prefix match determines the classification of the prefix. You cannot configure the default (0.0.0.0/0) prefix as VXLAN external subnet selector. A specific prefix configuration is the preferred approach. As a workaround, 0.0.0.0/1 or 128.0.0.0/1 can be used if the Catch All entry is required.

Procedure

Use this procedure to create a VXLAN external subnet selector and the VXLAN stretched bridge domain selector.

Post with information similar to the following, where 'bd1' is the VXLAN stretched BD:

POST to https://{{IP}}/api/node/mo/uni.xml

<fvTenant name=“tn1”>
    <fvAp name="ap">
      <fvESg name="esg1" status=''>
        <fvVxGwBdStretchSelector  bdName="bd1" status=''/>
        <fvExternalSubnetSelector ip="4.4.4.0/24" status=''/>
        <fvRsScope tnFvCtxName=“vrf1”/>
      </fvESg>
    </fvAP>
  </fvTenant>