Integrated DC Network, Infrastructure & Security Automation - Part 1

Overview

When deploying applications to a data center, there are certain elements of common configuration that would be used differently by different operational teams. Traditionally, each team would rely on this information being passed manually in ad-hoc manner. In a modern Infrastructure-as-Code (IaC) approach, this configuration can be defined once but leveraged consistently across multiple domains such as networking, infrastructure and security automatically. In this example, we are using a single Terraform plan to deploy a set of new DC networks, clone and configure virtual machines to use these networks and then create matching firewall objects and rules – all from the same simplified intent defined as a minimal set of variables. Finally, we will use the Intersight Service for Terraform to provide Terraform Cloud with secure managed API access to traditionally isolated domain managers within the on-premises data center. This will allow Terraform to reach both public and private infrastructure equally and build a consistent hybrid cloud infrastructure operating model.

Requirements

The Infrastructure-as-Code environment will require the following:

• GitHub Repository for Terraform plans, modules and variables as JSON files
• Terraform Cloud for Business account with a workspace associated to the GitHub repository above
• Cisco Intersight (SaaS) platform account with sufficient Advantage licensing
• An Intersight Assist appliance VM connected to the Intersight account above

This example will then use the following on-premise domain managers. These will need to be fully commissioned and a suitable user account provided for Terraform to use for provisioning.

• Cisco Data Center Network Manager (DCNM)
• VMware vCenter
• Cisco Firepower Management Center (FMC)

Note: The FMC security automation component has been moved to a 2nd GitHub repository and will now be run from a 2nd, linked Terraform workspace. This was required to allow the vCenter component to clone any number (count) of VMs. This prevented the FMC module from predicting the number of resources required until after the vCenter module has been executed. Instead the FMC component is now run from a 2nd linked workspace. Any updates to the main DCNM/vCenter workplace will trigger the FMC workplace to run and update as necessary.

The DC Networking module makes the following assumptions:

• An existing Nexus 9000 switch based VXLAN fabric has already been deployed and that it is actively managed through a DCNM instance.
• The DCNM server is accessible by HTTPS from the Intersight Assist VM.
• An existing VRF is available to use for new L3 VXLAN networks. Any dynamic routing/peering to external devices (including firewalls) have already be configured as necessary.
• In this example, the firewalls will be used as an external edge device and all necessary static or dynamic routing is already present (i.e., BGP used to learn a default route from the firewall and to advertise L3 network subnets back to the firewall). Alternatively, the firewall could be used in conjunction with policy-based redirect (PBR) or elastic service redirection (ESR) configuration.
• Suitable IP subnets (at least /29) are available to be assigned to each new L3 network.
• Suitable VLAN IDs are available to be assigned to each new L3 network.
• The following variables are defined within the Terraform Workspace. These variables should not be configured within the public GitHub repository files.
  • DCNM account username (dcnm_user)
  • DCNM account password (dcnm_password)
  • DCNM URL (dcnm_url)

The vCenter module makes the following assumptions:

• A group of VMware host servers are configured as a single VMware server cluster within a logical VMware Data Center, managed from an existing vCenter instance.
• The vCenter server is accessible by HTTPS from the Intersight Assist VM.
• VMware host servers have commissioned and are physically patched to trunked switch ports (or VPCs) on the VXLAN fabric access (leaf) switches.
• A distributed virtual switch (DVS) has been configured across all servers in the cluster and their physical ethernet uplink ports.
• A suitable VMware datastore is available for use by new cloned VMs’ virtual hard drives.
• A suitable virtual machine template is available to clone as required.
• The following variables are defined within the Terraform Workspace. These variables should not be configured within the public GitHub repository files.
  • vCenter account username (vcenter_user)
  • vCenter account password (vcenter_password)
  • vCenter server IP/FQDN (vcenter_server)

Note: The FMC security automation component has been moved to a 2nd GitHub repository and will now be run from a 2nd, linked Terraform workspace.

Link to Github Repositories

https://github.com/cisco-apjc-cloud-se/ist-dcn-vcenter https://github.com/cisco-apjc-cloud-se/ist-vm-fmc-sync

Steps to Deploy Use Case

1. In GitHub, create a new, or clone the example GitHub repository(s)
2. Customize the examples Terraform files & JSON variables as required

3. In Intersight, configure a Terraform Cloud target with suitable user account and token

4. In Intersight, configure a Terraform Agent target with suitable managed host URLs/IPs. This list of managed hosts must include the IP addresses for the DCNM, FMC and vCenter servers as well as access to common GitHub domains in order to download hosted Terraform providers.

5. In Terraform Cloud for Business, create a new Terraform Workspace and associate to the GitHub repository.

6. In Terraform Cloud for Business, configure the workspace to the use the Terraform Agent pool configured from Intersight.

7. In Terraform Cloud for Business, configure the necessary user account variables for the DCNM and vCenter servers.

Execute Deployment

In Terraform Cloud for Business, queue a new plan to trigger the initial deployment. Any future changes to pushed to the GitHub repository will automatically trigger a new plan deployment.

Results

If successfully executed, the Terraform plan will result in the following configuration for each domain manager.

DCNM Module

• New Layer 3 VXLAN networks, as defined in the “dc_networks” JSON object, within the existing VRF, each with the following configuration:
  • Name
  • Anycast Gateway IPv4 Address/Mask
  • VXLAN VNI ID
  • Layer 2 VLAN ID

• The new VXLAN networks trunked to the switches and ports defined in the “svr_cluster” JSON object

• [Updated Oct-2021] The DCNM module will now take a VPC Interface variable and build a set of vPC interfaces that will then be used by the Networks.

vCenter Module

• New distributed port groups, within the existing distributed virtual switch for each new network defined in the DCNM module.
  • Each new distributed port group will use the same name and VLAN ID as the DCNM L3 Network

• A number of cloned VMs as defined in the “vm_group_a” and “vm_group_b” JSON objects. These use the existing vCenter cluster, datastore and template details provided.
  • Each VM will be assigned a unique name and suffix combination based on the number of VMs defined (i.e. xxx-1, xxx-2).
  • Each VM will have a single virtual NIC (vNIC) connected to a specific new L3 Networks and associated new distributed port group above.
  • Each VM’s operating system will be configured with a unique hostname and suffix combination based on the number of VMs defined (i.e. xxx-1, xxx-2).
  • Each VM’s operating system will be configured with a unique static IP address automatically defined by the attached L3 Network’s IP subnet and an offset based on the number of VMs defined (i.e. .2, .3, .4 etc.)

Expected Day 2 Changes

Changes to the variables defined in the JSON files will result in dynamic, stateful updates across all domains. For example,

• Increasing the number of servers in group will result in a newly clone VM, and a matching host object in FMC and that object added to the existing server network group object. This means the new server would automatically inherit any existing security policies on the firewall(s) that reference this group object.

  "vm_group_a": {
    "group_size": 4,
    "name": "ist-svr-a",
    "host_name": "ist-svr-a",
    "num_cpus": 2,
    "memory": 1024,
    "network_id": "ist-network-a",
    "domain": "mel.ciscolabs.com",
    "dns_list": [
      "64.104.123.245",
      "171.70.168.183"
    ]
  },

• If a new server is added the VMware cluster, the DC networking configuration would need to be updated to ensure a consistent configuration across all members of the cluster. With this IaC lead approach, this can be achieved be allocating new switch ports on the N9Ks and adding these to the existing configuration and the Terraform plan executed.

  "svr_cluster": {
    "DC3-N9K1": {
      "name": "DC3-N9K1",
      "attach": true,
      "switch_ports": [
        "Ethernet1/1",  # Host 1 Uplink Port 1
        "Ethernet1/2"   # Host 2 Uplink Port 1
      ]
    },
    "DC3-N9K2": {
      "name": "DC3-N9K2",
      "attach": true,
      "switch_ports": [
        "Ethernet1/1",  # Host 1 Uplink Port 2
        "Ethernet1/2"   # Host 2 Uplink Port 2
      ]
    }
  },

• If a new server is added to the cluster, but is located in new rack (with its own switches) the existing configuration would be extended as follows:

  "svr_cluster": {
    "DC3-N9K1": {
      "name": "DC3-N9K1",
      "attach": true,
      "switch_ports": [
        "Ethernet1/1"  # Host 1 Uplink Port 1
      ]
    },
    "DC3-N9K2": {
      "name": "DC3-N9K2",
      "attach": true,
      "switch_ports": [
        "Ethernet1/1"  # Host 1 Uplink Port 2
      ]
    },
    "DC3-N9K3": {
      "name": "DC3-N9K3",
      "attach": true,
      "switch_ports": [
        "Ethernet1/1"  # Host 2 Uplink Port 1
      ]
    },
    "DC3-N9K4": {
      "name": "DC3-N9K2",
      "attach": true,
      "switch_ports": [
        "Ethernet1/1"  # Host 2 Uplink Port 2
      ]
    }
  },

• Each new server would require its own vPC interface.

  "vpc_interfaces": {
    "vpc5": {
      "name": "vPC5",
      "vpc_id": 5,
      "switch1": {
        "name": "DC3-LEAF-1",
        "ports": ["Eth1/5"]
      },
      "switch2": {
        "name": "DC3-LEAF-2",
        "ports": ["Eth1/5"]
      }
    }
  },

Integrated DC Network, Infrastructure & Security Automation - Part 1

Overview

When deploying applications to a data center, there are certain elements of common configuration that would be used differently by different operational teams. Traditionally, each team would rely on this information being passed manually in ad-hoc manner. In a modern Infrastructure-as-Code (IaC) approach, this configuration can be defined once but leveraged consistently across multiple domains such as networking, infrastructure and security automatically. In this example, we are using a single Terraform plan to deploy a set of new DC networks, clone and configure virtual machines to use these networks and then create matching firewall objects and rules – all from the same simplified intent defined as a minimal set of variables. Finally, we will use the Intersight Service for Terraform to provide Terraform Cloud with secure managed API access to traditionally isolated domain managers within the on-premises data center. This will allow Terraform to reach both public and private infrastructure equally and build a consistent hybrid cloud infrastructure operating model.

Requirements

The Infrastructure-as-Code environment will require the following:

• GitHub Repository for Terraform plans, modules and variables as JSON files
• Terraform Cloud for Business account with a workspace associated to the GitHub repository above
• Cisco Intersight (SaaS) platform account with sufficient Advantage licensing
• An Intersight Assist appliance VM connected to the Intersight account above

This example will then use the following on-premise domain managers. These will need to be fully commissioned and a suitable user account provided for Terraform to use for provisioning.

• Cisco Data Center Network Manager (DCNM)
• VMware vCenter
• Cisco Firepower Management Center (FMC)

Note: The FMC security automation component has been moved to a 2nd GitHub repository and will now be run from a 2nd, linked Terraform workspace. This was required to allow the vCenter component to clone any number (count) of VMs. This prevented the FMC module from predicting the number of resources required until after the vCenter module has been executed. Instead the FMC component is now run from a 2nd linked workspace. Any updates to the main DCNM/vCenter workplace will trigger the FMC workplace to run and update as necessary.

The DC Networking module makes the following assumptions:

• An existing Nexus 9000 switch based VXLAN fabric has already been deployed and that it is actively managed through a DCNM instance.
• The DCNM server is accessible by HTTPS from the Intersight Assist VM.
• An existing VRF is available to use for new L3 VXLAN networks. Any dynamic routing/peering to external devices (including firewalls) have already be configured as necessary.
• In this example, the firewalls will be used as an external edge device and all necessary static or dynamic routing is already present (i.e., BGP used to learn a default route from the firewall and to advertise L3 network subnets back to the firewall). Alternatively, the firewall could be used in conjunction with policy-based redirect (PBR) or elastic service redirection (ESR) configuration.
• Suitable IP subnets (at least /29) are available to be assigned to each new L3 network.
• Suitable VLAN IDs are available to be assigned to each new L3 network.
• The following variables are defined within the Terraform Workspace. These variables should not be configured within the public GitHub repository files.
  • DCNM account username (dcnm_user)
  • DCNM account password (dcnm_password)
  • DCNM URL (dcnm_url)

The vCenter module makes the following assumptions:

• A group of VMware host servers are configured as a single VMware server cluster within a logical VMware Data Center, managed from an existing vCenter instance.
• The vCenter server is accessible by HTTPS from the Intersight Assist VM.
• VMware host servers have commissioned and are physically patched to trunked switch ports (or VPCs) on the VXLAN fabric access (leaf) switches.
• A distributed virtual switch (DVS) has been configured across all servers in the cluster and their physical ethernet uplink ports.
• A suitable VMware datastore is available for use by new cloned VMs’ virtual hard drives.
• A suitable virtual machine template is available to clone as required.
• The following variables are defined within the Terraform Workspace. These variables should not be configured within the public GitHub repository files.
  • vCenter account username (vcenter_user)
  • vCenter account password (vcenter_password)
  • vCenter server IP/FQDN (vcenter_server)

Note: The FMC security automation component has been moved to a 2nd GitHub repository and will now be run from a 2nd, linked Terraform workspace.

Link to Github Repositories

https://github.com/cisco-apjc-cloud-se/ist-dcn-vcenter https://github.com/cisco-apjc-cloud-se/ist-vm-fmc-sync

Steps to Deploy Use Case

1. In GitHub, create a new, or clone the example GitHub repository(s)
2. Customize the examples Terraform files & JSON variables as required

3. In Intersight, configure a Terraform Cloud target with suitable user account and token

4. In Intersight, configure a Terraform Agent target with suitable managed host URLs/IPs. This list of managed hosts must include the IP addresses for the DCNM, FMC and vCenter servers as well as access to common GitHub domains in order to download hosted Terraform providers.

5. In Terraform Cloud for Business, create a new Terraform Workspace and associate to the GitHub repository.

6. In Terraform Cloud for Business, configure the workspace to the use the Terraform Agent pool configured from Intersight.

7. In Terraform Cloud for Business, configure the necessary user account variables for the DCNM and vCenter servers.

Execute Deployment

In Terraform Cloud for Business, queue a new plan to trigger the initial deployment. Any future changes to pushed to the GitHub repository will automatically trigger a new plan deployment.

Results

If successfully executed, the Terraform plan will result in the following configuration for each domain manager.

DCNM Module

• New Layer 3 VXLAN networks, as defined in the “dc_networks” JSON object, within the existing VRF, each with the following configuration:
  • Name
  • Anycast Gateway IPv4 Address/Mask
  • VXLAN VNI ID
  • Layer 2 VLAN ID

• The new VXLAN networks trunked to the switches and ports defined in the “svr_cluster” JSON object

• [Updated Oct-2021] The DCNM module will now take a VPC Interface variable and build a set of vPC interfaces that will then be used by the Networks.

vCenter Module

• New distributed port groups, within the existing distributed virtual switch for each new network defined in the DCNM module.
  • Each new distributed port group will use the same name and VLAN ID as the DCNM L3 Network

• A number of cloned VMs as defined in the “vm_group_a” and “vm_group_b” JSON objects. These use the existing vCenter cluster, datastore and template details provided.
  • Each VM will be assigned a unique name and suffix combination based on the number of VMs defined (i.e. xxx-1, xxx-2).
  • Each VM will have a single virtual NIC (vNIC) connected to a specific new L3 Networks and associated new distributed port group above.
  • Each VM’s operating system will be configured with a unique hostname and suffix combination based on the number of VMs defined (i.e. xxx-1, xxx-2).
  • Each VM’s operating system will be configured with a unique static IP address automatically defined by the attached L3 Network’s IP subnet and an offset based on the number of VMs defined (i.e. .2, .3, .4 etc.)

Expected Day 2 Changes

Changes to the variables defined in the JSON files will result in dynamic, stateful updates across all domains. For example,

• Increasing the number of servers in group will result in a newly clone VM, and a matching host object in FMC and that object added to the existing server network group object. This means the new server would automatically inherit any existing security policies on the firewall(s) that reference this group object.

  "vm_group_a": {
    "group_size": 4,
    "name": "ist-svr-a",
    "host_name": "ist-svr-a",
    "num_cpus": 2,
    "memory": 1024,
    "network_id": "ist-network-a",
    "domain": "mel.ciscolabs.com",
    "dns_list": [
      "64.104.123.245",
      "171.70.168.183"
    ]
  },

• If a new server is added the VMware cluster, the DC networking configuration would need to be updated to ensure a consistent configuration across all members of the cluster. With this IaC lead approach, this can be achieved be allocating new switch ports on the N9Ks and adding these to the existing configuration and the Terraform plan executed.

  "svr_cluster": {
    "DC3-N9K1": {
      "name": "DC3-N9K1",
      "attach": true,
      "switch_ports": [
        "Ethernet1/1",  # Host 1 Uplink Port 1
        "Ethernet1/2"   # Host 2 Uplink Port 1
      ]
    },
    "DC3-N9K2": {
      "name": "DC3-N9K2",
      "attach": true,
      "switch_ports": [
        "Ethernet1/1",  # Host 1 Uplink Port 2
        "Ethernet1/2"   # Host 2 Uplink Port 2
      ]
    }
  },

• If a new server is added to the cluster, but is located in new rack (with its own switches) the existing configuration would be extended as follows:

  "svr_cluster": {
    "DC3-N9K1": {
      "name": "DC3-N9K1",
      "attach": true,
      "switch_ports": [
        "Ethernet1/1"  # Host 1 Uplink Port 1
      ]
    },
    "DC3-N9K2": {
      "name": "DC3-N9K2",
      "attach": true,
      "switch_ports": [
        "Ethernet1/1"  # Host 1 Uplink Port 2
      ]
    },
    "DC3-N9K3": {
      "name": "DC3-N9K3",
      "attach": true,
      "switch_ports": [
        "Ethernet1/1"  # Host 2 Uplink Port 1
      ]
    },
    "DC3-N9K4": {
      "name": "DC3-N9K2",
      "attach": true,
      "switch_ports": [
        "Ethernet1/1"  # Host 2 Uplink Port 2
      ]
    }
  },

• Each new server would require its own vPC interface.

  "vpc_interfaces": {
    "vpc5": {
      "name": "vPC5",
      "vpc_id": 5,
      "switch1": {
        "name": "DC3-LEAF-1",
        "ports": ["Eth1/5"]
      },
      "switch2": {
        "name": "DC3-LEAF-2",
        "ports": ["Eth1/5"]
      }
    }
  },

Integrated DC Network, Infrastructure & Security Automation - Part 2

Overview

This repository is intended for use as 2nd, linked Terraform workspace. The 1st primary workspace (using the ist-dcn-vcenter GitHub repository) will share its output state to this workspace and be configured to trigger this workspace to run.

The output of the 1st workspace includes two group objects, one for each group of managed VMs. This Terraform plan will use these group details to generate host objects and network group objects dynamically.

Requirements

Note: This workspace expects to be triggered from another workspace using the "ist-dcn-vcenter" GitHub repository. That workspace must be configured first and then configured to share its state with this workspace as well as trigger this workspace to run.

The Infrastructure-as-Code environment will require the following:

• GitHub Repository for Terraform plans, modules and variables as JSON files
• Terraform Cloud for Business account with a workspace associated to the GitHub repository above
• Cisco Intersight (SaaS) platform account with sufficient Advantage licensing
• An Intersight Assist appliance VM connected to the Intersight account above

This example will then use the following on-premise domain managers. These will need to be fully commissioned and a suitable user account provided for Terraform to use for provisioning.

• Cisco Data Center Network Manager (DCNM)
• VMware vCenter
• Cisco Firepower Management Center (FMC)

The Firepower (FMC) module makes the following assumptions:

• An existing FMC server instance has been deployed
• The FMC server is accessible by HTTPS from the Intersight Assist VM.
• The following variables are defined within the Terraform Workspace. These variables should not be configured within the public GitHub repository files.
  • FMC account username (fmc_user)
  • FMC account password (fmc_password)
  • FMC server IP/FQDN (fmc_server)

Note: The FMC security automation component has been moved to a 2nd GitHub repository and will now be run from a 2nd, linked Terraform workspace.

Link to Github Repositories

https://github.com/cisco-apjc-cloud-se/ist-dcn-vcenter https://github.com/cisco-apjc-cloud-se/ist-vm-fmc-sync

Steps to Deploy Use Case

1. Complete the setup for the primary "ist-dcn-vcenter" workspace. This includes setting up Intersight Service for Terraform (IST).
2. In Terraform Cloud for Business, create a new Terraform Workspace and associate to this GitHub repository.

3. In Terraform Cloud for Business, configure the workspace to the use the Terraform Agent pool configured from Intersight.

4. In Terraform Cloud for Business, configure the necessary user account variables for the FMC servers.

5. In Terraform Cloud for Business, under Settings, configure this workspace to be triggered by the primary "ist-dcn-vcenter" workspace.

6. In Terraform Cloud for Business, in the 1st workspace (ist-dcn-vcenter) under Settings, configure this workspace to share its state file with this new workspace (ist-vm-fmc-sync).

Execute Deployment

Any successful runs in the primary "ist-dcn-vcenter" workspace will trigger this workspace to run. Any future changes to pushed to thist GitHub repository or the primary workspace repository will automatically trigger a new plan deployment.

Results

If successfully executed, the Terraform plan will result in the following configuration for each domain manager.

FMC Module

• New host type network objects for each VM defined the vCenter module.
  • Each VM’s name and static IP address will be used to define the host object.

• New network group objects for each group of VM servers

  • The IP addresses from the host objects above will be grouped into a single object. It is expected this object will be used for any firewall rule definitions.

  Note: The FMC provider has an issue removing objects from network groups. As a workaround, the IP addresses of the host objects will be used instead as literal objects in the group.

Intersight Service for Terraform Demo - Basic DCN & VMware Networking Automation

Overview

This is a simpler version of the “Integrated DC Network, Infrastructure & Security Automation” use case that focuses solely on DCNM and vCenter networking automation, specifically the automation of a DCNM-based VXLAN EVPN fabric connecting to a VMware ESXi cluster with a Distributed Virtual Switch.

Requirements

The Infrastructure-as-Code environment will require the following:

• GitHub Repository for Terraform plans, modules and variables as JSON files
• Terraform Cloud for Business account with a workspace associated to the GitHub repository above
• Cisco Intersight (SaaS) platform account with sufficient Advantage licensing
• An Intersight Assist appliance VM connected to the Intersight account above

This example will then use the following on-premise domain managers. These will need to be fully commissioned and a suitable user account provided for Terraform to use for provisioning.

• Cisco Data Center Network Manager (DCNM)
• VMware vCenter

Assumptions

The DC Networking module makes the following assumptions:

• An existing Nexus 9000 switch based VXLAN fabric has already been deployed and that it is actively managed through a DCNM instance.
• The DCNM server is accessible by HTTPS from the Intersight Assist VM.
• An existing VRF is available to use for new L3 VXLAN networks. Any dynamic routing/peering to external devices (including firewalls) have already be configured as necessary.
• Suitable IP subnets (at least /29) are available to be assigned to each new L3 network.
• Suitable VLAN IDs are available to be assigned to each new L3 network.
• The following variables are defined within the Terraform Workspace. These variables should not be configured within the public GitHub repository files.
  • DCNM account username (dcnm_user)
  • DCNM account password (dcnm_password)
  • DCNM URL (dcnm_url)

The vCenter module makes the following assumptions:

• A group of VMware host servers are configured as a single VMware server cluster within a logical VMware Data Center, managed from an existing vCenter instance.
• The vCenter server is accessible by HTTPS from the Intersight Assist VM.
• VMware host servers have commissioned and are physically patched to trunked switch ports (or VPCs) on the VXLAN fabric access (leaf) switches. The configuration of the switch ports is not included in this example, though is covered in other DCNM example use cases.
• A distributed virtual switch (DVS) has been configured across all servers in the cluster and their physical ethernet uplink ports.
• The following variables are defined within the Terraform Workspace. These variables should not be configured within the public GitHub repository files.
  • vCenter account username (vcenter_user)
  • vCenter account password (vcenter_password)
  • vCenter server IP/FQDN (vcenter_server)

Link to Github Repositories

https://github.com/cisco-apjc-cloud-se/ist-vcenter-dcnm

Steps to Deploy Use Case

1. In GitHub, create a new, or clone the example GitHub repository(s)
2. Customize the examples Terraform files & input variables as required
3. In Intersight, configure a Terraform Cloud target with suitable user account and token
4. In Intersight, configure a Terraform Agent target with suitable managed host URLs/IPs. This list of managed hosts must include the IP addresses for the DCNM server as well as access to common GitHub domains in order to download hosted Terraform providers. This will create a Terraform Cloud Agent pool and register this to Terraform Cloud.
5. In Terraform Cloud for Business, create a new Terraform Workspace and associate to the GitHub repository.
6. In Terraform Cloud for Business, configure the workspace to the use the Terraform Agent pool configured from Intersight.
7. In Terraform Cloud for Business, configure the necessary user account variables for the DCNM servers.

Workarounds

October 2021 In this example, both VLAN IDs and VXLAN IDs have been explicity set. These are optional parameters and can be removed and left to DCNM to inject these dynamically from the fabrics' resource pools. However if you chose to use DCNM to do this, Terraform MUST be configured to use a "parallelism" value of 1. This ensures Terraform will only attempt to configure one resource at a time allowing DCNM to allocate IDs from the pool sequentially.

Typically the parallelism would be set in the Terraform cloud workspace environment variables section using the variable name "TFE_PARALLELISM" and value of "1", however this variable is NOT used by Terraform Cloud Agents. Instead the variables "TF_CLI_ARGS_plan" and "TF_CLI_ARGS_apply" must be used with a value of "-parallelism=1"

October 2021 Due to an issue with the Terraform Provider (version 1.0.0) and DCNM API (11.5(3)) the "dcnm_network" resource will not deploy Layer 3 SVIs. This is due to a defaul parameter not being correctly set in the API call. Instead, the Network will be deployed as if the template has the "Layer 2 Only" checkbox set.

There are two workarouds for this

1. After deploying the network(s), edit the network from the DCNM GUI then immediately save. This will set the correct default parameters and these networks can be re-deployed.

2. Instead of the using the "Default_Network_Universal" template, clone and modify it as below. Make sure to set the correct template name in the terraform plan under the dcnm_network resource. Please note that the tag value of 12345 must also be explicity set.

   Original Lines #119-#123

   if ($$isLayer2Only$$ != "true") {
     interface Vlan$$vlanId$$
      if ($$intfDescription$$ != "") {
       description $$intfDescription$$
      }
   

   Modified Lines #119-#125

   if ($$isLayer2Only$$ == "true"){
    }
   else {
   interface Vlan$$vlanId$$
    if ($$intfDescription$$ != "") {
     description $$intfDescription$$
    }
   

Example Input Variables

{
  "vcenter_dc": "CPOC-HX",
  "vcenter_dvs": "CPOC-SE-VC-HX",
  "dcnm_fabric": "DC3",
  "dcnm_vrf": "GUI-VRF-1",
  "cluster_interfaces": {
    "DC3-LEAF-1": {
      "name": "DC3-LEAF-1",
      "attach": true,
      "switch_ports": [
        "Ethernet1/11"
      ]
    },
    "DC3-LEAF-2": {
      "name": "DC3-LEAF-2",
      "attach": true,
      "switch_ports": [
        "Ethernet1/11"
      ]
    }
  },
  "cluster_networks": {
    "IST-NETWORK-1": {
      "name": "IST-NETWORK-1",
      "description": "Terraform Intersight Demo Network #1",
      "ip_subnet": "192.168.1.1/24",
      "vni_id": 32101,
      "vlan_id": 2101,
      "deploy": true
    },
    "IST-NETWORK-2": {
      "name": "IST-NETWORK-2",
      "description": "Terraform Intersight Demo Network #2",
      "ip_subnet": "192.168.2.1/24",
      "vni_id": 32102,
      "vlan_id": 2102,
      "deploy": true
    }
  }
}

Execute Deployment

In Terraform Cloud for Business, queue a new plan to trigger the initial deployment. Any future changes to pushed to the GitHub repository will automatically trigger a new plan deployment.

Results

If successfully executed, the Terraform plan will result in the following configuration:

• New Layer 3 VXLAN network(s) each with the following configuration:

  • Name
  • Anycast Gateway IPv4 Address/Mask
  • VXLAN VNI ID
  • VLAN ID

• New Distributed Port Groups for each VXLAN network defined above

  • Name
  • VLAN ID

Expected Day 2 Changes

Changes to the variables defined in the input variable files will result in dynamic, stateful update to DCNM. For example,

• Adding a Network entry will create a new DCNM Network template instance and deploy this network to the associated switches, as well as trunk to the associated switch interfaces.
• Adding a Network entry will also create a matching distributed port group on the specific VMware distribute switch.
• Adding a new host to the VMware host cluster and distributed switch will ensure the hew host inherits the distributed port groups. Adding the new hosts' interfaces to the "cluster_interfaces" variable will ensure that all necessary VLANs are trunked to the new host.