CI/CD Example

Introduction

Continuous Integration/Continuous Deployment is a tool to automate the provisioning and management of the technology stack, it translates manual tasks into reusable, robust, distributable code, and relies on practices that have been successfully used for years in software development (version control, automated testing, release tagging, continuous delivery, etc.). What you can get from it are much higher delivery speed and a significant reliability boost. An example of using Gitlab with Terraform and ACI has been described in the section CI/CD Example. In this section, we will show you how to apply CI/CD pipeline to NX-OS environment with NDFC and ansible.

Note that the example that is shown in this section is proof of concept which is tooling agnostic, the same concepts can be applied to any automation infrastructure

Getting started

Version Control System(VCS) like Github is the crux of the CI/CD pipeline, keeping track of change and easily forwarding(merging) and reverting the change. Most VCSs have CI/CD pipeline system built-in which can be used to test your code and configuration before putting it on the production network. There is a wide range of tools available which can be used to serve the purpose. In this section, we will use GitLab to demonstrate how to do it.

In this example, the configuration (ansible vars) and playbooks are stored in the same repository. The idea of this example is to apply the configuration change on a dedicated testing environment or staging environment and verify it before applying it to the production environment.

The staging envrionment could be built with dedicate physical Nexus switches or with Cisco Modeling Labs and Nexus 9000v.

The above flow is one of the ways to implement the CI/CD pipeline on NX-OS fabric:

  1. Changes should always be committed to a branch before being merged into the protected branch. The protected branch is usually the main or master branch which developers can't make irrevocable changes as whatever goes to the protected branch will be applied to the production network.
  2. When changes are made, the DevOps team needs to create a pull request (PR) to merge the changes to the protected branch. The CI pipeline will be triggered along with the PR to apply the changes on the staging environment to test and verify the changes.
  3. At this point, DevOps teams have a chance to review the changes and make sure the CI pipeline is successful, then merge the changes to the protected branch.
  4. Merging the protected branch triggers the CD pipeline to deploy the changes to your production environment and also verify the changes in case there are any issues that are not caught in the previous pipeline.

Step 1: Initializing Gitlab Repository

Follow Step 1 to Step 2 of CI/CD Example in the ACI Section to initialize Gitlab Repo and register Gitlab Runner. In Step 2, give docker,nxos as tags for the runner.

Step 2: Cloning the example repository

Clone the example repository to local:

git clone https://github.com/dsx1123/ndfc-pipeline-example.git

Step 3: Creating the encrypted password

As this code will be stored centrally in a GitLab repository, it is a good practice to encrypt any credentials or sensitive data with ansible-vault or pass them as environment variables. This example uses ansible-vault to encrypt NDFC login and switch login. However, the vault password must be stored securely in GitLab and passed to the runner as environment variables when executing Ansible Playbooks.

Create the file vault.yaml in the folder group_vars/all, copy the below content to it and replace <your password> with your actual password:

ndfc_password: <your password>
switch_password: <your password>

Save the file and use the below command to encrypt the logins:

ansible-vault encrypt vault.yaml

enter the encryption vault password and take note of the password that is used.

Now that the logins have been encrypted, the same encryption password must be provided to Gitlab to decrypt the logins while playbooks are executed in the runner. Open your project on Gitlab, navigate to Settings > CI/CD, and expand Variables:

Click add variable to add variable ANSIBLE_VAULT_PASSWORD, add the vault encryption password that is used in the above step and uncheck the Protected variable:

Step 4: Modifying fabric setup detail

This example uses different fabrics for staging and production networks, NDFC addresses/hostnames are defined in host.stage.yml and host.prod.yml, and modify the ansible_host in each file to match with your environment:

---
ndfc:
  children:
    prod:
      hosts:
        shdu-ndfc-1:
          ansible_connection: ansible.netcommon.httpapi
          ansible_host: shdu-ndfc-1
          ansible_httpapi_use_ssl: true
          ansible_httpapi_validate_certs: false
          ansible_python_interpreter: auto_silent
          ansible_network_os: cisco.dcnm.dcnm
          ansible_user: admin
          ansible_password: "{{ ndfc_password }}"

A common configuration like VRF and Network are defined in group_vars/all/overlay.yml while environment-specific configurations are defined in group_vars/[stage|prod] folder. Modify them based on your environment.

Step 5: Pushing the code

You have managed to encrypt any sensitive data in your code with ansible-vault in Step 3 and modify the ansible variables to match your environment in Step 4. You are now ready to push the modified code to GitLab.

You can find the right remote destination by navigating to your repository in Gitlab and expanding Clone. Copy the link provided under HTTPS or SSH as shown in the following image:

Navigate to the folder that you cloned in step 1 in your terminal. Change the remote origin from GitHub to GitLab by executing the following commands:

git remote rename origin old-origin
git remote add origin https://gitlab.com/<id>/<repo>.git
git add .
git commit -am "initial gitlab commit"
git push -u origin --all

Step 6: Creating pipelines

Go to the project folder and create a new branch add-pipeline:

git checkout -b add-pipeline

Create a file .gitlab-ci.yml in the root of this folder and copy below to it:

---
variables:
  ENV: stage
default:
  before_script:
    - echo $ANSIBLE_VAULT_PASSWORD > .vault_pass
stages:
  - review
  - lint
  - deploy
  - verify

review:stage:
  stage: review
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  tags:
    - docker
    - nxos
  before_script:
  script:
    - echo "ENV=stage" >> review.env
  artifacts:
    reports:
      dotenv: review.env

review:prod:
  stage: review
  rules:
    - if: '$CI_COMMIT_BRANCH== "main"'
  tags:
    - docker
    - nxos
  before_script:
  script:
    - echo "ENV=prod" >> review.env
  artifacts:
    reports:
      dotenv: review.env

yamllint:
  stage: lint
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  image:
    name: cytopia/yamllint:1.26
    entrypoint: [""]
  tags:
    - docker
    - nxos
  script:
    - yamllint -d relaxed .

ansible-lint:
  stage: lint
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  image:
    name: cytopia/ansible-lint:5
    entrypoint: [""]
  tags:
    - docker
    - nxos
  script:
    - ansible-galaxy collection install -r collections/requirements.yml
    - ansible-lint -p build.yml

deploy:
  stage: deploy
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  image:
    name: cytopia/ansible:2.11
    entrypoint: [""]
  tags:
    - docker
    - nxos
  script:
    - echo "deploy on $ENV"
    - python3 -m pip install requests
    - ansible-galaxy collection install -r collections/requirements.yml
    - ansible-playbook --version
    - chmod -v 700 $(pwd)
    - ansible-playbook -i hosts.$ENV.yml build.yml --vault-password-file .vault_pass

verify:
  stage: verify
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: delayed
      start_in: '5 seconds'
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: delayed
      start_in: '5 seconds'
  image:
    name: cytopia/ansible:2.11
    entrypoint: [""]
  tags:
    - docker
    - nxos
  script:
    - echo "verify on $ENV"
    - python3 -m pip install requests
    - ansible-galaxy collection install -r collections/requirements.yml
    - ansible-playbook --version
    - chmod -v 700 $(pwd)
    - ansible-playbook -i hosts.$ENV.yml verify.yml  --vault-password-file .vault_pass

First step review is to set the deployment envrionment, as you can see review:stage is triggered only on merge request, while review:prod is triggered when the commit branch is main. The deploy and verfiy stage is always triggered but will read from different a host file based on the ENV that is set in review stage.

For more information about .gitlab-ci.yml and other examples, see: GitLab CI/CD

Add the file .gitlab-ci.yml to the local branch and then push it to the remote repository:

git add .gitlab-ci.yml
git commit -m "add pipeline"
git push --set-upstream origin add-pipeline

Step 7: Creating a Pull Request

Go to Gitlab and navigate to Merge Requests, click New merge Request:

Keep everything as default then click Create merge request

A CI pipeline is triggered once the merge request is created, check the running pipeline by navigating to CI/CD > Pipelines:

Click the Running status of the pipeline to view the detailed information:

Navigate to NDFC to verify that the configuration was deployed successfully:

Step 8: Merge to the main branch

Once CI pipeline is finished, go to Gitlab and navigate to Merge requests, click the merge request you just created then click merge:

A second pipeline will be triggered and the same configuration will be deployed to the production network:

Once it is finished, verify the same on the production network:

Step 9: Adding Configuration

Now that the initial configuration is pushed to both the staging and the production network. Modifying any configuration is similar to Step 7 and Step 8. Any change should be tested on the staging environment first and then deployed to the production environment. As an example, let's add a new network. First, create a new branch add-network:

git checkout main
git pull
git checkout -b add-network

Open file group_vars/all/overlay.yml and append the below configuration to the end of the file:

  - net_name: network_vnf
    vrf_name: *refvrf_red
    net_id: 30024
    vlan_id: 2203
    vlan_name: network_vnf_vlan2203
    gw_ip_subnet: "10.1.4.1/24"
    attach_group: esxi

Save the file and commit the branch to the remote repository:

git add overlay.yml
git commit -m "add network for VNF"
git push --set-upstream origin add-network

Then follow the same steps in Step 7 and Step 8 to create a Pull Request then merge the change to the main branch. The network_vnf_vlan2203 will be deployed to the staging environment and the production environment:

Congratulations! You have gone through the exercise of using the CI/CD pipeline to manage your code and network configuration with Ansible and NDFC.