dmmar/netascode - Cisco Code Exchange

Network as Code. NaC model.

Overview

This repository contains the source files of the student project.

The project introduction you can read from the following link:
http://www.netascode.com/1/

Objectives of the project were aimed at self-education and
improving practical skills in the area of computer networks
using an automation approach as the main tool.

You can use this repo as an example to play and to learn network automation.

Tested network OS: Cisco IOS (IOSv, IOSvL2), Cisco vASA (9.x), JunOS (vSRX 17.3R1) , and VyOS (1.1.8).

Table of project files

Ansible

1.1 Modules - folder contains common [Ansible playbooks] which can be used for dev/test/prod environments
1.2 Python_scripts - Please, refer to ### Step script-diff (optional)
1.3 Roles - folder contains [Ansible Roles and Jinja2 templates] which are used to generate configurations
1.4 inventories - folder contains an identical structure for dev/test/prod environments such as:
1.4.1 CONFIGS - folder contains generated partial and full configs for network devices, for example:
1.4.1.1 BR1-AC1:

[1_basic-config.conf] - a partial config that is specific for Ansible Role [basic_config],

[FINAL_pre.conf] - a full config that combines all partial configs in one,

[FINAL_pre_DIFF.conf] - contains the difference between a generated configuration and current on the device,

[FINAL.conf] that config will apply to the device using Ansible playbook [commit-merge-final-config.yml].

1.4.2 Modules - folder contains specific [Ansible playbooks] for env.
1.4.3 Python_scripts - used to apply a generated configuration from the file [FINAL_pre.conf] on Cisco vASA (HQ-FW1).
1.4.4 Topologies - contains network diagrams L1, L2, L3, etc.
1.4.5 group_vars - common variables for Ansible playbooks, for example: [Cisco.yaml] contains variables and values which are used for devices in [Cisco] group only. Groups can be seen in hosts file.
1.4.6 host_vars - specific variables to each network device which are used for Ansible playbooks.
1.4.7 hosts - Ansible inventory file.

Variables in group_vars and host_vars are used for Ansible Roles/Jinja2 templates to generate configurations.

Docker-runner - contains ansible.cfg for GitLab CI/CD runner, Please, refer to Step 4 (sending and applying new configurations)
GNS3 - folder contains startup configurations for network devices which are used to configure SSH access for Ansible.
Installation_components - Please, refer to Step Docker-containers (optional)
PyATS - contains examples of PyATS testcases
RobotFramework - contains examples of RobotFramework/PyATS united testcases
.gitlab-ci.yml - that file describes GitLab CI/CD pipeline

Overview diagram

Prerequisites

PyCharm CE
Python3
Python virtualenv
pip3
git
Debian10-amd64 server with [Docker] and [docker-compose]
(for NetBox, GitLab CE, and docker-gitlab-runner containers) - optional
GNS3

I personally tested on Ubuntu 18.04.2 LTS, Python3.7, PyCharm CE, and GNS3 2.1.21.

For Ubuntu also needs:

apt-get install python3-setuptools
apt-get install python3-pip

Installation

Step -1 (GNS3)

To play with project (without any changes in inventory files, playbooks and etc.) 
you need to have a GNS3 server for all environments or only for one 
and build exacly the same network which I used for the project.

- Topologies images can be found:

Ansible/inventories/production/Topologies
Ansible/inventories/test/Topologies
Ansible/inventories/development/Topologies

- GNS3 devices startup-configurations:

GNS3/PROD-Startup-CFGs
GNS3/TEST-Startup-CFGs
GNS3/DEV-Startup-CFGs

Also, all network devices in GNS3 must be connected to the out-of-band management network.
I have used a built in GNS3 ethernet switch.

For lazy deployment, you can use [gns3_auto_topology_builder]:

gns3_auto_topology_builder

Step 0

# git clone https://github.com/dmmar/netascode.git

# pip3 install -r requirements.txt

Step 1 (Ansible module napalm-ansible)

# Ansible/ansible.cfg

Configure ansible to use napalm
After napalm installation you need to configure proper path to library at ansible.cfg file. Example is below.

$ napalm-ansible

To ensure Ansible can use the NAPALM modules you will have
to add the following configuration to your Ansible configuration
file (ansible.cfg):

[defaults]
library = /Library/Python/2.7/site-packages/napalm_ansible/modules
action_plugins = /Library/Python/2.7/site-packages/napalm_ansible/plugins/action

For more details on ansible's configuration file visit:
https://docs.ansible.com/ansible/latest/intro_configuration.html

Step 2 (testing access network devices via SSH)

# cd Ansible

# ansible-playbook -i inventories/development/hosts inventories/development/Modules/PING/ping.yaml
# ansible-playbook -i inventories/test/hosts inventories/test/Modules/PING/ping.yaml
# ansible-playbook -i inventories/production/hosts inventories/production/Modules/PING/ping.yaml

Step 3 (Generation configuration using Ansible+NAPALM)

Each network device is represented as XX.yaml file.
Those files contain variables and its values to build configs files.
To build configs files, Ansible Roles and Jinja2 templates are used.

host_vars files located in:

Ansible/inventories/production/host_vars
Ansible/inventories/test/host_vars
Ansible/inventories/development/host_vars

To generate configurations for all network devices:

# ansible-playbook -i inventories/development/hosts inventories/development/Modules/generate/generate-all-config-and-make-diff.yml
# ansible-playbook -i inventories/test/hosts inventories/test/Modules/generate/generate-all-config-and-make-diff.yml
# ansible-playbook -i inventories/production/hosts inventories/production/Modules/generate/generate-all-config-and-make-diff.yml

Configs location:

Ansible/inventories/development/CONFIGS
Ansible/inventories/test/CONFIGS
Ansible/inventories/production/CONFIGS

Step 4 (sending and applying new configurations)

If you want to use only terminal to commit-merge generated configurations:

# ansible-playbook -i inventories/development/hosts inventories/development/Modules/commit-merge/commit-merge-final-config.yml 
# ansible-playbook -i inventories/test/hosts inventories/test/Modules/commit-merge/commit-merge-final-config.yml
# ansible-playbook -i inventories/production/hosts inventories/production/Modules/commit-merge/commit-merge-final-config.yml 

else: to use GitLab CI/CD pipeline to automate commit-merge process:

1) Please, read - ### Step Docker-containers (optional)
2) Import https://github.com/dmmar/netascode.git to GitLab CE
3) Check assigned and registered GitLab Runner to that project and tags
4) Create a new issue and a new branch without a merge request
5) Change a master branch to a new branch in PyCharm CE
6) Make some changes
7) git remote add gitlab-local http://[gitlab-docker-container-repo]
8) git add .
9) git commit -m "whatever"
10) git push --mirror gitlab-local
11) Create a merge request
12) Go to CI/CD in GitLab
(if you made everything correctly, you will see a working pipeline which will apply generated configurations)

.gitlab-ci.yml - that file describes GitLab CI/CD pipeline

Step 5 (run network verification testcases from the terminal)

=======
[PyATS]
=======
**TestCase-1 [connectivity]** - ping and trace from HQ-Clients to virtual (8.8.8.8) and HQ-S1 (10.255.255.2)
[Development environment]
# cd PyATS/ex4_pings
# pyats run job job1.py --testbed-file testbed.yaml

**TestCase-2 [vlans]** - checks VLANs on switches in HQ-site [HQ-DIS1,HQ-DIS2,HQ-AC1,HQ-AC2,HQ-AC3,HQ-AC4]
[Development environment]
# cd PyATS/ex5_vlans
# pyats run job job1.py --testbed-file testbed.yaml

**TestCase-3 [ntp1]** - checks NTP server (without a rollback, if NTP server is not correct) 
!!! default correct ntp server is 10.255.255.2 !!!
# cd PyATS/inventories/[environment]
# pyats run job job.py --testbed-file testbed.yaml --html-logs logs/ --ntp-server 10.255.255.2 --devices HQ-FW1 HQ-AC1 --archive-dir logs/ --runinfo-dir logs/

**TestCase-4 [ntp2]** - checks NTP server (DO a rollback to a 1 step back, if NTP server is not correct)
!!! default correct ntp server is 10.255.255.2 !!!
# cd PyATS/inventories/[environment]
open job.py
comment the next line -> run('ntp_check_v2_no_rollback.py', devices=devices, ntp_server=ntp_server)
uncomment the next line -> #run('ntp_check_v3.py', devices = devices, ntp_server = ntp_server)
# pyats run job job.py --testbed-file testbed.yaml --html-logs logs/ --ntp-server 1.1.1.1 --devices HQ-FW1 HQ-AC1 --archive-dir logs/ --runinfo-dir logs/

**TestCase-5 [rollback]** - DO a rollback to a 1 step back [works for Cisco IOS, Cisco ASA, JunOS, and VyOS]
[Development environment]
# cd PyATS/ex6_rollback
# python rollback.py

===========================
[PyATS and Robot Framework]
===========================
**TestCase-1 [connectivify and vlans]**
# cd RobotFramework/inventories/[environment]
# robot pyats1.robot

Step Docker-containers (optional)

WARNING: DO NOT use these containers in real production environment
(NetBox has predefined values for DB passwords, secrets, etc.)

**For correct installation in production - Please, read official documentation!**

WARNING2: [Installation_components/docker-compose.yml] has predefined values,
for example: gitlab.nac.local:192.168.1.100
For my project, host 192.168.1.100 was a VM Debian with installed Docker and docker-compose.


1. To install [docker-compose] and [disable firewall] on Debian server:

# cd Installation_components
# chmod +x docker_compose_setup.sh
# ./docker_compose_setup.sh

2. You need to change [192.168.1.100] in [Installation_components/docker-compose.yml] on your IP address of the debian server.
   Also, on the Debian server add [gitlab.nac.local] record to /etc/hosts file.

3. To install docker containers [GitLab CE], [docker-gitlab-runner] on Debian server using docker-compose.yml file:

# cd Installation_components
# docker-compose up -d
# docker ps

4. To install docker container [NetBox]:

https://github.com/netbox-community/netbox-docker

Step script-diff (optional)

If you want to make an ansible final commmand to generate configs,

only for devices in 'host_vars' directory which you changed manually.

The script is useful because you do not need manually write devices down to make a line.

You need to fix files in the following path:

netascode/Ansible/Python_scripts/generate_diff

For files:

generate-prod-diff.py
generate-test-diff.py

Need to change to your absolute path

# PATH TO DIR WHERE GIT WILL DOWNLOAD MASTER BRANCH
# EVERYTIME WHEN THE SCRIPT WILL RUN THAT DELETES 'MASTER' FOLDER AND CREATE A NEW ONE

GIT_MASTER_DIR = "/home/dmitrii/Desktop/netascode_master_branch/master"

# PATH TO FOLDERS WHERE 'MD5' WILL CHECK CHECKSUM 'host_vars' FILES BETWEEN 'MASTER' and 'BRANCH'
SRC_DIR_MASTER = os.path.abspath("/home/dmitrii/Desktop/netascode_master_branch/master/Ansible/inventories/production/host_vars/")
SRC_DIR_BRANCH = os.path.abspath("/home/dmitrii/PycharmProjects/nac/Ansible/inventories/production/host_vars")