Adding or Building New Docker Images

About Adding or Building New Docker Images

You can run automation tests in two ways in XPRESSO: in a locally-hosted (client-side) test environment via a Jenkins Execution Engine or via the Cloud using a Docker Image.

XPRESSO provides support for executing a Job via the Cloud using a Docker Image. The Docker image contains the virtual environment (with all required scripts, libraries, python environment, pyATS packages, and other dependencies/packages) needed to execute the Job Request.

You can associate one Docker Image to more than one Job at a time.

Why Docker Images Matter?

The following key reasons outlines why XPRESSO's cloud-equivalent test execution environment can help you with your test automation needs:

Eliminates the requirements of having to understand the complexity of how Docker Images work by providing a simple interface to add (register) existing pyATS-compliant Docker Images or build a new Docker Image.
Docker Images eliminates the need for a physical (Jenkins) Execution Engine or test harness.
Provides a faster, more stable and robust testing approach that's also portable.
Provides consistent and trustworthy test results since its executed in a reproducible and controlled testing environment.
Docker makes the process of application deployment very easy and efficient eliminating many deployment/upgrade issues; this results in higher productivity with all your testing requirements.

Ways to Build Docker Images and Register them in XPRESSO

There are three ways to build and register Docker Images for use in XPRESSO:

Raw (Ad hoc) method: You can build your own (do-it-yourself) Docker Images using a command-line program then link the image in XPRESSO. When you register the image in XPRESSO, it is checked to ensure it conforms to the pyATS Docker Image-schema requirements. See the "To Add (Register) New Docker Image" procedure below for information about how to add these Docker Image types to XPRESSO.
pyATS Image builder method: The pyATS Image builder offers a “with-training-wheels” way of building pyATS-compliant Docker images that uses its own special YAML file syntax which is much simpler than composing a raw Docker file. The YAML file is then parsed, actioned upon, and generates a docker image accordingly that works with both core-pyATS and XPRESSO. Since pyATS users are familiar with YAML files constructs, this allows them to by-pass the more complex method of composing the Docker Image in a command-line program. Docker images of this type are still validated when they are added to XPRESSO for registration to make sure they conform to the correct schema. See the "To Add (Register) New Docker Image" procedure below for information about how to add these Docker Image types to XPRESSO.
XPRESSO "Add" and "Build" Docker Image methods: You can add pyATS-compliant Docker image directly into XPRESSO or build them using the onboard (simplified) Docker Image Builder. As the Docker Image is built/compiled in XPRESSO, the output is translated/converted to the same YAML file requirements used in the pyATS Image Builder. You can preview the Docker Image build YAML at any stage of building a Docker Image to review all configured parameters. See the "To Build a New Docker Image in XPRESSO" procedure below for information.

Docker Image Hosting Requirements

When you create a Job to execute via the Cloud using a Docker Image, you are required to enter the name of a registered Docker Image into the image URL parameter field associated with the Job. This provides the linkage between the Job in XPRESSO and the Cloud Execution engine.

To facilitate linking the Job to the Cloud Execution engine, registered Docker images are required to be hosted on one of the following web sites so they can be "pulled" by XPRESSO when the Job executes:

Docker Hub repository public docker repository.
Any user-defined host site that is reachable by XPRESSO; you can configure the path requirement and any special hosting settings (e.g.; proxy) when you build the Docker Image in XPRESSO.

pyATS-compliant Docker Image Information/Schema Requirements/Repositories

Details on the requirements for pyATS-compliant Docker Images can be found at this websites: external and Cisco-internal.
Details on how to build pyATS-compliant Docker Images with the pyATS Image Builder can be found at this website.
You can find other pyATS-compliant Docker Images at this website.
You can Pull registered Docker Images from the Cisco pyATS Docker Hub repository; this website is publicly accessible.

NOTE:
This note only applies to users who are not using the default PYPI public repository. There is a dependency when working with the pyATS Image Builder using pip (the default Python package-management system used to install and manage additional libraries and dependencies that are not distributed as part of the Python standard library). You need to configure the default pip setting to work with the pyATS Image Builder; this is required to specify a different PYPI server, as well as other pip command line option defaults in a standard .ini style config file. Refer to the PyPA pip User Guide for information about how to customize the pip.conf file to point to a custom PYPI repository and not the default PYPI public repository.

Adding or Building Docker Images in XPRESSO: Which is Right for You?

You can register an existing pyATS-compliant Docker image in XPRESSO or build a pyATS-compliant Docker Image within XPRESSO. The following sub-topics outlines the use cases to consider to determine which option to take when registering a new Docker Image in XPRESSO.

Adding (Registering) an Existing Docker Image in XPRESSO

If you have experience working creating pyATS images, or you have a pre-existing Docker image built with another application, you can simply add (register) the existing image into XPRESSO. Follow the "To Add (Register) a New Docker Image" procedure below for details.

When an existing Docker image is registered in XPRESSO, it is validated to make sure it conforms to the necessary pyATS-schema requirements.

NOTE:
In the event you make a change or update the source version of a Docker image after you have registered the image in XPRESSO, you can update (Pull) the updated image details on-demand in XPRESSO. This syncs up the source version of the image with the version registered in XPRESSO. See the "To Perform Operations on a Docker Image" sub-topic below for details.

Building a New Docker Image in XPRESSO

If you are new to working with Docker Images or don't have experience/exposure to building images with the pyATS image builder, XPRESSO provides a onboard (simplified) Docker Image Builder to help you assemble all the required parts to build a valid Docker image. Follow the "To Build a New Docker Image" procedure below for details.

IMPORTANT:
If you use the pyATS Image Builder to build a Docker Image, it also includes the job in the image, whereas if you use the "official" pyATS Image only has the required pyATS elements and will execute jobs that are stored externally. XPRESSO always requires a job to be included in the image; the pyATS Image Builder highly simplifies the process of linking a job to be included the image since it has a predefined layout that XPRESSO knows how to handle.

How to Add (Register) a New Docker Image in XPRESSO

Only the Group Admin can add (register) an existing Docker Image. Group Members and Privilege Members are required to send an email to the Group Admin if they require a new Docker Image to be registered for their testing requirements.

To Add (Register) a New Docker Image in XPRESSO:

From the Main Navigation Bar, choose Main Menu→ENVIRONMENT→Docker Images to open the Registered Docker Images page.
Click the Add Image button located on the top-right of the page. The Add a New Docker Image form appears.
Configure the following required parameters on the Add a New Docker Image form:
- Image Name: Enter the name of the image which will be used to run the Docker container after it is built. Validation rules for this parameter include:
  - The name may contain lowercase letters, digits, and/or separators (hyphen, slashes, periods, underscore), and must not start or end with separators.
  - The image must reside in the defined image folder; XPRESSO attempts to retrieve the image for registration and if the image file exists, an internal verification is triggered/performed to validate the file contents against the required pyATS Docker Image schema. If the file is not present, it will transition to an "errored" state right away. See the "Docker Image Hosting Requirements" topic above for more information.
- Docker Tag: Enter a tag to logically tag the Docker Image in case you need to search for any test result details related to this Docker Image.
- Dockerhub URL: Enter the url to the dockerhub host and repository (example_domain.com/image_repository), for example: dockerhub.cisco.com/pyats-docker.
  
  NOTE:
  If the Dockerhub URL is not set it will default to the Default Dockerhub Repository set in the common settings.
- Username(Optional): Enter a username if pulling from a private repository.
- Password(Optional): Enter a password if pulling from a private repository.
Click the Save button. The new Instance appears in the list of Registered Docker Image Instances.

NOTE:
In the event your Docker Image does not register in XPRESSO successfully, you can determine the root cause of the failure by clicking on the Image Name on the Registered Docker Image page to open the Docker Image details page. Click the Error Info tab or Logs tabs (if log information exists) to see specific details as to why your registration/build failed.

How to Build a New Docker Image in XPRESSO

Only the Group Admin can build a new Docker Image using the provided interface. Group Members and Privilege Members are required to send an email to the Group Admin if they require a new Docker Image to be built for their testing requirements.

To Build a New Docker Image in XPRESSO:

From the Main Navigation Bar, choose Main Menu→ENVIRONMENT→Docker Images to open the Registered Docker Images page.
Click the Build New Image button located on the top-right of the page. The Building a new Docker Image wizard appears. Complete the following sub-steps associated with the Docker Image wizard sub-steps.

NOTE:
The XPRESSO Docker Image Builder follows the same design requirements defined in the pyATS Image Builder which is designed to standardize the building of pyATS test scripts and their corresponding environment dependencies into Docker images. See the pyATS Image Builder on-line user documentation if you require further information about specific parameters details/requirements.

Resources: You must provide either the file details or the path to the Docker Image repository.

Files: Include the list of files from the various sources to be copied into the Docker Image (including the syntax for copying files from the localhost (this server) and remote URL).
1. File Name or Directory (Optional): Enter the (first) file name or directory to be copied into the Docker Image. Only provide if you need to copy a localhost file to workspace directory OR rename it, for example: dirname/file3.
2. File Path: Enter the path to the file or the location of the remote URL that contains the file to be copied into the Docker Image; for example: /path/to/file3.py OR https://webaddress.com/path/to/file.
3. Add Files: Click this button if you need to add additional file name or directory details from other sources to be copied into the Docker Image; repeat as necessary to add all required sources. This button becomes active when the first source details are entered.
Repositories: Enter the path to the Git repositories to clone and include with/in the Docker image:
1. Repository Source: Enter the (first) URL source address to the Git repositories to clone and include in the Docker Image; for example: ssh://git@address/path/to/repo.git.
2. Destination: Enter the name of the folder or sub-folder to clone the Docker Image to.
3. Commit ID (Optional): Enter the commit-id/branch to checkout after cloning.
4. Private Repository Settings (Optional): Enter the username/password or SSH Key (without passphrase) to clone a private git repository.
5. Ignore SSL certificates: Enable this option to ignore SSL certificates on the git pull.
6. Add Repository: Click this button if you need to add additional URL source addresses to the Git repositories to clone and include with/in the Docker Image; repeat as necessary to add all required URL addresses. This button becomes active when the first URL source address is entered.

Basic Info: Configure the following general information parameters associated with the Docker Image you are building:

Image Name: Enter the name of the image used to run the Docker container. See the Image Name parameter details in the "To Add (Register) a New Docker Image" procedure above for the validation attributes that apply to this parameter.
Tag: Enter a tag to logically tag the Docker Image in case you need to search for any test result details related to this Docker Image.
Push Dockerhub URL: Enter the URL address to share (Push) your images to the Cisco pyATS Docker Hub repository or to a self-hosted website.

NOTE:
If the Push Dockerhub URL is not set it will default to the Default Dockerhub Repository set in the common settings.
Python Version: Specify the Python version used to build the Docker Image:

NOTE:
The pyATS Image Builder currently uses a base image of python:(version: 3.6.9)-slim to build Docker images. This same value is set as the default version in XPRESSO but you can set your Python version.
Package 1 (details): Provide the name of the (first) Python package associated with the Docker Images to be installed into the virtual environment.
Add Package: Click this button if you need to add additional Python package details; repeat as necessary to add all required Python packages to the virtual environment. This button becomes active when package 1 details are entered.
Pin Version: Provides the option to not pull the docker image on execution runs.

Arguments & Configuration: As required, configure the following Environment Variables, Proxy Arguments, and Docker Commands to be set within the Docker Image:

Environment Variables (Optional): Provide any flags to set the package environment variables in the Docker Image you are running:
1. Name: Provide a name for the Python package environment variable you want to install in the Docker Image; for example: 'JOBFILE: ${WORKSPACE}/somejob/somejobfile.py'
2. Value: Enter the appropriate flag for the Python package environment variable, for example:
  - pyats
  - pyats[full]
  - ${WORKSPACE}/my-custom.whl
3. Add Environmental Variables: Click this button if you need to set additional package environment variables in the Docker Images you are running; repeat as necessary to add all required environment variables. This button becomes active when the first environment variables details are entered.
Proxy Arguments (Optional): Configure the following proxy variables if your host server(s) is located behind a proxy. These variables are required for pip installations using public PyPI servers.
1. Proxy: Select the appropriate Proxy type (HTTPS_Proxy | HTTP_Proxy | FTP_Proxy | No_Proxy) for your host server location from the pull-down menu. These values are hard-coded and cannot be changed.
2. Value: Enter the appropriate proxy address, for example:
  - https_proxy: https://proxy.esl.cisco.com:80/
  - http_proxy: http://proxy.esl.cisco.com:80/
  - ftp_proxy: ftp://proxy.esl.cisco.com:80/
  - no_proxy: .cisco.com
3. Add Proxy Variables: Click this button if you need to add additional proxy variables for other host server(s) located behind a proxy; repeat as necessary to add all required proxies. This button becomes active when the first proxy address details are entered.
Docker Commands (optional): As required, provide any additional pre-or-post commands to be inserted into the Docker Image to be executed before and after the pip install process.

WARNING!
The Docker Command parameters are designed for users with advance knowledge working with Docker Images. These parameters can cause unexpected behaviors with your network testing and with XPRESSO; do not use these parameters unless you are absolutely sure of the consequences.

Pre/Post Docker Commands: Select [pre] for the Docker command(s) to be executed before the pip installation or [post] for the Docker command(s) to be executed after the pip installation process.
Docker Commands: Enter the Docker command(s) to execute in a string format.
Add Docker Commands: Click this button if you need to add additional Docker command details; repeat as necessary to add all required Docker commands. This button becomes active when the first Docker command details are entered.

Preview: You can preview the Docker Image build YAML file at any stage of building a Docker Image to review all configured parameters. Preview just presents a visual method for you to review the contents of the Docker Image file configurations; it does not allow you to edit the file directly in the build YAML file. In the event you need to modify any parameter details, you must change the configuration in the source parameter field.
Click the Save button. The button appears when XPRESSO determines all required fields are configured and have met the required criteria to be considered to be a pyATS-compliant Docker Image.

Performing Operations on Registered Docker Images

Group Admins can perform the following operations on a registered Docker Image: Pull | Delete.

NOTE:
Once you register a new Docker Image, you cannot edit them. This is a limitation of Docker Images in general due to how they are created and stored. In the event you need to modify the parameters associated with a Docker file, you will need to delete it from XPRESSO and re-create it with the new parameter settings.

To Perform Operations on a Registered Docker Images:

From the Main Navigation Bar, choose Main Menu→ENVIRONMENT→Docker Images to open the Registered Docker Images page.
Locate and highlight the Registered Docker Image that you want to perform the operation on and click the appropriate floating action button located to the right of the selected Instance: Pull | Delete.

NOTE:
The Pull function allows you to sync up the source version of the image with the version registered in XPRESSO in the event you make a change or update the source version of a Docker image after you have registered the image in XPRESSO.

Refer to the following topics for other related Docker Image information:

See "Creating a New Job" for information about how to configure a Job to specify which Docker Image to use for a Job Run.
See "Docker Preferences" for information about how to set all preferences related to the configuration of Docker Images such as the Pip Config Index URL and trusted host name, and the Python version used.
See "Operational and Configuration States" for information about the configuration states that apply to Docker Images.
See "Communications" for informations about when the events that apply to Docker Images would be notified to you.
See "Manifests" for how pyATS uses a file with YAML syntax (the “manifest” file) to capture the runtime requirements, script arguments and execution profile for test scripts.