Creating XPRESSO Jobs

About Creating XPRESSO Jobs

In XPRESSO, a "Job" defines the common entry point for one or more pyATS test scripts that can be aggregated together to prepare a Job Run in advance of execution within the same runtime environment. XPRESSO "Jobs" follows the same design concept as an Easypy Jobfile.

IMPORTANT:
When you add/create a Job in XPRESSO, they are associated with a least one Job Profile (the DEFAULT Job profile) but can have many which means the Job can be run in a different environment or against a different set of testbeds.

A Job (and it's associate Default Job Profile) associates all required elements to run the pyATS scripts such as:

• Your test environment (test harness, Execution Engine, and location-details).
• Your lab resources (testbeds or topologies).
• Any localized arguments/environment variables.
• You can also optionally set:
  • The condition(s) for the test results for a Job run to be marked as a baseline Job Run.
  • For Jenkins Job Profiles; specify the path and file to the Easypy Configuration YAML File to be used.

Once you create a Job, they are registered in XPRESSO and saved with a unique name for identification and tracking purposes.

Defining XPRESSO Job-related Terms

The following Job-related terms are used to define the progression of a Job as it moves through the various transition states in XPRESSO: creating, scheduling, queuing, execution, viewing, and management of the Job.

Job-related term	Term Definition
Job Profile	XPRESSO adds an additional layer on top of the Job layer in order to perform varying degrees of tests/coverages using the same pyATS test script but with different parameters/arguments/environment variables. This layer is called a "Job Profile. A Job Profile contains the metadata information associated with a Job that defines how to run your Job such as your test environment (test harness, Execution Engine, and location-details); your lab resources (testbeds or topologies); and any localized arguments/environment variables that apply to a particular test run. You can also optionally set the condition(s) for the test results for a Job run to be marked as a baseline Job Run. When you add/create a Job in XPRESSO, they are associated with a least one Job Profile (the DEFAULT Job profile) but can have many which means the Job can be run in a different environment or against a different set of testbeds. You can create or clone additional Job profiles and associate them to your Job so you can run your Job in different testing environments. When you create a Job Profile, they are registered in XPRESSO and saved with unique name for identification and tracking purposes. Job Profiles are located under each registered Job on the Job details page.
Job Request	When you save a Job in XPRESSO and the Job is submitted for execution and queued (therefore, the job is set to “Run DEFAULT” on the Registered Jobs page), the Job is now considered to be a "Job Request" (also known as a Run Request and is truncated to just "Request" on the XPRESSO dashboard) You do not directly create Job Requests via the XPRESSO main menu; they are created as a result of performing a "Run" request. You can however, configure additional parameter settings with the Job Request to further define the test environment your Job will be execute in. Job Requests can be run individually, in Job Groups, or in Job Bundles. When XPRESSO creates a Job Request, they are registered in XPRESSO and saved with a unique ID for identification and tracking purposes. The Job Request will execute when the testbed resources become available or when the Job is scheduled to run. Once the Job Request completes the run, it contains your test results. You can optionally view job execution statistics in real-time using LiveView when a Job Request is running.
Baseline Job Request	You can optionally configure (set) a Job as a Baseline Job Request; you can then compare the test results from one or more other Job Requests against the Baseline Job Request. Baseline Job Requests also have the same attributes as a standard Job Requests.
Job Test Results	Once a Job Request completes the run, it contains your Job test results and (if set), the comparison data against a baseline set of test results. You can then: Search your Job test results. View a quick-look Results Summary that provides a Pass/Fail breakdown for all test cases in your Job Request. View Job test result details that displays a line-by-line log of all test cases in your Job Request broken down to each section, sub-section, and each discrete executed device command. View your Job test results in Cisco TRADe.
Job Bundle	Job Bundles; (also truncated to just "Bundle" on the XPRESSO Dashboard) are grouped combinations of "saved" Jobs (multiple jobs can be selected) and Job Profiles so they can be re-run at a later date. Job Bundles lets you group any combination of Jobs together as a "single" entity (along with as many Job Profiles as required) but lets you schedule them as a "discrete or single" object. Job Bundles are in essence, a "batch" processing tool that has the same attributes of a "testing-template".
Job Group	A Job Group is when you select more than one Job Request on-the-fly so you can run them concurrently; you cannot save Job Groups. See the "Post-registration Job Operations" sub-topic below for information about how to run a Job Group.

Ways to Execute Jobs in XPRESSO

XPRESSO supports running automation tests (Jobs) in two ways: client-side (via Jenkins) or cloud-side execution (via a Docker image):

• Jenkins Execution Engine The default execution engine in XPRESSO is Jenkins which serves as the workflow engine to manage/provide a CI/CD solution/process to execute each XPRESSO Job Run on a server using the provided Test Harness. Jenkins is a self-contained, open source, orchestration and automation execution engine used to automate the various tasks related to building, testing, delivery, and deployment of software.

• Cloud Execution Engine Cloud Execution use a Docker Image which provides a virtual equivalent of a test execution environment to execute Job Runs in the container.

How Jobs are Executed in XPRESSO

Each pyATS script run is launched via its corresponding Job in XPRESSO. You must create and register your jobs on the Jobs page in XPRESSO along with:

• The test environment to run it (harness and engine).
• Arguments to run with (environment variables, command line arguments).
• Resources to run it on (testbeds/topologies).

If your Job supports various execution options; it can:

• Run on multiple testbed/topologies.
• Support many different argument and environment variables.
• Support various execution engines/test harnesses.

Creating Multiple Job Profiles

You can create multiple Job profiles for each parameter setting that the job can be run with. Job Profiles adds an additional layer on top of the job layer to provide the means to perform varying degree of tests/coverages using the same pyATS test script but with different parameters/arguments thus saving you the effort of having to configure the right settings before every single Job Run.

Stacking Jobs to Run Together

In addition to creating single Jobs, you can also stack multiple Jobs to run together in a single Job Request. Two methods are supported:

• Job Groups: Are created on-the-fly and cannot be saved; see the "Running a Job Group" sub-topic below for more information.

• Job Bundles: Likewise to Job Groups, Job Bundles allow multiple Jobs to be selected so they can be run concurrently; the key difference is that Job Bundles can be saved. See "Creating Job Bundles" for more information.

Creating a New Job

To Create a New Job:

1. From the Main Navigation Bar, choose Main Menu→EXECUTION→Jobs to open the Registered Jobs page.

2. Click the Add Job button located on the top-right of the page. You are presented with two options. In both cases, the Job creation wizard opens.

   NOTE:
   Jobs can be created for both Jenkins Jobs and Clouds Jobs; note the differences between them (the differences are noted in each step): (1) the required steps in two wizards are similar (Jenkins Job Profiles have one additional step) (2) some of the parameters in each step are unique to the specific execution engine type.

   1. Jenkins Job: Traditional execution via the Jenkins Engine
   2. Cloud Job: Cloud execution via a Docker Image.

3. Configure each wizard step as required.

4. Step 1 - Job Info:

   • Job Name: Provide a unique identifier for the Job Name.
     
   • Description: Provide a meaningful description to describe the purpose of the Job.
     
   • Job File: Provide the absolute path to the job file in the file system, for example: [full path + file name] /path/to/job/file/myjob.py
     
   • Docker Image: (Cloud-only). Select which Docker Image to run the Job with.
     
   • Execution Engine: (Jenkins-Only) Select which Jenkins execution engine to run the Job on.
     
   • Harness Instance: (Jenkins-Only) Select which test harness to use for the Job Run.
     
   • Supporting TCL Harness Instance (Jenkins-Only and optional): Select the appropriate TCL
     Harness Instances; only required if you intend to connect Tcl-ATS trees.

5. Step 2 - Baseline Criteria: Allows you to set the condition(s) for the test results for this Job run to be marked as a baseline based on the following criteria:

6. The overall percentage of passed test cases.

7. The minimum number of passed test cases.

8. If the test results match or have improved when compared to the latest baseline.

NOTE 1:
Baseline criteria are cumulative; all set criteria must be met or exceeded for a Job Run to be marked as a baseline.
NOTE 2:
You can set multiple conditions for the Job Run. To ignore a condition, leave the parameter field blank.

• Pass Rate (%): Allows you to set the overall percentage of passed test cases required to mark the test results as a baseline.

• Min. No. of Passed TCs: Allows you to set the minimum number of passed test cases (TCs) required to mark the test results as a baseline.

• Match or Improved Compared to Baseline: Mark the test results as a baseline if the test results match or have improved when compared to the latest baseline.

6. Step 3 - Job Arguments: As required, configure the following custom command line arguments/variables to be set when running this job file; you can configure multiple arguments by clicking the + Add string.

   • Job Arguments
   • Environment Variables
   • Harness Arguments: Select the pyATS specific arguments to set for this job profile.
   • Cloud Mount Points

7. Step 4 - Testbed/Topologies:

   • Testbeds: Select the testbed(s) on which the Job should be run against; (OR)

   • Topologies: Select the topology (or topologies) on which the Job should be run against.

     NOTE 1:
     You can only assign testbeds/ topologies that are located in the same site/building as the selected execution engine and harness instance assigned to the job.
     NOTE 2:
     You can select multiple testbeds or multiple topologies but not both.

8. Step 5 - Configuration YAML: (applies to Jenkins Job Profiles only)

   • (Optional) Copy/paste (and if required, edit), the Easypy Configuration YAML File to be used.

9. More Job options: You can also optionally configure the following parameters for the Job details as required to meet your specific testing requirements:

• Maximum Runtime: Specify the maximum time period the job/profile should be completed within. If not completed within the maximum runtime, the job will be either terminated or allowed to expire.

• Priority: Defines the priority of your job run based on the testbed queueing rules.

• Support Mailer: Enter/update the email address used for the destination address for those users who are notified when the Job completes.

• Interest List: Specify which users the Job results to be carbon copied (Cc) when completed.

• Restrict the execution on Jenkins Node: (Jenkin-only) XPRESSO Jobs are executed on the master Jenkins execution engine and any connected slave nodes by default. This parameter allows you to designate which slave Jenkins execution nodes should be used for offloading instead of the default mode of executing to any slave nodes that have less resources or test cases waiting to execute.

  NOTE:
  You must enter the exact name(s) of the alternate Jenkins execution node(s) as used (defined) within the Jenkins dashboard. Names associated with a single slave node cannot have spaces in them. Names associated with multiple slave nodes must be separated by a space. Avoid special characters with slave node names such as these: !&|<>(), as other Jenkins features allow for defining label expressions.

• Coverage Type: Allows you to tag or categorize the type of test the Job applies to, for example, Sanity test, Performance test, or Regression test.

  NOTE:
  The Coverage Type option provides a method to append a test-type label to the Job details; XPRESSO does not implicitly do anything differently based on the selected test-type label. The label however, can be useful when searching for and analyzing test results returned by the Job execution.

Viewing Current Jobs

All Jobs currently registered to you Group, can be viewed on the Registered Jobspage (Main Navigation Bar→Main Menu→EXECUTION→Jobs).

Viewing Job Profile Information Associated with a Job

In addition to the Job details you added in the registration, a job will list the profiles which are directly associated to the job.

Jobs require at least one profile for execution purpose. Hence a job will be automatically created with the DEFAULT Profile. This ensures the system has at least the minimum viable information required to run the job. This includes:

• A Test Harness
• An Execution Engine
• One or more Testbeds

This information will be used to automatically create the DEFAULT profile for this job. You can add more profiles to the job as required. See More about Profiles.

Post-registration Job Operations

Group Admins can perform the following post-registration management operations with Jobs after they are created:

To Perform Post-registration Operations on a Job:

1. From the Main Navigation Bar, choose Main Menu→EXECUTION→Jobs to open the Registered Job page.

2. Locate and highlight the Job that you want to perform the post-registration operation on and click the appropriate following floating action button located to the right of the Job.

   • Run DEFAULT: Provides the launch point to configure a new job request via the New Job Request wizard for each Job. See "Configuring and Executing Job Requests" for more information.

   • Schedule DEFAULT: Provides a launch point to schedule a Job via the New Schedule wizard. See "Job and Job Bundle Scheduling" for more information.

   • Edit: Allows you to edit all pre-configured parameters on the Registered Job page with the exception of the Job Name parameter. To change the Job Name parameter, delete the Job and create a new one.

   • Delete: Allows you to delete a registered Job from the XPRESSO database.

   • Disable: Allows you to temporarily disable a Job from being used by other Group Members.

   • Maintenance: This state has the same attributes as a Job in a Disabled state except Group Admins and System Admins are able to use them to troubleshoot or leverage them in emergency cases.

   • Enable: Allows you to re-active a Job if the Job was disabled or place into a Maintenance state.

Running a Job Group

3. Select the checkbox adjacent to two or more Jobs to include in the Job Group. The Group Action selection bar appears at the top-right side of the Registered Jobs page (this also indicates the number of jobs selected).

4. Click the Group Run icon button. The button provides a launch point to the New Group Job Request wizard. See "Configuring and Executing Job Requests" for the rest of the steps required to complete the procedure to run a Job Group.