Viewing Job Test Results

About Viewing Job Test Results

A Job Request will execute when the testbed resources become available or when the Job is scheduled to run. Once a Job Run successfully completes, it contains your test results.

You can then view the test result details (if the Job Passed) and/or any other Job Request details for all other Jobs that failed to Passed using the following levels of granularity:

A quick-look test results summary provides:
- A high-level pass/fail breakdown (and all other possible states) for all test cases in your Job Request; the overall state is determined by analyzing the statuses of all sub-ordinate test cases. See the Job Request States sub-topic in "Configuring and Executing Job Requests" for a breakdown of the different states.
- For Job Requests that failed to complete, you can determine the percentage of which test cases passed (vs. failed) from the Success Rate column.
- If the Job Request was used/set as a Baseline test run; this is indicated by: Set-Auto | Requested state information in the Baseline column.
- General information about the Group Request: was the Job Request part of a bundle | any tags applied | submitter | submission date.
You can view a detailed analytic report of your test statistics on the "Result" page that displays a line-by-line log of all test cases in your Job Request. Click the status filter to view the log results by specific status. See the "To View your Test Results" procedure below for details.
You can also view the progression of Job Runs (includes concise log details and test case results) as they happen via LiveView in XPRESSO; see the "Viewing Job Request Executions using LiveView" sub-topic below for details.
For convenience, Cisco personnel can optionally view test results in Cisco TRADe or Jenkins Log Viewers from the XPRESSO dashboard. See the "Viewing Test Results Posted to Cisco TRADe or Jenkins Log Viewers" sub-topic below for details.

Job Test Results Retention Period

There is no expiration date for Job test results in XPRESSO. Job Requests and test results are archived on a regular basis to maintain efficiency of XPRESSO and as a backup. All archived test results can be accessed through the Requests page in XPRESSO (this is the same way as accessing the latest test results).

Viewing your Test Results

To View your Test Results:

This procedure assumes you have previously submitted your Job Request to execute and your test results have been generated; see "To Configure and Execute a Single Job Request" if required prior to starting this procedure.

From the Main Navigation Bar, choose Main Menu→EXECUTION→Requests to open the Request page. This page shows a summary of all Jobs Requests submitted for execution listed by their ID number (listed highest to lowest ID). See the "About Viewing Job Test Results" sub-topic above for a breakdown of the information available via the summary page.
As required, use one of the following options to view/list/organize your summary details (also see the "Searching Local Registered Objects" topic for complementary information about how to search for information using a secondary search field, custom filters, and sorting schemes).
- If you know the ID of the Job Request, enter the value into the Search pane located at the top of the Request page.
- Apply a pre-set "State" filter located to the right of the search panel (drop-down menu). You can also view Bundle or Individual Job Requests by clicking the appropriate filter button located on the top-right side of the page.
- Click the Sort icon button located to the right of each column heading to sort a table column in ascending/descending order using a column value to create a logical listing of the data you can scroll through.
- Click the Filter icon button located to the right of each column heading to apply a filter to a pre-set value in an individual column.
Perform one of the following to view detailed statistics, states, and results of your requests on each sub-request page:
1. Click the chevron button located beside the appropriate Job Request ID to see each Job Profile included in the Job Run.
2. Click the appropriate Job Request ID to open the Job Request details page. Each Job Profile included in the Job Run is located at the bottom of the page.
Click the appropriate Job Profile (sub-request) to open the Job Request/Request Details page; the following tabs are available (the information presented below represents what you would see if the Job Request is in a "Passed" state.)

NOTE:
The availability of information contained on these tabs is subject to the "finished" state of the Job Request. For example, for Job Requests that have a success rate of zero per cent, no test results would be available on the Results tab.
- Overview: Provides a high-level summary of the Job Request details itself (with no test results) such as when the Job executed, the time to complete, Job Request state (Passed | Failed | etc.), and other key parameter settings such as which Profile, Testbed, Execution Engine and Test harness was used for the Job Request.
  
  NOTE 1:
  For Job Requests that did not complete successfully, the Overview tab also states if enough data is available in order to use the Debug tab to determine what went wrong.
  NOTE 2:
  The Overview tab also provides the launch points to
- Results: (the default landing tab): Provides a test case by test case result summary of each of the Job Requests tasks, their start time, and run time. As required, click on each line item on the Results tab to display the execution details for just that item. You can also access the following floating actions buttons located to the right of the Result column:
  - Log (shows detailed log information for each line item).
  - Light-Log Viewer (opens a new tab and shows detailed log information for each line item).
  - Commands (shows any executed command the applied at that stage of the Job Request).
- Run Console Log: Provides a complete listing of all system message logs associated with the execution of the Job Request.
  
  NOTE:
  After you click the Run Console Log tab, there may be a short delay (5-10 seconds) before the Console opens.
- Files: Provides a listing of all files associated with the execution of the Job Request such as the pyATS script, all YAML files, and the ENV file used. As required, click on each file type on the File tab to display the discrete log associated for just that line item. You can also access the following floating actions buttons located to the right of the Result column:
  - Log (shows detailed log information with each file type).
  - Light-Log Viewer (opens a new tab and shows file information).
  - Download (allows you to download the log details from that line item to a text editor).
- Parameters: Provides a listing of all Arguments and Resources parameters associated with the Job Request.
- History: Provides a history tracking record of all significant events and background actions associated with the Job Request.
- Debug: Provides information (when available/when required) about what went wrong when running a Job Request so you can take the necessary corrective action.

Viewing Job Request Executions using LiveView

You can access a LiveView of your Job Runs that provides detailed information (live logs and live test results) of how the Job Run is progressing. The advantage (value-add) of this feature; it gives you the ability to watch your Job Run in real-time rather than having to wait until the Job completely finishes to see your test results or what caused the Job Request to terminate in the event it failed to execute.

LiveView has the following attributes/characteristics: (see the arrows in the LiveView example below as required for locations)

LiveView supports the displaying of real-time progress logs for Job Runs executed via a:
- Cloud Execution Engine using a Docker image.
- Jenkins Execution Engine.
You can only watch the progression of a Job Request in LiveView; all other operational functions that can be configured on the Job Request/Request Details page such as setting Baseline and Baseline Comparison or Comparing Job Test Results is only available when the Job Request is finished executing.
Click the Results tab on the Job Request/Request Details page to display a real-time progress log for a Job Request that are in a Running state. See the procedure above for information about how to access the Job Request/Request Details page.

NOTE:
After you click the Results tab on the Job Request/Request Details page, there may be a short delay (5-10 seconds) before the LiveView Console shows any data.

You can determine the state of the Job Run in the Result column on the Job Request details page.
Click the Files tab to see the progression of any detailed file execution (for example: the pyATS script, all YAML files, and ENV files) associated with the Job Request.
Each file within the Files tab can also be clicked to display "live" information as it runs for just that file.
You can limit which test results to view by setting the appropriate filter (for example: Passed, Failed, Running) in the Results column.
LiveView is not supported over HTTPS connections.
For Job Requests that are complete, click the Run Console Log tab on the Job Request/Request Details page to display a complete listing of all system message logs associated with the execution of the Job Request. This is the same information as shown by the LiveView log viewer but is presented as a complete file rather than discrete entities.

LiveView Example:

Viewing Test Results Posted to Cisco TRADe or Jenkins Log Viewers

For convenience, Cisco personnel can optionally view test results in Cisco TRADe or Jenkins Log Viewers from the XPRESSO dashboard.

IMPORTANT:
While a convenience link is provided on the XPRESSO dashboard to view test results posted to Cisco TRADe or Jenkins Log Viewers, test report details are limited to those details/features supported by those log viewers. XPRESSO provides more robust/detailed reports, and many other additional features such a test run comparisons, triage, and baseline comparisons/checks, etc. not available in these alternate log viewers.

To View Test Results Posted to Cisco TRADe or Jenkins Log Viewers:

NOTE:
A TRADe Log link or Jenkins link is only provided for Requests in a "Passed" or "Failed" state; all other Request states (Errored, Expired, or Stopped) return a "Not Applicable" value.

Complete steps 1 to 3 in the "To View Test Results" procedure above for information about how to access the Job Request/Request Details page.
Click on the Overview tab.
As required, click one of the following links located at the bottom-right of the Job Request/Request Details page to view XPRESSO test results in the alternate log viewer:
- TRADe link
- Jenkins