XPRESSO

Baseline and Baseline Comparison

This topic details:

How the baseline feature differs from the compare feature (baseline set vs. no baseline set; manual vs. auto); use cases are provided to explain the differences.
Why/how setting a baseline can assist with your test automation requirements.
The various baseline features that can be configured in XPRESSO and when to use them.
How to set the conditions for a Job to be marked as a baseline.
How to set a Run Request as a Baseline; multiple options are available.
How to compare test results against your Baseline.
How to Propagate and Unset a Baseline.

Also see "Comparing Job Test Results" for information about Run Result Comparison with no baseline set.

Baseline Comparison Use Cases

The following table outlines the use cases for the two comparison features (baseline set vs. no baseline set) that XPRESSO supports and the shared and different attributes between them:

Method	Comparison Type	Use Case	Unique Attributes
Manual XPRESSO Request page; REST/CLI	Comparing two Run Request Results	You can compare the results from two Run Requests from the same Job & Profile to determine the differences between the test case results. Use for quick one-off comparisons between two different Run Results of the same Job & Profile. Used to determine if the test scenario has been influenced by script or image changes.	Shows a side-by-side comparison for the two selected Run Request test results and respective generated files/logs from the run.
Manual XPRESSO Request page; REST/CLI	Comparing a Run Request Result with its Baselined Run Request Result	You can compare the result of a Run Request already completed to the result of a Run Request that is set as a Baseline for the same Job & Profile with a matching Branch and Label. Use this comparison type when you have established a Baseline for a given Job & Profile, and for subsequent runs of the same Job & Profile you want to compare the outcome of each test case result to the set Baseline. Used to show trending, cumulative, or incremental changes resulting from any changes you made.	Results show a side-by-side comparison for one or more selected Job runs compared to the baseline Job run.
Automatic Run Request set with Baseline enabled and Compare to Baseline option set to None.	No Comparison	The Run Request is initiated specifically to have its Run Request result to be set as a Baseline. The Request Run Result will be set as a baseline when the run has successfully completed (not Errored) and does have test case results. Used to established an unconditional Baseline Job Request Result.	—
Automatic Run Request set as Conditional Baseline option enabled and some Baseline Criteria set.	Compares Run Request test results with its latest Baselined Results	The Run Request is initiated specifically to have its results compared to the latest Baselined results. If the results are matched or improved, the Run Request is set as baseline. Used to established an Baseline Job Request Result only if it passes certain criteria, or has the same or better results than the latest baseline for the same Job & Profile.	—
Automatic Run Request set as Conditional Baseline option enabled and no Baseline Criteria set.	Comparing the Run Request set to latest or a specific result, but not selecting any other baseline options.	The Run Request runs a baseline comparison and displays it in the test results, but does not attempt to set the result as a new baseline. Used to generate persistent comparison data instead of using the “one-off” compare feature in the UI.	—
Shared Attributes of Both Comparison Types Only Jobs with the same Job profile can be compared; comparing dissimilar Jobs results in a No Match error. Both Jobs must contain test results; if one Job Run is missing results, you will get an "Unable to Compare" error. You can compare test results from any Job Run that has successfully ran to a Completion-Passed or Completion-Failed operational state. For all other operational states, a best-effort to compare is attempted; if no useable results can be extrapolated, you will receive an "Unable to Run" error.

About Setting Baseline and Baseline Comparison

You can configure (set) the result of a Job Request as a baseline in order to establish a starting point for comparison purposes to compare the test results from one Job Run against the test results from another Job Run.

Baseline test results are saved (archived) with the Job test results and can be accessed by clicking on the Archive Comparison tab on the Results page.

Why Setting a Baseline Matters?

Once you establish your baseline Job Profile/Job Run, you can use this baseline as a means to:

Test new device features to prove their conformance to all functional, performance, reliability, and security requirements. This greatly improves/expedites Cisco Software development workflow by automatically catching regression errors and highlights areas for improvement.
Allows you to easily validate new software updates from Cisco as part of your validation process prior to upgrading your network.
From a Networking testing perspective:
- Test any proposed changes or pending fixes to your network to perform a verification/sanity check to prove or assess the effectiveness of a specific implementation.
- Determine the health of your network at a certain point in time or at regular intervals to ensure that your network is working at an optimal level.
- Determine the current utilization of network resources and to help you plan for future expansion.
- Identify current network problems and to predict future problems.

Which Baseline Conditions (Types) can be Set?

Baselining in XPRESSO is a rigorous process which includes setting associate baseline conditions and parameters to ensure meaningful/effective test results are returned between the Job Runs being compared.

To set a Job Run as a baseline, you can configure the following parameters to set the baseline conditions (available on multiple forms; see the procedures below for details):

Baseline Condition Options

Baseline Criteria: Allows you to set the condition(s) for the test results for this Job run to be marked as a baseline based on the following criteria:
1. The overall percentage of passed test cases.
2. The minimum number of passed test cases.
3. If the test results match or have improved when compared to the latest baseline.
NOTE 1:
Baseline criteria are cumulative; all set criteria must be met or exceeded for a Job Run to be marked as a baseline.
NOTE 2:
You can set multiple conditions for the Job Run. To ignore a condition, leave the parameter field blank.
Pass Rate (%): Allows you to set the overall percentage of passed test cases required to mark the test results as a baseline.
Min. No. of Passed TCs: Allows you to set the minimum number of passed test cases (TCs) required to mark the test results as a baseline.
Match or Improved Compared to Baseline: Mark the test results as a baseline if the test results match or have improved when compared to the latest baseline.

How to Set a Run Request as a Baseline

Configure the following parameters (available on multiple forms; see the procedures below for details) to set a Run Request as a Baseline.

Baseline (checkbox): Allows you to set the test results of this Job run as a baseline for comparison purposes. (OR)
Conditional Baseline (checkbox): Allows you to set the test results of this Job run as a baseline if the results meets or exceeds the conditions set in the Baseline Criteria parameter (see the Profile Options below for condition details).

NOTE:
If the Conditional Baseline parameter is enabled and the Baseline Criteria parameter (see the Profile Options below) has the "Match or Improved Compared to Baseline" parameter also set to enabled, a baseline comparison is run against the selected comparison option or latest baseline that matches with the branch and label.
Compare to Baseline: Specify which baseline to compare the test results from this Job Run against; two options are available:
- Baseline of your choice: If you specified a Label with the Job Request, select an existing baseline image from the dropdown list.
- Latest: If you do not provide a specific baseline to compare with, XPRESSO uses the latest baseline that matches with the branch (if provided) and label (if provided) and profile of your Job Request to perform the baseline comparison.

Setting Conditions for a Job to be Marked as a Baseline

When you create a new Job (includes Jobs that can be executed in the Cloud via a Docker image or the Jenkins Execution Engine), you can Set the condition(s) for the test results for the Job run to be marked as a baseline based on the conditions identified in the "Which Baseline Options (Types) can be Set?" sub-topic above.

See "Creating a New Job" (specifically, Step 2) for detailed information about how to mark the Job as a baseline.

Setting a Baseline to Compare Test Results

In addition to the above method of setting the conditions for a Job to be marked as a Baseline, there are three other methods to set a baseline for a Job in XPRESSO after you create a Job.

Method 1 - Via the Job Request Wizard
Method 2 - On the Request Summary page
Method 3 - On the Request Details page

CAUTION!
When you create a baseline comparison, the comparison data is attached to the test results. If you delete your test results, you also delete the associate baseline comparison.

NOTE:
Although XPRESSO has the ability to compares and archive the test results of one or more Job runs against the test results from a baseline Job run, you must run each Job run individually.

To Set a Baseline Test Run:

NOTE:
This procedure focuses on how to set a baseline which is performed when you configure a Job Request; see the Configuring and Executing Job Requests topic for all other required wizard steps, parameter settings/descriptions that pertains to configuring Job Requests.

Method 1 - Via the Job Request Wizard

From the Main Navigation Bar, choose Main Menu→EXECUTION→Jobs menu to open the Jobs page.
Hover over the Job that you want to set as your baseline and click the Run DEFAULT floating action button on the right-side of the page. The New Group Job Request Wizard opens with the Request Options step displayed.
Optional (but recommended): Select the appropriate Branch identifier from the Branch pull-down menu.

NOTE:
When you designate this Job Request as a baseline, you are recommended to associate a Branch identifier with the baseline in order to track comparison results/create a usable baseline. This is because a single Job Request can be used to compare against many Job Requests; the data results from each Job run needs to be mapped to the specific Job Request.
Optional (but recommended): Enter a Label for the Branch to facilitate further refinement of comparison results and to aid with searching the comparison data with the Global Search tool.
Set the following baseline-related parameters (refer to the "Which Baseline Conditions (Types) can be Set?" sub-topic above for parameter details)
- Baseline (checkbox)
- Conditional Baseline (checkbox)
- Compare to Baseline
Click submit to run your Job Profile. Once your Job has executed and the results generated, your result will be validated with baseline criteria. If the baseline criteria check is valid, your result will be set as a baseline.
You can confirm your Job Profile was set as the baseline by noting the REQUESTED | SET-AUTO | SET-FORCED status icons in the Baseline column on the Requests page (Main Navigation Bar→Main Menu→EXECUTION→Requests).

Method 2 - On the Request Summary Page (and)

Method 3 - On the Request Details Page

From the Main Navigation Bar, choose Main Menu→EXECUTION→Request menu to open the Requests page.
Locate the Job Request that you want to set as your baseline and click the right-arrow adjacent to the Job ID to expand the Job Request details.
You can set a baseline at two different levels/pages:
- On the Request page: Hover over the specific Job that you want to set as your baseline Job Profile and click the Set Baseline floating action button on the right-side of the page. A message window appears.
- On the Request Details page: Click the Job Request item number to launch the Request Details page and click the Set Baseline button on the top right-side of the page. A message window appears.
Enter a reason for setting the baseline and click the Set button. If the baseline criteria check is valid, your result will be set as a baseline.
You can confirm your Job Profile was set as the baseline by noting the REQUESTED | SET-AUTO | SET-FORCED status icons in the Baseline column on the Requests page (Main Navigation Bar→Main Menu→EXECUTION→Requests).

Propagating Baseline Attributes

The Propagate function allows you to take all of the baseline attributes (or lack thereof) from a baseline result from a particular Job Run/Profile that has occurred, and retroactively apply them (going forward) to all existing baseline results that are run after the designated baseline.

You can optionally specify that only specific baseline attributes to be propagated to all later baselines rather than all attributes.

The Propagate function has the following characteristics:

Baseline Propagate is applied to all baselines within the same Job Runs/Profiles (and branch and label if applied).
Baseline Propagate has no effect (no application) on the first baseline you make and is meant to be used on older baseline test results.
If you perform this operation on the latest baseline that has been run, it will have no effect.
Propagate is run once when the action is requested by a User is not a continuous background process; you must continuously apply it between Job Runs.

Propagate "Simplified" Use Case

You execute a Job Run; an issue is identified with your test case.
You determine where/when the problem first started in your test case.
You determine the cause is legitimate.
You can then mark the first occurrence of the problem to indicate the cause is legitimate (this is your baseline).
You can then use the Propagate function to take all of the baseline attributes (or selected attributes) to retroactively apply them so all subsequent baseline results that were run after the designated baseline have the same flags (attributes)
When you apply the Propagate function, you can set specific baseline attributes to indicate: what is Ignored | Must Pass | Is Intermittent | etc.

To Propagate Baseline Attributes:

From the Main Navigation Bar, choose Main Menu→EXECUTION→Request to open the Requests page.
Locate the Job Request that you want to propagate the Baseline attributes for and click the right-arrow adjacent to the Job ID to expand the Job Request details.
You can propagate baseline attributes at two different levels/pages:
- On the Request page: Hover over the specific Job that you want to propagate the Baseline attributes for and click the Propagate Baseline floating action button on the right-side of the page. A message window appears.
- On the Request Details page: Click the Job Request item number to launch the Request Details page and click the Propagate button located on the top right-side of the page. A message window appears.
To propagate a single attribute, select the Results tab, click the baseline button on the desired test case, and click the Propagate button in the right baseline pane.
Enter a reason for propagating the Baseline attributes and click the Set button.

Unsetting a Baseline

In the event your baseline for the current Job run is no longer useful or you want to establish/create a new baseline, you need to remove (unset) your current test result image so it is no longer designated as your baseline.

To Unset a Baseline:

From the Main Navigation Bar, choose Main Menu→EXECUTION→Request to open the Requests page.
Locate the Job Request that you want to unset as your baseline and click the right-arrow adjacent to the Job ID to expand the Job Request details. Refer to the baseline status indicators in the Baseline column to assist you with locating the appropriate Job Request.
You can unset a baseline at two different levels/pages:
- On the Request page: Hover over the specific Job that you want to unset as your baseline Job Profile and click the Un-set Baseline floating action button on the right-side of the page. A message window appears.
- On the Request Details page: Click the Job Request item number to launch the Request Details page and click the Un-set Baseline button located on the top right-side of the page. A message window appears.
Enter a reason for unsetting the baseline and click the Set button.

NOTE:
You may need to refresh your browser window to view the result of unsetting your Job profile.
You can confirm your Job Profile was unset by noting the Set icon in the Baseline column has changed to Requested on the Requests page.

Comparing Test Result against your Baseline

After you set a baseline to compare your test results, you need to indicate (set) if any subsequent Job runs should be compared against your baseline. There are two methods to Compare Baseline Results in XPRESSO:

Method 1 - On the Job Request Options page (use this method if your intend to compare the results of many Job runs against a baseline).
Method 2 - On the Request page (use this method if you only intend to compare the results of one Job run against the baseline Job run).

To Compare Baseline Results:

Method 1 - On the Job Request Options Page

From the Main Navigation Bar, choose Main Menu→EXECUTION→Jobs to open the the Registered Jobs page.
Hover over the Job that you want to compare against the baseline and click the Run DEFAULT floating action button on the right-side of the page. The New Request Page opens showing multiple tabs.
If required, click the Request Options tab. The Request Options page opens.
Select the appropriate baseline to compare your test results against from the Compare To: pull-down menu:
- Compare with latest baseline (default): If you do not provide a specific baseline to compare with, XPRESSO uses the latest baseline that matches with the branch and label (if provided) of your request and perform the baseline comparison.
- Compare with your choice of baseline: Select an existing baseline image from the dropdown list (when you submit request/baseline comparison).
Click submit to run your Job Profile. Once XPRESSO finishes the comparison, you are directed to the Results Comparison page; a summary of the two Job runs is presented on the top part of the page.
To see the differences between the specific test results, hover over a specific Job Profile parameter to query/compare and select the Testcase Diff floating action button on the right-side of the Results Comparison page.

The specific test results differences display in a side-by-side comparison format. You can scroll up or down as required or use the Quick Navigator on the right-side of page to locate specific differences.

Method 2 - On the Request Page.

From the Main Navigation Bar, choose Main Menu→EXECUTION→Requests to open the Requests page.
Click the checkbox adjacent to the Job Request ID column on the Job Request that you want to compare against the baseline Job Run.
Click the Compare icon button located at the top-right side of the Request page.

Once XPRESSO finishes the comparison, you are directed to the Results Comparison page; a summary of the two Job runs is presented on the top part of the page.
To see the differences between the specific test results, hover over a specific Job Profile parameter to query/compare and select the Testcase Diff floating action button located on the right-side of the Results Comparison page.

The specific test results differences display in a side-by-side comparison format. You can scroll up or down as required or use the Quick Navigator on the right-side of page to locate specific differences.

Baseline Test Comparison Results and Summary

Baselined results are clustered by;

Group
Profile
Branch
Label

When you request a baseline comparison a result, it will be strictly compared only to a Baseline in your Group with exact Job, Profile, Branch and Label. If an exact baseline match is not found in XPRESSO, baseline comparison will not be run for your request.

The Baseline summary and the Baseline results will be displayed on the Results Comparison page of a request, and the Results Comparison overview page will indicate the detailed differences.

Baseline Compare and Test Result Comparison Messages

XPRESSO reports the following test result comparison messages: (shows the baseline comparison results/settings vs. the results your are comparing and what the state would be.)

Test Comparison Result Message	Baseline Result	Job Run Result
Same	Must Pass & Passed	Passed
Fixed	Must Pass & Failed	Passed
Broken	Must Pass	Failed (Job did not run)
Ignored	Ignored	Passed/Failed
Added	Not Run	Passed/Failed
Missing	Passed/Failed	Not Run
Same	Passed	Passed
Same	Failed	Failed
Broken	Passed	Failed
Fixed	Failed	Passed

Baseline compare and results definitions:

Passed is defined as test results that are either Pass or Passed with an Exception.
Failed is defined as test results that are neither Pass nor Passed with an Exception.
Not Run is defined as the test case never presented a baseline result.
Ignored is the baseline attribute indicates the comparison result will be ignored.
Must pass is the baseline attribute that indicates the comparison result must pass, otherwise the test level result will be "broken" and overall will be "mismatched".
Added means there are new testcases in test results comparing to baseline results. With added testcases, overall result will be "Comparison not applicable" with reason "Mismatched with additional cases" or "Improved with additional cases" or "Matched with additional cases", pass/fail number of added testcases will be counted and returned as well.

Baseline Configuration State Definitions

Once you set a baseline in your Job Request, XPRESSO reports the following configuration states in the Baseline column on the Requests page (Main Navigation Bar→Main Menu→EXECUTION→Requests); the following defines what each state message:

REQUESTED: A baseline was requested/set by a user for the Run Request. If the test results meet the baseline criteria, they are set as a baseline. If they do not meet the criteria, no baseline comparison is set.
SET-AUTO: The baseline was set automatically since the prescribed/defined conditions were met within the test results.
SET-FORCED: A baseline was forced-set (manually applied) by a user after the Run Request completed as a non-baseline run.

Refer to the following topics for other related Baseline and Baseline Comparison information:

See "Creating a New Job" (specifically, Step 2) for information about how to set the condition(s) for a Job run to be marked as a baseline based the conditions identified in the "Which Baseline Options (Types) can be Set?" sub-topic above.
See "Creating a New Job Profile" for information about how to set the condition(s) within a Job Profile to be marked as a baseline based the conditions identified in the "Which Baseline Options (Types) can be Set?" sub-topic above.
See "Configuring and Executing Single Job Requests" for information about how to set the condition(s) within a Job Request to be marked as a baseline based the conditions identified in the "Which Baseline Options (Types) can be Set?" sub-topic above.
See "Job and Job Bundle Scheduling" for information about how to establish baseline Job runs with all subsequent Run Requests (and at a regular scheduled intervals) to prove the stability of the system, and/or image builds.
See "Viewing Job Test Results" for information about how to determine if a Job Request was used/set as a Baseline test run.
See "Operational and Configuration States" for information about the configuration states that apply to Baseline Comparison.
See "Requests, Bundle, Results, and Baseline CLI APIs" for information about the following Baseline-related CLI APIs: Compare | List | Propagate | Set | Set-attribute | Unset | Unset-attribute | Update.