XPRESSO

Working with Testbeds

About Working with Testbeds

Testbeds represent the network devices that work together as a test platform that your tests will be executed against. XPRESSO supports the following testbed types:

Static testbeds: Represents the physical network devices in your lab registered in XPRESSO (and are not managed by LaaS).
LaaS testbeds: Are testbeds create on-the-fly based on LaaS-NG topologies; the testbed can be either virtual or physical.

How Testbeds are Managed by XPRESSO?

Testbeds are scarce resources in a test environment; they are used for both manual testing as well as running automated tests. XPRESSO actively manages testbed resources using testbed (priority) queues and reservations.

Once you have registered your testbed in XPRESSO, they can be associated with your Job Run when you create a Job.

What Gets Modeled?

When you register your testbed in XPRESSO, the following is fully modeled on the XPRESSO dashboard:

Testbed representation:
- Testbed YAML file
- Testbed Clean YAML file (Optional)
- Testbed CONFIG file (Optional); used to support running legacy TCL Testbed Clean operations.
Testbed time allocations: Includes the testbed Queue management for Run Requests and Reservation Requests.

A list of all testbeds currently registered to your Group can be accessed from the Registered Testbeds page.

Testbed YAML File

The testbed YAML file describes the physical devices and how they link together to form the testbed network topology.

When the YAML file is uploaded to XPRESSO as part of the testbed registry, it is checked to make sure it complies with the pyATS testbed YAML schema requirements.

If the file conforms, it is uploaded into XPRESSO. If validation errors are detected, the file is tagged to point out the errors. You can manually correct any errors directly in XPRESSO to complete the upload/verification process.

NOTE:
You can download a sample YAML file from this site to help you determine the Testbed YAML file requirements.

Testbed Clean YAML File

The testbed clean YAML file describes the clean requirements for the devices in the testbed.Cleaning a testbed describes the process of loading new images on to each testbed device, and wiping out the current configuration. Clean operations returns the device to a sane state.

As part of the Job/Profile Run Request, you have the following options to run a clean operation with the devices on the associated testbeds:

Loading a given image on the DUT.
Reloading the device with the same image the device is operating with.

Additional information regarding Clean requirements can be accessed at these locations:

For Clean YAML file; refer to this site
For Image loading file markup requirements; refer to this site.
For general pyATS testbed and clean requirements; refer to this site.

Testbed CONFIG file for Legacy TCL Clean Operations

XPRESSO currently provides support for legacy Cisco TCL scripts and config files (also referred to as Tcl-ATS scripts) that are used to run tests and clean devices in your testbed. You can upload a TCL CONFIG file to XPRESSO when you create (register) the testbed. Support for uploading TCL scripts will be deprecated from XPRESSO in a future release and no other information is provided about TCL Config scripts. If required, contact the XPRESSO Technical Support team for information about how to upload a TCL CONFIG file.

Static Testbed Support

Static testbeds in XPRESSO have the following attributes:

There is no limit on the number of testbeds that can be registered in XPRESSO.
You can determine which testbeds are registered with your Group on the Registered Testbeds page.
The registered testbed name (as it appears on the Registered Testbed page) is derived from the testbed name field used in the Testbed YAML file:
- To change a testbed name in XPRESSO, change the name field in the associate YAML file.
- If the name field is not defined in the Testbed YAML file, the file name is used instead.
You can perform the following management functions on a registered testbed: Reserve, Edit, Delete, Maintenance, Disable and Clean Testbed. See the "Post-registration Testbed Operations" sub-topic below for details.

LaaS Testbed Support

If your Group uses LaaS to manage their lab and testing resources, you need to register the LaaS instance and Domain with XPRESSO to use the topologies within that Domain for your Job Runs.

See "LaaS Instances and Domains" for information about:

How LaaS can be used with/in XPRESSO to remove the burden of having to create physical testbeds for each and every test.
How to request that a System Admin add (register) a LaaS Instance in XPRESSO for a Group.
How to determine which LaaS Domains are registered for a Group.

When the LaaS Domain registry is completed by the System Admin, the topology in a given LaaS Domain is associated with a working Group. You can then associate one or more topologies within a registered LaaS Domain when you create a Job.

NOTE 1:
If you create a topology in XPRESSO, the topology is associated with a registered LaaS Domain. When you associate a topology to a Job, all topologies registered to the LaaS Domain are then selectable via a checkbox in the Creating Job wizard (via the Testbed/Topologies sub-step). See "Creating Jobs" for detailed information about how to choose topologies within a given LaaS Domain.
NOTE 2:
If you create a topology in LaaS and the topology is within a LaaS Domain that is registered in XPRESSO, all topologies within the specified LaaS for a Group will be automatically propagated to XPRESSO. Likewise to the above note, when you associate a topology with a Job, all topologies registered to the LaaS Domain are then selectable.

LaaS Testbed Attributes

LaaS testbeds in XPRESSO have the following attributes:

LaaS-NG testbeds are transient; you do not create them directly in XPRESSO but it is necessary to create and register the associate topology.
Groups can be associated with one or more LaaS-NG domains (particular to one instance).
LaaS will dynamically orchestrate the testbeds based on the reserved LaaS topology; then release and delete the testbed from XPRESSO when the reservation expires.
When a topology is Reserved and is in an active state, a LaaS testbed is associated to that topology and returned to XPRESSO. The returned LaaS Testbed is listed on the Registered Testbeds page.
When a Job & Profile associated to a Topology is selected to run, XPRESSO with interface with LaaS to orchestrate the testbed based on Topology. The Run Request resumes execution when the LaaS testbed is active. Upon completion of the Run Request, the LaaS testbed is released and returned to the available pool of devices in LaaS.
LaaS testbeds are orchestrated during a Job Run Request and released once the Job Run Request completes.
The LaaS Testbed name (as it appears on the Registered Testbed page) is auto-created by XPRESSO and follows this patterning: TB0000023, TB0000024. When the testbed reservation ends, the testbed name is purged from the Registered Testbed page.
The LaaS Testbed on the Registered Testbed page:
- Cannot be edited.
- Can have their configuration state changed to a Disabled state; then returned to an Enabled state.
- Can be associated to a Job & Profile Run Request with a Clean option.
After you create a LaaS-NG topology and it has an reservation in an active state applied to it, you can take a snapshot of an existing LaaS topology and it's configuration at that point in time so it can be reused to re-create the test environment at a later time. This significantly decreases the time to set up a LaaS-NG topology since the entire test infrastructure is replicated. See the "Post-registration Reservation Operations" sub-topic in the Reservations topic for more information.

CAUTION!
If you associate a testbed created from a LaaS-NG Topology Reservation directly with any object, this association is temporary since the testbed is deleted at the end of the reservation. Instead, associate the topology itself to any required objects if you have this requirement.

Registering a Static Testbed

You need to add (register) your testbed in XPRESSO so you can execute your Job Runs on it.

To Register a Static Testbed:

From the Main Navigation Bar, choose RESOURCES→Testbeds to open the Registered Testbeds page.
Click the Add Testbed button located on the right side of the page. The New Testbed page appears.
You have two options to upload the required testbed YAML file to XPRESSO. Each testbed is represented by a corresponding testbed YAML file.

Click the Add Testbed File button to locate your YAML file and click Open.
You can drag and drop the testbed YAML file onto the Add Test File button pane.

Once the YAML file is uploaded, a check mark appears to confirm the file successfully uploaded.

You can optionally upload or drag and drop the testbed Clean file at this stage or later. Refer to the Clean Details in the Post-registration Testbed Operations topic below for details for a latter deployment.
Click Next to proceed to the Testbed YAML tab. As part of the YAML testbed file upload, the content of the YAML file are validated. If there are validation errors, they are flagged in the Warnings window; you can correct any errors directly in the dashboard browser page if required. You can re-validate any changes made or choose to ignore any errors by clicking the appropriate button; clicking either button cycles you to the next tab.
If you imported a testbed Clean file, click Next to proceed to the Clean YAML tab to fix any validation errors in the Clean YAML otherwise, proceed to step 7.
On the Testbed info tab, enter the Location Info (Site and Building) for the testbed; this is a required parameter.
You can optionally configure the following parameters:
- Testbed Name: When adding (registering) a testbed, you cannot change the testbed name directly in XPRESSO. To change a testbed name, change the name field in the associate YAML file. If the name field is not specified in the YAML, the testbed name is derived from the file name of the YAML.
NOTE:
Once a testbed is registered in XPRESSO, you cannot change the testbed name; this is design intent to maintain equipment tracking integrity. If you change the name of the testbed in the name field in the associate YAML file, XPRESSO continues to track the testbed with the name it was previously registered with.
- Maximum Testbed Queue Retention: Set in your current Group's default preferences (the default is 12 hours). See "Testbed Queuing Settings" for information about testbed queues. See "Configuring Group Preferences" for information about how to change the defaul value.
- Maximum Reservation Duration: Set in your current Group's default preferences (the maximum is one day). See "Reservation Settings" for information about reservations. See "Configuring Group Preferences" for information about how to change the default.
- Testbed is Reservable | Not Reservable: Use this parameters in case you need to temporarily suspend reservation operations on the testbed.
- Configuration State - Enable | Maintenance | Disable: Use this parameters to change the current "Configuration State" of the testbed. The default is Enable.

If you specified a Clean YAML file as part of the testbed definition, complete the next step, otherwise, go to step 10.

Click Next to proceed to the Clean Info tab. You have three options you can configure on this tab:

If you specified a platform in your Clean YAML, you can specify which platform the Clean YAML can be applied to.
If you did not specify a platform in your Clean YAML, click the Select All checkbox to choose which devices to clean.
For either option, you can specify the Maximum Clean Request Runtime duration for the testbed.

Click Save to proceed. The testbed appears on the Registered Testbed page.

Viewing Testbed Details

After you register your testbed in XPRESSO, it appears on the Registered Testbed page listed by the creation date.

You can locate/isolate/list testbeds on the Registered Testbed page by selecting the appropriate details tab: All Testbeds | Static Testbeds | LaaS Testbeds.

You can also use the Search panel to locate a specific testbed by name or use the Sort button adjacent to the field name to further refine which testbeds to view.

Drilling Down on Testbed Details

Three layers of information are provided about testbeds:

Layer 1 - Top level; always visible on the Registered testbed page: Provides the Name, Operational/Configuration State, Reservation Status, and Creation Date of the testbed.
Layer 2 - Access via the chevron button to right of the testbed name: Provides additional testbed details such as the device name, image type, OS, and platform name. You can drill down to additional information such as the connection type, IP address, port, and protocol via the chevron button to the right of each device name.
Layer 3 - Access by single-clicking on a testbed: Provides access to the testbed details page that displays information on the following tabs:
- Overview: Provides a summary of the parameters configured on the testbed such as the Maximum Testbed Queue Retention time, Min./Max. Reservation Duration, and specified Location. Also displays additional details if a Clean operation is specified for the testbed such as the device name, platform, and image used.
- Devices: Lists all devices used by the testbed; additional information can be accessed by clicking the chevron button to the right of the device type.
- Testbed YAML: Provides a read-only view of the Test YAML file; to change any details, click the Edit Testbeds button.
- Reservations: Provides a visual timeline of all current/future reservation for the testbed. You can also view past reservation details. Single-click on a Reservation (if set) to access the Reservations and Job Request details for the testbed.
- Queues: Provides access to all testbed Queue information (both recent and stacked Queues) registered to run on the testbed.
- History: Provides an historical record of all significant events/operations that occurred or were set on the testbed.

Post-registration Testbed Operations

You can perform the following post-registration management operations with each testbed type: (see the "Viewing Testbed Details" sub-topic above for information about how to access the different information layers for each testbed).

Static testbeds:
- Layer 1 and 2 Operations: Reserve, Edit (or Edit Testbed), Delete, Maintenance, and Disable.
- Layer 3 Operations: Reserve (duplicated for ease-of-access) and Clean Testbed.
LaaS-NG testbeds:
- Layer 1 and 2 Operations: Maintenance, and Disable.
- Layer 3 Operations: Clean Testbed

NOTE:
Information that follows is limited to unique testbed operations (Reserving and Cleaning); other actions you can perform such as Edit or Delete are standard operations you can perform with most objects in XPRESSO.

Reserving a Testbed

Click the Reserve icon button to reserve a testbed for your Job Runs to book time-slots in XPRESSO in advance of your testing requirements to guarantee system resources for a specific date, time, and duration. See "Creating a Reservation" topic for information about on testbed reservations.

Cleaning a Testbed

You can clean a testbed anytime provided a clean YAML is associated with the testbed. XPRESSO supports associating multiple clean files (and their configurations) to a testbed, and specifying which one will be used by default for all executions. However, this can be overwritten at profile level. That is, you can configure each profile of a job to use a different clean, eventhough they have the same testbed and would have otherwise used the default clean. You have the following options to clean a testbed:

Option 1: Upload clean YAML file(s) when you add (register) a new testbed; see step 4 in the "To Add a New Static Testbed" procedure above.
Option 2: See the procedure below.

To Add a Clean YAML (Post-Registration):

From the Main Navigation Bar, choose the RESOURCES→Testbeds menu to open the Registered Testbeds page.
Click on the testbed that you want add a clean YAML file to and select Edit Testbed OR Use the quick Edit Testbed action on the list view
On the Cleans tab, you can:
1. Upload new clean file(s).

Delete existing clean file(s).

View and modify the clean YAML of all uploaded cleans. This is shown by default (YAML Preview)

Configure each clean file uploaded. i.e. specify image loading mechanism (Device/Platform), request runtime, e.t.c.

Set/Change default clean - which will automatically be used for execution, except specified otherwise in profile/at runtime. The first uploaded file is made the default (until changed).

Rename clean file to something memorable or unique - The file name is the default name (until changed).

Download each of the previously uploaded clean files.

You have two options to upload Clean YAML file to XPRESSO.

Click the Add Clean File button to locate your YAML file(s) and click Open.
You can drag and drop the Clean YAML file(s) onto the Add Clean File button pane.

Once the Clean YAML file is uploaded, a checkmark symbol appears to confirm the file successfully uploaded.

The content of the YAML file(s) are validated. If there are validation errors, they are flagged with a warning icon; You can expand to see full validation error/warning message, and make appropriate changes to the YAML, upon which, it is revalidated.
When configuring each clean File, bear in mind that:

If you have chosen "Platform" as your image loading mechanism, you can specify which platform the Clean YAML can be applied to.
Otherwise, if you have chosen "Device" as your image loading mechanism, you can specify which devices the Clean YAML can be applied to.
Alternatively, you can specify this in your clean YAML and set the sequence in which devices should be cleaned. This is done by adding the `clean_devices` key to your YAML. See details of usage here
For either option, you can specify the Maximum Clean Request Runtime duration for the testbed.

Click Save to update your testbed with latest clean configuration.
For a demo on pyATS clean on XPRESSO, check out pyATS Clean VOD

Standalone Clean - Without Job/Profile execution:

You can choose to clean a testbed, by itself, without necessarily running a job as part of the request.
To do this, in the Testbed detail view, click on 'Clean'. Configure environment variables, request options and execution environment - Select Docker Image for cloud execution or Execution engine and Harness environment for Jenkins execution.
Since there is no concept of profile in standalone clean, this execution will automatically use the 'Default' clean associated to the testbed.
You can overwrite images, specify sequence to clean devices, e.t.c.
Once clean is complete, you will be able to view and download request clean results, logs and output files.

Associating Clean to a Profile:

When creating or editing a testbed, you have the option to associate multiple clean files, however, only one of these clean YAML and configuration can be set as the default to be used by all profiles for execution.
Each profile will automatically use the default clean associated to the selected testbed(s). However, you can edit the profile to always use a different clean from the default one.
To configure this, Edit the profile, and after selecting testbed(s), in the clean tab, expand each testbed. You will notice that the default clean is checked. Select an alternative clean, which will always be used by the profile instead.
Click save to submit changes.
All subsequent executions on that profile after these changes have been saved, will always use the clean you have specified. If no changes were made, it will use the default clean for the selected testbed.

Auto integrate ABS/S2S nightly/greenlable images to scheduler

If your image path changes periodically per build, pyATS now supports providing callables in clean YAML. You can follow the steps below to ensure your executions are run with the correct image:

For ABS nightly greenable images, the "get_last_green_image" function has been added. To leverage, in your clean YAML, rather than specifying a static image value, set this as the value of your image - '%CALLABLE{pyats.cisco.abs.get_last_green_image(argument1,argument2)}', where argument1 is your branch, and argument2 is your target. This gets built into the URL for the API call.
Alternatively, you can write your own function to retrieve periodic images, and pass in whatever arguments you want (if required). To do this:
1. Create a file, with your function, in your harness environment's $PYTHON_PATH.
2. In your clean YAML, invoke by using %CALLABLE. E.g. '%CALLABLE{your_file_name.your_method_name(optional_args)}'
At the time of execution, you can also overwrite images by providing the callable method, which will return the right image, instead of the previous static image value
For ABS nightly greenable images, to implement this flow seamlessly so that your regular runs are not affected, it is recommended that you do the following:
1. Associate a new clean file to your testbed, which will be integrated with ABS to retrieve latest images. This is different from your other clean files used for regular runs. XPRESSO allows you associate as many clean files to your testbed as possible, and specify which one each job profile will use.
2. Then, create a new profile for ABS executions. This is different from your DEFAULT profile, which is used for regular executions. While creating, configure it to use the clean YAML written with callable method to ABS. That is, the clean file you created for ABS integration exclusively. Refer to "Associating Clean to Profile" section above for detailed instruction on this.
3. This way, your DEFAULT profile, for regular runs, will use the DEFAULT clean file associated to your testbed, but your profile, created exclusively for cleaning with ABS integration, will use your clean yaml, which was created for the same purpose, and already references a callable method to retrieve the image(s).

Specifying which Testbed(s) are Used for Job Runs

You have the following options to control which testbeds are used when you execute a Job Run with XPRESSO:

You can specify which testbeds to use in the Job Profile associated with your Job Run. See "Creating a Job Profile" for more information.
Once your test beds have been specified in your Job Profile, you can limit which testbeds are used by selecting/de-selecting the appropriate testbed in the Job Profile associated with your Job Run; see "Configuring Job Request Attributes" for more information.
You have three testbed options to execute your Jobs Runs in XPRESS0:
- On a Single Testbed: You can specify that a Job Run executes on one testbed defined in your Job Request; the Job Run executes when the testbed becomes available according to the testbed priority queuing rules. This is the default testbed option.
- On Multiple Testbeds: You can specify that a Job Run executes on several testbeds defined in your Job Request; the Job Run executes when the first testbed becomes available and is subsequently purged from any other testbed queues once the Job Run starts.
- On all Specified Testbeds: You can specify that a Job Run executes on each specified testbed defined in your Job Request using the "Execute on all Testbeds" parameter located on Request Options tab on the Job Request page. Job execution is still subject to the testbed availability and priority queues for each specified Testbed.
See "Executing Job Requests" for more information on all three options.

Refer to the following topics for other related testbed information:

See "Configuring Job Request Attributes" for information about how to configure a Job Request to specify which testbed(s) to use for a Job Run, specifying if a single or multiple testbed should be used, and how to execute a Job Run so it runs on all specified testbeds.
See "Creating Reservations" for information about how to create reservations for your Job Runs to book time-slots for your testbeds and topologies in advance of your testing requirements and; how to view all "booked" reservations and queue info for a specific testbeds
See "Testbed Queuing" for information about how testbeds and testbed queuing interrelate, why Testbed Queuing is important, Testbed Queuing Rules, and the different methods you can use to control Testbed Queues.
See "Operational and Configuration States" for information about the different testbeds states, what they mean, or how to set them (Configuration states only).
Testbed Availability Status can be obtained via "Webex Bot Notifications". It posts the status as Xpresso Bot.