CML 2.7 Release Notes

Cisco Modeling Labs (CML) is a network simulation platform. CML 2.7 is the latest feature release of CML. This release adds support for running IOL and IOL-L2 images in CML. Starting with this release, CML includes access to the latest IOL-XE image and to the Catalyst SD-WAN images. Other new features include the External Connectivity Editor, the ability to use node types with multiple config files and to customize the SMBIOS parameters for a node's VM, compute host pinning for CML clusters, and the addition to Azure support in cloud-cml.

New and Changed Information

This table shows any published changes to this CML 2.7 release notes document.

Date Changes
August 19, 2024 CML 2.7.2 maintenance release. See Bug Fixes and Known Issues for the list of changes in CML 2.7.2.
July 22, 2024 CML 2.7.1 maintenance release. See Bug Fixes and Known Issues for the list of changes in CML 2.7.1.
April 15, 2024 Updated to warn about the Insufficient space error when upgrading from 2.6.x, make minor corrections, add some FAQs, and add pages for IOL and IOL-L2.
March 27, 2024 Updated with the CML 2.7.0 FCS release.
January 31, 2024 CML 2.7.0 Beta test begins, build.1643.

Installing or Migrating to CML 2.7

Upgrading to CML 2.7

If your CML instance is already running CML 2.3.0 or later, you can upgrade it to CML 2.7.0. See the In-Place Upgrade page for detailed upgrade instructions.

  • We recommend reviewing the Known Issues and Caveats before upgrading to a new CML release. The latest release may include bugs that impact specific features or CML installation types. Depending on how you deploy and use CML, you may want to skip a release and wait until a specific bug is fixed in a future maintenance release.
  • If your CML instance is already running CML 2.3.0 or later, you can upgrade it to CML 2.7.0. If you haven't upgraded to CML 2.6.x yet, see the CML 2.6 Release Notes for more details about changes to the upgrade process starting in CML 2.6.0.
  • Don't forget to disable maintenance mode once the upgrade is complete.
  • Note that if you are upgrading from CML 2.6.x, you must follow the steps in the order documented in In-Place Upgrade. Otherwise, you may encounter an Insufficient space available error when uploading the PKG file. See also the Troubleshooting FAQ.

In-place upgrades from CML version 2.2.3 or earlier are not supported. In that case, a migration is required. The migration process from CML 2.2.3 is detailed in the CML 2.6 Release Notes.

Upgrading a CML Cluster to CML 2.7

If you have deployed a CML cluster, you may upgrade it to CML 2.7.0.

When upgrading from CML 2.4.0, which introduced cluster support, you must manually run the CML upgrade first on the controller and then separately on each compute host.

If you are upgrading from CML 2.5.0+, you only need to run the CML upgrade on the controller. The CML controller's upgrade automatically initiates a CML upgrade on the cluster's compute hosts.

Installing CML 2.7

The CML Installation Guide provides detailed steps for installing your new CML 2.7 instance.

  • In CML 2.7, all the reference platform VM images must be copied to the local disk of the CML instance. Be sure to provision enough disk space for your new CML instance to accommodate the reference platforms.
  • Be sure to check the System Requirements for updated system requirements.
  • Important: After the reference platforms are copied to disk during the Initial Set-Up, wait a couple of minutes for the CML services to finish restarting before clicking OK to continue. If you click OK too soon, the initial set-up can fail. (See the Known Issues below.)

Summary of Changes

This section details new and changed features in the new release. Bug fixes are listed in the subsequent section.

IOL Support and Images

Starting in CML 2.7, the standard ISO of reference platform images includes two IOL node definitions and their associated images. Node definitions and images are included both for an L3 router (IOL) and for an L2/L3 switch (IOL-L2) that run IOS XE 17.12.01. Nodes of these two types do not run as full virtual machines (VMs). Therefore, they consume fewer resources, and using these new IOL node types will permit you to run more nodes at once (up to your license limit) on your existing CML server.

Node Types with Multiple Configuration Files

CML 2.7 expands the support for editing a node's day-0 configuration to permit modifying multiple day-0 configuration files on a node in your lab.

The first time you start a new node in a lab (or the first time you start a node after wiping it), CML can provision a day-0 configuration for the VM. The node definition associated with the node defines which file or files CML should provision for the new VM. You can specify the contents of each node's day-0 configuration in the Workbench, and the day-0 configuration contents are stored with the associated node in your lab.

In earlier versions, CML could provision multiple day-0 configuration files and provide them to the running node's VM, but CML only permitted you to edit one of those files on each node. Starting in CML 2.7, a CML administrator may now mark multiple filenames in a node definition's Provisioning section as Editable. You can then select which configuration file to edit in the Workbench and edit the day-0 configuration text for that file. You may specify the contents for each editable configuration file on all nodes of that type.

Please note that adding or renaming configuration files is not trivial, and you cannot simply select a configuration file to use. The node is configured to expect only specific filenames, and will ignore any files other than those hard-coded into the image. For details, see the note at the top of Node Configuration Files.

Examples of node definitions that can benefit from multiple editable configuration files are Ubuntu (user-data and meta-data files) and Cat9000v (iosxe_config.txt and conf/vswitch.xml files).

Specify SMBIOS Parameters

CML 2.7 introduces an option to specify SMBIOS parameters for a node definition or for a node itself. SMBIOS customization adds a layer of control, enabling users to tailor the virtualized environment to their specific requirements. This feature allows you to simulate different hardware profiles for nodes or influence how operating systems identify the virtualized environment by adjusting SMBIOS parameters, such as manufacturer, product name, version, and serial number. It also allows you to run VM images that require specific SMBIOS parameters, such as a Juniper Switch.

For more information, see Node parameters specification and Node definition parameters specification.

CML supports setting any of the SMBIOS system information values on a node.

UI Changes

Other than UI changes to support the new features mentioned above, there are some other small UI changes and bug fixes in CML 2.7:

  • You can now drag the Sidebar to change its width in the Workspace. In particular, this feature lets you make the Config pane wider while you're editing or reviewing the 0-day configurations for nodes in your lab.
  • Double-clicking a node no longer hides / unhides the node's links.
    • The functionality is still available in CML 2.7: right-click the node and select Hide Links from the node's context menu.
    • In previous releases, you could Hide Links by double-clicking a node, but CML users found it confusing when they accidentally double-clicked a node.
  • When you have enabled Hide Links on a node, all of its links are shown when the hidden node is selected.
  • You may now right-click a bundled link in the Workbench and invoke an action on all of the links in the bundle. For more about link bundling, see the Hiding Links page.
  • The Workbench saves your annotations when you leave the page.
    • If you are changing an annotation's property in one of your labs, and you navigate away from the page (for example, to return to the Dashboard), CML 2.7 saves your latest change to the annotation.
    • You no longer need to click elsewhere in the lab after updating an annotation property to ensure that the change will be saved.

External Connectivity Editor

CML 2.7 extends the functionality of L2 external connectors with the ability to modify the built-in NAT network IP range. CML system administrators can also use the new External Connectivity Editor to add new network bridges to the CML host. These managed bridges are attached to an internal DHCP server and optionally use network address translation to allow connected lab nodes to reach outside resources.

For more information, see External Connectors in the CML Administrator's Guide.

SD-WAN Support and Images

With the release of CML 2.7.0, CML provides Catalyst SD-WAN images to our users on a new ISO file of supplemental VM images. While previous versions of CML could run the Catalyst SD-WAN images, a CML license did not provide access to those VM images. Therefore, only users who had access to the SD-WAN images via some other entitlement could download them and add them to CML.

Note that if you were using the contributed SD-WAN node definitions from the cml-community repository, the node definitions have been updated along with this new release. The CML 2.7 upgrade will not migrate labs that use the old viptella-vmanage node definition to the new cat-sdwan-manager node definition. If you want to use the new node definitions and SD-WAN images, you will need to migrate your labs manually.

Compute Pinning

CML 2.7 introduces the ability to pin a node to a specific compute host.

When you start a lab on a CML cluster, the CML controller assigns nodes to specific compute hosts automatically. This automatic compute host assignment can sometimes lead to suboptimal distribution of nodes to compute hosts. For example, multiple resource-intensive nodes may all be assigned to the same compute host instead of being distributed evenly across the cluster's compute hosts. The compute pinning feature allows administrators to bypass the automatic compute assignment and choose the best compute host themselves.

A CML administrator may only pin a node to a compute host if the node has no persistent state on a compute host. That is, you can only pin a node to a compute host if the node is new and has never been started or if the node was wiped and has not been started since being wiped. A pinned node will only run on the specified compute host even if it is wiped, and a pinned node will fail to start if its associated compute host does not have the capacity to run the node, even if there is capacity to run the node on another compute host.

For more information, see Compute Pinning in the CML Administrator's Guide.

Running CML on AWS or Azure

CML 2.7 makes running a CML instance on Azure a supported deployment option. This option builds on the support for deploying CML to Amazon Web Services (AWS) that was introduced in CML 2.6.

For more information, see the cloud-cml page.

Smart Licensing Transport

CML 2.7 introduces a new transport module, designated as Smart Transport, within Smart Licensing. This module replaces the previous transport module, Call Home, utilized in CML 2.6 and below. The transition to Smart Transport is designed to ensure that CML can use the latest versions of Cisco's Smart Licensing technology.

The upgrade process to CML 2.7 seamlessly completes the transition from Call Home to Smart Transport. This ensures a smooth and hassle-free experience for CML administrators adopting the latest version. As part of the CML upgrade process, default direct transport settings are automatically updated. This involves switching the URL from Call Home to Smart Transport, aligning the system with the enhanced transport capabilities offered by the new module.

This change only impacts CML instances that utilize the default direct transport settings or a transport gateway (e.g., SSM OnPrem). CML instances configured to not communicate with CSSM (i.e., instances configured to use SLR) remain unaffected by this transition.

After you upgrade your CML instance to 2.7, we recommend using Actions > Renew License Authorization Now on the Tools > Licensing to verify that the new license settings are working.

API Changes

CML 2.7 introduces several changes to the REST-based web service APIs. For details about a specific API, log in to the CML UI and click Tools > API Documentation to view the CML API documentation page.

APIs Removed in CML 2.7

This table summarizes the API endpoints that have been removed in CML 2.7.

Removed API Endpoint Use This API Endpoint Instead
GET /licensing/certificate None
POST /licensing/certificate None
DELETE /licensing/certificate None

APIs Deprecated in CML 2.7

This table summarizes the API endpoints that have been deprecated in CML 2.7.

Deprecated API Endpoint Use This API Endpoint Instead
GET /diagnostics Any diagnostics endpoint mentioned in New APIs section.
GET /labs/{lab_id}/nodes/{node_id}/tags/ GET /labs/{lab_id}/nodes/{node_id}

New APIs in CML 2.7

This table summarizes the API endpoints that are new in CML 2.7.

API Endpoint Description
GET /diagnostics/computes Return diagnostic information for compute hosts.
GET /diagnostics/labs Return diagnostic information for labs.
GET /diagnostics/lab_events Return diagnostic information for lab events.
GET /diagnostics/node_launch_queue Return diagnostic information for the node launch queue.
GET /diagnostics/services Return diagnostic information for services.
GET /diagnostics/node_definitions Return diagnostic information for node definitions.
GET /diagnostics/user_list Return diagnostic information for the user list.
GET /diagnostics/licensing Return diagnostic information for licensing.
GET /diagnostics/startup_scheduler Return diagnostic information for the startup scheduler.

APIs Changed in CML 2.7

This table summarizes the APIs that have been changed in CML 2.7.

API Endpoint Applies to request/response Summary of Change
GET /users
GET /users/{user_id}
POST /users
PATCH /users/{user_id}		 response
response
request, response
request, response		 Added property pubkey_info.
PUT /sample/labs/{lab_name} request Set minLength, maxLength and pattern for valid lab_name parameter.
PUT /sample/labs/{lab_name} response Finalized response schema for labs returned.
GET /system/compute_hosts
GET /system/compute_hosts/{compute_id}
PATCH /system/compute_hosts/{compute_id}		 response
response
response		 Removed pattern from property hostname and server_address.
GET /system/external_connectors
PUT /system/external_connectors
GET /system/external_connectors/{connector_id}
PATCH /system/external_connectors/{connector_id}
GET /labs/{lab_id}/connector_mappings
PATCH /labs/{lab_id}/connector_mappings		 response
response
response
response
response
response		 Added new property allowed.
GET /system/external_connectors
PUT /system/external_connectors
GET /system/external_connectors/{connector_id}
PATCH /system/external_connectors/{connector_id}		 response
response
response
response		 Removed pattern from property device_name
Removed pattern from operational property ip_networks.
PUT /system/external_connectors response Schema now recognizes multiple different types of data (string, object).
GET /node_definitions
GET /node_definitions/{def_id}
GET /simplified_node_definitions
GET /node_definition_schema
PUT /node_definitions
POST /node_definitions		 response
response
response
response
request
request		 Added new enum values to libvirt_domain_driver: iol, lxc.
GET /node_definitions
GET /node_definitions/{def_id}
GET /simplified_node_definitions
GET /node_definition_schema
PUT /node_definitions
POST /node_definitions		 response
response
response
response
request
request		 Added new property min_count.
GET /node_definitions
GET /node_definitions/{def_id}
GET /simplified_node_definitions
GET /node_definition_schema
PUT /node_definitions
POST /node_definitions		 response
response
response
response
request
request		 Added new property parameters (optional list of smbios key:value parameters).
PUT /node_definitions
POST /node_definitions
GET /node_definitions
GET /node_definitions/{def_id}
GET /simplified_node_definitions
GET /node_definition_schema		 request
response
response
response
response
response		 Removed driver enum values: iol, iol-2
POST /labs/{lab_id}/interfaces
GET /labs/{lab_id}/nodes/{node_id}/interfaces		 response
response		 Changed mac_address property from maxLength to pattern.
POST /labs/{lab_id}/interfaces
GET /labs/{lab_id}/nodes/{node_id}/interfaces
GET /labs/{lab_id}/interfaces/{interface_id}		 response
response
response		 Updated schema with operational property.
GET /labs/{lab_id}/topology
GET /nodes
GET /labs/{lab_id}/nodes
GET /labs/{lab_id}/nodes/{node_id}
POST /import
POST /labs/{lab_id}/nodes		 response
response
response
response
request
request		 Changed value of property configuration from False to null.
GET /labs/{lab_id}/topology
GET /nodes
GET /labs/{lab_id}/nodes
GET /labs/{lab_id}/nodes/{node_id}
PATCH /labs/{lab_id}/nodes/{node_id}
POST /import
POST /labs/{lab_id}/nodes		 response
response
response
response
response
request
request		 Added new parameters property in topology's "nodes" dict.
GET /labs/{lab_id}/simulation_stats response Updated schema for response code 404 (Not Found).
GET /nodes
GET /labs/{lab_id}/nodes
GET /labs/{lab_id}/nodes/{node_id}		 Updated schema parameters by including property exclude_configuration.
GET /nodes
GET /labs/{lab_id}/nodes
GET /labs/{lab_id}/nodes/{node_id}		 Updated schema parameters by updating property operational.
GET /nodes
GET /labs/{lab_id}/nodes
GET /labs/{lab_id}/nodes/{node_id}		 response
response		 Added new property iol_app_id.
GET /labs/{lab_id}/nodes/{node_id}
PATCH /labs/{lab_id}/nodes/{node_id}		 response
response		 Added new property pinned_compute_id.
GET /labs/{lab_id}/interfaces/{interface_id} response Changed mac_address property from maxLength to pattern.
PUT /labs/{lab_id}/links/{link_id}/capture/start request Updated schema with request body.

Other Changes

Summary Release Notes
Enhanced LDAP Error Reporting The LDAP test authentication API now provides detailed error messages for incorrect LDAP configurations, rather than generic errors.
Node Double-Click Behavior Removal The confusing double-click functionality that hid/showed node links and altered node outlines has been removed. This feature is now only accessible through the node's right-click context menu.
Cytoscape Performance Enhancements Optimizations have been made to improve the lab preview and page loading times, as well as the rendering of links in the workbench.
Improved Sample Lab Error Messaging The sample lab loader now displays a specific error message in the UI when required node definitions are missing, improving user feedback.
LXC and Network Migration Script A new migration script supports transferring CML lab data and compute hosts within clusters for version 2.7.0. It is not designed for other upgrades or merging multiple CML instances.
Simplified Workbench Pane Closure The cross icon in the workbench's PANES pane now closes the pane when only one is open for a more intuitive user experience.
PCL Annotation Support Annotations are now supported in the client library, enhancing documentation and note-taking capabilities.
Enhanced Dark Mode Text Selection Text selection within the node configuration tab is now more visible when using the workbench in dark mode.
SSH Public Key Authentication in virl2-sshd The virl2-sshd service now supports SSH Public Key authentication for secure and convenient access.
Workbench Sidebar Resizability The sidebar in the workbench has been improved to allow for both vertical and horizontal resizing, aiding in configuration tasks.
Anonymized Node Configuration in Events The Diagnostics -> Events section now anonymizes node configuration content, displaying only the length and alteration status rather than the full details.
Node Configuration Exclusion Option API calls for node data can now exclude the node configurations, reducing payload sizes where configurations are unnecessary.
VLAN Zero Tag Cleaning Service A new service is available for customers affected by 802.1p tags on bridged networks, particularly on UCS bare metal deployments, to correct external connectivity issues.
Reduced Node Data Requests in System Admin The automatic periodic GET requests for node data on the System Admin page have been replaced with data processed from websocket events to reduce unnecessary traffic.
Cat9kv DD Reshipment The Cat9000v series has been updated, with the reintroduction of the Cat9000v UADP (formerly Cat9000v DD) and an update to the Cat9000v Q200 model.
Intelligent CML Disk Selection During Install The baremetal installation process for CML has been improved to automatically select the appropriate disk, with a longer wait time for user input when multiple disks are present.
Specific Low Resource Launch Fail Messages Messages indicating high system load now differentiate between CPU, memory, and disk space constraints for clearer troubleshooting.
Linux Login Message Update Compute host console login messages have been updated to display more relevant system information.
Deprecated GET Tags Endpoint The GET /labs/{lab_id}/nodes/{node_id}/tags endpoint has been deprecated in favor of more modern and efficient alternatives.
CML Version Display in Cockpit UI The Cockpit UI has been updated to display the current CML version and logo for better user awareness.
Improved Service Health Status Handling Health status reporting and diagnostics have been improved for scenarios where compute host services are experiencing issues.
Enhanced NFS Health Detection The process for detecting NFS health has been improved to perform periodic checks and handle disconnections properly.
Automatic Image Re-mounting Service A new service has been introduced to re-mount reference platform images automatically after CML services restart or a host reboot in a cluster environment.
virl2-sshd SFTP Protocol Support SFTP protocol support has been implemented in the virl2-sshd service for secure file transfer capabilities.
Expanded Diagnostics API Endpoints The diagnostics API has been expanded with multiple endpoints, such as GET /diagnostics/computes and GET /diagnostics/labs, to provide targeted diagnostic data.

Bug Fixes

Bug Description Resolution
Upgrades to CML 2.7 did not replace default expired certificates. Fixed in 2.7.2. Fixed a bug that caused the upgrade process not to replace the default self-signed certificates. The default self-signed certificates provided by earlier CML releases for the UI and the System Administration Cockpit have now expired. CML 2.7 replaces those expired self-signed certificates during upgrade with valid certificate. This change doesn't affect CML instances that are using custom certificates.
Upgrades to CML 2.7 failed when the latest OS updates had already been installed. Fixed in 2.7.2. Fixed a bug that prevented in-place upgrade to CML 2.7.x when latest OS updates had been installed. The fix ensures that CML "extras" are installed during upgrade.
Upgrades from CML 2.3 to CML 2.7.1 broke the underlying CML instance. Fixed in 2.7.2. Fixed a bug that did not allocate a COMPUTE_ID to a CML 2.3 instance upgraded to 2.7.1 and caused the CML instance to be unusable.
Database backups could be read by unprivileged users. Fixed in 2.7.2. Since CML systems normally have only a single privileged system administrator account, this fix should have little impact in practice. This fix was made to follow the principle of least privilege: there's no reason for unprivileged accounts on the CML server to have read access to the sensitive data in a CML database backup.
In-place upgrades to CML 2.7.1 froze and never completed. Fixed in 2.7.2. Fixed a bug that caused the upgrade script to execute a process that could have been already executed in parallel which resulted into a frozen upgrade.
Unable to import lab which was exported from the same CML 2.6.1 instance Fixed in 2.7.1.
  • Improved the error message to elide the middle part of error content for long error messages so that the UI shows the root cause of the error.
  • Fixed a bug that didn't trim the lab notes of imported VIRL 1.x topologies to the field's size limit.
  • Increased the limit of lab notes to 32768 characters.
The bridge protection service is down after a restart or power cycle of the CML instance. Fixed in 2.7.1. After you restart or power cycle your CML instance, the bridge protection service may not come up. You can see the failed service in the CML Cockpit Services tab. Bridge protection service failure can cause L2 traffic from virtual nodes inside CML to reach your network outside the CML instance and can result in network disruptions. This bug affects all 2.6.x and 2.7.x instances.

Workaround: After power cycling or restarting CML, open the CML Cockpit UI, navigate to the Services tab, search for virl2-protect-bridges@bridge0.service and make sure that it is up and active. If it is down, use the toggle button to start and enable the service.
virl2-protect-bridges@bridge service fails after CML shutdown Fixed in 2.7.1. Fixed an issue where commands setting up firewall for bridge protection on Ubuntu 20.04 fail if they are executed too close together.
NetworkManager ignores vNIC bridge configurations created by Cockpit. Fixed in 2.7.1. Worked around a bug that caused the configuration for a L2 bridge member interface (as created through Cockpit) to be ignored after reboot on Ubuntu 20.04. The problem required the sysadmin to re-add the the port manually after a reboot.
Failure to load a node or image definition yaml file used to break CML service initialization. Fixed in 2.7.1. Fixed a bug that led to not loading all node and image definitions if one of the files was corrupt. As improvements had been done previously, this problem was happening only in rare cases.
Fabric and pcapdemux panic when running a packet capture. Fixed in 2.7.1. Fixed a bug in pcapdemux which mishandled logging in very verbose mode of transferred packet capture data, causing a crash in fabric service.
The back button doesn't work on the network settings page of the CML Initial Setup process. Fixed in 2.7.1. Fixed button labels and allow going back during the Initial Setup if only one interface exists on the host.
Fix Create Link dialog accessibility. Fixed in 2.7.1. Fixed aria labels that were not getting attached properly to the dropdown elements when adding a new link.
UI does not always reflect topology changes (removed nodes) performed in another session. Fixed in 2.7.1. Fixed a bug in the UI that prevented nodes from being removed from the canvas when a selected node was deleted by another session.
The initial tour of the Workbench did not respect the Dark Mode preference setting. Fixed in 2.7.1. Fixed the color schema of the Workbench Tour in the dark mode.
Large CML systems had poor performance when opening the Diagnostics page in the UI. Fixed in 2.7.1. A few small changes were made to the Diagnostics page in UI to improve performance. The most significant change was the separation of API endpoints and individual loading of tabs. Thus, less data is fetched when you initially open the Diagnostics page in the UI, and CML loads the specific data needed for a specific tab when you access it.
Improve UI performance. Fixed in 2.7.1. Implemented a few small UI performance improvements, e.g. labels are now hidden when they are unreadably small and the number of render calls is reduced which causes blur.
Add Node dialog doesn't disappear when you drag a node. Fixed in 2.7.1. Fixed a bug that prevented the Add Node dialog from disappearing when a node is being dragged onto the canvas.
Certain markdown tags weren't displayed in the Lab Guide for a lab opened in the Workbench. Fixed in 2.7.1. Fixed multiple UI elements to display markdown correctly.
Nodes sometimes got stuck in QUEUED state with no indication that they will never start. Fixed in 2.7.1. Fixed a bug where some errors which prevent node starts were not propagated properly, causing nodes to stay in the launch queue instead of rejecting the start.
Changing the configuration file name in a node definition resulted in nodes that failed to start (i.e., node VM crashed on start). Fixed in 2.7.1. Changing the name of the main configuration file in a node definition no longer causes preexisting nodes associated with that node definition to fail to start.
Client library (virl2-client) cannot load labs with annotations. Fixed in 2.7.1. Fixed a bug in the client library that prevented labs with annotations from loading.
Increase number of interfaces for IOL/IOL-L2 nodes from 16 to 32. Fixed in 2.7.1. The maximum number for IOL and IOL-L2 nodes have been increased to 32.
IOL images that need a license will not start if the hostname is greater than 26 characters. Fixed in 2.7.1. Shorten the hostname of IOL containers in case the hostname is longer than 26 characters to fix issues with licensing such IOL containers.
When you selected any of the links between two nodes and pressed the DELETE key, the CML UI was always deleting all links between those two nodes. Fixed in 2.7.1. CML now correctly deletes only the selected link(s).
TRex 3.19.1 bundled with CML 2.7.0 does not connect to GUI Fixed in 2.7.1. Re-built TRex image to support TRex GUI client. If you previously used the refplat-20240225-fcs-iso.zip file from the 2.7.0 release, you can replace the TRex image with the one from the the updated refplat-20240623-fcs-iso.zip file.
NX-OS 9000 10.4(2)(F) image def had incorrect RAM. Fixed in 2.7.1. The default memory of NX-OS 9000 images has been increased to the new minimum of 12 GB.
The default day0 configuration of NX-OS 9000 nodes was using the wrong key for the loader workaround. Fixed in 2.7.1. A workaround script that enables nodes to boot again after being stopped first determines a path to a versioned boot file on the node. A key used by this script changed as far back as the 7.x releases of Nexus 9000v from kick_file_name to nxos_file_name. After upgrade, when you add a new NX-OS 9000 node to a lab, the default day0 configuration for node will use the correct key.
If you have used this loader workaround in your existing lab nodes and yaml topology files, you'll need to update your day0 configurations for NX-OS 9000 nodes accordingly. The updated default only applies to nodes that you add to labs after the upgrade to 2.7.1.
The default bootstrap configuration for Catalyst SD-WAN Edge nodes does not work. Fixed in 2.7.1. The updated Catalyst SD-WAN Edge node definition has a corrected default day0 configuration. New Catalyst SD-WAN Edge nodes added to a lab after the upgrade will get the new day0 configuration. You will have to manually update and correct the day0 configuration of any existing Catalyst SD-WAN Edge nodes in your labs.
Fixed the SD-WAN Edge initial boilerplate configurations. Fixed in 2.7.1. Fixed the invalid default SD-WAN Edge configuration provided by the node definition and also invalid SD-WAN Edge configurations generated by Build Initial Configurations.
In an SD-WAN lab, the vManage NMS application is not coming up. Fixed in 2.7.1. Some CML users have reported that the NMS application may not start up after they boot the vManage node in their CML lab.

Workaround: The SD-WAN vManage node definition that is bundled with refplat-20240322-supplemental-iso is configured to use 16 GB RAM. Increase the RAM to 32 GB and try again on a new vManage node.

If you have already assigned 32 GB RAM to the node, and the NMS application is still down, run request nms all restart on the vManage node CLI.
Convert cat9kv S1 and DD into Q200 and UADP on import. Fixed in 2.7.1. Fixed a bug that broke the conversion of renamed Catalyst 9000v Q200 and UADP (formerly S1 and DD) in imported labs.
Unable to successfully boot legacy IOL images which require 32-bit architecture. Fixed in 2.7.1. Added i386 architecture to the CML hosts for running some 32-bit legacy IOL images.
Errors when upgrading a CML cluster because of failed node definition patches on compute hosts. Fixed in 2.7.1. CML now applies node def patches in the postinst step only on the controller.
On a CML cluster, it was possible to register two compute hosts with the same hostname, causing problems. Fixed in 2.7.1. In a CML cluster, the controller now refuses to register a compute host with the same name as another compute. This issue is typically caused by an installation or upgrade failure, and you should contact TAC for help to resolve the problem. If a new compute host should legitimately replace another compute host with the same hostname, please decommission the old compute host first.
On a CML cluster, libvirt upgrades failed on compute hosts. Fixed in 2.7.1. Fixed an issue where on cluster compute hosts, updates to the libvirt-daemon-system Ubuntu package failed. Users who have updated the package outside of CML upgrades may fix the issue on their existing CML instances by running
sudo dpkg-statoverride --add root root 0755 /var/lib/libvirt/images; sudo apt install -f
CML permitted Group14 SHA1 keys to be used to connect to the CML Terminal Server. Fixed in 2.7.1. Improve security by forbidding the diffie-hellman-group14-sha1 algorithm in the Terminal Server.
Node parameters should not be supported on external connectors or unmanaged switches. Fixed in 2.7.1. Node parameters are no longer configurable on External Connector or Unmanaged Switch nodes where those parameters are not relevant.
Node parameters are not cloned. Fixed in 2.7.1. Fixed a bug in the Clone action on a node that prevented node parameters from being cloned.
GET link conditioning API is not available to users with read-only permissions. Fixed in 2.7.1. Fixed a bug that prevented link conditioning from being displayed with read-only permissions.
When you upload the CML upgrade PKG file on the Tools > Upgrade System page, the upload fails with an error indicating Insufficient space available on controller and Bytes Available 0. A bug in CML 2.6.x will always cause you to get this error if you put the controller into Registered state before you upload the PKG file. If you experience this problem while upgrading from CML 2.6.x, see the Troubleshooting FAQ for workarounds.
Delete Key Behavior in Settings Tab Resolved an issue where pressing the Delete key within settings tab input fields or text areas no longer triggers the node deletion dialog.
Swagger Interface Schema Improvement Fixed the API schema for topology interface responses to ensure accurate representation in Swagger documentation.
Multiple Link Selection Highlight Addressed a visual bug where occasionally, after deselecting multiple links, one link would remain highlighted.
Early Console Opening in Firefox Corrected a bug that caused Firefox to slow down when consoles were opened shortly after a node entered the started state.
Devicemux Recursive Libvirt Close Fixed a devicemux bug that prevented console start and stop operations following a restart of the Libvirt service.
Removal of Vulnerable SSH Algorithm Updated SSH server configurations to remove or disable the chacha20-poly1305@openssh.com algorithm due to CVE-2023-48795, thereby mitigating the Terrapin attack.
Reference Platform ISO Instructions Update Clarified instructions in the CML Install wizard for copying the reference platform ISO, indicating that it's not critical to have the ISO during installation as images can be added afterward.
Persistence of Deleted Custom Bridges in Cockpit Ensured that when a custom bridge is deleted, it is also removed from the list of available external connectors in the CML UI Administration Tools > External Connectors section.
Non-Admin User Removal with LDAP Enabled Fixed a bug that previously blocked CML administrators from removing users while LDAP authentication was enabled.
Username Case Sensitivity in CML vs. Active Directory Unified username case sensitivity by making all usernames case-insensitive, matching Active Directory's behavior. This may affect pre-upgrade phases of existing CML deployments.
NFS3 Usage with QEMU Base Images Improved the detection of NFS-related issues in cluster deployments, particularly concerning the use of QEMU base images with NFS3.
Concurrent Map Read/Write in Dispatcher Resolved a bug in the console dispatcher service that could cause a crash during concurrent map reads and writes, affecting lab node boot progress monitoring.
Debug Log Level and Fabric Service During Restart Separated component log levels to allow administrators to set LOG_LEVEL=DEBUG without breaking the fabric during virl2.target restarts.
Loss of Annotations Fixed an issue where annotations would be lost if a user left the workbench without clicking on the canvas; annotations are now automatically saved upon exiting.
UMS Start Behavior with Unavailable Compute Hosts Modified the start behavior for UMS and other nodes when compute hosts are disabled or unavailable in both single host and cluster deployments. Lab nodes will not start until compute hosts are available and a new start request is made.
Custom Column Reset in Nodes Pane Addressed a bug that caused custom column settings to reset when the resource pool column was enabled in the Nodes pane.
Node Start Error Messaging in Clusters Improved error messaging when a node start is rejected due to controller limitations, ensuring that errors are relevant and specific to the issue at hand.
Annotation Border Style Update on WebSocket Events Fixed a bug that prevented real-time updates to the annotation border style when changes were broadcasted via WebSocket events.
Validation Against Invalid Elements on Canvas Enhanced validation to prevent errors from invalid or random strings dropped onto the canvas; such input is now silently ignored if not valid.
Smart License Registration Enablement Post-Licensing Reset Resolved a bug that hindered the re-enabling of Smart License Registration (SLR) after a licensing reset while nodes were running.
VNC CTRL + ALT + DEL Function Restoration Restored the non-functional CTRL + ALT + DEL keyboard command for VNC access.
Node.remove() Method Behavior in Client Library Corrected the Node.remove() method in the Client Library to prevent incorrect NodeNotFound exceptions.
Suppressing Warnings for Old Lab Schemas in Client Library Eliminated unnecessary warnings when creating labs with old schemas using the Client Library.
Log Download During Service Failures Fixed the log download feature to function even when CML services are in a failed state.
PyATS Integration Test Handling Addressed a compatibility bug with PyATS in the Python Client Library, ensuring proper test-bed loading and disabling unnecessary variable replacement based on CLI arguments.
Migration Script Compute ID Consideration Improved the migration script to maintain compute host identifiers during migrations between hosts of version 2.4.0 or later.
ASAv Image Definition Configuration Update Updated the ASAv image definition configuration to ensure accuracy.
Enhanced LLD NFS Handling in CML Enterprise Clusters Improved NFS health checks in CML Enterprise clusters for more robust detection and handling of issues with reference platforms.
Removal of Incomplete Image Definitions Fixed a bug that prevented the proper removal of image definitions when their associated image file was missing.
Custom Bridge Protection Default State Correction Corrected a bug where the L2 bridge protection service for external connectors was reported as enabled by default but was not actually started.
Client Library Performance Syncing Addressed a performance issue in the client library when syncing operational parameters while listing lab nodes.
Improved Add Nodes Pop-out Search Functionality Enhanced the search tool in the Add Nodes pop-out to better match on labels.
External Connector Configuration with Resource Pool Restrictions Fixed an issue that now allows enterprise users to see which external connectors are disallowed by resource pool limits set by an administrator.
Disk Resizing While VM Is Running Fixed a bug to allow the maximization of lab storage after the disk is resized while the VM is running.
Deletion Order of Links and Nodes in UI Corrected a bug to ensure links are deleted before nodes when removing multiple objects in the UI simultaneously.
Configuration Upgrade on Installation Retry Resolved an issue that impeded configuration upgrades when attempting to reinstall the cml2 package after a previous failure.
AWS Installation and Libvirt Default Network Creation Fixed a bug in AWS installations/upgrades that prevented the default network from being properly recreated.
LDAP User Assignment to Resource Pool Fixed a bug that prevented existing users from being assigned to a resource pool when LDAP was enabled.
Local Authentication Data Handling Corrected an issue where LDAP data was unnecessarily included when configuring local authentication methods in System Administration.
Cat9kv Node Definition Patch Reapplication Resolved a bug that caused repeated failed attempts to reapply a patch to the cat9kv node definition.
IP Address Retention in VMware Fusion Addressed a problem where network connections could become misconfigured after a shutdown in VMware Fusion, with the fix applied to new deployments or those rerunning initial configuration.
IOSv and IOSvL2 Node Boot with Software RAID Fixed an issue preventing the boot of IOSv and IOSvL2 nodes on deployments with a block size of 4096.
Fabric Remotemux Bind Error After Reboot Corrected a bug in cluster deployments that prevented lab traffic from being received from other cluster compute hosts, impacting external connectors and unmanaged switch nodes.
Pre-Login Message Enhancement for Installations Implemented an informative pre-login message to guide users to support if the CML installation dialog fails to start.
Unmanaged Switch State Reflection in Controller Fixed a discrepancy in the displayed state of unmanaged switches for certain situations, such as when the corresponding bridge fails to activate.
Default GRPC Service Ports Adjustment Updated internal processes to bind to registered port ranges instead of dynamic ones to avoid port conflicts.
Copy Reference Platform Script Robustness Improved the reference platform copy script to handle cases where the ISO is already mounted or specified in an unexpected location.
Reference Platform Reload in Installation Script Fixed an issue allowing for the reload of the reference platform during installation if it's attached after the CML VM boots.
Workbench External Connector Mappings Update Resolved a bug that required a page refresh to update external connector mappings in the Workbench.
Diagnostics Page Display Fix Fixed the Diagnostics page to maintain content visibility when the screen width is reduced.
Licensing Feature API Validation Enhanced the licensing feature API to disable the save button when no license is selected in the "Choose Licenses" dialog.
File Upload Availability in REGISTERED State Fixed a bug that blocked file uploads when the controller was in the REGISTERED state, such as during a system upgrade.
Console Size Settings Consistency in Firefox Corrected inconsistencies with maximal console resolution settings in Firefox.
Persistent UI Indicator on Links Fixed a bug to ensure the UI indicator for links remains persistent and does not disappear in certain scenarios.
Link Menu Node Navigation Fix Corrected a navigation issue where links to nodes in the link menu did not open the node side menu.
Stopping an Already Stopped Link Modified behavior so that attempting to stop an already stopped link is treated as a no-op instead of producing an error.
Unmanaged Switch Configuration Handling Clarified that Unmanaged Switches (UMS) no longer accept configurations since such settings were not utilized.
Error Response Format for Large Image Uploads Adjusted API error responses (429, 413, 502) to return JSON instead of HTML during large image file uploads.
Annotations Input Validation Consistency Aligned the UI input validation for annotations with the API to ensure consistency.
Interface Deletion Restriction Fixed a bug that allowed deletion of interfaces while the node was running.
Dashboard Annotations WebSocket Update Resolved an issue where changes to annotations did not update in real-time on the lab preview in the Dashboard.

Known Issues and Caveats for CML 2.7

Bug Description Notes
External connectivity does not work in CML bare metal installations. This bug impacts CML bare metal deployments regardless of release.

In some deployments, most notably on UCS bare metal servers in which the vNIC's VLAN is set to access mode, the ingress traffic is tagged with vlan 0. This VLAN is meant to carry priority information as per the 802.1p standard.

Network services can normally ignore this tag and handle vlan 0 packets transparently as untagged (no VLAN), but some node types in CML do not handle it gracefully, ignoring the incoming packets. As a result, the virtual nodes connected to an external connector node may not acquire an IP address from an external DHCP server or may not be able to communicate with external devices.

Workaround: Follow the instructions from the Handling 802.1p Frames in UCS section of the Cisco Modeling Labs Administrator's Guide.
After a restart, you cannot reach the CML UI. The console of the CML instance shows a login prompt, but it also indicates that CML does not have an IP address. After you shut down and restart a CML instance, it sometimes loses its IP address. When that happens, you will no longer be able to reach the CML UI even though the CML services are running. This bug affects all 2.7.X instances.

Workaround: When you encounter this problem, open the CML console (e.g., the CML VM's console in VMware or the CML server's remote management console). Log in as the sysadmin user and run this command:

sudo netplan apply

You can verify the current/new IP address of the CML instance using this command:

ip addr show bridge0

If the issue persists, run the following command to call the initial setup script and complete the initial setup wizard to resolve the issue:

sudo HOME=/var/local/virl2 /usr/local/bin/virl2-initial-setup.py --reconfigure
Cannot run simulations on a macOS system that uses Apple silicon (e.g., the Apple M1 chip). CML and the VM Images for CML Labs are built to run on x86_64 processors. In particular, CML is not compatible with Apple silicon (e.g., the Apple M1 chip).

Workaround: There are no plans to support ARM64 processors at this time. Deploy CML to a different system that meets the system requirements for running CML.
Even if you were able to run network simulations with CML 2.2.x on VMWare Fusion and macOS Big Sur or above, the same labs no longer start after migrating to CML 2.3.0+. A default setting in the CML VM changed between CML 2.2.3 and CML 2.3.0. Because of the new virtualization layer in macOS 11+, this setting causes errors when running the CML VM on VMware Fusion.

Workaround: follow the instructions to disable nested virtualization in the CML VM.
Bug in macOS Big Sur. After upgrading to Big Sur and VMware Fusion 12.x, simulations no longer start. MacOS Big Sur introduced changes to the way macOS handles virtualization, but the problems were resolved in later versions of macOS Big Sur and VMware Fusion.

Workaround 1: If you are still experiencing problems, be sure to upgrade to the latest version of VMware Fusion 12.x and the latest bug fix release of your current macOS version.

Workaround 2: If you are using CML 2.3+, then follow the instructions to disable nested virtualization in the CML VM.
Deleting or reinstalling a CML instance without first deregistering leaves the license marked as in use. Your new CML instance indicates that the license authorization status is out of compliance. If you have deleted or otherwise destroyed a CML instance without deregistering it, then the Cisco Smart Licensing servers will still report that instance’s licenses are in use.

Workaround 1: Before you destroy a CML instance, follow the Licensing - Deregistration instructions to deregister the instance. Properly deregistering an instance before destroying it will completely avoid this problem.

Workaround 2: If you have already destroyed the CML instance, see the instructions for freeing the license in the CML FAQ.
When the incorrect product license is selected, the SLR (Specific Licensing Reservation) can still be completed. It eventually raises an error, but the SLR reservation will be complete with no entitlements. Workaround: If you accidentally select the wrong product license, then you will need to release the SLR reservation and then correct the product license before trying to complete the SLR licensing again.
If you have read-only access to a shared lab, some read/write actions are still shown in the UI. Attempting to change any properties or to invoke these actions raises error notifications in the UI. Workaround 1: Do not attempt to invoke the read/write actions when working with a lab for which you only have read-only permissions.

Workaround 2: Ask the lab owner to share the lab with read/write permissions so that you can make changes to the lab.
If you have read-only access to a shared lab, you will still not be able to see the current link conditioning properties in the link's Simulate pane. Workaround: ask the lab owner to share the lab with read/write permissions if you need to access the link conditioning properties.
The algorithm for computing the available memory does not take into account maximal possible memory needed by all running nodes. It only checks the currently allocated memory. CML will permit you to start nodes that fit within the current available memory of the CML server. Later, if the memory footprint of the already-running nodes grows and exceeds the available memory on the CML server, those nodes will crash. Workaround 1: Launch fewer nodes to stay within your CML server's resources.

Workaround 2: Add additional memory to the CML server to permit launching additional nodes. Of course, this workaround just moves the point where the CML server will eventually hit its limit, so you must remain aware of your CML server's resource limits or ensure that the CML server has sufficient memory to accommodate all labs that will be running on your CML server simultaneously.
When starting a CSR 1000v node for the first time or after a Wipe operation, the interfaces are all in shutdown state. Workaround 1: Add no shut to each interface configuration in each CSR1000v node's Edit Config field in the UI.

Workaround 2: Use the EEM applet on the CSR 1000v page.
New nodes cannot be started when CPU usage is over 95%, when RAM is over 80%, or when disk usage is over 95%. Workaround: Allocate additional resources to the CML server.

Workaround 2: If you're hitting the disk usage limit, check the Remove Orphaned VMs function in Cockpit. Orphaned VMs can consume a significant amount of disk space even though CML is no longer using or managing those VMs.
The journalctl output shows errors like this: cockpit-tls gnutls_handshake failed: A TLS fatal alert has been received This log message only occurs with Chrome browsers and has no functional impact on the use of Cockpit. No workaround needed.
In CML Cluster, the Cockpit Storage table for NFS mounts looks broken. This is a display bug of the Cockpit UI when it encounters an NFS share using an IPv6 address. The single mount is split into two rows, only the first of which is indicated as mounted. Your NFS setup is probably fine, as long as the CML UI shows in its Cluster Status view that each compute host's Images indicator is green. Trying to edit or repair the NFS mount in any way in Cockpit is more likely to break the configuration. Please contact TAC if you experience networking issues with clusters which cannot be resolved by verifying basic connectivity on the cluster network or by restarting CML services.