CML 2.9 Release Notes
Cisco Modeling Labs (CML) is a network simulation platform. CML 2.9 is the latest feature release of CML. This release introduces Docker container support, external lab repositories, node disk image cloning, and fine-grained group permissions for lab sharing. Starting with this release, CML provides access to an IOL node type and image that supports serial interfaces and a variety of container images. The supplemental reference platform ISO image now includes images for Catalyst 9800-CL Cloud Wireless Controller, Cisco Identity Services Engine (ISE), and Splunk Enterprise. Other new features include custom font for the Console Panes, display of snooped IP addresses in the Sidebar, new sample labs, and the ability to specify the default console in the node definition.
New and Changed Information
This table shows any published changes to this CML 2.9 release notes document.
Date | Changes |
---|---|
August 20, 2025 | Updated Known Bugs with newly-found issues and removed or updated fixed issues. |
July 24, 2025 | Added notices about the missing / corrupt supplemental reference platform ISO. |
July 18, 2025 | Initial version for CML 2.9.0 FCS release. |
Installing or Migrating to CML 2.9
Upgrading to CML 2.9
If you are still running CML 2.7.1 or older, you cannot upgrade it directly to CML 2.9. Your CML instance must be running CML 2.7.2 or CML 2.8.x before you can upgrade it to CML 2.9. If you have not upgraded to CML 2.7.2 yet, see CML 2.7 Release Notes for more details about the process to upgrade to CML 2.7.2.
See the In-Place Upgrade page for detailed upgrade instructions.
- Online upgrades require access to Docker repositories hosted at
https://download.docker.com
. Your network environment must allow connections to this host over the Internet, otherwise upgrade will fail. - We recommend reviewing the Known Issues and Caveats before upgrading to a new CML release. The latest release may include bugs that impact specific features or CML installation types. Depending on how you deploy and use CML, you may want to skip a release and wait until a specific bug is fixed in a future maintenance release.
- Do not forget to disable maintenance mode once the upgrade is complete.
In-place upgrades from CML version 2.2.3 or earlier are not supported. In that case, a migration is required. The migration process from CML 2.2.3 is detailed in the CML 2.6 Release Notes.
Upgrading a CML Cluster to CML 2.9
If you have deployed a CML Cluster, you may upgrade it to CML 2.9 provided that your CML cluster is already running CML 2.7.2 or later.
When upgrading from CML 2.4.0, which introduced cluster support, you must manually run the CML upgrade first on the controller and then separately on each compute host.
If you are upgrading from CML 2.5.0+, you only need to run the CML upgrade on the controller. The CML controller's upgrade automatically initiates a CML upgrade on the cluster's compute hosts.
Installing CML 2.9
The CML Installation Guide provides detailed steps for installing your new CML 2.9 instance.
- In CML 2.9, all the reference platform virtual machine (VM) images must be copied to the local disk of the CML instance. Be sure to provision enough disk space for your new CML instance to accommodate the reference platforms.
- Be sure to check the System Requirements for updated system requirements.
- Important: After the reference platforms are copied to disk during the Initial Set-Up, wait a couple of minutes for the CML services to finish restarting before clicking OK to continue. If you click OK too soon, the initial set-up can fail. (See the Known Issues below.)
Summary of Changes
This section details new and changed features in the new release. Bug fixes are listed in the subsequent section.
Support for Container Images
With the CML 2.9.0 release, we have added support for Docker containers in CML.
Before version 2.9, CML supported running the following node types:
- KVM virtual machines — full-fledged x86 virtual machines (VMs) that bring their own kernel and virtualized devices
- IOL — using libvirt LXC containers, these node types provided a way to start up IOL and IOL-L2 devices in CML. The use of LXC containers was limited to the IOL and IOL-L2 node types.
- built-in supporting node types - External Connector and Unmanaged Switch
Starting with CML 2.9.0, when you create or import a new node definition, the Domain Driver can be KVM virtual machine, IOL, or Docker container. You can create a node definition to use your own Docker container within a CML lab. We have also added several container node types to the reference platform ISOs for CML 2.9:
Container | Description |
---|---|
chrome | Chrome web browser, which can be accessed via the VNC console in CML |
dnsmasq | DNS (default), DHCP (off by default) and TFTP (off by default). |
firefox | Firefox web browser, which can be accessed via the VNC console in CML. |
freeradius | A service container with FreeRADIUS |
frr | A Free Range Routing (FRR) container |
net-tools | A container with a variety of network troubleshooting tools installed, such as tcpdump , tshark , and traceroute |
nginx | A service container for the nginx web server |
splunk | A service container with Splunk Enterprise |
syslog | A service container with syslog-NG |
tacplus | A service container with TACACS+ |
thousandeyes-ea | A service container with a ThousandEyes Enterprise agent |
Sample Labs Improvements
We have included more sample labs with CML 2.9. See Tools > Sample Labs in the CML UI. Even better, CML 2.9 makes it easier for CML administrators to update the sample labs in between CML releases. You can also supplement or replace the default list of sample labs that are included with CML by default.
Prior to CML 2.9, a fixed list of sample labs were included with the CML controller. You weren't able to replace them with your own custom list of sample labs. Starting in 2.9.0, the sample labs included with the CML controller are tied to a lab repository. CML administrators can add or customize lab repositories and link them to Git repositories. The sample lab content is synced from the git repository and cached on the CML controller host. Users of the CML system can then browse this catalog of sample labs just as they do today in Tools > Sample Labs. The CML 2.9 release is pre-configured with the cml-community repository, and a snapshot of the sample labs from that repository are already bundled in the release.
Custom Font for Console Pane
The Console pane for accessing the nodes' terminals in the CML Workbench now has a configurable font. You can select the font in the Global Console Settings dialog.
Address Display in Interfaces Pane
When a node is L2 adjacent to an external-connector that has an IP address snooper running, such as the NAT external connector, CML will display the IP addresses learned by the IP address snooper. You can see the IP address in the Interfaces page of the Sidebar when you select the node in the Workbench.
The Interfaces page of the Sidebar will also show the MAC addresses in use by the node's interfaces if CML has randomly assigned a MAC address to the interfaces. In previous releases, CML only showed a MAC address in the Workbench if you had set the MAC address to the interface explicitly.
Node Disk Image Cloning
To make it easier to create custom VM images from an existing image on your CML server, we have added the node disk cloning feature in CML 2.9.0. While you have always been able to add third-party and custom VM images to your CML instance, the CML administrator had to use a separate system or access CLI tools on the CML server itself to create a custom image. With Node Disk Image Cloning, a CML administrator can convert the disk image of a specific node in an existing, instantiated lab and convert it to a new image that's available to use with an existing or new node type. This feature provides a simpler way to create, for example, a custom Ubuntu Linux VM image with specific packages already installed and configured. All users on your CML instance could then use that custom VM image for their Ubuntu nodes, and their nodes will boot up with those packages already installed and configured.
Lab Sharing and Advanced Group Permissions
Since we introduced Lab Sharing for multi-user CML deployments in CML 2.2, the permission level when you shared a lab with a group was a simple selection between "read-only" or "read-write". With the 2.9 release, we're introducing more fine-grained permissions: View, Exec, Edit, and Admin.
Starting in CML 2.9, you may also share a lab directly with an individual user, not just a group. This improvement enables a user to share their lab with another user on the same CML instance without the assistance of a CML administrator to create a lab sharing group.
IOL Nodes with Serial Interfaces
A year ago, we were happy to provide IOL-XE node definitions and images for you to use within your CML labs. The IOL-XE node types and images were well-received, but many of you asked us about serial interfaces with IOL. You asked, and we listened!
CML 2.9.0 provides a new IOL node type and image that supports serial interfaces. If you are upgrading an existing CML instance to CML 2.9, you will also need to add the new iol-xe-serial-4eth node definition and the corresponding image from the reference platform ISO image to your CML instance. You will then be able to add one of these new nodes to your lab. These IOL serial nodes have a static 4-port Ethernet "module" installed, but the rest of the interfaces on the IOL node are of type "serial". These serial interfaces support protocols like FrameRelay and HDLC / PPP. See the IOL (Serial) page for more details.
Attention: Remember that Cisco VM and container images that are provided for download with CML are only licensed for use within CML labs.
Catalyst 9800-CL and ISE Images
Starting with CML 2.9.0, the updated supplemental reference platform ISO includes two new virtual machine (VM) images. The first is the Catalyst 9800-CL Cloud Wireless Controller, an IOS-XE-based virtual wireless controller. The second is the Cisco Identity Services Engine (ISE) image.
Note: CML-Free provides a more limited set of reference platform VM images and does not include access to the supplemental reference platform ISO. The supplemental reference platform ISO and associated images are available to all CML customers who have an active license for the Personal, Personal Plus, Enterprise, or Education offering.
API Changes
CML 2.9 includes several changes to the REST-based web service APIs. For details about a specific API, log in to the CML UI and click Tools > API Documentation to view the CML API documentation page.
The most notable API changes are related to the revamped group permissions. In previous CML releases, you could share a lab with named groups of users. In CML 2.9, you can still share a lab with a group, but you can also share a lab directly with an individual user on your CML instance. In both cases, you can use the new fine-grained lab/group permissions. (See the Lab Sharing and Advanced Group Permissions section above.)
The existing data structure for the groups
property and API endpoints could not accommodate this change in a backwards-compatible way. We decided to add new endpoints and deprecate the groups
property for now, replacing it with a new associations
property that capture the details of the expanded lab sharing enabled by this release.
CML 2.9 also removes some redundant lab endpoints that handled simple data values when there are already more general lab API endpoints that accomplish the same purpose. For example, instead of a specific endpoint, PUT
/labs/{lab_id}/title, to update the lab title, use the more general PATCH
/labs/{lab_id} and specify the title field and its new value in the request body.
APIs Removed in CML 2.9
This table summarizes the deprecated API endpoints that have been removed in CML 2.9.
API Endpoint | Alternative API Endpoint |
---|---|
GET /diagnostics |
GET /diagnostics/computes,GET /diagnostics/labs,et cetera |
GET /labs/{lab_id}/nodes/{node_id}/tags |
GET /labs/{lab_id}/nodes/{node_id} |
POST /labs/{lab_id}/nodes/{node_id}/tags/{tag} |
PATCH /labs/{lab_id}/nodes/{node_id} |
DELETE /labs/{lab_id}/nodes/{node_id}/tags/{tag} |
PATCH /labs/{lab_id}/nodes/{node_id} |
DELETE /licensing/reservation/discard |
POST /licensing/reservation/discard |
PUT /labs/{lab_id}/description |
PATCH /labs/{lab_id} |
PUT /labs/{lab_id}/title |
PATCH /labs/{lab_id} |
PUT /labs/{lab_id}/notes |
PATCH /labs/{lab_id} |
GET /labs/{lab_id}/nodes/{node_id}/image_definition |
GET /labs/{lab_id}/nodes/{node_id} |
PUT /labs/{lab_id}/nodes/{node_id}/image_definition |
PATCH /labs/{lab_id}/nodes/{node_id} |
GET /labs/{lab_id}/nodes/{node_id}/config |
GET /labs/{lab_id}/nodes/{node_id} |
PUT /labs/{lab_id}/nodes/{node_id}/config |
PATCH /labs/{lab_id}/nodes/{node_id} |
PUT /labs/{lab_id}/links/{link_id} |
None |
APIs Deprecated in CML 2.9
This table summarizes the API endpoints that have been deprecated in CML 2.9.
API Endpoint | Alternative API Endpoint |
---|---|
GET /users/{user_id}/groups |
GET /users/{user_id} |
GET /groups/{group_id}/labs |
GET /groups/{group_id} |
GET /groups/{group_id}/members |
GET /groups/{group_id} |
GET /system_archive |
Use combination of other existing endpoints yielding specific subset of archive data. |
GET /labs/{lab_id}/groups |
GET /labs/{lab_id}/associations |
PUT /labs/{lab_id}/groups |
PATCH /labs/{lab_id}/associations ; PATCH /labs/{lab_id} |
APIs to be Removed in CML 2.10
This table summarizes the API endpoints that were deprecated in earlier releases and that will be removed in CML 2.10.
API Endpoint | Alternative API Endpoint |
---|---|
GET /licensing/features |
GET /licensing |
GET /build_configurations |
PUT /labs/{lab_id}/bootstrap |
PUT /system/auth/config |
PATCH /system/auth/config |
New APIs in CML 2.9
This table summarizes the API endpoints that are new in CML 2.9.
API Endpoint | Description |
---|---|
PUT /labs/{lab_id}/nodes/{node_id}/clone_image |
Allows admin users to create a new image definition by cloning the disk image of an existing, stopped node. |
GET /labs/{lab_id}/associations |
Shows new user-lab and user-group associations. |
PATCH /labs/{lab_id}/associations |
Adds new or updates existing user-lab and user-group associations. |
GET /pcap/{link_capture_key}/packets |
Returns a list of all captured packets for a specific PCAP. |
GET /lab_repos |
Add new lab repository. |
POST /lab_repos |
Get the list of configured lab repositories. |
PUT /lab_repos/refresh |
Performs a git pull on each configured lab repository and returns the result. |
DELETE /lab_repos/{repo_id} |
Delete the specified lab repository. |
GET /sample/labs/{sample_lab_id} |
Get the list of available sample labs. |
APIs Changed in CML 2.9
This table summarizes the APIs that have been changed in CML 2.9.
API Endpoint | Summary of Change |
---|---|
GET /usersGET /users/{user_id}POST /users/{user_id}PATCH /users/{user_id}GET /groupsGET /groups/{group_id}POST /groupsPATCH /groups/{group_id}POST /labsPATCH /labs/{lab_id} |
Added new property associations. |
GET /labs/{lab_id}/tileGET /labs/{lab_id} |
Added property effective_permissions, which represents a union of user/lab and lab/group permissions for the user user whose API token was used to make the API request. |
POST /usersPATCH /users/{user_id} |
Added property pubkey, which is the user's public key that is used in key-based authentication to CML's console server. |
GET /system/compute_hostsGET /system/compute_hosts/{compute_id}PATCH /system/compute_hosts/{compute_id} |
Added property node_counts. |
GET /system_information |
Added property allow_ssh_pubkey_auth, which indicates whether key-based authentication to CML's console server is enabled. |
GET /node_definition_schemaGET /node_definitionsPOST /node_definitions |
Added property sim.linux_native.enable_tpm. Added property device.interfaces.default_console. Added property device.interfaces.iol_static_ethernets. |
GET /image_definitionsPUT /image_definitionsPOST /image_definitionsGET /image_definitions/{def_id}GET /image_definition_schemaGET /node_definitions/{def_id}/image_definitions |
Added property efi_boot. Added property sha256. Removed property disk_subfolder. |
POST /import |
Property lab.owner is no longer accepted. |
Other Changes
Summary | Release Notes |
---|---|
Permit SSH key-based authentication to the console server | Added a user setting for setting an SSH public key. If the CML administrator enables it, users may set one SSH public key that will be accepted by the SSH service (for the console server and SCP uploads) that runs on the CML server's port 22, complementing password-based authentication. |
Display interface MAC and IP addresses in the Workbench on the node interface pane | Added a new column in the interface tab of the node pane to display allocated MAC and IP addresses and a new button with the ability to refresh addresses. |
Display selected image definition for running nodes | When you select a node in the Workbench, the node details in the sidebar pane show the image definition used and the resource allocation even when the node is running or stopped but not wiped. In previous releases, the values were only shown before you started the node or after you wiped the node. |
Permit link type selection for packet capture (PCAP) | You can now select the link encapsulation for PCAP, which defaults to ethernet. If you are using serial links with IOL, for example, this change allows you to specify frame relay as the link encapsulation. The PCAP display both in the CML UI's Packet Capture pane and in the resulting PCAP files have the proper link encapsulation set so that packets are displayed properly. |
Allow Windows 11 VM images to be used with CML | Added support for secure boot and TPM enablement in node definitions. These options are required for running certain virtual machine images, including Windows 11 VMs. |
Add node definition default console property | Added an ability to select the default serial port in node definitions. |
Add support for node types that read their day0 configuration from an ext4-formatted disk | Added the ability to use an ext4-formatted disk for provisioning a node's day0 configuration. This functionality is needed by certain VM's like Cisco ISE. CML has a variety of ways to provide the day0 configuration to a node the first time it boots. See the Provisioning section of the node definition. CML 2.9.0 adds the Media Type ext4 for passing the day0 configuration file(s) to the running node. |
Remove deprecated disk_subfolder attribute from the image definition. |
Removed deprecated disk_subfolder attribute from the image definition schema. Upgrade to 2.9 will ensure that existing image definitions are migrated as well. |
Auto-generate and renew certs | Starting with CML 2.9.0, a unique self-signed certificate is automatically generated for each deployment during installation. This process replaces the previous approach where the same certificate was used for all deployments of a particular CML release. CML will also automatically regenerate the self-signed certificate before its expiration date, removing the need to manually update it by installing a newer version of the product. |
Cockpit admin for some customer-controlled system options | CML administrators can now configure a small set of CML-specific settings on the CML controller directly in the Cockpit UI. These settings include the MAC address OUI prefix, enablement of SSH key-based authentication for the Console Server, and configuration customizer scripts. Previously, these settings required manually editing files in the terminal. |
Expose resource thresholds in /etc/virl2/defaults | More of CML's default settings, including resource thresholds, are now part of a configuration file for easier customization. NOTE: we do not recommend modifying the settings in /etc/virl2/defaults unless instructed to by Support. The defaults should work for most installations. But if Support instructed you to customize the resource thresholds for your CML instance, such as setting custom values for CPU_THRESHOLD or MEMORY_THRESHOLD , you may now configure those settings more easily in /etc/virl2/defaults . |
Make the statistics collection interval configurable | The statistics collection interval affects how frequently CML updates the information about your running nodes and in the CML UI's status bar. CML now exposes that setting in /etc/virl2/defaults . In some cases, Support may recommend setting a larger (i.e., slower) collection interval for performance reasons. NOTE: we do not recommend modifying the settings in /etc/virl2/defaults unless instructed to by Support. |
Show CML administrators all running nodes in the Compute Hosts and Node Administration pages under Tools > System Administration | We have made multiple improvements to the Tools > System Administration pages, mostly to ensure that CML displays the accurate number of running nodes and so-called orphaned nodes. While orphan nodes are uncommon and mostly result from bugs or system failures, if your CML system has orphaned nodes, it could impact your system performance. |
Allow stopping a link even if the node on either side of it is running on a disconnected compute host. | You can now stop links even when their nodes are running on CML compute host(s) that are in a disconnected (broken) state without generating an error response. This change only affects CML cluster deployments. |
Warn users about running nodes when moving computes to REGISTERED state | In the Compute Hosts page under Tools > System Administration, CML will show a warning to inform you about any running nodes on a compute host when you move compute hosts to REGISTERED state. |
Enable SSH to CML server during Initial Setup script | Starting with CML 2.9, the Initial Setup process now includes a screen that allows you to enable the SSH service on port 1122 and the PATty port-forwarding service. Because those are optional services, they are disabled by default. (Following security best practices, a newly-installed CML instance only enables required services.) |
Bug Fixes
Bug Description | Resolution |
---|---|
CML AWS and Azure deployments are not supported with 2.9.0. | The AWS and Azure cloud deployment options provided by the cloud-cml tools should now work with CML 2.9.0. |
Cannot find or cannot unzip the supplemental reference platform ISO in the initial 2.9.0 release. | You can now download the "supplemental" refplat ISO file from the the download site. Note: the 20250718 version of the supplemental reference platform ISO was corrupt and should not be used. If you downloaded that original version of that optional file, discard it and download the 20250724 version of the supplemental-iso.zip file instead. |
Node positions are not updated on the backend when using the align buttons in the Workbench. | Fixed a bug in node alignment in the Workbench canvas that prevented the new node positions from being saved after a reload. |
The node definition attribute Visible in the User Interface section is not respected by the CML UI. | The Visible flag in node definitions works correctly, making a node with the flag disabled disappear from the Add Node menu in the Workbench. |
Invalid MAC Addresses are configured on IOL devices. | Various improvements have been made to MAC address representation in the Workbench. |
You are unable to log in to the the CML UI and get the "Go make some coffee" message. | Normally this message indicates that the services are still starting error, but some users encountered a more persistent problem with authentication. This bug was caused by malformed HTTP headers. We improved notifications around missing source files in CML 2.9, especially after upgrading a CML instance. |
Buttons in the UI use different vIcon sizes. | CML 2.9.0 includes consistency improvements on buttons across all CML UI components. |
You start an unmanaged-switch in CML, but the node fails to start. | The node start was timing out in libvirt. We fixed a bug where wiped bridges for unmanaged switches were not removed from firewall rules by libvirt, causing performance degradation and failure to start new unmanaged-switch nodes. |
When you remove a node or link, the CML UI still showed the panes for deleted element. | When a node or link is removed from the topology, CML closes the associated UI panes, such as VNC, Console, and Packet Capture. |
Using a USB stick to perform a bare metal deployment of CML doesn't work. | The artifacts on the CML ISO image are now installable via USB sticks. |
The number of running VMs reported in the status bar doesn't match the number of nodes in the lab. | We now include Unmanaged Switch nodes and orphan nodes in node counts, and those nodes are removed together with other nodes. |
Interface names are changed after upgrade to 2.8 on Bare metal installation | After upgrading to CML 2.8.x, you could not access CML. This issue only affects some bare metal deployments. Upgrading a CML 2.7.2 instance to CML 2.8.0+ also upgrades the underlying Ubuntu operating system to version 24.04, which includes major updates for some drivers. For some NICs, the new driver changes the interface naming schema to expose newly-supported features, but that change breaks the networking configuration for the CML server. If your system is affected, neither the CML UI nor the Cockpit admin UI will accessible after the upgrade. In CML 2.9, we added a boot-time service that migrates some CML server networking configurations if it detects that Ethernet interfaces were renamed from the names used in previous boots. |
Fix resizing problems on the Packet Capture pane in the UI. | When you resized the UI, CML was not adjusting the split between the list of captured frames and the frame details for the selected frame when you resized the pane in the UI. The Packet Capture pane now adjusts the split when you resize the UI. |
IOSv CVAC only reads the first 4096 bytes of the day-0 configuration | Fixed a bug where IOSv and IOSvL2 nodes were only able to read the first 4096 bytes of their day0 configuration. If you previously applied a workaround to your CML system for this bug, you should undo that workaround before upgrading. |
Refresh VNC display after switching panes | We fixed an issue with VNC where the screen used to go black when you switched panes between Nodes and Panes. |
The display of the Lab Notes in the Workbench does not resize with its pane | We fixed problems with the Lab Notes editor/viewer and display resolution in CML 2.8.x, especially when there is a large amount of content being present in the Lab Notes text area. |
CML indicated that the CML server or a CML compute host was OK even when nested virtualization was not enabled in the BIOS. | In the status and system health panes in the CML UI, the KVM column indicated a green checkmark for a compute host or for the CML server even when nested virtualization was not enabled in the BIOS settings In CML 2.9.0, we expanded the KVM health status indicators to provide more accurate information about the virtualization state. |
Node occasionally can't be selected | We fixed a bug that caused nodes not to be selectable or to require multiple clicks to select them. The previous workaround was to page reload, and that should no longer be necessary. |
Initialize interface in stopped state if unlinked | For docker and KVM nodes, nodes now report proper disconnect state after restart. When docker and KVM nodes are first started, any interfaces without a connected link are put into disconnected state. You should use the link connect function when you add a new link to a running node interface. |
When reconfiguring a CML compute host in a CML cluster, the compute node is getting stuck while setting up the NFS share for base VM images. | We have fixed multiple bugs in the Initial Setup script, specifically when running it after initial installation with the --reconfigure flag enabled. |
Missing in-field UI validations on smart annotation attributes | We made various improvements to smart annotation properties and to the validators in the CML UI. The most significant changes are the reduction of the smart annotation label size to just 256 characters and an increase in the limit on the distance field to 10000. |
Right-clicking on disconnected nodes doesn't show menu | CML now shows the right-click context menu even for disconnected nodes in the Workbench's canvas. |
The Smart Annotation pane in the Workbench, the select all checkbox has an incorrect initial state. | When you opened the Smart Annotation pane in the Workbench for the first time after a (re)load, the select all checkbox could show the incorrect initial state. |
Expanded footer affects split panes in the workbench | We fixed a bug in the expanded footer that affected other panes in the Workbench. |
PATty device access was not working in CML 2.8.x. | We fixed various bugs in PATty that prevented it from working properly in CML 2.8.x. The problem was related to changes to the underlying system between CML 2.7, which is based on Ubuntu 22.02, and CML 2.8, which is based on Ubuntu Linux 24.04. |
Replace deprecated function call os.fork . |
CML 2.9.0 uses an improved image health check on cluster computes to better detect stale condition. |
CML 2.8 runs NFS services on a standalone instance | Fixed a bug where, on All-in-One CML deployments, the installed NFS services were left automatically and unnecessarily enabled. Since there were no NFS shares configured, these services were idle but needlessly running. NFS services are now only enabled in cluster deployments. The firewall remains configured to only allow the necessary sharing of base images across the internal cluster network. |
The migration script doesn't work in CML 2.8 | We fixed a few bugs in the migration-tool that prevented it from working properly when performing a same-to-same-version migration. |
The IOS XRv 9000 node type should be configured for EFI boot. | Image definitions now may enable use of EFI booting even when the node definition does not. This change was made specifically to support newer IOS XR 9000v VM images. IOS XRv 900 EFI booting is required for VM images for IOS XRv 9000 version 24.4+. Existing, older IOS XRv 9000 VM images continue to use the non-EFI boot. |
Known Issues and Caveats for CML 2.9
Bug Description | Notes |
---|---|
After upgrading Ubuntu packages on the CML server, you are no longer able to SSH to port 1122 over IPv4. | This issue only impacts CML administrators who enabled the (optional) ssh service (OpenBSD Secure Shell server) that listens on port 1122 on your CML server. The problem affects CML versions 2.8.0, 2.8.1, and 2.9.0. If you applied the Ubuntu package upgrades on your CML server after July 24, 2025, the ssh service will no longer be listening on your CML server's IPv4 addresses. The cause of this problem is CML's own configuration for the SSH socket and a change in base configuration of Ubuntu 24.04's openssh-server package, which now causes that configuration to be interpreted as IPv6-only. Workaround: You can check the installed version of the package by logging in to Cockpit, clicking the Terminal tab, and running dpkg -l openssh-server . If your CML instance has openssh-server package version 1:9.6p1-3ubuntu13.13 or later, it is impacted by this issue, and you will need to apply the workaround. Do NOT apply the workaround if your CML server has openssh-server version 1:9.6p1-3ubuntu13.12 or lower. To apply the workaround, run the following commands to remove CML's redundant socket configuration and reload the service: sudo rm -r /usr/lib/systemd/system/ssh.socket.d |
Upgrade to CML 2.9.0 fails with a message that references download.docker.com , such as Err:5 https://download.docker.com/linux/ubuntu noble InRelease |
Online upgrades require access to Docker repositories hosted at https://download.docker.com . Ensure that your CML Server can reach that URL and then log in to Cockpit and restart the upgrade. |
Upgrading a bare metal CML deployment to CML 2.9.0 can fail with a message that indicates *** Updating package sources FAILED... please check the output above preceded by errors that refer to a cml_disk file path, such as file:/tmp/tmp.Am4VyzogBJ/cml_disk . |
We believe that this error only impacts in-place upgrades of CML bare metal deployments. Workaround: if you encounter this error, log in to Cockpit and try removing this file: sudo rm /etc/apt/sources.list.d/cml-upgrade-offline.list Then try starting the in-place upgrade again from the CML2 page in Cockpit. |
Terraform Provider for CML does not support 2.9.0 | The latest CML release is not supported by the Terraform provider yet. |
Login prompt before a CML bare metal installation completes is confusing. | During a CML bare metal installation, you can type into the console and the login prompt will appear. The login prompt is unusable and there's no account configured at this time. You need to wait for the installation to finish and reboot automatically, then the CML installation dialog should appear, and you will not need to log in until after you complete this initial setup. The installation can end in an error prompt on a failure, but in that case you also will not need to log in. |
The user contact information collection dialog keeps re-appering. | To be fixed in 2.9.1. When you refresh the CML UI, you will see the dialog for collecting user contant information even if you have dismissed it already. |
Missing animation while drawing annotations is missing. | To be fixed in 2.9.1. When you draw an annotation, the annotation won't appear until you release your mouse. |
A false notification is raised in User's Settings page. | To be fixed in 2.9.1. The false error notification is displayed when a user accesses the Settings page. |
Create annotation feature should be disabled without lab admin permissions. | To be fixed in 2.9.1. You can attempt to create an annotation without lab admin permissions, but your request will be rejected by the CML server and you will get an error notification. |
The default cml-community lab repository with sample labs cannot be refreshed. | To be fixed in 2.9.1. You won't be able to pull any updates from the cml-community repository in 2.9.0. |
The lab repository refresh feature blocks other operations. | To be fixed in 2.9.1. When you attempt to refresh your lab repositories and your CML instance is not configured properly to reach all respective Git URLs, the refresh operation may take up to a few minutes, especially if there are new repositories that have not been synchronized yet. The consequences are that one of the CML services becomes unavailable for that period of time, and while the UI and CML APIs will be responsive, no one will be able to start new start nodes until the lab repository refresh completes. |
Factory reset doesn't clean lab repositories. | To be fixed in 2.9.1. If you perform a factory reset in Cockpit, the lab repositories remain on the server. Sample labs from the these repositories will be still available after the factory reset, but you won't be able to refresh their contents anymore to pull down new updates unless you remove the lab repositories and add them again. |
Clean up feature doesn't remove docker containers. | To be fixed in 2.9.1. If you run the cleanup feature in Cockpit to remove all nodes from the CML server, the docker nodes will remain in the system as orphaned nodes. Workaround: Run the Remove Orphaned Nodes feature to clean them. |
A docker image is not removed with its image definition. | To be fixed in 2.9.1. When you remove a docker image definition, the image is not removed from the docker registry. |
Docker wipe fails as a stopped container actually still running. | To be fixed in 2.9.1. When you stop a docker container, it may appear as stopped in the CML UI, but it still may be running on the CML server. This is more likely to happen in CML deployments running many nodes. |
Nginx container node stops unexpectedly. | To be fixed in 2.9.1. When you run a nginx container, it may stop unexpectedly. This is more likely to happen in CML deployments running many nodes. |
A reasonable exception for misconfigured docker nodes is not raised. | To be fixed in 2.9.1. When you create a custom docker node type and do not configure metadata for the connected image correctly, you will get an unclear error notification. |
Reduce the number of lab stats events sent. | To be fixed in 2.9.1. CML sends unnecessary updates to all active clients which can have performance impacts on the CML UI responsiveness especially in heavy-loaded CML deployments. |
A KVM domain with NVRAM cannot be undefined while removing orphaned nodes | To be fixed in 2.9.1. If you remove orphaned KVM nodes, nodes with a NVRAM defined, e.g., IOS XRv 9000, will be stopped, but they won't be removed. |
The client library (virl2_client) should ensure compatible imports for older scripts | To be fixed in 2.9.1. The client library may not work with older CML servers. |
Copy reference platform ISO wrong size requirement check. | To be fixed in 2.9.1. When you copy a reference platform ISO and forget to provide enough disk space, the installation will continue, but it will fail due to insufficient disk space. Workaround: Increase the disk space and repeat the copy process. |
Copy reference platform ISO overwrites existing files. | To be fixed in 2.9.1. The copy reference platform ISO process overwrites any custom changes that you made to the default node and image definitions with the ones present on the copied reference platform ISO. |
Unmanaged switches may fail to start. | If you have tens of unmanaged switches running at once, additional unmanaged switches may fail to start. |
Docker disk usage collection CPU spikes in Azure deployments. | To be fixed in 2.9.1. If you deploy the latest CML release in Azure and start docker nodes, you may see CPU spikes caused by disk usage collection for those nodes. |
IOL ARP drops and doesn't ping when ACL outbound is configured. | When you configure an outbound ACL on an interface of an IOL node connected to another IOL node, ARP is automatically dropped and the nodes don't reach each other. |
When starting the CML VM on VMware Workstation or VMware Player, you receive an error saying that "Intel VT-x/EPT is not supported" or that VMWare "does not support nested virtualization on this host." | Recent (Nov / Dec 2024) changes in the Windows operating system have made it more difficult to disable Hyper-V so that VMware Workstation can enable nested virtualization for the CML VM. We are working on impacted customers to ensure that our updated documentation works for all supported deployment options. Workaround 1: See the steps in the latest version of CML Personal Intel VT-x/EPT Issue. If that does not resolve the problem, please contact CML support. Workaround 2: If you're running Windows Pro, you may be able to deploy CML directly to Hyper-V by following the community-contributed instructions for CML on Hyper-V. While we do not currently test CML on Hyper-V, customers have reported that the CML VM works well when running on Hyper-V. |
External connectivity does not work in CML bare metal installations. | This bug impacts CML bare metal deployments regardless of release. In some deployments, most notably on UCS bare metal servers in which the vNIC's VLAN is set to access mode , the ingress traffic is tagged with vlan 0 . This VLAN is meant to carry priority information as per the 802.1p standard. Network services can normally ignore this tag and handle vlan 0 packets transparently as untagged (no VLAN), but some node types in CML do not handle it gracefully, ignoring the incoming packets. As a result, the virtual nodes connected to an external connector node may not acquire an IP address from an external DHCP server or may not be able to communicate with external devices. Workaround: Follow the instructions from the Handling 802.1p Frames in UCS section of the Cisco Modeling Labs Administrator's Guide. |
After a restart, you cannot reach the CML UI. The console of the CML instance shows a login prompt, but it also indicates that CML does not have an IP address. | After you shut down and restart a CML instance, it sometimes loses its IP address. When that happens, you will no longer be able to reach the CML UI even though the CML services are running. Workaround: When you encounter this problem, open the CML console (e.g., the CML VM's console in VMware or the CML server's remote management console). Log in as the sysadmin user and run this command: sudo netplan apply You can verify the current/new IP address of the CML instance using this command: ip addr show bridge0 If the issue persists, follow the instructions from the Editing the Management IP address via the Console section of the Cisco Modeling Labs Administrator's Guide. |
Cannot run simulations on a macOS system that uses Apple silicon (e.g., the Apple M1 chip). | CML and the VM Images for CML Labs are built to run on x86_64 processors. In particular, CML is not compatible with Apple silicon (e.g., the Apple M1 chip). Workaround: There are no plans to support ARM64 processors at this time. Deploy CML to a different system that meets the system requirements for running CML. |
Even if you were able to run network simulations with CML 2.2.x on VMWare Fusion and macOS Big Sur or above, the same labs no longer start after migrating to CML 2.3.0+. | A default setting in the CML VM changed between CML 2.2.3 and CML 2.3.0. Because of the new virtualization layer in macOS 11+, this setting causes errors when running the CML VM on VMware Fusion. Workaround: follow the instructions to disable nested virtualization in the CML VM. |
Bug in macOS Big Sur. After upgrading to Big Sur and VMware Fusion 12.x, simulations no longer start. | MacOS Big Sur introduced changes to the way macOS handles virtualization, but the problems were resolved in later versions of macOS Big Sur and VMware Fusion. Workaround 1: If you are still experiencing problems, be sure to upgrade to the latest version of VMware Fusion 12.x and the latest bug fix release of your current macOS version. Workaround 2: If you are using CML 2.3+, then follow the instructions to disable nested virtualization in the CML VM. |
Deleting or reinstalling a CML instance without first deregistering leaves the license marked as in use. Your new CML instance indicates that the license authorization status is out of compliance. | If you have deleted or otherwise destroyed a CML instance without deregistering it, then the Cisco Smart Licensing servers will still report that instance’s licenses are in use. Workaround 1: Before you destroy a CML instance, follow the Licensing - Deregistration instructions to deregister the instance. Properly deregistering an instance before destroying it will completely avoid this problem. Workaround 2: If you have already destroyed the CML instance, see the instructions for freeing the license in the CML FAQ. |
When the incorrect product license is selected, the SLR (Specific Licensing Reservation) can still be completed. It eventually raises an error, but the SLR reservation will be complete with no entitlements. | Workaround: If you accidentally select the wrong product license, then you will need to release the SLR reservation and then correct the product license before trying to complete the SLR licensing again. |
The algorithm for computing the available memory does not take into account maximal possible memory needed by all running nodes. It only checks the currently allocated memory. CML will permit you to start nodes that fit within the current available memory of the CML server. Later, if the memory footprint of the already-running nodes grows and exceeds the available memory on the CML server, those nodes will crash. | Workaround 1: Launch fewer nodes to stay within your CML server's resources. Workaround 2: Add additional memory to the CML server to permit launching additional nodes. Of course, this workaround just moves the point where the CML server will eventually hit its limit, so you must remain aware of your CML server's resource limits or ensure that the CML server has sufficient memory to accommodate all labs that will be running on your CML server simultaneously. |
When starting a CSR 1000v node for the first time or after a Wipe operation, the interfaces are all in shutdown state. |
Workaround 1: Add no shut to each interface configuration in each CSR1000v node's Edit Config field in the UI. Workaround 2: Use the EEM applet on the CSR 1000v page. |
New nodes cannot be started when CPU usage is over 95%, when RAM is over 80%, or when disk usage is over 95%. | Workaround: Allocate additional resources to the CML server. Workaround 2: If you are hitting the disk usage limit, check the Remove Orphaned VMs function in Cockpit. Orphaned VMs can consume a significant amount of disk space even though CML is no longer using or managing those VMs. |
The journalctl output shows errors like this: cockpit-tls gnutls_handshake failed: A TLS fatal alert has been received |
This log message only occurs with Chrome browsers and has no functional impact on the use of Cockpit. No workaround needed. |
In CML Cluster, the Cockpit Storage table for NFS mounts looks broken. | This is a display bug of the Cockpit UI when it encounters an NFS share using an IPv6 address. The single mount is split into two rows, only the first of which is indicated as mounted. Your NFS setup is probably fine, as long as the CML UI shows in its Cluster Status view that each compute host's Images indicator is green. Trying to edit or repair the NFS mount in any way in Cockpit is more likely to break the configuration. Please contact TAC if you experience networking issues with clusters which cannot be resolved by verifying basic connectivity on the cluster network or by restarting CML services. |