CML 2.10 Release Notes
Cisco Modeling Labs (CML) is a network simulation platform. CML 2.10 is the latest feature release of CML. This release introduces Model Context Protocol (MCP) Server integration for your AI tools, packet capture streaming, lab autostart, and node staging improvements. The reference platform image set now includes XRd, Meraki vMX, Snort3, and Wireless images. Other notable updates include custom icon enhancements, unmanaged switch improvements, and experimental support for wireless networking and RADIUS user authentication to CML.
New and Changed Information
This table shows published changes to this CML 2.10 release notes document.
| Date | Changes |
|---|---|
| May 13, 2026 | Initial version for CML 2.10.0 FCS relase. |
Installing or Migrating to CML 2.10
Upgrading to CML 2.10
If you are still running CML 2.7.1 or older, you cannot upgrade it directly to CML 2.10. Your CML instance must be running CML 2.7.2 or later before you can upgrade it to CML 2.10. If you have not upgraded to CML 2.7.2 yet, see CML 2.7 Release Notes for more details about the process to upgrade to CML 2.7.2.
See the In-Place Upgrade page for detailed upgrade instructions.
- Review Known Issues and Caveats before upgrading.
- Online upgrades require access to Docker repositories hosted at
https://download.docker.com. Your network environment must allow connections to this host over the Internet, otherwise upgrade will fail. - Disable maintenance mode after the upgrade is complete.
In-place upgrades from CML version 2.2.3 or earlier are not supported. In that case, a migration is required. The migration process from CML 2.2.3 is detailed in the CML 2.6 Release Notes.
Upgrading a CML Cluster to CML 2.10
If you have deployed a CML Cluster, you may upgrade it to CML 2.10 provided that your CML cluster is already running CML 2.7.2 or later.
Installing CML 2.10
The CML Installation Guide provides the required installation workflow.
- Verify the latest System Requirements.
- In CML 2.10, all the reference platform virtual machine (VM) images must be copied to the local disk of the CML instance. Be sure to provision enough disk space for your new CML instance to accommodate the reference platforms.
- During Initial Set-Up, wait for services to stabilize before continuing.
Summary of Changes
This section details new and changed features in the new release. Bug fixes are listed in the subsequent section.
Model Context Protocol (MCP) Server for Better AI Integration with CML
MCP enables you to interact with CML using English, or one’s local language, commands using your preferred AI tool. MCP is a protocol that helps AI understand how to interact with APIs. Once you configure your AI tool with the CML MCP Server, you will be able to create topologies with configured nodes, run commands on the nodes, and evaluate the network without complex scripting or calling the CML APIs yourself. Overall, this feature will allow users to Speak their lab into Existence with AI-Driven Cisco Modeling Labs and MCP.
Custom Icons
We have gotten numerous requests for custom node icons ever since the first CML release. In CML 2.10, this feature is now available for users to be able to set and assign custom icons within lab topologies. When you override the default node icons, you also have the option to include custom icons when you export or import the lab.
![]()
Packet Capture Streaming
Prior to CML 2.10, you could only see active packet capture traffic in the CML UI. The CML UI can show active packet captures in the Link Packet Capture pane, which provides the ability to expand each packet to see the decoded messages. In CML 2.10, you are now able to configure the Breakout UI and stream the captured data through it into Wireshark instances that run on your local computer. This feature allows more convenient packet analysis as the data arrives.
Lab Autostart

If you would like a specific lab to start automatically as soon as the CML server boots up, check out the Lab AutoStart feature in CML 2.10. If you’re a CML administrator, you can now flag a lab for "autostart." You can even flag multiple labs and set a stage value to dictate the order that CML should start the flagged labs. Lab AutoStart is especially useful in an education environment, where you may be using CML to simulate the same sample network for each student. CML-Enterprise administrators who always want CML to run a simulation of their company’s production network or individual learners who are using a particular lab for their studies may also find Lab AutoStart useful.
Node Staging

The Node Staging feature in CML 2.10 allows you to set priorities on nodes in your lab to control the order that CML starts the nodes when you start the lab. If you are using resource-intensive node types, such as IOS XRv 9000 or NX-OS 9000, in your CML labs, node staging may help you ensure that all of the nodes in your lab reliably boot up when you start the lab. When you use this feature, CML starts nodes with a higher priority first, and it waits for all nodes with a higher priority to finish booting up before it starts any nodes with a lower priority.
While Node Staging priorities are not generally needed to start labs in CML, specific labs may start up more reliably when you use this feature. In particular, if you have a lab where some nodes fail to boot up to the prompt when you start the lab, but all of the nodes seem to run well and without error once you stop and restart the nodes that initially failed to boot up, you should consider using this feature. By tuning the order in which CML starts the lab’s nodes and the number of resource-intensive nodes that CML starts up simultaneously, you may be able to ensure that all nodes boot up reliably when you start the lab.
Unmanaged Switch Improvements
CML 2.10 adds an additional option for your unmanaged switches that allows you to toggle it to behave as a hub for SPAN support. This feature will allow traffic from a device to go to another device for troubleshooting or learning purposes.
RADIUS User Authentication (beta)
CML 2.10 Enterprise and Education deployments now support RADIUS as a method for user authentication. RADIUS authentication is currently a beta feature, and it is disabled by default while functionality is still being expanded. Early customer feedback is requested, and you can submit your feedback on this feature by sending e-mail to cml-info@cisco.com with subject "CML RADIUS feedback".
Enable the RADIUS Authentication Option
Before you can use RADIUS as an authentication method, you must first enable this feature for your CML instance. Edit /etc/default/virl2 and set CML_FEATURES to include radius.
# Optional feature toggles are configured via CML_FEATURES as a
# comma-separated feature list (for example: "foo,bar,baz").
CML_FEATURES="radius"
After this change, restart controller services with systemctl restart virl2.target. RADIUS now appears as a selectable option in the Authentication Method dropdown.

Wireless Support
Wireless support is available starting in CML 2.10. We will be introducing this feature in phases across multiple CML releases. CML's wireless support will provide network engineers, students, and instructors with a realistic, comprehensive, and flexible virtual environment to design, test, troubleshoot, and learn about wireless network technologies.
CML 2.10 adds beta support for simulated Wi‑Fi in the lab so you can run wireless clients and access points alongside your wired nodes--no physical radios required. Wireless-capable nodes use a reference image based on Ubuntu 24.04 with wireless support included. This support includes a virtual wireless radio and either wpa_supplicant and hostapd, which facilitates the wireless client and wireless AP's, respectively.
Updated Docker Images on GitHub
In this release, Docker images will be released via GitHub with updates for container images. This approach is beneficial for images that are fast-moving, like Chrome and Firefox, and will include all CML Docker-based node and image definitions that are built from open source, It will also include Docker images that are publicly available on hub.docker.com, like ThousandEyes and Splunk.
New Reference Platform Images
Starting with CML 2.10, the updated reference platform content includes additional images to support expanded lab scenarios and newer product workflows. The image set now includes IOS XRd, Meraki vMX, Snort3, and wireless images aligned with the 2.10 feature set. The 2.10 documentation cycle also includes FAQ and reference-platform page refreshes to keep image/version guidance consistent, including dedicated page updates for each newly added image.
| Image | Description |
|---|---|
| XRd | Containerized XRd image support for XR-focused lab topologies and API-driven workflows. |
| Meraki vMX | Virtual Meraki vMX image availability for SD-WAN and cloud-edge use cases in CML labs. |
| Snort3 | Snort3 container image support for security and traffic inspection scenarios. |
| Wireless | New wireless image coverage aligned with 2.10 experimental wireless workflows and dedicated refplat image-page documentation. |
API Changes
CML 2.10 includes several changes to the REST-based web service APIs. For details about a specific API, log in to the CML UI and click Tools > API Documentation to view the CML API documentation page.
APIs Removed in CML 2.10
| API Endpoint | Alternative API Endpoint |
|---|---|
GET /build_configurations |
PUT /labs/{lab_id}/bootstrap |
GET /licensing/features |
GET /licensing |
PUT /system/auth/config |
PATCH /system/auth/config |
POST /feedback |
None |
APIs Deprecated in CML 2.10
| API Endpoint | Alternative API Endpoint |
|---|---|
GET /authok |
GET /api/v0/authentication |
GET /diagnostic_event_data |
None |
GET /image_definition_schema |
None |
GET /simplified_node_definitions |
GET /node_definitions |
GET /node_definition_schema |
None |
GET /labs/{lab_id}/links/{link_id}/capture/key |
No alternative. Capture key is now the same as the link ID. |
APIs to be Removed in CML 2.11
| API Endpoint | Alternative API Endpoint |
|---|---|
GET /users/{user_id}/groups |
GET /users/{user_id} |
GET /groups/{group_id}/labs |
GET /groups/{group_id} |
GET /groups/{group_id}/members |
GET /groups/{group_id} |
GET /system_archive |
Use combination of other existing endpoints yielding specific subset of archive data. |
GET /labs/{lab_id}/groups |
GET /labs/{lab_id}/associations |
PUT /labs/{lab_id}/groups |
PATCH /labs/{lab_id}/associationsPATCH /labs/{lab_id} |
New APIs in CML 2.10
| API Endpoint | Description |
|---|---|
GET /ai/mcp/configuration |
Get MCP (Model Context Protocol) client configuration for AI tools. |
POST /wireless/pcap |
Start new PCAP capture session on a wireless node. |
GET /wireless/pcap/{capture_key} |
Get status of a wireless PCAP capture session. |
DELETE /wireless/pcap/{capture_key} |
Stop wireless PCAP capture session. |
GET /wireless/pcap/{capture_key}/download |
Download wireless PCAP file. |
GET /keys/pcap |
Return keys for active PCAP packet captures. |
GET /authentication |
Return authentication information about currently logged-in user. |
GET /system/auth/config/{method} |
Return configuration for the specified authentication method. |
GET /lab_repos/{repo_id} |
Get details about the specified lab repository. |
PUT /lab_repos/{repo_id}/refresh |
Request CML to pull updates from the specified lab repository. |
GET /labs/{lab_id}/interfaces |
Return list of all interfaces in the specified lab. |
DELETE /labs/{lab_id}/layer3_addresses |
Clear the discovered L3 addresses for all nodes in the lab from the snooper. |
DELETE /labs/{lab_id}/nodes/{node_id}/layer3_addresses |
Clear the node's discovered L3 addresses from the snooper. |
PUT /reload_definitions |
Reloads all node and image definitions. |
APIs Changed in CML 2.10
| API Endpoint | Summary of Change |
|---|---|
| Authentication configuration endpoints restructured | The GET /system/auth/config response schema has been restructured. Individual per-method properties have been removed from the top-level response. Use the new GET /system/auth/config/{method}. |
| Node schema restructured across multiple endpoints | The node response schema has been restructured. Properties have been moved or reorganized. The error response for a missing node has changed from code 400 to 404. |
| Interface schema restructured | The interface response schema returned by GET and PATCH /labs/{lab_id}/interfaces/{interface_id} has been restructured. The properties src_udp_port, dst_udp_port, device_name, operational.device_name, operational.mac_address, operational.src_udp_port, and operational.dst_udp_port have been removed or relocated. |
| Node definition schema restructured | The /node_definitions request schemas no longer include these ui properties: has_configuration, show_ram, show_cpus, show_cpu_limit, show_data_volume, show_boot_disk_size, has_config_extraction. |
| Image and node definition endpoints now support YAML | The /image_definitions/ and /node_definitions endpoint now supports application/yaml as an additional response media type alongside application/json. |
| Lab download and pyATS testbed endpoints return YAML | The GET /labs/{lab_id}/download and GET /labs/{lab_id}/pyats_testbed endpoints now return application/yaml instead of application/json. |
| Lab schema restructured | The GET /labs/{lab_id}/tile response no longer icludes topology.nodes, topology.links, topology.annotations, and topology.smart_annotations. The GET /populate_lab/{lab_id} response no longer includes effective_permissions. |
New query parameter with_data |
The GET /labs endpoint now accepts a with_data query parameter. |
opt_in property type changed to enum |
Affected endpoints: /users and /telemetry |
| PCAP endpoints parameter change | pcap endpoints have replaced the link_capture_key parameter with link_id. |
| Lab repository endpoints schema changed | id property was removed. name property now uniquely identifies lab repository. |
Several endpoints now return 204 No Content instead of 200 OK |
GET /authok, PUT /labs/{lab_id}/links/{link_id}/capture/stop, PATCH /web_session_timeout/{timeout}, PUT /labs/{lab_id}/bootstrap |
Compute_hosts endpoints now include list of nodes |
GET /system/compute_hosts, GET /system/compute_hosts/{compute_id}, |
| Lab import endpoint updated | node_staging property added for each lab. Each node in a list of nodes now includes properties priority and pyats. |
Other Changes
| Summary | Release Notes |
|---|---|
| Add "Since: CML 2.10" lifecycle metadata to the OpenAPI schema | The OpenAPI schema now carries lifecycle metadata for endpoints, request fields, and response fields introduced or changed in this release. The 2.10 release covers items new in 2.10; future releases will extend coverage to older versions. |
| Set pyATS testbed credentials per image and per node definition | PyATS credentials and testbed parameters can now be configured per image and per node definition rather than defaulting to cisco/cisco. This unblocks platforms such as CoreOS whose console prompts behave differently and lab environments where uniform credentials across nodes cannot be assumed. |
| Include the per-node enable password in generated pyATS testbeds | Generated pyATS testbeds now include a per-node enable password. Nodes whose enable password differs from the login password can be driven by pyATS-based tooling without manual edits to the testbed YAML file generated by CML. |
| Make the LDAP authentication timeout configurable | The LDAP authentication timeout is now configurable. CML can be used behind multi-factor authentication proxies (for example, OpenLDAP behind a push-MFA gateway) where the user-approval step legitimately exceeds the previous 10-second timeout. |
| Regenerate the nginx and Cockpit TLS certificates independently | The CML certificate utility now lets you regenerate the nginx and Cockpit TLS certificates independently. Previously a single virl2-generate-cert.sh run always reissued both, disrupting browser trust caches and active Cockpit sessions even when only one chain needed renewal. |
| Configure key exchanges, ciphers, and MACs for the console SSH server | The console SSH server (virl2-sshd) accepts new command-line flags for selecting which key exchanges, ciphers, and MACs are offered. Operators can now match organisational security policy by disabling individual algorithms instead of accepting the built-in defaults. |
| Wipe multiple selected nodes from the Workbench | The multi-node selection menu in the Workbench now includes a wipe action alongside start/stop/delete, so multiple nodes can be wiped in a single operation. |
| Unify the node definition and image definition editors | The node definition and image definition editors in the CML UI have been refactored for consistency. Both editors now share the same layout (the legacy "save button only at the bottom of the page" pattern is gone), and the node definition editor's Linux native simulations section has been moved into a dedicated component for easier navigation. |
| Auto-focus the serial console pane on open | Serial console panels now auto-focus when opened and when the change serial line dialog closes, so users can type immediately without first clicking inside the console. VNC panels already had this behaviour. |
| Filter nodes by node type in the administrator's Nodes tab | The Nodes tab in the administrator UI now shows a node-type column and allows filtering by node type, making it easier to locate nodes of a specific kind (for example, all NXOSv nodes) in larger topologies. |
| Reach the CML API over HTTPS from lab nodes on a NAT-mode external connector | Nodes running on a NAT-mode external connector can now reach the CML API over HTTPS. Previously the libvirt firewall zone only allowed DNS, DHCP, TFTP, and SSH, forcing users to switch to bridge-mode external connectors if devices needed to call the CML API. |
Pull container images by docker_tag at node start time |
Image definitions for Docker-based nodes can now reference an image by docker_tag and skip uploading a tar.gz archive. When docker_tag is set, the CML host pulls the referenced image from the corresponding Docker registry (Docker Hub at docker.io by default) at node start time. Launches are deferred until the pull completes; progress is currently visible only in the docker-shim log and not in the UI. |
| Reference platform improvements | CML 2.10 includes a number of refinements to the bundled reference platforms - refreshed images, updated default configurations, and reliability fixes for IOL, Cisco container images, and several other node types. See Reference Platform Images for the full list of images shipped with this release. |
virl2_client Python library improvements |
The CML Python client library virl2_client has received numerous improvements in this release: coverage for the new APIs introduced in CML 2.10, additional typed helpers and convenience methods, and a number of bug fixes. See the virl2-client release notes for the complete change list. |
| User interface and accessibility improvements | The CML web UI has received accessibility and usability polish across the Workbench, administrator pages, and configuration dialogs - including improved keyboard navigation, refreshed iconography, and a number of smaller layout fixes. |
| Upgrade and installation reliability improvements | The upgrade and migration tooling has been hardened against a number of edge cases reported by customers, including better recovery when libvirt or Docker are not yet running, additional pre-checks before destructive operations, and more informative progress reporting. |
Bug Fixes
| Bug Description | Resolution |
|---|---|
CVE-2026-31431 ("Copy Fail"): a privilege-escalation vulnerability in the Linux kernel's authencesn AEAD wrapper. An unprivileged user with a local shell on the CML controller or compute could escalate to root. |
CML 2.10 ships with kmod 31+20240202-2ubuntu7, which installs a modprobe drop-in that blocks loading of the vulnerable algif_aead kernel module. This blocks the Copy Fail primitive on CML controllers and computes until a kernel update carrying the upstream fix is available.Workaround: On earlier releases, install the latest distribution updates and reboot to pick up the patched kmod package. |
CVE-2025-62727: a denial-of-service vulnerability in Starlette's FileResponse allowed an attacker to trigger O(n²) work by crafting overlapping Range headers. CML's API server uses Starlette to serve static files (including the Swagger UI under /api/v0/ui) and was potentially affected. |
CML 2.10 ships a Starlette version that includes the fix for CVE-2025-62727. Affects: CML releases before 2.10 that used Starlette versions prior to 0.49.1. The practical impact was already limited by the nginx defaults that cap request sizes. |
| A CML administrator (not a system administrator) logged in over SSH on port 22 with an SFTP client could traverse the underlying Linux filesystem and read files outside the intended dropfolder. The SFTP session could not upload, move, or delete files outside the dropfolder. | The virl2-sshd SFTP subsystem is now confined so a CML administrator cannot read files outside the dropfolder. |
Some endpoints under /api/internal/ and /ws/ were reachable from unauthenticated, off-host clients instead of being restricted to inter-service traffic. |
The internal endpoints are now restricted to loopback and cluster-internal addresses. The updated nginx configuration rejects unauthenticated remote requests. |
ZAP and SOLIS TLS scans against Cockpit on port 9090 reported a missing HSTS header, a missing or insufficient Content-Security-Policy (allowing the data: scheme, unsafe-inline, and no frame-ancestors), and acceptance of weak TLS ciphers. |
The Cockpit configuration now sets the missing security headers, and a systemd socket override restricts Cockpit's TLS server to a modern cipher set. Workaround: On earlier releases, contact Cisco TAC for assistance. Modifying the CML controller's nginx or Cockpit configuration directly is not supported. |
Clickjacking protection on the CML UI was incomplete because the Content-Security-Policy was delivered through a <meta> tag, and the frame-ancestors directive is ignored when set that way. |
The CML UI now sets Content-Security-Policy: frame-ancestors 'none' as an HTTP response header, so the directive takes effect and the UI cannot be framed by other origins.Workaround: On earlier releases, contact Cisco TAC for assistance. Modifying the CML controller's nginx or Cockpit configuration directly is not supported. |
Static assets served by the CML UI under /assets/ on port 443 were returned without security response headers - Strict-Transport-Security, X-Content-Type-Options, Content-Security-Policy, and related headers were all absent. |
The nginx configuration now includes security-headers.conf on the /assets/ location, so static assets are served with the same security headers as the rest of the CML UI.Workaround: On earlier releases, contact Cisco TAC for assistance. Modifying the CML controller's nginx or Cockpit configuration directly is not supported. |
An in-place upgrade from CML 2.7.2 to CML 2.10 could fail while installing the libgdk-pixbuf2.0-0 package. |
In-place upgrades from CML 2.7.x to CML 2.10 now resolve the libgdk-pixbuf2.0-0 package dependency cleanly.Affects: Upgrades from CML 2.7.x. |
Docker upstream changed its default container storage driver to overlayfs, which is not compatible with CML. Affected CML instances failed to start Docker nodes due to mismatched image checksums and other disk-related issues. |
The CML upgrade flow now forces the overlay2 driver during all upgrades from versions prior to 2.9.0, so Docker nodes continue to start as expected.Affects: CML instances upgraded online to 2.9.x after November 2025, which picked up the new overlayfs default. CML system administrators can check the active storage driver by logging in to Cockpit and running docker info | grep Storage on the Terminal.Workaround: On affected hosts, switch back to overlay2. Note that this workaround wipes all Docker nodes in all users' labs on the host. In the Cockpit Terminal, run the following commands as root:systemctl stop virl2.target docker.socket docker containerd |
| After an in-place upgrade from CML 2.9.x, some labs reported "image definition not found" for nodes whose image definitions had been renamed during the upgrade. Affected nodes could not be edited from the Workbench, and the "set image definition to automatic" option was rejected for already-deployed nodes. | Upgrading to CML 2.10 repairs affected nodes automatically: the CML 2.10 migration keeps node references consistent when image definition IDs are renamed during an upgrade. Affects: Upgrades from CML 2.9.x with labs containing nodes whose image definitions were renamed. |
| The migration-tool script that you may have used during a CML upgrade was failing on systems where libvirt or Docker were unavailable (for example when those packages were part of a broken upgrade chain after a reboot), because the script depended on those services to gather state. No usable backup was produced. | The migration-tool no longer requires libvirt and Docker to be available. If either service is unavailable, it skips the corresponding state-gathering steps with a clear log message and still produces a usable backup. Workaround: On earlier releases, contact Cisco TAC for assistance running the migration when libvirt or Docker are unavailable. |
| Re-running the upgrade on the controller after a previous upgrade had failed (typically while contacting compute hosts) did nothing, because the controller already had all upgrade packages installed. | The upgrade workflow now continues past the package-installation step when packages are already present, so a re-run on the controller is able to complete the upgrade on compute hosts as well. Affects: Upgrades that failed in later stages with the same target version. Workaround: On earlier releases, run the upgrade on each compute host manually once the controller is up and the upgrade packages have been distributed. |
| An in-place upgrade could fail or produce inconsistent state if the controller's group table contained two groups whose names differed only in case. | The upgrade flow now detects case-only group-name collisions and refuses to start the upgrade until the conflict has been resolved. |
Cockpit logins for the sysadmin user failed because ~sysadmin/.config existed but was owned by root (typically a residue from an earlier sudo -E -s shell), so Cockpit's session setup could not write to it. |
The Cockpit session setup now ensures that the sysadmin user owns ~sysadmin/.config, so logins succeed without manual intervention.Workaround: On earlier releases, run sudo chown -R sysadmin:sysadmin /home/sysadmin/.config as root, then retry the Cockpit login. |
| During ISO-based deployment, a login prompt appeared on the console while installation scripts were still running, leading users to think installation had completed. | The console no longer offers a usable login prompt until ISO installation has finished. Affects: ISO-based deployments on bare metal or virtual machines. OVA deployments are not affected. Workaround: On earlier releases, wait for installation to complete before interacting with the login prompt; the initial setup scripts finish in the background. |
| The ISO installer did not detect pre-existing LVM volumes on the target disk, which could lead to duplicate volume-group names and installation failures. | The ISO installer now detects and clears conflicting pre-existing LVM setups before installation. Affects: Occurs when a previous LVM installation exists on the target install disk, or when an LVM installation using the same labels as CML exists on another disk attached to the host. |
The LVM resize script failed when a CML ISO was attached to the controller, preventing the /var partition from being resized during reference-platform copies. |
The LVM resize script now works correctly even when a CML ISO is attached. Affects: Occurs when copying reference platform images from an attached ISO. |
| The Copy Reference Platform ISO feature did not reject the operation when there was not enough free space on the target disk. | The Copy Reference Platform ISO workflow now calculates the required free space correctly when the ISO is attached as a virtual CD-ROM and rejects the operation if there is not enough space. Affects: Cases where the ISO is attached as a virtual CD-ROM device, instead of an .iso file uploaded to the CML controller via scp.Workaround: On earlier releases, manually verify that /var has at least the (unzipped) ISO size available before starting the copy. |
| The CML installer reported an incorrect disk size for the controller during the initial setup, which could mislead administrators when sizing the deployment. | CML now detects and reports the controller's disk size correctly. |
| After releasing a Specific License Reservation (SLR), the base license count did not reset to 1 and could not be changed. | The base license count now correctly resets to 1 after an SLR is released, and the value can be modified afterwards. Affects: CML instances using Specific License Reservation (SLR) licensing. |
| When a CML instance was out of compliance (OOC), it emitted licensing-status warnings on a recurring timer. | CML no longer periodically emits licensing-status warnings while it is out of compliance. Affects: CML instances in an out-of-compliance (OOC) licensing state. |
| The licensing quotas reported by CML were invalid when a registered CML instance was out of compliance. | CML now reports valid licensing quotas while a registered instance is out of compliance. Affects: CML instances in an out-of-compliance (OOC) licensing state. |
| In a cluster deployment, a lab link that connected nodes on different compute hosts could neither be stopped nor started again if its start had previously failed on one of the computes. | A failed link start now wipes the link from both computes and surfaces the error, allowing the link to be cleanly restarted afterwards. |
| Connecting a disconnected interface on a Docker node did not restore forwarding on the link when the affected interface was the second (B-side) port of the link. | Fabric state is now updated correctly on both ends of a link, so reconnecting a Docker node interface restores forwarding regardless of which end of the link it is. |
| Errors raised on compute hosts in a CML cluster were not surfaced to the controller, so administrators could not see why a compute-side operation had failed. | Compute-side errors are now propagated to the controller and surfaced through the standard error response. Affects: CML cluster deployments. |
| The IP snooper occasionally discarded IP addresses held by lab nodes on NAT-connected interfaces, which caused PATty to remove DNAT entries and external connectivity to the affected nodes to be lost. | The IP snooper now keeps NAT-side address assignments stable across DHCP lease renewals and other transient conditions. Affects: CML instances where lab nodes use NAT-connected interfaces with DHCP - most likely with longer DHCP lease renewals or high node churn. Workaround: On earlier releases, ensure that the node responds to ARP requests using its configured IP address, or restart the affected node / renew its DHCP lease to re-validate the assignment. |
Some CML instances experienced repeated restarts of the patty service. |
patty error handling has been improved, and the service now retries failed operations instead of shutting down. |
| The dispatcher service could crash with a concurrent map read/write error, breaking console and PCAP connections on a busy CML instance. | This defect has been fixed, and console or PCAP connections no longer fail when many sessions are opened or torn down simultaneously on a busy CML instance. |
| The Clean Up action in Cockpit restarted the CML services twice in succession. | The Clean Up action in Cockpit no longer restarts the CML services twice. |
| Nodes that were queued for start were briefly displayed as orphans while the controller was waiting for their startup transitions. | The orphan-detection logic has been improved so that queued and starting nodes are no longer reported as orphans. |
| Starting a lab from System Administration > Lab Administration left the Compute column empty for the lab's nodes in the Node Administration view. | Lab nodes started from System Administration > Lab Administration now have the Compute value populated correctly in the Node Administration view. |
| When a node was started with a broken node definition, the controller log included an unnecessary Python traceback at exception level. | Broken node definitions are now logged at error level without the spurious traceback. |
| It was not possible to move a node on the Workbench canvas if its node definition was missing or invalid. | Nodes whose node definition is missing or invalid can now be repositioned on the Workbench canvas. |
| Docker image definitions could not be deleted from the CML controller while any Docker node referenced them, even when the node had never been started - non-Docker image definitions could already be deleted in that case. | Docker image definitions can now be deleted as long as no Docker node referencing them is currently running or stopped. Wiped Docker nodes, or nodes that have never been started, no longer block deletion. |
| A misconfigured Docker node definition could cause the controller to loop indefinitely while attempting to start the node. | The controller now detects misconfigured Docker node definitions, reports a clear error, and exits the start workflow instead of looping. |
| The CML cleanup workflow removed images from the local Docker registry, which could leave node definitions referencing images that no longer existed. | The cleanup workflow no longer removes images from the local Docker registry. Container images are now cleared only by the factory-reset workflow. |
Console SSH sessions could leave termserv child processes running on the CML controller after the SSH client disconnected. Over time these accumulated child processes consumed memory and could exhaust controller resources, requiring an administrator to restart the virl2-sshd service or reboot the controller. |
virl2-sshd now cleans up its termserv child processes when a session ends, even if the SSH connection terminates abruptly. |
Automated SSH interactions with the virl2-sshd console server failed unless the SSH client requested a pseudo-terminal - a non-PTY ssh ... command invocation did not run the requested command at all. |
The console server now accepts non-PTY SSH commands and executes them as expected. Workaround: On earlier releases, force a PTY by running SSH with -tt or -o RequestTTY=force. |
| JSON responses from the CML 2.9.x API omitted properties whose values were equal to their defaults, so the response shape varied per resource and per request. | JSON responses now consistently include all documented fields, including those at their default values. YAML responses remain in their pre-2.10 minimal shape that only contains explicitly-set properties. Affects: Any HTTP client consuming JSON responses from the CML 2.9.0 / 2.9.1 API. Workaround: On 2.9.x, treat any documented field as potentially absent and fall back to the documented default value. |
| LDAP-authenticated login failed with HTTP 500 (surfaced in the UI as "Authentication failed") on a CML controller that had LDAP and a default Resource Pool both configured under the LDAP settings. | LDAP logins now succeed when LDAP authentication is configured together with a default Resource Pool. Affects: External authentication set to LDAP, with a non-NULL default Resource Pool, on every LDAP login by a user not yet present in the local user table. Workaround: On 2.9.x, clear the LDAP-level default Resource Pool and assign resource pools per user or per group instead (System Admin > User Management > External Authentication > LDAP > Resource Pool = "None", Save). |
After wiping or restarting an ASAv node, the day-0 configuration extracted from the node disk reset the admin user's enable password. The extracted configuration contained masked password placeholders (enable password *****) instead of the hashed password, so subsequent boots applied the masked placeholder and the original credentials no longer worked. |
ASAv configuration extraction now preserves the hashed enable password instead of writing back the displayed ***** mask, so admin credentials survive wipe/restart cycles. |
| Configuration extraction failures returned a generic error, making it hard to diagnose the cause. | Configuration-extraction failures now return improved and unified error messages. |
Image files uploaded to the CML controller over scp could not be selected when creating a new image definition through the UI. |
Files placed in the dropfolder over scp are now available for selection when creating an image definition. |
| The Diagnostics API returned an error response when a compute host was disconnected or not yet synchronised, instead of returning the diagnostics payload. | The Diagnostics API now returns a valid response that flags the affected compute, even when a compute is disconnected. |
| Setting an OUI prefix outside the allowed range produced a generic error in the CML UI. | The CML UI now displays a specific error message when an out-of-range OUI prefix is entered. |
| Moving an active PCAP tab into a different pane stopped new packets from appearing in the tab. | Moving a running PCAP tab between panes now keeps the live packet stream connected and packets continue to be displayed. Workaround: On earlier releases, drag the tab back to its original pane (a page reload does not help). |
| After editing a field in a node's sidebar, switching to a different node could leave the sidebar showing the previously-edited value instead of the newly-selected node's value. | Sidebar fields are now correctly refreshed to the newly-selected node's value. |
| Repeatedly switching between a node and a link in the Workbench eventually prevented the link side-bar menu from opening. | The link side-bar menu now opens reliably regardless of the previous selection. Workaround: On earlier releases, refresh the page and avoid selecting a link while a node was selected. |
| The Wireless LAN Controller (WLC) node type did not show its icon in the Workbench. | The WLC node definition now ships with its icon and the icon is rendered correctly on the Workbench. |
| Creating an annotation in the Workbench opened the annotation's sidebar in every other browser tab that had the same lab loaded, not just in the tab where the annotation was drawn. | A new annotation now opens its sidebar only in the tab where it was created. Other open tabs of the same lab no longer switch their selection. |
| An annotation with a higher layer value could render on top of the currently-selected annotation's selection handles and outline, making it difficult to interact with the selected annotation in the Workbench canvas. | Selection handles and outlines for the currently-selected annotation are now always rendered above other annotations. |
| The Delete key triggered the delete-confirmation dialog in the Workbench even when focus was inside an editable text field, so users could accidentally delete a node or annotation while editing one of its properties. | The Delete key no longer opens the delete-confirmation dialog when focus is on an editable element. Press Alt+Delete if you want to delete the currently-selected node or annotation while focus is on a text field. |
| Switching to a different lab in the Workbench reset the open pane and tab layout, even when the user later returned to the original lab. | Pane and tab layouts are now stored per-lab inside the current browser tab, so they survive switching between labs and persist until the browser tab is closed. |
| Right-clicking inside a right-click menu in the Workbench opened a nested context menu, instead of being ignored. | Right-click is now disabled inside Workbench context menus, so nested menus are no longer opened by accident. |
| The node sidebar in the Workbench always rendered a vertical scrollbar, even when the content fit inside the pane. | The node sidebar in the Workbench no longer displays an unnecessary vertical scrollbar. |
Known Issues and Caveats for CML 2.10
| Bug Description | Notes |
|---|---|
| Signing in can be slow when LDAP group management is enabled. | When LDAP group management is enabled, group membership is refreshed for every LDAP-linked user at each login. If the directory contains many LDAP-linked users and the LDAP server is slow or geographically remote, the per-login refresh can noticeably delay sign-in. |
| Discarding an unused Specific License Reservation (SLR) authorization code from the CML UI fails. | After cancelling an SLR reservation request and re-entering Register mode, clicking Discard Unused Authorization Code and submitting the previously-issued authorization code is rejected by the UI with the error 'Input validation failed. Input should be a valid string.' The authorization code therefore remains marked as in use. Workaround: Use the licensing API to discard the unused authorization code: POST /licensing/reservation/discard, with the authorization code in the request body. The endpoint can be reached from the Swagger UI at /api/v0/ui or with curl using a CML admin session token. |
| The Breakout web UI cannot be used to perform its initial password configuration. | The breakout config and breakout ui commands require a password to be supplied via the BREAKOUT_PASSWORD environment variable; without it, they refuse to run. Breakout does not write the password to any configuration file.Workaround: Set BREAKOUT_PASSWORD in the shell before launching Breakout. On Linux and macOS run BREAKOUT_PASSWORD="..." ./breakout-yoursystem-yourarch ui (terminals typically do not save commands starting with a space to history). On Windows use PowerShell: $env:BREAKOUT_PASSWORD = "..."; .\breakout-windows-amd64.exe ui. As an alternative, edit the Breakout configuration file manually and add a valid password to the configured CML user — be aware that storing the password in the file makes it readable to anyone with filesystem access. |
| The Decommission button on the Cluster Status page is unresponsive when a compute host is selected. | When you select one or more compute hosts on the Cluster Status page in the CML UI, the toolbar containing the Decommission button briefly disappears and the button never enables. The browser console logs a TypeError accessing the length property of an undefined nodes field.Workaround: Decommission the compute host through the API by sending DELETE /system/compute_hosts/{compute_host_id} (for example via the Swagger UI at /api/v0/ui or with curl using a CML admin session token). Make sure all nodes on the compute host have been wiped first. |
| Various actions requiring communication with a remote server fail due to the system DNS server being unreachable. | When you try to configure LDAP authentication, the process may block other CML operations for a couple of minutes if the system DNS server is unreachable. |
| A Docker container never starts and remains in the queued state. | When you create a custom Docker node type and do not configure metadata for the connected image correctly, the container remains in the queued state until it's moved back to the fresh state after an hour. |
| Nginx container node stops unexpectedly. | When you run an nginx container, it may stop unexpectedly. This problem is more likely to happen in CML deployments running many nodes. |
| IOL ARP drops and doesn't ping when ACL outbound is configured. | When you configure an outbound ACL on an interface of an IOL node connected to another IOL node, ARP is automatically dropped and the nodes don't reach each other. |
| When starting the CML VM on VMware Workstation or VMware Player, you receive an error saying that "Intel VT-x/EPT is not supported" or that VMWare "does not support nested virtualization on this host". | Recent (Nov / Dec 2024) changes in the Windows operating system have made it more difficult to disable Hyper-V so that VMware Workstation can enable nested virtualization for the CML VM. We are working on impacted customers to ensure that our updated documentation works for all supported deployment options. Workaround 1: See the steps in the latest version of CML Personal Intel VT-x/EPT Issue. If that does not resolve the problem, please contact CML support. Workaround 2: If you're running Windows Pro, you may be able to deploy CML directly to Hyper-V by following the community-contributed instructions for CML on Hyper-V. While we do not currently test CML on Hyper-V, customers have reported that the CML VM works well when running on Hyper-V. |
| External connectivity does not work in CML bare metal installations. | This bug impacts CML bare metal deployments regardless of release. In some deployments, most notably on UCS bare metal servers in which the vNIC's VLAN is set to access mode, the ingress traffic is tagged with vlan 0. This VLAN is meant to carry priority information as per the 802.1p standard. Network services can normally ignore this tag and handle vlan 0 packets transparently as untagged (no VLAN), but some node types in CML do not handle it gracefully, ignoring the incoming packets. As a result, the virtual nodes connected to an external connector node may not acquire an IP address from an external DHCP server or may not be able to communicate with external devices. Workaround: Follow the instructions from the Handling 802.1p Frames in UCS section of the Cisco Modeling Labs Administrator's Guide. |
| After a restart, you cannot reach the CML UI. The console of the CML instance shows a login prompt, but it also indicates that CML does not have an IP address. | After you shut down and restart a CML instance, it sometimes loses its IP address. When that happens, you will no longer be able to reach the CML UI even though the CML services are running. Workaround: When you encounter this problem, open the CML console (e.g., the CML VM's console in VMware or the CML server's remote management console). Log in as the sysadmin user and run this command: sudo netplan apply You can verify the current/new IP address of the CML instance using this command: ip addr show bridge0 If the issue persists, follow the instructions from the Editing the Management IP address via the Console section of the Cisco Modeling Labs Administrator's Guide. |
| Cannot run simulations on a macOS system that uses Apple silicon (e.g., the Apple M1 chip). | CML and the VM Images for CML Labs are built to run on x86_64 processors. In particular, CML is not compatible with Apple silicon (e.g., the Apple M1 chip). Workaround: There are no plans to support ARM64 processors at this time. Deploy CML to a different system that meets the system requirements for running CML. |
| Even if you were able to run network simulations with CML 2.2.x on VMWare Fusion and macOS Big Sur or above, the same labs no longer start after migrating to CML 2.3.0+. | A default setting in the CML VM changed between CML 2.2.3 and CML 2.3.0. Because of the new virtualization layer in macOS 11+, this setting causes errors when running the CML VM on VMware Fusion. Workaround: follow the instructions to disable nested virtualization in the CML VM. |
| Bug in macOS Big Sur. After upgrading to Big Sur and VMware Fusion 12.x, simulations no longer start. | MacOS Big Sur introduced changes to the way macOS handles virtualization, but the problems were resolved in later versions of macOS Big Sur and VMware Fusion. Workaround 1: If you are still experiencing problems, be sure to upgrade to the latest version of VMware Fusion 12.x and the latest bug fix release of your current macOS version. Workaround 2: If you are using CML 2.3+, then follow the instructions to disable nested virtualization in the CML VM. |
| Deleting or reinstalling a CML instance without first deregistering leaves the license marked as in use. Your new CML instance indicates that the license authorization status is out of compliance. | If you have deleted or otherwise destroyed a CML instance without deregistering it, then the Cisco Smart Licensing servers will still report that instance’s licenses are in use. Workaround 1: Before you destroy a CML instance, follow the Licensing - Deregistration instructions to deregister the instance. Properly deregistering an instance before destroying it will completely avoid this problem. Workaround 2: If you have already destroyed the CML instance, see the instructions for freeing the license in the CML FAQ. |
| When the incorrect product license is selected, the SLR (Specific Licensing Reservation) can still be completed. It eventually raises an error, but the SLR reservation will be complete with no entitlements. | Workaround: If you accidentally select the wrong product license, then you will need to release the SLR reservation and then correct the product license before trying to complete the SLR licensing again. |
| The algorithm for computing the available memory does not take into account maximal possible memory needed by all running nodes. It only checks the currently allocated memory. | CML will permit you to start nodes that fit within the current available memory of the CML server. Later, if the memory footprint of the already-running nodes grows and exceeds the available memory on the CML server, those nodes will crash. Workaround 1: Launch fewer nodes to stay within your CML server's resources. Workaround 2: Add additional memory to the CML server to permit launching additional nodes. Of course, this workaround just moves the point where the CML server will eventually hit its limit, so you must remain aware of your CML server's resource limits or ensure that the CML server has sufficient memory to accommodate all labs that will be running on your CML server simultaneously. |
When starting a CSR 1000v node for the first time or after a Wipe operation, the interfaces are all in shutdown state |
Workaround 1: Add no shut to each interface configuration in each CSR1000v node's Edit Config field in the UI. Workaround 2: Use the EEM applet on the CSR 1000v page. |
| New nodes cannot be started when CPU usage is over 95%, when RAM is over 80%, or when disk usage is over 95%. | Workaround: Allocate additional resources to the CML server. Workaround 2: If you are hitting the disk usage limit, check the Remove Orphaned VMs function in Cockpit. Orphaned VMs can consume a significant amount of disk space even though CML is no longer using or managing those VMs. |
The journalctl output shows errors like this: cockpit-tls gnutls_handshake failed: A TLS fatal alert has been received. |
This log message only occurs with Chrome browsers and has no functional impact on the use of Cockpit. No workaround needed. |
| In CML Cluster, the Cockpit Storage table for NFS mounts looks broken. | This is a display bug of the Cockpit UI when it encounters an NFS share using an IPv6 address. The single mount is split into two rows, only the first of which is indicated as mounted. Your NFS setup is probably fine, as long as the CML UI shows in its Cluster Status view that each compute host's Images indicator is green. Trying to edit or repair the NFS mount in any way in Cockpit is more likely to break the configuration. Please contact TAC if you experience networking issues with clusters which cannot be resolved by verifying basic connectivity on the cluster network or by restarting CML services. |