CML 2.2 Release Notes

Cisco Modeling Labs (CML) is a network simulation platform. CML 2.2 is the latest feature release of CML, and these release notes include information for all CML 2.2.x maintenance releases.

New and Changed Information

This table shows any published changes to this CML 2.2 release notes document.

Date Changes
May 12, 2021 Initial version for the CML 2.2.1 release.
June 8, 2021 Updated bug fixes and known issues for CML 2.2.2 release.
August 30, 2021 Updated bug fixes and known issues for CML 2.2.3 release and various other corrections.
November 19, 2021 Added Cockpit certificate issue to the list of known issues.
December 15, 2021 Added known issue for using CML with the latest versions of the Smart Licensing On-Prem Software.

Installing or Upgrading to CML 2.2

The CML Installation Guide provides detailed steps for installing or upgrading to CML 2.2.x.

If you already have an earlier CML 2.x instance, you can upgrade it to CML 2.2. You may upgrade directly to CML 2.2 from any CML 2.0.x or CML 2.1.x release. See the In-Place Upgrade page for step-by-step upgrade instructions. Upgrades from CML 1.x are not supported.

If you choose to migrate an existing, licensed CML 2.x instance to a new installation of CML 2.2, you must deregister the old CML instance before decommissioning it. See the note at the end of the Licensing - Deregistration page for instructions on deregistering a CML installation.

If you are creating a new CML instance, first choose one of these two CML installation options:

  • Deploy as a virtual machine (VM) on a supported VMware hypervisor
  • Deploy as a bare metal server

Check the System Requirements for supported hypervisors and other system requirements. Download the installation files, and then follow the installation and initial set-up instructions for your selected installation option.

Summary of Changes

CML 2.2.1 was a feature release of CML. This section explains the features that were added or changed since CML 2.1.2.

Link labels are now available in the workbench. The labels are automatically generated based on the interface name. The labels can be toggled on/off via the workbench toolbar. See the CML User Guide - Workbench.

Note: Link labels are not shown for the external connector and unmanaged switch node types. Link labels are also hidden when there are two or more links added between two nodes.

Show/Hide link labels in the workbench

The link style can now be changed from the default curved links to straight links via the workbench toolbar. See CML User Guide - Workbench

Curved/Straight link style in the workbench

Console Pane in the UI

This release includes changes to the console in the UI.

  • By default, when you connect to a node's console, the Console pane will now display up to 300 lines of history from the activity on that node's console before you connected to it.
  • The Console spans the entire width of the lower pane of the Workbench.
  • The settings, such as the number of rows and columns for the console or the number of lines of history to display, are now configured via a separate console settings dialog.
  • You may also enable auto-connect in the console settings dialog. With that setting enabled, the UI will auto-connect to a node's console as soon as you open the Console pane or select a different node in the Workbench while the Console is visible. Note that even with auto-connect enabled, you may need to click in the Console to change the focus to that pane of the UI before you can type in the console.
  • As long as you are using the same web browser, the console settings will apply to all consoles for all labs even if you log out and log back into the UI.

Other Notable UI Changes

  • The Show All button on the Dashboard is now available in both list and tiles mode and for all users. Toggle this button to change the lab list from showing only your labs vs. all labs you can access. See Changing the Lab List View and Multi-Selecting Labs and Lab Sharing.
  • As long as you are using the same web browser, the current state of the Show All and Show List toggle buttons will be preserved when you return to the Dashboard page, even after you log out and log back into CML.
  • Download System Archive has been disabled. This feature was never fully supported, and it will be added back in a future release.
  • The Force Wipe button for a node has been removed from the Workbench page of the UI. Note: It is still possible to force-wipe a node via the CML APIs.
  • The System Administration page has been updated to use a tabbed layout.
  • The web browser's window or tab title now shows the current page name. In previous releases, the window or tab title was the same for all pages. Page name in the web browser title
  • When you edit a lab's title in the Dashboard or in the lab Workbench, the input field has confirm (✔) and cancel (✖) buttons. Confirm/cancel changes to lab title
  • Since CML 2.2.2, the VNC pane for a node has new buttons for showing the VNC in full screen and for sending the CTRL+ALT+DEL keyboard sequence to the node.

Lab Sharing

CML 2.2 introduces the ability to share labs with other users on the same CML instance. You can grant other users read-only or read/write access to the lab. This lab permissions feature is available for all non-personal license levels, such as CML-Enterprise.

For details on this feature, see the CML User Guide - Lab Sharing.

Creating a group for lab sharing

Base OS and Security Improvements

The base operating system (OS) for new CML installations has been updated to CentOS 8.3.2011 for both OVA and bare metal deployments.

CML 2.2 has an improved security posture. For example, the SSH and TLS configurations have been updated to prevent vulnerable ciphers from being used. The CML UI has been updated to prevent potential XSS vulnerabilities. To improve password strength, new or updated CML user passwords will be checked against a list of thousands of common, easy-to-guess passwords.

CML 2.1.x was restricted to running an older version of the Linux kernel to avoid problems with specific reference platforms. CML 2.2.1 relaxes the list of packages that are excluded from updates, enabling your CML server to use the latest kernel from the upstream Linux repositories. The CML Smart Licensing module has also been updated, making it compatible with the current base system libraries. These changes make it safer to for administrators to apply CentOS security updates to a CML server.

Node Definition Improvements

The default Server image that is provided on the new refplat ISO file includes IPv6 support.

CML's boot monitor can now use regular expressions. When creating a custom node definition, you may use regular expressions in the boot lines that you add to the node definition's Boot section. See the CML User Guide - Creating A New Node Definition.

The node definition editor has a new Video Model dropdown for selecting the video driver. Select Tools > Node and Image Definitions from the navigation menu. In a node definition, the Video Model dropdown is part of the Linux Native Simulation group.

Video adapter drop-down list

Updates for Admin Users

With administrator privileges, you can modify the role of an existing user. Select Tools > System Administration from the navigation menu, and click on the User Administration tab.

Modify a user's role

You may now search for users in the User Administration page. Select Tools > System Administration from the navigation menu, and click on the User Administration tab. Enter text in the Search box to filter the list of users.

Search the user list

When you delete a user, a confirmation dialog will be shown before the user is removed.

When downloading logs in the CML2 page in Cockpit, you may now select the number of days of log data to include.

The Cockpit CML2 page now includes exposes additional actions:

  • There's a new Licensing Reset button under the Factory Reset group. Normally, you would configure licensing on the Tools > Licensing page in the CML UI, but in some licensing failure cases, it is necessary to reset all licensing data before relicensing the instance.
  • The new Copy Refplat ISO button replaces the old script for copying the reference platforms to the CML server's hard drive. See Copy the Refplat ISO to Disk for more details.
  • The new Remove Orphaned VMs button in Cockpit permits you to remove old VMs that CML is no longer using. It is normal for the Force Wipe action to leave these orphaned files or VMs on the CML server, but they will continue to consume disk space on your CML server. The Remove Orphaned VMs button enables the sysadmin account to clean up the system.

API Changes

CML 2.2.1 introduces several changes to the REST-based web service APIs. For details about a specific API, log into the CML UI and click Tools > API Documentation to view the CML API documentation page.

New APIs in CML 2.2

This table lists the API endpoints that were added in CML 2.2.1.

API Endpoint Description
DELETE /logout Logs out a user. Under the hood, current token is invalidated
GET /users/{user_id}/groups Get the groups the user is member of.
PUT /users/{user_id}/roles Update the user roles.
GET /groups Get the list of available groups.
POST /groups Creates a group.
GET /groups/{group_id} Gets the info for the specified group.
PUT /groups/{group_id} Updates a group.
DELETE /groups/{group_id} Deletes a group.
GET /groups/{group_id}/labs Get the list of labs that are associated with this group.
GET /groups/{group_id}/members Get members of the group.
GET /labs/{lab_id}/groups Returns the groups this lab is associated with.
PUT /labs/{lab_id}/groups Modifies lab / group association
GET /labs/{lab_id}/nodes/{node_id}/consoles/{console_id}/log Returns the log for the specified console

APIs Changed in CML 2.2

This table lists existing API endpoints that were modified in CML 2.2.1:

API Endpoint Summary of Change
GET /users The response object was changed. The id property replaces user_id. The admin property was added, and the context property was removed.
GET /users/{user_id} The response object was changed. The id property replaces user_id. The admin property was added, and the context property was removed.
POST /users/{user_id} The request object was changed. The groups property was added, and the context property was removed. The API now returns an HTTP 201/Created to indicate success.
GET /licensing/certificate The response may be empty--i.e., a null object.
DELETE /licensing/certificate No longer returns a response body on success.
PATCH /labs/{lab_id}/nodes/{node_id} The request object now accommodates changing additional properties on the node.

Bug Fixes

A list of the significant bugs fixed in CML 2.2.

Bug Description Resolution
Stopping a lab while it is starting could leave the lab's nodes in a bad state. After the lab stops, the affected nodes would not start. Fixed in 2.2.0. CML properly removes non-started nodes from the launch queue when you press Stop Lab before all nodes are actually started.
Nodes with video or using a frame buffer, such as the included Desktop image or a user-defined Windows image, would not properly configure the video card model and the video memory. In addition, the USB tablet for proper mouse pointer location was attached to the wrong bus inside the node VM. Fixed in 2.2.0. The node definition now provides a drop down list for the video card selection instead of a free-form input field.
Unexpected low-level events inside the CML controller could cause event handling to stop. All users would stop receiving updates in the UI. Fixed in 2.2.0.
Stopping a node from within the guest OS--for example, by running shutdown -h node in the console of an Ubuntu Linux node--would cause the virl2-devicemux service to lock up. When that happened, all nodes for all users would stop working. Fixed in 2.2.0.
If a link was stopped twice, it could cause an error that would prevent the link or the associated nodes from starting again until the lab was wiped. The most common cause was multi-selecting the nodes on both ends of a link and stopping the nodes. Fixed in 2.2.0. Improvements in the low-level driver and fabric services to handle duplicate link deletions gracefully and prevent errors from concurrent requests.
The default DHCP client used with NetworkManager in CML 2.1.x did not work with certain DHCP servers due to different padding and specific options set in the DISCOVER packet. In particular, a CML 2.1.x server could not get a DHCP address from the DHCP server on a Cisco Virtual Office (CVO) router. Fixed in 2.2.0. CML 2.2 switches to the dhclient standalone / standard DHCP client instead of NetworkManager's internal DHCP client.
In the UI's Packet Capture pane, the Packet Details display would only show the topmost layer of packets that had multiple layers with the same layer name. This bug was typically seen with encapsulated / tunneled payloads, such as an MPLS VPN. Fixed in CML 2.2.1.
In some cases, CML 2.1.x failed to create the bridge0 interface and the underlying plumbing, which resulted in a lack of network connectivity to the CML server. Fixed in CML 2.2.1. An additional check has been added to make sure the first connected ethernet interface has a valid connection UUID.
The IP address snooper for interfaces connected to an external network could sometimes fail to acquire the address of a host that uses a static IP address. Fixed in CML 2.2.1. Improved the ipsnooper service by adding functionality to acquire the IP address by looking at ARP request/replies and at gratuitous ARPs.
The controller service would periodically crash with a Smart Licensing error. Fixed in 2.2.0. The Smart Licensing component has been improved and made more robust.
If you set and then cleared the CPU Limit on a node in the UI, CML would refuse to start the lab, indicating an illegal cpu_limit value. Fixed in CML 2.2.1.
If the image associated with a node is not available, attempting to start the node will cause it to get stuck in a Queued state. Fixed in CML 2.2.1. If the refplat ISO is not mounted, and the refplat images have not been copied to the CML server, then the nodes will simply fail to start, and you will see an error notification in the UI.
When you try to start a node that requires more vCPU or RAM resources than the CML instance has available at that moment, the UI does not display any error or warning notification. The node was still in the launch queue, but the UI displayed no warning notification, and the node state in the UI still showed CREATED state. Fixed in CML 2.2.1. The node will still fail to start until resource are available, but the UI will show an error notification that indicates insufficient resources.
It was possible to start nodes that would fill up the CML server's disk with no warning or notification. The CML server now checks available disk space, and when the available disk space drops below 5%, the CML server will stop launching new nodes. The user will receive an error notification in the UI explaining why the node was not started.

The threshold was changed from 20% of disk space free in CML 2.2.1 to 5% of disk space free.
Deleting a user who is still logged into CML triggers, causes the controller service to stop sending events. All other users who are currently logged into the CML UI are impacted. They can continue working with the UI, and all actions are correctly saved, but the UI will not update to show the result of their actions, e.g., adding a node to a lab or starting a lab, unless the user refreshes the browser. Fixed in CML 2.2.1.
After you deregistered a CML instance, the System Status dialog and status bar indicator did not reflect the updated status until you refreshed the UI. Fixed in CML 2.2.1. The licensing status now reflects the correct status immediately.
The virl2-devicemux and virl2-dispatcher services sometimes crashed, especially when the CML server was under load. Fixed in CML 2.2.1. Improved validation checks and fixed bugs in handling concurrent requests.
If you set a custom value for the node's simulation properties, such as the RAM allocation or the CPU limit, and then reset the value back to the default, you could no longer download the lab. The default value was being set incorrectly, causing the lab download to fail silently. Fixed in CML 2.2.1.
Setting a bridge interface higher than 99 on an External Connector node caused the node to fail to start. Fixed in CML 2.2.1. The bridge interface may now be any value from 0 to 9999.
When clicking on Build Initial Bootstrap Configurations, the configuration generated for the XRv9k nodes did not include the default user accounts. Fixed in CML 2.2.1. When configurations are generated for XRv9k nodes, the cisco, admin, and lab users are part of the configuration.
When a user is updated or deleted, his json token was valid for the next 24 hours or until the next restart of the CML controller. Fixed in CML 2.2.1. The token is invalidated as soon as the user is deleted.
When the password change fails because, for example, the specified old password is incorrect, the error message was unclear. Fixed in CML 2.2.1. The error messages were improved and now indicate the reason for the failure.
When using the Nodes pane in the Workbench, clicking the checkbox next to a node in the list was not correctly checking the checkbox or adding the node to the selection. Fixed in CML 2.2.1.
VM Images uploaded to the CML server from the Python Client Library could not be used in a new image definition. Fixed in the virl2-client library version 2.2.
After running Copy Refplat ISO, CML could fail to start custom VM images that you added to the CML server. The Copy Refplat ISO action was fixed in CML 2.2.2.
The external connector node could not be configured in NAT mode when running CML as a VM. Fixed in CML 2.2.2.
There was no way to send CTRL+ALT+DEL to nodes with a graphical desktop via the VNC pane in the CML UI. Fixed in CML 2.2.2. There are now buttons to show the VNC in full screen and to send the CTRL+ALT+DEL key sequence to the node.
The Tools > API Documentation page in CML 2.2.1 still showed CML 2.1. The page content was correct, but the title and API version were not. Fixed in CML 2.2.2.
Upgrading CML-Personal from 2.1.x to 2.2.1 with the RPM broke licensing. Fixed in the CML 2.2.2 RPM. Upgrading a licensed CML 2.1.x CML-Personal instance directly to CML 2.2.2 will result in a licensed CML 2.2.2 instance.
The Workbench's Nodes pane in the CML UI did not show the number of days of uptime. Fixed in CML 2.2.2. If a node has been running for more than 24 hours, the uptime includes the number of days.
The Remove Orphaned VMs button could remove persistent data for some nodes that were still in active use, not just the orphaned data. This bug could result in data loss: for example, any changes made in the consoles of a running simulation would be lost unless they had previously been extracted back to the lab with the Extract Configuration button. Fixed in CML 2.2.3.
The Packet Capture pane for a selected link could become unresponsive when running a large packet capture. Fixed in CML 2.2.3.
There were no limits on the settings for the number of rows and columns for the Console pane in the lab Workbench page had no limits. Fixed in CML 2.2.3. CML validates the console settings for the number of columns and rows with a valid range based on the display resolution.
No error was shown in the UI when CML failed to import an invalid node definition. Fixed in CML 2.2.3.
Various known security vulnerabilities in packages shipped with CML 2.2.2. CML 2.2.3 includes the latest package updates and security fixes. Note that if you are upgrading an existing CML installation, you must also run follow the instructions for "Upgrading the Base OS" to get security updates that are not included in the CML RPM file.

Known Issues and Caveats for CML 2.2

Bug Description Notes
Unable to perform CentOS 8.x software updates in CML cockpit. CentOS Linux 8.x reached End Of Life (EOL) on December 31st, 2021. Performing software updates in CML Cockpit will return the following error:

Loading available updates failed. Cannot update repo 'appstream': Cannot prepare internal mirrorlist: No URLs in mirrorlistcannot update repo 'appstream': Cannot prepare internal mirrorlist: No URLs in mirrorlist

Workaround: CentOS has decommissioned and archived the CentOS 8.x package repositories, but your CML instance is still referring to the now-defunct repository URLs. Log into the CML server's console as the sysadmin user and then run:
sudo sed -i 's/^mirrorlist/#mirrorlist/; s/^#baseurl/baseurl/; s/mirror.centos.org/vault.centos.org/' /etc/yum.repos.d/CentOS-Linux-AppStream.repo 
sudo sed -i 's/^mirrorlist/#mirrorlist/; s/^#baseurl/baseurl/; s/mirror.centos.org/vault.centos.org/' /etc/yum.repos.d/CentOS-Linux-BaseOS.repo
sudo sed -i 's/^mirrorlist/#mirrorlist/; s/^#baseurl/baseurl/; s/mirror.centos.org/vault.centos.org/' /etc/yum.repos.d/CentOS-Linux-Extras.repo

Please note that the workaround sets your CML instance to use the archived CentOS 8 package repositories and permits you to install updates that were released up to the CentOS 8.x EOL on Dec. 31, 2021. Those repositories will have no new updates or security fixes after that date..

When using the latest versions of the Smart On-Prem software, the CML server cannot refresh its license registration or authorization. This problem only impacts CML Servers that are registered and licensed with a Cisco Smart Manager On-Prem instance. The CML server will eventually go into a license OUT_OF_COMPLIANCE state. CML was tested and works with On-Prem version 8-202008. CML 2.2.3 may not be fully compatible with later versions of Cisco Smart Manager On-Prem software.

Workaround: In practice, you can register and authorize your CML server with the On-Prem instance, but this problem will cause the CML server's license authorization renewals and registration renewals to fail. Both automatic and manual renewals are affected. The temporary fix is to reset licensing data. Log into Cockpit, and on the CML2 page, click the Factory Reset group to expand it. Click the Licensing Reset button. After the licensing reset completes, follow the Licensing instructions to license your CML server again. The CML server should report that the license status is Authorized. This workaround may need to be repeated after about 90 days when the CML server again fails to reauthorize its license with the On-Prem instance.
After the services restart or after a software update, Cockpit fails to start. If you navigate with your web browser to your CML server, port 9090, you will get a connection refused error. Since Cockpit is not working, you will need to use the CML server's console or, if you had previously enabled the OpenSSH service, SSH to the CML server on port 1122. If you look at the journalctl output in the CML server's console, you will see repeated errors:

cockpit-certificate-ensure: unable to check expiry of chained certificates

Workaround: Log into the CML server's console as the sysadmin user and then run:
sudo rm /etc/cockpit/ws-certs.d/0-self-signed.cert 
sudo systemctl restart cockpit

Cockpit will generate a new self-signed certificate when it restarts, and it should start up successfully. The next time you open the Cockpit page in your web browser, you will probably need to accept the new self-signed certificate first.

Deleting a CML VM without first deregistering leaves the license marked as in use. New instances indicate that the license authorization status is out of compliance. If you have deleted or otherwise destroyed a CML instance without deregistering it, then the Cisco Smart Licensing servers will still think that instance’s licenses are in use.

Workaround 1: Before you destroy a CML instance, follow the Licensing - Deregistration instructions to deregister the instance. Properly deregistering an instance before destroying it will completely avoid this problem.

Workaround 2: If you have already destroyed the instance, see the instructions for freeing the license in the CML FAQ's Licensing topic.
Bug in OS X Big Sur. After upgrading to OS X Big Sur and VMware Fusion 12.x, simulations no longer start. Big Sur introduced changes to the way OS X handles virtualization. While VMware Fusion 12 works with the new changes in OS X Big Sur, there is a bug that impacts some Apple systems. On those systems, after you upgrade the host OS to OS X Big Sur, labs will not start, and nodes just stay in QUEUED state. While not all systems exhibit this problem, we currently recommend that all CML customers remain with OS X Catalina until Apple and VMware fix this bug.

Workaround 1: Do not upgrade to OS X Big Sur. Continue running the CML VM on OS X Catalina and Fusion version 10+. Note that CML appears to work correctly with Fusion 12 as long as the host OS is OS X Catalina.

Workaround 2: If you already upgraded to Big Sur, and you experience the specified symptoms, then you may need to roll back your entire system to the last backup before you upgraded to Big Sur.
Ubuntu images will use a different interface name from what was defined in the node definition. This problem is mostly cosmetic. In the definitions, the interfaces are named enp0s2, enp0s3, etc. With changed VM definitions, the interface names are ens2, ens3, etc. Note: you may need to change the cloud-init scripts for your Ubuntu nodes if they specifically reference the interface names.
When the refplat ISO is not mounted until after the initial set-up steps have started, CML will not load the node and image definitions after you complete the initial set-up. Workaround: Log into Cockpit, and on the CML2 page, click the Restart Controller Services group to expand it, and click the Restart Services button. CML will load the node and image definitions when the services restart.
When the incorrect product license is selected, the SLR (Specific Licensing Reservation) still can be completed. It eventually raises an error, but the SLR reservation will be complete with no entitlements. Workaround: If you accidentally select the wrong product license, then you will need to release the SLR reservation and then correct the product license before trying to complete the SLR licensing again.
If the same reservation authorization code is used twice, the CML Smart agent service crashes. Workaround: Do not use the same reservation authorization code more than once.
If you have read-only access to a shared lab, some read/write actions are still shown in the UI. Attempting to change any properties or to invoke these actions raises error notifications in the UI. Workaround 1: Do not attempt to invoke the read/write actions when working with a lab for which you only have read-only permissions.

Workaround 2: Ask the lab owner to share the lab with read/write permissions so that you can make those changes to the lab.
If you have read-only access to a shared lab, you will still not be able to see the current link conditioning properties in the link's Simulate pane. Workaround: ask the lab owner to share the lab with read/write permissions if you need to access the link conditioning properties.
The algorithm for computing the available memory does not take into account maximal possible memory needed by all running nodes. It only checks the currently allocated memory. CML will permit you to start nodes that fit within the current available memory of the CML server. Later, if the memory footprint of the already-running nodes grows and exceeds the available memory on the CML server, those nodes will crash. Workaround 1: Launch fewer nodes to stay within your CML server's resources.

Workaround 2: Add additional memory to the CML server to permit launching additional nodes. Of course, this workaround just moves the point where the CML server will eventually hit its limit, so you must remain aware of the your CML server's resource limits or ensure that the CML server has sufficient memory to accommodate all labs that will be running on your CML server simultaneously.
On a CML server or VM with multiple NICs, the CML services are active and running and do not report an error, but the CML UI and APIs are not reachable. The CML services are listening on the wrong interface because, when the CML instance has multiple NICs, CML chooses the first alphabetically-ordered interface and configures it as the CML server's management interface. That may not be the correct interface. For example, an interface name like enp10s0 would be selected before enp9s0. Workaround 1: Change the network connections to the CML server so that the interface that CML selects is the one that you'd like it to use for its management interface.

Workaround 2: After the CML server boots up, configure the network interface manually in the Cockpit Terminal:
nmcli con
That command shows which interface bridge0-p1 is actually connected to. In this example, let's say that bridge0-p1 connected to ens10p0, and we want it to be connected to ens9p0. We need to remove the bridge0-p1 connection and then recreate it with a connection to the right interface. Adjust the following commands based on your interface IDs.
check con del bridge0-p1 
nmcli con add type bridge-slave \
con-name bridge0-p1 \
ifname enp9s0 master bridge0
nmcli con down enp10s0
# verify that the set up is now correct
nmcli con
In the end, you should see only 3 active connections (green with a device set). The first two are bridge0 and virbr0, and we do not touch these; the third one is bridge0-p1 which should now have the correct interface listed as the device.
When starting a CSR 1000v node for the first time or after a Wipe operation, the interfaces are all in shutdown state. Workaround 1: Add no shut to each interface config in each CSR1000v node's Edit Config field in the UI.

Workaround 2: Use the EEM applet on the CSR 1000v page.
New nodes cannot be started when CPU usage is over 95%, when RAM is over 80%, or when disk usage is over 95% (over 80% in CML 2.2.1). Workaround: Allocate additional resources to the CML server.

Workaround 2: If you're hitting the disk usage limit, check the Remove Orphaned VMs function in Cockpit. Orphaned VMs can consume a significant amount of disk space even though CML is not longer using or managing those VMs.
The external connector node cannot be configured in NAT mode when running CML 2.2.1 as a VM. This problem was fixed in the CML 2.2.2 OVA. If you upgraded your CML VM to 2.2.1 from an earlier CML 2.x release, your CML VM should also not be affected by this problem.

This problem only affects new deployments that used the CML 2.2.1 OVA.

Workaround 1: Run the following commands in the Cockpit Terminal:
sudo firewall-cmd --permanent --change-zone=virbr0 --zone=libvirt 
sudo systemctl reload firewalld
Workaround 2: Set the external connector node's config to use Bridge or Custom.
VM Images uploaded via the API can only be used in an image definition if they are transmitted via a multipart/form-data request. The problem is that the SELinux context is not set correctly. Workaround: Use the Python Client Library, which handles image uploads correctly.

Workaround 2: Create a multipart/form-data request when calling this API.
If you ran the Copy Refplat ISO step in CML 2.2.1, or if you ran the copy-refplat-iso-to-disk.sh script in earlier CML releases, the SELinux security context for the VM images may be incorrect. Workaround 1: In CML 2.2.1, before using the Copy Refplat ISO button in Cockpit, run
sudo sed -i 's/ahvz/ahvzX/' 
/usr/share/cockpit/virl2/copy-refplat-iso-to-disk.sh
. This workaround is no longer needed in CML 2.2.2.

Workaround 2: If you have already copied the refplat ISO in a release before CML 2.2.2, run the following commands in the Cockpit > Terminal to repair your system. Ensure that all labs are stopped before running these commands.
sudo systemctl stop virl2.target 
source /etc/default/virl2
sudo semanage fcontext -a -t virt_content_t "$REF_PLAT_ISO_IMAGE(/.)?"
sudo restorecon -vvRF "$REF_PLAT_ISO_IMAGE"
sudo semanage fcontext -a -t virt_content_t "$REF_PLAT/diff(/.
)?"
sudo restorecon -vvRF "$REF_PLAT/diff"
sudo systemctl start virl2.target

Workaround 3: Do not copy the refplat ISO until after upgrading to CML 2.2.2, which fixes this bug.
If you used the default product configuration when installing an earlier CML-Personal release, your CML instance may experience licensing errors after upgrading the system to CML 2.2.1. The Tools > Licensing page indicates the you must select a product configuration. Workaround: After upgrading to CML 2.2.1, run the following commands in the Cockpit Terminal:
sudo -E -s 
echo "CML_Personal" > /var/local/virl2/smart/product.txt
chmod 644 /var/local/virl2/smart/product.txt
chown nginx:nginx /var/local/virl2/smart/product.txt
Then restart the controller services on the CML2 page in Cockpit. Once the services restart, licensing should be working again, and the Tools > Licensing page no longer shows an error.
The CML UI's VNC pane for nodes with a graphical desktop does not permit you to send CTRL+ALT+DEL to the node. Upgrade to CML 2.2.2, which provides this capability.
In CML 2.2.1 and 2.2.2, the Remove Orphaned VMs button could remove persistent data for some nodes that were still in active use, not just the orphaned data. This bug could result in data loss: for example, any changes made in the consoles of a running simulation would be lost unless they had previously been extracted back to the lab with the Extract Configuration button. Upgrade to CML 2.2.3 before running Remove Orphaned VMs to avoid this problem.

Workaround: If you previously ran Remove Orphaned VMs, and it removed data for active nodes, those nodes will fail to start. To recover, wipe the node before starting it. Note that all persistent data associated with the node will be lost.
The Packet Capture pane for a selected link could become unresponsive when running a large packet capture. The packet summary table would continue to be updated, but none of the other buttons or selections in that pane would take effect until after the packet capture completed. In particular, it was not possible to stop a packet capture that was already in progress. Upgrade to CML 2.2.3, which fixes this bug.