Proxied, Air-gapped and Offline Environments¶
This section provides information on allowing the CML’ underlying Ubuntu OS to receive security and bugfix updates from its official package repositories.
In CML cluster deployments, each host’s OS package configuration and updates are independent of each other, and therefore any steps taken must be repeated on each host.
Package management and upgrades are performed by APT, with all configuration stored
in the /etc/apt directory on each Ubuntu host. The most important configuration there
is of package sources, i.e. where APT is looking what package versions are available
for installation.
By default, the primary official sources from the Ubuntu project are configured in all CML deployments, and they do not need to be changed. You can check for available upgrades and apply them whenever it is convenient. It is vital for this information to be accurate during upgrades; this includes removing any configuration when sources are not readily available, i.e. the deployment cannot make requests over public Internet.
In case your CML deployment is not able to directly access the remote repository over the Internet, consult first with your lab infrastructure administrators if there are any alternative sources readily available with an Ubuntu package repository, or what policies and approaches may be applicable for updating Linux-based systems.
In CML releases from 2.3.0 until 2.7.2 on Ubuntu 20.04 LTS, the configuration file
is named sources.list. During online upgrades to 2.8.0 or later, which is based on
Ubuntu 24.04 LTS, this file will be automatically edited to point to the newer OS release
and remains in use. New Ubuntu 24.04 LTS deployments use the sources.list.d/ubuntu.sources
file with a new shorter format. With both file formats, the most important information which
may require adjustment is the source HTTP URL of the respective repository involved.
By default, Ubuntu repositories are configured to be accessed over the HTTP protocol. Each package file is cryptographically signed, therefore it is safe to download packages even from insecure mirrors. However, you may change the source URLs to use the HTTPS protocol instead; the official Ubuntu OS repositories are also available that way, and the overhead of securing the connections is considered small.
The procedures below can be performed in a console, e.g., after you log into the System Administration Cockpit as the system administrator account. See Logging into the System Administration Cockpit. You can then use the Terminal tab from the panel on the left side of the page.
Configuring APT with Proxy and use of forcing IPv4¶
Procedure
You can configure a proxy URL for APT in deployments where access to the internet is available through a proxy server. APT ignores any other proxies configured elsewhere in the system or user environments and only uses its own configuration files.
Proxy servers may have configured access lists only allowing specific domains to be reachable.
The Ubuntu system sources are hosted on several subdomains of ubuntu.com. Since release
2.9.0, the software has a new dependency on Docker hosted at https://download.docker.com.
Online upgrades to 2.9.0 or later must ensure that this domain is reachable in the same way.
Some proxy servers disallow connecting to remote servers over IPv6. If your CML server resolves the APT repositories to an IPv6 address, but the remote site is not actually reachable, you can force APT to use IPv4 only. This applies even if you do not configure a proxy.
In deployments where many Ubuntu hosts are installed, it may be possible to use dedicated
caching servers as intermediaries, such as squid-deb-proxy or apt-cacher-ng. These
servers would be used in the same fashion as any other proxy server. They can themselves use
the main proxy server to contact the public repositories. Creating such setups is outside
of scope for this guide.
Create a new configuration file at /etc/apt/apt.conf.d/proxy.conf:
sudo nano /etc/apt/apt.conf.d/proxy.conf
Add the following lines to the file; skip
[username]:[password]@if not applicable:Acquire::http::Proxy "http://[username]:[password]@[proxy-web-or-IP-address]:[port-number]/"; Acquire::https::Proxy "http://[username]:[password]@[proxy-web-or-IP-address]:[port-number]/";
In case your CML server network is dual-stacked, and resolves the remote repository to an IPv6 address, but it is unable to connect using that address (directly or through the proxy), add the following line to force use of IPv4:
Acquire::ForceIPv4 "true";
Save the edited file using Ctrl+X and Y.
The APT commands can now use the configured proxy to reach the repository. Verify this with:
sudo apt update
Configuring an alternative APT mirror URL¶
Procedure
In some corporate environments, you may have access to a local Ubuntu APT mirror server. You can then use its URL instead of the official public repositories in the source configuration. You may also select alternate public APT mirrors, which may be located more closely to your lab.
Save a backup of one or both of the configuration files (whichever exist in your deployment).
sudo cp /etc/apt/sources.list{,.bak}
sudo cp /etc/apt/sources.list.d/ubuntu.sources{,.bak}
Edit the original configuration file:
sudo nano /etc/apt/sources.list
sudo nano /etc/apt/sources.list.d/ubuntu.sources
Replace any occurrences of the public repository URLs with the mirror server URL. Consult the comments inside the file for additional information on the meaning of the configuration options.
Save the edited file using Ctrl+X and Y.
The APT commands can now use the configured proxy to reach the repository. Verify this with:
apt update
Air-gapped and Offline Environments¶
Procedure
If your CML environment is permanently unable to reach any public or internal Ubuntu package repository, then you should remove the default package source configuration. You will not be able to install regular security and bugfix updates as published by Ubuntu in this situation.
Individual packages updates can be downloaded, copied onto the CML host and installed in case
you require a particular update to be installed in an urgent fashion. The apt-offline tool can
be downloaded and installed in the CML hosts if regular updates are required. A different
(Ubuntu) host connected to the Internet is then used to download updated packages from the public
repositories. The details of the procedures involved are described in the apt-offline manuals
available online.
Since CML 2.8.0 (i.e., the in-place upgrade from release 2.7.2 to 2.8.0), you can download and use
the CML .iso image that is used for baremetal installations as a package source. The .iso
for each release includes the latest available updates for those packages which would be installed
in a new deployment. These packages will be installed as part of the .pkg-based upgrade.
It is strongly recommended but not required to provide the .iso for ugprades for every maintenance
release of CML. It is usually required to provide the file for every feature upgrade of
CML, e.g. when upgrading from 2.7.2 to 2.8.x or 2.9.x, or from 2.8.x to 2.9.x. In either case,
the release notes for each individual version will provide guidance on this requirement.
To allow the underlying APT update function to complete correctly, the default remote package source must be removed from configuration, otherwise the tool will raise an error and the procedure will abort.
Move the configuration file to a backup location ignored by APT (whichever exists in your deployment):
sudo mv /etc/apt/sources.list{,.bak}
sudo mv /etc/apt/sources.list.d/ubuntu.sources{,.bak}
Prior to performing in-place upgrades to newer CML release, download both the .pkg and
.iso files for that release. Attach the .iso as a virtual CD-ROM device to each of your hosts.
In Cluster deployments, the .iso must be attached to all compute hosts at the same time.
The CML hosts are now ready for the in-place upgrade procedure using the .pkg file to
drive the process in the same way an online procedure would be performed.