Zenith Quick Start Guide

These new nodes entered Early Access service in August 2026.

Important

  • This is a service that is in early access and not yet available for projects. Please keep an eye on official channels of communication from the Cambridge RCS team for further information.
  • During the Early Access phase, this page will be updated and improved regularly, if you notice something not quite right, please contact support@hpc.cam.ac.uk.

Getting Access

As mentioned above, this is in early access and so you will be invited to the relevant early access project via our Cambridge Access Portal, Waldur, please follow the guidance found in User Dawn Access as the procedure is the same for Zenith projects.

First Time Accessing Zenith

MultiFactor Authentication (MFA) is required to login, as with all our services. If you are already a user of our services (CSD3, Dawn or RDS/RCS/RFS) you will already have the appropriate MFA setup.

Otherwise, it is strongly recommended that you read through the MultiFactor Authentication (MFA) user documentation to assist with the setup of your MFA.

Please note that the MFA required in order to login via SSH to Zenith/Dawn/CSD3 is distinct from the MFA required to access Waldur (access.hpc.cam.ac.uk), and also from the MFA required by Open OnDemand (login-web.hpc.cam.ac.uk).

Login

All Zenith users should initially use the following login nodes to access Zenith resources (Zenith-specific login nodes will be forthcoming):

ssh <username>@login-dawn.hpc.cam.ac.uk

The first time you login, you will be asked to check that the host key fingerprints are correct. Please check that the fingerprints reported match those on the page CSD3 Host Keys before responding yes (NB not y).

You will then be asked for your TOTP Verification code; this is the MFA step mentioned above.

If you cannot connect to the above login nodes, first check that you have received and completed your Zenith project invite, as described above, and have received confirmation that you have been added to the project. If you have checked and received confirmation, please contact support@hpc.cam.ac.uk for further assistance.

Once you have logged in, you will have immediate access to your storage, but for development work you will likely want to request an interactive job using Slurm, as detailed below. This is because the login nodes are currently a different architecture to the Zenith nodes themselves, and also to lessen load on the login nodes.

Filesystems

Please see here for a summary of available filesystems and the rules governing them.

Hardware

Each Zenith (MI355X) node consists of:

  • 8x AMD Instinct MI355X GPUs.
  • 2x AMD EPYC 9575F processors (64 CPU cores per processor).
  • 288 GiB of GPU memory per GPU.
  • 64 TB/s aggregate GPU memory bandwidth per node.

Slurm Partition

The MI355X (gpu-u-N) nodes are in the mi355x Slurm partition.

Zenith Slurm projects follow the usual CSD3 naming convention for GPU projects and follow the pattern NAME-ZENITH-GPU, containing units of GPU hours.

Modules

To load the Zenith-specific modules explicitly (they are loaded by default on the Zenith nodes themselves) run:

module purge
module load rhel9/default-amdgpu-zenith

This will make available modules tailored to the Zenith architecture. In particular there are modules for the AMD compilers:

module load rocm

The rocm module gives access to the amdclang, amdclang++, and amdflang compilers. These are strongly recommended for CPU codes, and essential for GPU codes.

It also gives access to the ROCm command line tools. To check which version of ROCm is currently available, run the following on a Zenith compute node:

amd-smi version

amd-smi is also useful for checking that the GPUs are being used while an application is running.

The GCC compiler suite is also available via the gcc module.

For MPI codes, there is the openmpi module. This is tied to the rocm compiler suite. If you believe you need MPI with the GCC suite, please contact support.

Recommendations for running on Zenith

To run an interactive job on Zenith, use the sintr command, e.g. to request 16 CPU cores on 1 node, with 1 GPU for 10 minutes run:

sintr -p mi355x -N1 -n1 --gres=gpu:1 -A NAME-ZENITH-GPU -t 00:10:00

Note that you will be billed for Zenith GPU hours on interactive jobs, and that you will automatically receive 1/8 of the CPUs (16 CPUs) for each GPU requested.

When compiling codes to run using the Zenith GPUs you should specify the particular AMD architecture by passing the --offload-arch=gfx950 flag to the AMD compiler you are using. You may also use --offload-arch=native when compiling on a node.

Python

There is no central set of Python packages on Zenith: you should install the packages you need yourself. Doing so inside a virtual environment is strongly encouraged, as this keeps projects isolated from one another and makes them straightforward to rebuild. Possible ways of doing this include:

  • the python module together with the built-in venv module;
  • the miniforge3 module and conda environments;
  • uv, which manages the Python interpreter and the environment together.

Tip

Python environments, and the package caches used to build them, can easily run to several GB. Your home directory has a 50 GB quota, so keep both on your RDS storage, for example under /rds/user/$USER/<project-dir> (also reachable more conveniently as ~/rds/<project-dir>).

Existing Cambridge CSD3 users may also use ~/rds/hpc-work.

Creating the environment on RDS is not sufficient on its own, as the package managers cache downloads under ~/.cache by default. Redirect the caches as well, for example:

export UV_CACHE_DIR=/rds/user/$USER/<project-dir>/.cache/uv
export PIP_CACHE_DIR=/rds/user/$USER/<project-dir>/.cache/pip
export CONDA_PKGS_DIRS=/rds/user/$USER/<project-dir>/.conda/pkgs

The same applies to any models or datasets downloaded at run time, e.g. via HF_HOME for Hugging Face.

Running PyTorch

These instruction assume that you will be running PyTorch on a compute node. Having loaded the Zenith-specific modules above, load openmpi:

module load openmpi

As well as Open MPI itself, this makes available the ROCm libraries and command line tools installed on the system, including amd-smi.

Next, create a virtual environment to install PyTorch into. Any of the options in the Python section above will work; the example below uses uv:

uv init --python 3.14
uv venv
source .venv/bin/activate

Following the official AMD Documentation adapted for uv, to install the torch, torchvision, and torchaudio packages:

uv add --index rocm=https://repo.amd.com/rocm/whl-multi-arch/ \
    "torch[device-gfx950]==2.12.0+rocm7.14.0" \
    "torchvision[device-gfx950]==0.27.0+rocm7.14.0" \
    "torchaudio==2.11.0+rocm7.14.0"

You can verify your installation on a compute node by running

uv run python -c "import torch; n = torch.cuda.device_count(); print(f'visible GPUs: {n}')"

You should see an output corresponding to however many GPUs you requested in your submission.

The primary source of information about PyTorch, including installation instructions, is the PyTorch website. Note, however, that it is sometimes not yet updated for the most recent ROCm release; where that is the case, the AMD documentation linked above is likely to be more current.

Note

In PyTorch, AMD GPUs use the same device type ("cuda") and the same distributed backend ("nccl") as NVIDIA GPUs. Code written for NVIDIA GPUs therefore generally runs unchanged.

Multi-node PyTorch jobs

Multi-node PyTorch jobs can be launched with either srun or mpiexec. Note that mpiexec must come from the openmpi module; the version bundled with Slurm does not work for this.

torchrun is currently only supported for single-node (multi-GPU) jobs, as inter-node communication fails. If you need torchrun across multiple nodes, please contact support.

Problems

Please contact support@hpc.cam.ac.uk if you encounter any problems or have any questions