Running Lethe on Digital Research Alliance of Canada Clusters

Setting-up the Folder Structure

In your $HOME, create a “dealii” folder and a “lethe” folder, each containing “build” and “inst” folders:

mkdir -p {dealii,lethe}/{build,inst}

The deal.II and Lethe projects can then be cloned in their corresponding folders, as indicated later in this tutorial.

After installation is complete, the folder structure will be, for deal.II (and likewise for Lethe):

• $HOME/dealii/dealii for deal.ii git,

• $HOME/dealii/build for compilation (cmake command),

• $HOME/dealii/inst for installation (ninja install command)

Folders can be open with the cd command (cd $folder_path).

For the sake of clarity, this is the folder structure considered for the rest of this tutorial.

Installing deal.II

All operations must be performed on login nodes.

Load Trilinos, Parmetis and P4est, and their prerequisite modules and set the appropriate environment variables. It is convenient to create a .dealii file in your $HOME directory that contains the following lines to source the appropriate libraries:

module load CCEnv #Only if on the Niagara cluster
module load StdEnv/2023
module load trilinos/15.1.1
module load parmetis/4.0.3
module load p4est/2.8.6
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$EBROOTTRILINOS/lib64/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$EBROOTP4EST/lib64/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$EBROOTFLEXIBLAS/lib64/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$EBROOTIMKL//mkl/2023.2.0/lib/intel64/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$EBROOTOPENMPI/lib/

export DEAL_II_DIR=$HOME/dealii/inst/
export PATH=$PATH:$HOME/lethe/inst/bin/

This file needs to be sourced every time you launch a job or you compile deal.II and/or Lethe. Once the file has been created, you can then source it on the terminal with:

source $HOME/.dealii

and use it in your .sh script when launching a job (see Launching Simulations below).

Although Lethe always supports the master branch of deal.II, we maintain an identical deal.II fork on the CHAOS laboratory organization. This fork is always tested to make sure it works with Lethe. To clone this deal.II fork, execute in $HOME/dealii directory:

git clone https://github.com/chaos-polymtl/dealii.git

We can compile dealii in the $HOME/dealii/build folder, by defining the paths to installation folders of Trilinos, Parmetis and P4est. To increase the speed of this step, we skip dealii tests and compile in release mode only.

cmake ../dealii -DDEAL_II_WITH_MPI=ON -DDEAL_II_WITH_TRILINOS=ON   -DTRILINOS_DIR=$EBROOTTRILINOS  -DDEAL_II_WITH_P4EST=ON -DCMAKE_INSTALL_PREFIX=$HOME/dealii/inst/ -DDEAL_II_WITH_METIS=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../inst/ -DDEAL_II_COMPONENT_EXAMPLES=OFF  -DCMAKE_CXX_FLAGS="-march=native" -G Ninja

Tip

The -DCMAKE_CXX_FLAGS=”-march=native” works on both Rorqual, Nibi and Niagara. To ensure that the flag has worked correctly, the cmake output should contain the following information : Vectorization level:    512 bit (sse2 avx2 avx512*).

Warning

If you wish to run simulations with over 4B (\(4\cdot 10^9\)) degrees of freedom, you must compile with the DEAL_II_WITH_64BIT_INDICES = ON flag. Such large simulations should be carried out using the lethe-fluid-matrix-free application.

and:

nice ninja -j6 install

The argument -jX specifies the number of processors used for the compilation. On login nodes, a maximum of 6 cores should be used in order to ensure that other users can continue using the cluster without slowdowns. If you use more than 6 cores, your compilation may be terminated automatically.

Installing Lethe

After installing deal.II, compiling Lethe is relatively straightforward. To compile Lethe, the Trilinos, Parmetis and P4est modules should be loaded.

In the $HOME/lethe directory, download Lethe:

git clone https://github.com/chaos-polymtl/lethe.git

To install Lethe in the $HOME/lethe/inst directory (applications will be in inst/bin), run in the $HOME/lethe/build directory:

cmake ../lethe  -DDEAL_II_DIR=$HOME/dealii/inst -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../inst -DCMAKE_CXX_FLAGS="-march=native" -G Ninja
nice ninja -j6 install

Copying Local Files

On Linux, use scp (for secure copy) to copy needed files for the simulation (prm, msh):

scp /home/path/in/your/computer/*.prm username@clustername.calculcanada.ca:/scratch/path/in/cluster

If you need to copy a folder, use scp -r.

Simulation files must be in scratch. To get the address of your scratch folder, in your cluster account run:

cd $SCRATCH
pwd

On Windows, use third-party, such as PuTTY or WSL (see the wiki page on Transferring data).

Launching Simulations

Simulations are sent to the scheduler via batch scripts. Visit the Digital Research Alliance of Canada (Alliance) wiki page for more information about the scheduler and running jobs. For your convenience, an example of job.sh used on Beluga is given below:

#!/bin/bash
#SBATCH --account=$yourgroupaccount
#SBATCH --ntasks-per-node=$X #number of parallel tasks per node.
#SBATCH --nodes=1 #number of whole nodes used
#SBATCH --time=1:00:00 #maximum time for the simulation (hh:mm:ss)
#SBATCH --mem=120G #memory usage per node. See cluster specification for maximal amount.
#SBATCH --job-name=$yourjobname
#SBATCH --mail-type=ALL
#SBATCH --mail-user=$your.email.adress@email.provider

source $HOME/.dealii
srun $HOME/lethe/inst/bin/$lethe_application_name_wanted $parameter_file_name.prm

Tip

The --ntasks-per-node option is the number of parallel tasks per node. When using a full node, this should correspond to the number of cores available on the node. For example, on Narval, this should be set to 64.

The job is sent using:

sbatch job.sh

Status can be followed with the sq command: under ST, PD indicates a pending job, and R a running job.

Console outputs are written in slurm-$jobID.out. For instance, to display the 20 last lines from this file, use:

tail -n 20 slurm-$jobID.out

Clusters Specifications

Please consult the documentation for the machine you are using for the specification of the nodes:

Cluster	Tasks per Node	Memory per Node	URL
Narval	64	248 Go	https://docs.alliancecan.ca/wiki/Narval/en
Niagara	40	200 Go	https://docs.alliancecan.ca/wiki/Niagara/en
Rorqual	192	760 Go	https://docs.alliancecan.ca/wiki/Rorqual/en

Saving a SSH Key (Linux)

To save your key on the cluster, so that it is not asked for each log or scp, generate your ssh-key with:

ssh-keygen

which defaults to an RSA key. If you want to specify the key type you want to generate (i.e. ED25519 key), type

ssh-keygen -t ed25519

Note

ED25519 keys are preferred to RSA keys since they are more secure and performant. Seek more information in the GitLab Documentation.

To upload this local key to your Compute Canada Database account (CCDB) use:

ssh-copy-id username@clustername.computecanada.ca

Warning

This command does not work on Niagara anymore. You may use the following:

cat ~/.ssh/$KEY_ID.pub

where $KEY_ID.pub is the public key file located in ~/.ssh/. For more information, see SSH documentation.