Learning CFD with OpenFOAM and Linux

OpenFOAM is Linux software. So the best option would be to run it on a Linux system. There are many different variations of Linux out there, so called distributions. The one distribution I would recommend for a beginner (and which we will use in one form or the other) is Ubuntu because it is currently the most popular and is (in my view) the easiest to install. Another reason is that OpenFOAM is available as an installation package for this distribution.

Linux can be installed on most machines, so if you have a computer that has enough room on a hard drive (or you have a spare hard drive) you can install Linux alongside the Windows system in a so-call “dual-boot” configuration, where you can select the operating system in a boot menu. Ubuntu comes as a bootable image that you can put on a usb drive and boot from it. The cool thing is that that image is a “live system”, so you can boot into Ubuntu directly from the usb-drive and test the system before you install it. Select the option to install it alongside the existing Windows installation and the installer will do the rest¹.

To run OpenFOAM on Windows you can use Windows Subsystem for Linux (WSL2) which comes with Windows 10. In WSL2 you can then install a Linux system (Ubuntu 20.04LTS) and then run OpenFOAM in that. Detailed instructions on the Microsoft website. Follow these and select Ubuntu 20.04LTS as the Linux distribution to install². On some systems you will need to activate the virtualisation in the BIOS³ See this page - on AMD systems the setting may also be called AMD-V.

You can also install a docker image containing both Linux and OpenFOAM. This requires you to install docker and then install a docker image with a Linux system and OpenFOAM.

This is an interesting method, but I have not tested it. You may want to give it a go. Please let me know and we can try that. On Windows the docker container runs in a small virtual box, so it may have some performance penalty over WSL2.

The university has a subscription to Azure Labs from Microsoft. Azure is a remote desktop system, so you can run everything in the cloud through your browser. This is an excellent solution if your computer is not fast enough or does not have enough RAM to run OpenFOAM directly. It should even work on a Chromebook. We are currently finalising the setup, and I will update this entry with the details.

OpenFOAM v2106 is the ESI version of OpenFOAM. You can also use their Installation Instructions. It is not in an included repository, so before you can install it you need to add the repository and the security keys:

curl -s https://dl.openfoam.com/add-debian-repo.sh | sudo bash

Simply copy and paste the above line into the terminal. This runs a script that will add the repository to your system configuration. After that you can install the OpenFOAM packages using:

    sudo apt update
    sudo apt install openfoam2106-default

To initialise OpenFOAM, you open a terminal and type:

openfoam2106

OpenFOAM 7 is the Foundation version of OpenFOAM⁶. You can also use their Installation Instructions.

sudo sh -c "wget -O - https://dl.openfoam.org/gpg.key | apt-key add -"
sudo add-apt-repository http://dl.openfoam.org/ubuntu

Simply copy and paste the above line into the terminal. This runs a script that will add the repository to your system configuration. After that you can install the OpenFOAM packages using:

    sudo apt update
    sudo apt install openfoam7

This will install version 7 of Foundation-OpenFOAM. If you want to install the newest version as well (not required):

    sudo apt install openfoam9

To initialise OpenFOAM 7, you open a terminal and type:

    source /opt/openfoam7/etc/bashrc

You will have to install Paraview on the Windows system.

Then you need to create an empty file called casename.foam in the case directory. The easiest way to do that is:

    touch casename.foam

This is a file that will tell ParaView to read the data directly from the FOAM file structure.

Then you can open Paraview and open that file from the “File” menu. But before you can do that, you will need to set up access to the Linux Filesystem in WSL2 from Windows (below).

Then we can open ParaView using the command:

    paraFoam

This is a command that sets up the case to be read in ParaView and then calls ParaView.

Oops, something went wrong? What’s the error message?

    Cannot use ParaView reader module library (PVFoamReader)
    The PV_PLUGIN_PATH environment value is not set

    Continuing with builtin reader:  paraFoam -vtk

    Created temporary 'elbow.foam'
    /opt/openfoam/openfoam7/bin/paraFoam: 327: paraview: not found

The first is not an error, it can’t use one reader, but falls back to the builtin one, that’s fine. The error is paraview: not found.

So we need to install that first. Luckily that’s easy on a Ubuntu system:

    sudo apt install paraview

will download and install ParaView and automatically download and install a lot of other packages that ParaView depends on.

You can start Paraview with:

    paraFoam

and have a look at the data. Information about Paraview can be found on the ParaView website. Have a look under “Resources”.

Have fun. You may want to have a look at some of the other tutorials as well.

We will have a closer look at the anatomy of OpenFOAM cases soon. In the meantime check the user guide.

Also interesting

Python is an object-oriented programming and scripting language that is widely used in data and engineering science. In is often used to control computational systems, as well. We will use PyFOAM, which is a set of utilities for OpenFOAM which make case handling and setup easier.

HaeoFoam is the code that I wrote for blood flow simulations. It is written in C++ using the OpenFOAM framework and includes some additional features:

• non-Newtonian rheology models, in particular the Quemada model Quemada1977,Quemada1978,Quemada1978b and a modified Krieger-Dougherty model Hund2017, as well as the original Krieger model Krieger1959,Woods1970,Papir1970.
• haematocrit transport model based on Phillips1992.
• windkessel boundary conditions Westerhof2009, adapted from code written by colleagues at Imperial College, London Piebalgs2017,Piebalgs2018,Pirola2017

haemoFoam is hosted on github, and if you have a github account and know how to compile code, I can give you access to it. But to make things easier, I have precompiled this code for you and you can simply download an archive file OF2106_apps.tgz from the projects Google Drive, move it to the Linux filesystem (into your home directory) (see How to access the Linux file system from Windows) and then install it as follows:

    cd ~/OpenFOAM/$USER-v2106/
    tar xvfz ~/OF2106_apps.tgz

Test that the installation worked with:

Initialise OpenFOAM (unless already initialised):

    openfoam2106

Call a haemoFOAM solver:

    haemoPimpleFoam

This should output something like:

    /*---------------------------------------------------------------------------*\
    | =========                 |                                                 |
    | \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
    |  \\    /   O peration     | Version:  v2106                                 |
    |   \\  /    A nd           | Website:  www.openfoam.com                      |
    |    \\/     M anipulation  |                                                 |
    \*---------------------------------------------------------------------------*/
    Build  : 14410a73-20200806 OPENFOAM=1912 patch=200727
    Arch   : "LSB;label=32;scalar=64"
    Exec   : haemoPimpleFoam
    Date   : Oct 31 2020
    Time   : 16:27:12
    Host   : TS-ZBook-17
    PID    : 2860549
    I/O    : uncollated
    Case   : /home/acests3/OpenFOAM/acests3-v1912
    nProcs : 1
    trapFpe: Floating point exception trapping enabled (FOAM_SIGFPE).
    fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
    allowSystemOperations : Allowing user-supplied system call operations

    // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
    Create time



    --> FOAM FATAL ERROR: 
    cannot find file "/home/acests3/OpenFOAM/acests3-v1912/system/controlDict"

        From function virtual Foam::autoPtr<Foam::ISstream> Foam::fileOperations::uncollatedFileOperation::readStream(Foam::regIOobject&, const Foam::fileName&, const Foam::word&, bool) const
        in file global/fileOperations/uncollatedFileOperation/uncollatedFileOperation.C at line 548.

    FOAM exiting

It fails, since there is no case file to run, but if you see the header file, then the installation was successful and your system is ready for the project.

The dimensions line tells the code what the dimensions of the object are. The numbers denote the exponent in the SI system in the order of:

[kg m s K mol A cd]

This means that the velocity has:

dimensions [0 1 -1 0 0 0 0];

i.e. \(m^1 s^{-1}\).

The next line describes the initial conditions. In most cases it will be enough to initialise with a uniform value as is done here:

   internalField   uniform (0 0 0);

This initialised the whole field with zero velocity.

The rest of the file contains the boundary conditions, in a nested block:

     boundaryField
     {
       patchName
         {
           type            boundaryConditionType;
           value           boundaryConditionValues;
         }
     }

Make sure that you keep track of the opening and closing curly brackets.

We will discuss the different boundary condition types here: OpenFOAM BCs. A full list of all available boundary conditions (most of which will not be of interest in our context) can be found here.

Most common pressure boundary conditions used in our simulations are:

• fixedValue: set boundary condition to a fixed value (very often zero relative pressure at an outlet ¹³)
• zeroGradient: this is the typical pressure BC that goes with a velocity BC that prescribes a fixed value for the velocity.

• Windkessel boundaries are a special case, so see the Windkessel boundary documentation: Windkessel BCs

  /*--------------------------------*- C++ -*----------------------------------*\
  | =========                 |                                                 |
  | \\      /  F ield         | foam-extend: Open Source CFD                    |
  |  \\    /   O peration     | Version:     4.0                                |
  |   \\  /    A nd           | Web:         http://www.foam-extend.org         |
  |    \\/     M anipulation  |                                                 |
  \*---------------------------------------------------------------------------*/
  FoamFile
  {
      version     2.0;
      format      ascii;
      class       volScalarField;
      location    "0";
      object      p;
  }
  // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
  
  dimensions      [0 2 -2 0 0 0 0];
  
  internalField   uniform 0;
  
  boundaryField
  {
      ICA
      {
          type            fixedValue;
          value           uniform 0;
      }
      ECA
      {
          type            fixedValue;
          value           uniform 0;
      }
      wall
      {
          type            zeroGradient;
      }
      APEX
      {
          type            zeroGradient;
      }
      SINUS
      {
          type            zeroGradient;
      }
      CCA
      {
          type            zeroGradient;
      }
  }
  // ************************************************************************* //
  

Important: OpenFOAM uses kinematic pressures, not static pressures. Note that the dimensions [0 2 -2 0 0 0 0]; are in \({m^{2} s^{-2}}\), not in \({Pa}\).

Kinematic pressure is related to static pressure as kinematic viscosity to dynamic viscosity:

\[p_{k} = \frac{p_{s}}{\rho}\]

See also: OpenFOAM User Guide: Kinematic Pressure

Velocity boundary conditions also have the two basic modes of setting a value and setting a gradient. But there are many more special use boundary conditions derived from these.

• fixedValue boundary conditions need to set the components of the velocity vector.
• fixedGradient boundary conditions are mostly used in the special form of a zeroGradient which does not need any parameters to be set.

The following shows these basic boundary condition in their native habitat:

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | foam-extend: Open Source CFD                    |
|  \\    /   O peration     | Version:     4.0                                |
|   \\  /    A nd           | Web:         http://www.foam-extend.org         |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       volVectorField;
    location    "0";
    object      U;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

dimensions      [0 1 -1 0 0 0 0];

internalField   uniform (0 0 0);

boundaryField
{
    ICA
    {
        type            zeroGradient;
    }
    ECA
    {
        type            zeroGradient;
    }
    WALL
    {
        type            fixedValue;
        value           uniform (0 0 0);
    }
    APEX
    {
        type            fixedValue;
        value           uniform (0 0 0);
    }
    SINUS
    {
        type            fixedValue;
        value           uniform (0 0 0);
    }
    CCA
    {
        type            fixedValue;
        value           uniform (0 0 1);
    }
}


// ************************************************************************* //

As you can see, the outlets have zeroGradient conditions, while both walls and inlet have fixedValue conditions. For the walls, the value is (0 0 0), while for the inlet the components are given in vector form (0 0 1), since the inlet in this case is in normal-z direction. Take care that you get the direction right! One of the most common mistakes is to set the flow direction to point out of the volume - which essentially sucks … the fluid out of the inlet, which rarely is what you want.

If you work on a lot of different machines, or want to automate processes (like copying the results back to your machine), then it is cumbersome to have to type your password every time. ssh has another login method, ssh-key. An ssh-key is a pair of keys, one secret/private and one public key, that together allow authentification. Make sure to keep your keys safe, and don’t accidentally give away the secret/private key! If you do keep the key safe then key authentification can be safer than password authentication, because I need to get my (digital) hands on your private key, and cannot simply guess your password (believe me, guessing passwords is ridiculously easy sometimes). Many servers don’t allow password authentification, but demand that you have a key.

In order to set this up we need to:

1. create the key:

thor@TS-ZBook-17-G2:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/thor/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/thor/.ssh/id_rsa
Your public key has been saved in /home/thor/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:Gi5AhJ7qfVmI5kZ2W0nBsvMVuiXZeeT2WyFzzLpSJGM thor@TS-ZBook-17-G2
The key's randomart image is:
+---[RSA 3072]----+
| ..   .+ ooo     |
|..    oE..+o.    |
|. o   ..+...     |
| +   .  .oo      |
|. . + *.So.      |
|.  * X *=..      |
|. * o  +.+       |
| . + *  =        |
|  . .  o         |
+----[SHA256]-----+
thor@TS-ZBook-17-G2:~$

if you cannot guarantee that you keep your key safe, you should enter a passphrase.

1. and then copy the public key to the server:

thor@TS-ZBook-17-G2:~$ ssh-copy-id acests3@hpc.hallam.shu.ac.uk
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/thor/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
acests3@hpc.hallam.shu.ac.uk's password:
Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'acests3@hpc.hallam.shu.ac.uk'"
and check to make sure that only the key(s) you wanted were added.

thor@TS-ZBook-17-G2:~$

you will have to enter the password again (one last time). Then you will be able to log into the cluster without a password:

thor@TS-ZBook-17-G2:~$ ssh acests3@hpc.hallam.shu.ac.uk
Last login: Thu Feb 11 17:17:31 2021 from 10.5.52.139
Welcome to Bright release         7.3

                                                    Based on CentOS Linux 7
                                                                ID: #000002

Use the following commands to adjust your environment:

'module avail'            - show available modules
'module add <module>'     - adds a module to your environment for this session
'module initadd <module>' - configure module to be loaded at every login

-------------------------------------------------------------------------------
[acests3@hpc ~]$

No idea why you’d want to use a Windows machine ;-)

… but, nevertheless, there are easy ways to do that, but you’ll need to install am ssh client, like putty or bitwise ssh client. I prefer putty but bitwise is available on AppsAnywhere, so you can use it on SHU machines.

These should be more or less self-explanatory, you need to enter your username, password and the machine name. Then you will get a terminal window with a shell on the cluster.

While I do recommend the shell commands scp and rsync,

For those of you who have embraced the command line more fully, there are, of course, several command line tools to copy files between machines. The most powerful is the command rsync. It has many options so you may want to look at the documentation man rsync. Many of the options won’t make sense at first, and we will only use a few. The simplest is copying a case directory to the cluster:

~/OpenFOAM/acests3-7/run
❯ rsync -avp HT-FSI3 acests3@hpc.hallam.shu.ac.uk:OpenFOAM/acests3-7/run/
sending incremental file list
HT-FSI3/
HT-FSI3/Allclean
HT-FSI3/Allrun
HT-FSI3/HT-FSI3.foam
HT-FSI3/PyFoamHistory
HT-FSI3/0/
HT-FSI3/0/fluid/
HT-FSI3/0/fluid/U
HT-FSI3/0/fluid/p
HT-FSI3/0/fluid/pointMotionU
HT-FSI3/0/solid/
HT-FSI3/0/solid/D
HT-FSI3/0/solid/DD
HT-FSI3/0/solid/pointD
HT-FSI3/0/solid/pointDD
HT-FSI3/constant/
HT-FSI3/constant/fsiProperties
HT-FSI3/constant/physicsProperties
HT-FSI3/constant/fluid/
HT-FSI3/constant/fluid/dynamicMeshDict
HT-FSI3/constant/fluid/fluidProperties
HT-FSI3/constant/fluid/transportProperties
HT-FSI3/constant/fluid/polyMesh/
HT-FSI3/constant/fluid/polyMesh/blockMeshDict
HT-FSI3/constant/fluid/polyMesh/blockMeshDict.first
HT-FSI3/constant/fluid/polyMesh/boundary
HT-FSI3/constant/fluid/polyMesh/faces
HT-FSI3/constant/fluid/polyMesh/neighbour
HT-FSI3/constant/fluid/polyMesh/owner
HT-FSI3/constant/fluid/polyMesh/points
HT-FSI3/constant/solid/
HT-FSI3/constant/solid/dynamicMeshDict
HT-FSI3/constant/solid/g
HT-FSI3/constant/solid/mechanicalProperties
HT-FSI3/constant/solid/solidProperties
HT-FSI3/constant/solid/polyMesh/
HT-FSI3/constant/solid/polyMesh/blockMeshDict
HT-FSI3/constant/solid/polyMesh/boundary
HT-FSI3/constant/solid/polyMesh/faces
HT-FSI3/constant/solid/polyMesh/neighbour
HT-FSI3/constant/solid/polyMesh/owner
HT-FSI3/constant/solid/polyMesh/points
HT-FSI3/history/
HT-FSI3/history/0/
HT-FSI3/history/0/solidPointDisplacement_pointDisp.dat
HT-FSI3/system/
HT-FSI3/system/controlDict
HT-FSI3/system/decomposeParDict
HT-FSI3/system/fluid/
HT-FSI3/system/fluid/decomposeParDict
HT-FSI3/system/fluid/fvSchemes
HT-FSI3/system/fluid/fvSolution
HT-FSI3/system/solid/
HT-FSI3/system/solid/controlDict
HT-FSI3/system/solid/decomposeParDict
HT-FSI3/system/solid/fvSchemes
HT-FSI3/system/solid/fvSolution

sent 1,109,151 bytes  received 996 bytes  444,058.80 bytes/sec
total size is 1,105,750  speedup is 1.00

~/OpenFOAM/acests3-7/run
❯ 

Use an uppercase P instead of the lower case p if you want the [P]rogress to show with more detail (like transfer speeds).

rsync is very economic, since it will only copy files that have changed on the source, so if you only have edited a boundary condition since the last rsync, it will only copy that one file over.

The basic syntax is like cp, so rsync -avp SOURCE TARGET, where both SOURCE and TARGET can be remote locations, which have the syntax USER@MACHINE:DIRECTORY, with DIRECTORY being relative to the user’s home directory, so if you want to copy to the home directory you end with the colon.

IMPORTANT: The / at the end is used when you want to copy into that directory, without it, you copy the source to be the target directory. So the command

rsync -avp HT-FSI3 acests3@hpc.hallam.shu.ac.uk:OpenFOAM/acests3-7/run/FSI3

would create a directory FSI3 on the remote machine, which will have the content of the source HT-FSI3 on the local machine.

TIP: If you have set up an ssh-key, then tab-completion will work for the remote machine’s file system as well!.

A very elegant and transparent way to work with files on the cluster (or on any other unix based computer) is to mount the filesystem directly. Together with an automated mounting system, this makes it as convenient as if the files were stored locally.

This will work on a full Ubuntu install, but should also work on a WSL install (not tested, please let me know if you try this.)

Linux does not have drive letters, but different drives get mounted into the directory tree. E.g. the home directory is often on a separate drive to the system files, but this will not be obvious to the user, because in Unix systems /home simply becomes part of the directory tree, so it doesn’t matter to the user, where that filesystem is located physically.

Using three tools, fuse, autofs, and sshfs, we can configure a Linux system such that it will automatically mount the remote drives, when we access them.

All cases have to be prepared to run as a batch process (without user interaction). Descriptions in this document are mostly for OpenFOAM, refer to the documentation of your code for instructions how to prepare cases for batch processing.

OpenFOAM cases will be ready to run in batch mode without any special measures. The only thing you should do is prepare them for parallel runs, i.e. run decomposePar (this can be done on the cluster, or on the pre-processing machine).

The cluster runs the SLURM workload manager. All cases have to be run through this. No interactive runs are allowed on the cluster.

When you log into the cluster, you will be on a login node. This is the only node that will allow you to use an interactive session directly. For everything else you need to ask SLURM to allocate resources to you. Resources are

• CPU/GPU
• memory
• time

I will only explain the basics here, please refer to the SLURM documentation for more.