Build

Contents

This page discusses various ways of building Quinoa, including requirements, optional libraries, defaults, and build system options.

Quick start

See section Build on the main page.

Minimum requirements

  • Fortran compiler
  • C and C++ compilers, the latter implementing the C++17 standard
  • Message Passing Interface library, e.g., OpenMPI

Routinely tested

Compilers:

  • GNU
  • Clang
  • Intel

C++ standard library:

  • libc++
  • libstdc++

MPI:

  • OpenMPI
  • Intel MPI

Operating systems:

  • Debian Linux
  • Alpine Linux
  • RedHat Enterprise Linux
  • MacOS

Two-stage build

Quinoa is built in two stages, of which the first one may be optional:

  1. Build third-party libraries (TPLs)
  2. Build Quinoa

Both stages consist of the following two steps

  • Use cmake to configure the build
  • Perform the build

The first step may be optional if all TPLs are found on the system or are pre-installed and/or configured, e.g., via environment modules.

Build using the defaults

The easiest, but not necessarily the quickest, way to build Quinoa is to use the defaults.

Install the prerequisites on a Debian-based Linux do

   sudo apt-get install cmake gfortran gcc g++ openmpi-bin libopenmpi-dev

Clone the code with the external packages (third party libraries, TPLs):

   git clone --recurse-submodules https://github.com/quinoacomputing/quinoa.git

Build the TPLs in an empty directory:

   cd quinoa
   mkdir external/build
   cd external/build
   cmake -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 ..
   make -sj$(grep -c processor /proc/cpuinfo)

If the above went fine, we build the code, in a new empty build directory:

   cd quinoa
   mkdir build
   cd build
   cmake -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc ../src
   make -sj$(grep -c processor /proc/cpuinfo)

If this also went fine, let's run the unit-, and regression tests in the build directory:

   cd quinoa/build
   ./charmrun +p$(grep -c processor /proc/cpuinfo) Main/unittest -v -q
   ctest -j$(grep -c processor /proc/cpuinfo) -LE extreme

This will run all tests, except those that would take a long time. If all of the above steps finished without error, you have successfully built Quinoa. You can run the executables from build/Main or install them with

    make install

The default install directory is /usr/local, which usually needs root permissions. You can change that by configuring a different path in the CMAKE_INSTALL_PREFIX cmake variable.

Working with a non-recursive clone

The default build process, described in section Build using the defaults, instructs you to do a recursive clone, which clones not only the source code for Quinoa but also all external packages (third-party libraries, TPLs). The TPLs are pulled in to directory external/ as git submodules and contain significantly more data and files than the Quinoa source. If you plan to do code development, it is quicker to work with the repository (changing branches, stashing, etc.) if you work with a non-recursive clone.

To make this work, we need to clone the TPLs separately:

   git clone --recurse-submodules https://github.com/quinoacomputing/quinoa-tpl.git

Then build the TPLs using a similar process as in Build using the defaults :

   cd quinoa-tpl
   mkdir build
   cd build
   cmake -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 ..
   make -sj$(grep -c processor /proc/cpuinfo)

By default, this will install all TPLs in quinoa-tpl/install/<compiler>-<arch>/, e.g., quinoa-tpl/install/gnu-x86_64/. Now we can clone Quinoa without cloning the TPLs:

   git clone https://github.com/quinoacomputing/quinoa.git

To build the code using the TPLs installed above, we tell that location to cmake using the cmake variable TPL_DIR:

   cd quinoa
   mkdir build
   cd build
   cmake -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DTPL_DIR=<path-to-quinoa-tpl>/install/gnu-x86_64/ ../src
   make -sj$(grep -c processor /proc/cpuinfo)

Since the Quinoa clone no longer contains initialized submodules, git operations are faster. This difference can be significant on network-mounted or shared drives.

If the need arises, e.g., to pull in changes from the TPL submodules into such a non-recursive clone, the external/ directory can still be populated under the quinoa clone by initializing the submodules:

   cd quinoa
   git submodule init
   git submodule update --recursive
   cd external
   git submodule init
   git submodule update --recursive
   cd ..
   git submodule status --recursive

As you can see form the output of the commands above, there are two layers of git submodules. You can also just use the above as a single, chained command:

   cd quinoa
   git submodule init && git submodule update --recursive && cd external && git submodule init && git submodule update --recursive && cd .. && git submodule status --recursive

After this under quinoa/external/ the TPLs can be updated, e.g., git pull, etc. When the TPL sources are, again, no longer needed, we de-populate it:

   cd quinoa
   git submodule deinit --all

Build specific executables only

The default build process, described in section Build using the defaults, does a recursive clone of most TPLs and builds all executables. If you only care about a particular tool, e.g., Inciter, you can

  1. Clone only those TPLs that are required for the given tool (instead of cloning them all), e.g.,

     git clone https://github.com/quinoacomputing/quinoa.git && cd quinoa
 git submodule init && git submodule update && cd external
 git -c submodule."src/pugixml".update=none -c submodule."src/rngsse2".update=none -c submodule."src/testu01".update=none -c submodule."src/tut".update=none -c submodule."src/backward-cpp".update=none -c submodule."src/omega_h".update=none -c submodule."src/doxygen".update=none -c submodule."src/m.css".update=none -c submodule."src/root".update=none -c submodule."src/aec".update=none submodule update --init --recursive

  2. Build only those TPLs that are required for the given executable by setting the <executable>_ONLY cmake variable to true, e.g.,

     cd quinoa && mkdir external/build && cd external/build
 cmake -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DINCITER_ONLY=true ..
 make -sj$(grep -c processor /proc/cpuinfo)

    This will build only the TPLs required for Inciter. The rest of the build process is the same as in Build using the defaults : running cmake configuring Quinoa will enable only those build targets for which it finds the required TPLs. Such *_ONLY cmake variables are defined for all executable targets in <quinoa-tpl>/CMakeLists.txt, in the section starting with the comment line

       # Set which libraries are built/searched by default

    Here are the git submodule update commands that pull in only the TPLs required for the given tool for each executable:

Walker

   git -c submodule."src/pugixml".update=none -c submodule."src/rngsse2".update=none -c submodule."src/aec".update=none -c submodule."src/testu01".update=none -c submodule."src/tut".update=none -c submodule."src/root".update=none -c submodule."src/backward-cpp".update=none -c submodule."src/omega_h".update=none -c submodule."src/doxygen".update=none -c submodule."src/m.css".update=none -c submodule."src/sol2".update=none submodule update --init --recursive

Inciter

   git -c submodule."src/pugixml".update=none -c submodule."src/rngsse2".update=none -c submodule."src/testu01".update=none -c submodule."src/tut".update=none -c submodule."src/backward-cpp".update=none -c submodule."src/omega_h".update=none -c submodule."src/doxygen".update=none -c submodule."src/m.css".update=none -c submodule."src/root".update=none -c submodule."src/aec".update=none -c submodule."src/sol2".update=none submodule update --init --recursive

RNGTest

   git -c submodule."src/hdf5".update=none -c submodule."src/netcdf".update=none -c submodule."src/pugixml".update=none -c submodule."src/h5part".update=none -c submodule."src/trilinos".update=none -c submodule."src/tut".update=none -c submodule."src/numdiff".update=none -c submodule."src/root".update=none -c submodule."src/backward-cpp".update=none -c submodule."src/highwayhash".update=none -c submodule."src/omega_h".update=none -c submodule."src/doxygen".update=none -c submodule."src/m.css".update=none -c submodule."src/aec".update=none -c submodule."src/sol2".update=none submodule update --init --recursive

UnitTest

   git -c submodule."src/rngsse2".update=none -c submodule."src/testu01".update=none -c submodule."src/numdiff".update=none -c submodule."src/root".update=none  -c submodule."src/backward-cpp".update=none -c submodule."src/omega_h".update=none -c submodule."src/doxygen".update=none  -c submodule."src/m.css".update=none -c submodule."src/aec".update=none -c submodule."src/sol2".update=none submodule update --init --recursive

MeshConv

   git -c submodule."src/random123".update=none -c submodule."src/rngsse2".update=none -c submodule."src/lapack".update=none -c submodule."src/aec".update=none -c submodule."src/h5part".update=none -c submodule."src/testu01".update=none -c submodule."src/tut".update=none -c submodule."src/root".update=none -c submodule."src/backward-cpp".update=none -c submodule."src/omega_h".update=none -c submodule."src/doxygen".update=none -c submodule."src/m.css".update=none -c submodule."src/sol2".update=none submodule update --init --recursive

FileConv

   git -c submodule."src/random123".update=none -c submodule."src/rngsse2".update=none -c submodule."src/lapack".update=none -c submodule."src/aec".update=none -c submodule."src/h5part".update=none -c submodule."src/testu01".update=none -c submodule."src/tut".update=none -c submodule."src/numdiff".update=none -c submodule."src/backward-cpp".update=none -c submodule."src/omega_h".update=none -c submodule."src/doxygen".update=none -c submodule."src/m.css".update=none -c submodule."src/sol2".update=none submodule update --init --recursive

Using system-wide packages

The default build process, described in section Build using the defaults, attempts to find (and if not found, build) all required and most optional packages that are enabled by default. See the section on Optional third-party libraries for what packages are enabled by default. On some platforms, it is advantageous (and could be faster) to use the packages provided by the system.

On a Debian-based Linux you can pre-install the packages given in

   quinoa/tools/docker/Dockerfile.quinoa-build-debian

On Alpine Linux you can pre-install the packages given in

   quinoa/tools/docker/Dockerfile.quinoa-build-alpine

Build on Mac OS

On Mac OS we do not recommend using the system-wide compilers. Instead, we use spack to install compilers and OpenMPI:

   spack install cmake libtool autoconf automake
   spack install environment-modules cmake gcc openmpi%gcc hdf5%gcc+hl

The above will use install gcc and OpenMPI on top of it. You will have to load the environment modules as, e.g.,

   module load gcc-9.2.0-clang-10.0.0-apple-vcpkolh openmpi-3.1.4-gcc-9.2.0-my2rkcv hdf5-1.10.5-gcc-9.2.0-prdkirn

Then follow Build using the defaults.

Build using Clang or Intel compilers

The default build process on Linux, described in section Build using the defaults, uses the system-wide gnu compiler suite. You can also use the clang or the Intel compilers. We do this by either building OpenMPI using clang or Intel or by using the machine-specific MPI wrappers pointing to the compiler and then use the build process given in section Build using the defaults.

Build using Charm++'s randomized message queues

For serious development work, especially involving asynchronous parallel programming, we also build and test with Charm++ using randomized message queues. This enables more thorough testing of the available space for scheduling messages and thus more heavily testing the asynchronous logic. We do this by configuring the TPL build as

   cmake -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DCHARM_EXTRA_ARGS="--enable-error-checking;--with-prio-type=int;--enable-randomized-msgq;--suffix;randq-debug" <path-to-TPL-CMakeLists.txt>

Build using Charm++'s SMP mode

For better performance computing large problems using many threads, Quinoa can also be used using Charm++'s symmetric multi-processing (SMP) mode. This can be done by configuring the TPL build as

   cmake -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DCHARM_EXTRA_ARGS="smp" <path-to-TPL-CMakeLists.txt>

You can run the unit-, and regression tests in SMP mode as, e.g.,

   ./charmrun +p 34 --bind-to none Main/unittest -v -q +ppn 17
   ctest -j 36 --output-on-failure -LE extreme

The above will use 36 PEs, distributed into two logical (compute) nodes. See also the Charm++ manual on how to run in SMP mode.

Static build

Some machines, e.g., Crays, prefer builds using static libraries. This can be done by passing the cmake argument -DBUILD_SHARED_LIBS=off to both TPL and Quinoa configure steps. See also section Build using the defaults.

Build with Charm++'s Projections enabled

To use Charm++'s performance analysis tool, Projections, you can build Charm++ with

   cmake -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DCHARM_EXTRA_ARGS="--enable-tracing;--enable-commthread-tracing" <path-to-TPL-CMakeLists.txt>

Then configure the Quinoa build with the extra link arguments:

   cmake -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DEXTRA_LINK_ARGS="-tracemode projections" <path-to-Quinoa-src-CMakeLists.txt>

See also the Charm++ Projections manual.

Using ninja instead of make

Similar to other CMake/C++ projects, Quinoa can also be built with ninja instead of make, by configuring the Quinoa build with

   cmake -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -GNinja <path-to-Quinoa-src-CMakeLists.txt>

Optional third-party libraries

The default build process, described in section Build using the defaults, will attempt to build a set of default TPLs. Not all TPLs are required: some are optional, some must be specifically enabled if needed. Then Quinoa can be configured to build whatever it can based on the libraries it can find. See also the section on Build specific executables only.

All libraries, required or optional, are listed on page Libraries.

Build in Debug mode

Similar to other cmake projects, Quinoa can be configured for Debug, Release, or other types of builds cmake supports. As usual, this is done by setting the CMAKE_BUILD_TYPE variable, e.g., -DCMAKE_BUILD_TYPE=Debug. The default is Release.

Disabling packages

Sometimes it may be necessary to disable searching for packages, e.g., due to broken or unsuitable system-wide installs, etc. This can be done by configuring the TPL build by passing to cmake, e.g.,

   -DCMAKE_DISABLE_FIND_PACKAGE_Boost=true

Configuring the C++ standard library

We routinely use and test libc++ and libstdc++. Which one to use can be configured by the cmake variable STDLIBCPP, as

   -DSTDLIBCPP=libc++

or

   -DSTDLIBCPP=libstdc++

Build system features, requirements, and defaults

General requirements for configuring the TPL as well as Quinoa builds:

  • In-source builds are not allowed.
  • The default build type for both TPLs and Quinoa is Release.
  • Shared cmake code is in https://github.com/quinoacomputing/cmake-modules.git, pulled in as a git submodule into the TPL build and as a git subtree into the Quinoa build. See also the page on Modules.
  • The TPL build requires a Fortran, C, and a C++ compiler.
  • The Quinoa build requires only a C and a C++ compiler.
  • If unspecified, the default install directory, configured by the cmake variable CMAKE_INSTALL_PREFIX, for the TPL build is <root>/install/<compiler>-<architecture>/, where <root> is the root of the TPL repository clone. In a recursive clone <root> is quinoa/external. See also the section on Working with a non-recursive clone.
  • If unspecified, the default TPL directory, configured by the cmake variable TPL_DIR, for the Quinoa build is <root>/external/install/<compiler>-architecture>/, where <root> is the root of the Quinoa repository clone. See also the section on Working with a non-recursive clone.

  • For the list of TPLs searched (and if not found, built) by default, see <quinoa-tpl>/CMakeLists.txt, section starting with the comment line

     # Set which libraries are built/searched by default
  • MPI is required and Charm++ is built on top of MPI.
  • The default C++ standard library is libstdc++ for gnu compilers, and libc++ for clang and Intel.

Helpful docker image configurations

You can get more examples on ways we build Quinoa by browsing the following docker images:

Main cmake files

The most important (top-level) files responsible for configuring the build system are

If you have questions not answered here, consult the above.

Feedback

If you would like to improve this page, contact us using the links in the footer below or submit a pull request on github.