Third-party libraries and external tools using git submodules and subtrees

Quinoa uses git submodules as well as git subtrees to pull in third-party libraries (TPLs) and auxiliary tools. This allows easy update to a TPL's (or tool's) new or custom version, record all TPLs' and tools' histories (including branches and tags), and also allows contributing fixes upstream, all within git. There are some differences between git's submodule and subtree functionality each with its own pros and cons. This page explains how to use git submodules and subtrees in Quinoa.

TPLs and external tools

An important distinction between TPLs and tools is that TPLs are libraries whose source we directly use, e.g., via include files and/or (static and dynamic) linking, while from a tool we do not include source code nor link its object files, only build it and use it as a separate (external) executable. To learn more about TPLs and tools, check out the page on Third-party libraries, compilers, and tools.

Git subtree for sharing cmake code

Currently there is a single git subtree in quinoa and is used to reuse/share cmake code between two repositories

Both of the above git repositories are setup to pull in the cmake code from the https://github.com/quinoacomputing/cmake-modules repository, in quinoa as a subtree, in the TPL repository as a submodule.

Pull changes from the cmake repository

To pull in the latest updates from the cmake repository (from its branch master) into the quinoa repository, do

git fetch cmake
git subtree pull --prefix=cmake cmake master --squash

Push changes to the cmake repository

If you have committed changes in the quinoa repository under the cmake subtree directory cmake, here is how you push the changes upstream to the cmake git repository.

  • Verify the changes you are about to push by examining the diff output:
git diff cmake/master master:cmake
  • If you are happy with the diff, push the changes upstream (to the master branch of the cmake repository):
git subtree push --prefix=cmake cmake master

Useful links on git subtree

Git submodule for incorporating TPLs

To ease and automate building the third-party libraries that are not always available on a system, we maintain a git repository that consists of a list of git submodules pointing to the libraries. Check the Build section of the main page for a list of libraries that may be installed by your operating system's package manager, which then can accelerate and ease installing the TPLs.

When you do a clone of quinoa

git clone --recurse-submodules https://github.com/quinoacomputing/quinoa.git

all submodules are cloned and initialized. This is probably the simplest way to clone the code with all dependencies, however it is not necessarily the quickest to work with, since git operations will take longer when submodules are present and most developers will only need the submodules once (to build and install the TPLs).

Another way of cloning (and working with) quinoa is to not clone the submodules, but work with the git repositories for quinoa, cmake, and the TPLs separately. Since the cmake code and the TPLs are much less frequently updated compared to the code in quinoa, the TPLs can be cloned, built, and forgotten about. This approach, however, will require the specification of the -DTPL_DIR=<path-to-installed-tpls> command line argument to quinoa's cmake command. This way, the quinoa repository, which still has the cmake directory as a git subtree, is kept minimal, and thus git operations, e.g., switching branches, stashing, are quicker. This can be especially useful on large HPC machines where the large parallel file systems are shared among many users who may be doing heavy I/O. Another option is to deinitialize the submodules once not needed if they were recursively cloned and initialized.