Third-party libraries and external tools using git submodules and subtrees
Quinoa uses git submodules as well as git subtrees to pull in third-party libraries (TPLs) and auxiliary tools. This allows easy update to a TPL's (or tool's) new or custom version, record all TPLs' and tools' histories (including branches and tags), and also allows contributing fixes upstream, all within git. There are some differences between git's submodule and subtree functionality each with its own pros and cons. This page explains how to use git submodules and subtrees in Quinoa.
An important distinction between TPLs and tools is that TPLs are libraries whose source we directly use, e.g., via include files and/or (static and dynamic) linking, while from a tool we do not include source code nor link its object files, only build it and use it as a separate (external) executable. To learn more about TPLs and tools, check out the page on Third-party libraries, compilers, and tools.
Currently there is a single git subtree in quinoa and is used to reuse/share cmake code between two repositories
/ github.com/ quinoacomputing/ quinoa under directory cmake/ , and
/ github.com/ quinoacomputing/ quinoa-tpl under directory cmake/ , and
Both of the above git repositories are setup to pull in the cmake code from the https:/
To pull in the latest updates from the cmake repository (from its branch master) into the quinoa repository, do
git fetch cmake git subtree pull --prefix=cmake cmake master --squash
If you have committed changes in the quinoa repository under the cmake subtree directory
cmake, here is how you push the changes upstream to the cmake git repository.
- Verify the changes you are about to push by examining the diff output:
git diff cmake/master master:cmake
- If you are happy with the diff, push the changes upstream (to the master branch of the cmake repository):
git subtree push --prefix=cmake cmake master
/ hpc.uni.lu/ blog/ 2014/ understanding-git-subtree
/ getlevelten.com/ blog/ tom-mccracken/ smarter-drupal-projects-projects-management-git-subtree
/ github.com/ git/ git/ blob/ master/ contrib/ subtree/ git-subtree.txt
/ blogs.atlassian.com/ 2013/ 05/ alternatives-to-git-submodule-git-subtree
/ www.codeproject.com/ Articles/ 562949/ ManagingplusNestedplusLibrariesplusUsingplustheplu
To ease and automate building the third-party libraries that are not always available on a system, we maintain a git repository that consists of a list of git submodules pointing to the libraries. Check the Build section of the main page for a list of libraries that may be installed by your operating system's package manager, which then can accelerate and ease installing the TPLs.
When you do a clone of quinoa
git clone --recurse-submodules https://github.com/quinoacomputing/quinoa.git
all submodules are cloned and initialized. This is probably the simplest way to clone the code with all dependencies, however it is not necessarily the quickest to work with, since git operations will take longer when submodules are present and most developers will only need the submodules once (to build and install the TPLs).
Another way of cloning (and working with) quinoa is to not clone the submodules, but work with the git repositories for quinoa, cmake, and the TPLs separately. Since the cmake code and the TPLs are much less frequently updated compared to the code in quinoa, the TPLs can be cloned, built, and forgotten about. This approach, however, will require the specification of the
-DTPL_DIR=<path-to-installed-tpls> command line argument to quinoa's cmake command. This way, the quinoa repository, which still has the cmake directory as a git subtree, is kept minimal, and thus git operations, e.g., switching branches, stashing, are quicker. This can be especially useful on large HPC machines where the large parallel file systems are shared among many users who may be doing heavy I/O. Another option is to deinitialize the submodules once not needed if they were recursively cloned and initialized.