Table of Contents
NOTE: This documentation assumes you are familiar with the module system. See this page for details.
Compiling and installing software
We have hundreds of high performance, scientific computing applications installed and available through our modules system. If there's a software app that you'd like to use, is commonly used by a group of labs or your scientific domain, and is not available on the cluster, we encourage you to request that we install it as a module by submitting a help request on the RC Portal. We want the cluster to be as immediately usable as possible for all users, and having all the common apps available is part of that.
If you are looking to use a 'bleeding edge' version of software or something specific only to your lab, you can install apps yourself in your home directory, any other space to which you can write, or a shared lab directory. Writing and installing your own software may be part of your research, too. Below you will find several approaches to compiling and installing personal software on the cluster. If you are having problems during your installation, submit a help request on the RC Portal or stop by Office Hours.
As explained in our documentation on modules, the cluster environment is very basic by default. You may wish to load newer or alternative versions of compilers such as
icc. See this page for more details about compiler choices.
Installing GNU-toolchain-style apps
Many apps are provided in source code form and require conversion to binary form -- a process called
building. Such apps are often packaged to work with the GNU toolchain -- you use the commands
make install to build and install them. The important thing is that you specify a
--prefix in order to write the software to a non-default location, i.e. a location writable by you. The examples below install personal software to a directory
Software source is often distributed as a zipped tarball, such as
APP-X.Y.Z.tar.gz. The first step is to unpack it. This can be done in whatever directory you like, even a temporary space that you later delete, since the installation step below will copy stuff out of this particular directory and into the prefix you specify:
tar xvf APP-X.Y.Z.tar.gz
Next, cd into the unpacked directory:
(Often the newly created directory won't exactly match the tarball, or the files will have been dumped to the current working directory; adjust accordingly.)
Next, configure the software, including telling it to install to somewhere in your home directory:
If you want to install every app in its own directory, instead of having them all share one (e.g. if you want to have different versions of the same software available), use a more specific prefix, like
Some apps don't actually use the full GNU toolchain, and don't have a configure script; they have just a Makefile. In that case, you'll have to look at the Makefile to see if it has an adjustable installation location. If not, you'll have to manually install files.
Next, build the software:
This is the step that actually compiles the software. Once you work through any issues that may come up and the software compiles successfully, install it:
If you get any
Permission denied issues at this point, it's possible the software did not properly use the
--prefix you have it. Likewise, you may get the error
No rule to make target `install'. In fact, many packages only mimic the GNU-toolchain style but actually work slightly differently. In such cases, you'll have to manually copy the build outputs to their destination.
As a final step, you'll want to modify your environment to make the software available for use. See this section below for details.
Adding packages to Python, Ruby, Perl, R, etc.
Please see our dedicated pages on these languages and environments:
Developing your own software
See this doc for information about software development on the cluster.
Making software available
The shell environment relies on environment variables that name where apps and libraries are located. When you install software to non-default locations such as your home directory, you'll need to update these variables. Without doing so, you'll get errors like
command not found, or
error while loading shared libraries.
For example, the environment variable
PATH names directories in which the apps you invoke on the command line reside. Conventionally, when you install software under a
--prefix, the apps are in a sub-directory of that prefix named
bin. Similarly, the variable
LD_LIBRARY_PATH controls what libraries are available for dynamic dependency loading, and those libraries are often put in a directory named
We provide a convenience script to search a directory and print these common environment modifications. For example, if you've installed software using
--prefix=~/sw/, you may find:
$ module load fasrc
$ generate_setup.sh --action echo ~/sw/
You can copy these variable definitions to the end of your
~/.bashrc file, and then the software will be available by default. The script is very aggressive, so you may wish to prune some if its output (e.g.
FPATH is only relevant if your app provides fortran source code). Also, the output is only applicable to sh-family shells (e.g. bash), you'll have to translate for csh-family shells (e.g. tcsh).
Alternatively, you can save this output to a file such as
~/sw/setup.sh, and then every time you want to use this software, you run:
This is necessary if you want to install multiple incompatible apps, or different versions of the same app, and pick and choose between them at will (you'll have to use a different
--prefix for each).
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. Permissions beyond the scope of this license may be available at Attribution.