Skip to content

Using alternative compilers or dependencies

Using an alternative compiler

Intel

GCC is the most supported compiler

The ITSR Applications Team have compiled the majority of applications available via modules on Apocrita using GCC. Whilst other compilers are available on Apocrita as outlined below, you will receive the best support from us should you choose to use GCC as well.

If you don't specify a compiler when installing something from Spack, Spack will use what it considers to be the best compiler available (usually GCC). But what about if we prefer to use one of the other compilers in our personal compilers.yaml file?

First, we can check which compilers are available:

spack compilers output 
$ spack -C ${HOME}/spack-config-templates/0.22.0 compilers
==> Available compilers
-- gcc rocky9-x86_64 --------------------------------------------
gcc@12.2.0  gcc@11.4.1

-- oneapi rocky9-x86_64 -----------------------------------------
oneapi@2024.1.0

So, we can see a list of available compilers as previously defined.

To use a specific compiler, you need to add it to the install command after the % sigil. So, let's spec nano@6.3, but this time we will use the Intel compiler (so add %oneapi@2024.1.0 to the end of the command):

spack spec output 
$ spack -C ${HOME}/spack-config-templates/0.22.0 spec nano@6.3 %oneapi@2024.1.0
Input spec
--------------------------------
 -   nano@6.3%oneapi@2024.1.0

Concretized
--------------------------------
 -   nano@6.3%oneapi@2024.1.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[e]      ^glibc@2.34%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^gmake@4.4.1%gcc@11.4.1~guile build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^gcc-runtime@11.4.1%gcc@11.4.1 build_system=generic arch=linux-rocky9-x86_64_v4
 -       ^intel-oneapi-runtime@2024.1.0%oneapi@2024.1.0 build_system=generic arch=linux-rocky9-x86_64_v4
[^]      ^ncurses@6.5%gcc@11.4.1~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-rocky9-x86_64_v4
[^]          ^pkgconf@2.2.0%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4

So, a similar output to before - most of the dependencies are already upstream and the only things that need to be compiled and installed personally are intel-oneapi-runtime@2024.1.0 and the actual binary for nano itself. So let's go ahead and run an install:

spack install output (click to expand) 
$ spack -C ${HOME}/spack-config-templates/0.22.0 install -j ${NSLOTS} nano@6.3 %oneapi@2024.1.0
[+] /usr (external glibc-2.34-xri56vcyzs7kkvakhoku3fefc46nw25y)
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc-runtime/11.4.1-llid4hw
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gmake/4.4.1-xchit5a
==> Installing intel-oneapi-runtime-2024.1.0-hcmhmi4v7ou7vii6g3kka5r2jwcxd4bj [4/6]
==> No binary for intel-oneapi-runtime-2024.1.0-hcmhmi4v7ou7vii6g3kka5r2jwcxd4bj found: installing from source
==> No patches needed for intel-oneapi-runtime
==> intel-oneapi-runtime: Executing phase: 'install'
==> intel-oneapi-runtime: Successfully installed intel-oneapi-runtime-2024.1.0-hcmhmi4v7ou7vii6g3kka5r2jwcxd4bj
  Stage: 0.00s.  Install: 0.81s.  Post-install: 0.58s.  Total: 1.49s
[+] /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/oneapi-2024.1.0/intel-oneapi-runtime/2024.1.0-hcmhmi4
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/ncurses/6.5-4n2uzha
==> Installing nano-6.3-kwyxa3xqqzarnz7zyk3wu7yipvkfdedy [6/6]
==> No binary for nano-6.3-kwyxa3xqqzarnz7zyk3wu7yipvkfdedy found: installing from source
==> Using cached archive: /data/scratch/abc123/spack/cache/_source-cache/archive/eb/eb532da4985672730b500f685dbaab885a466d08fbbf7415832b95805e6f8687.tar.xz
==> No patches needed for nano
==> nano: Executing phase: 'autoreconf'
==> nano: Executing phase: 'configure'
==> nano: Executing phase: 'build'
==> nano: Executing phase: 'install'
==> nano: Successfully installed nano-6.3-kwyxa3xqqzarnz7zyk3wu7yipvkfdedy
  Stage: 0.19s.  Autoreconf: 0.00s.  Configure: 32.96s.  Build: 1.38s.  Install: 0.61s.  Post-install: 0.28s.  Total: 35.71s
[+] /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/oneapi-2024.1.0/nano/6.3-kwyxa3x

So, once again:

  • The existing available upstream dependencies have been used from the central location as opposed to reinstalled
  • Spack has noticed that intel-oneapi-runtime@2024.1.0 and nano@6.3 compiled against Intel are missing and has thus pulled down a source code tarball to the source_cache directory defined in config.yaml for intel-oneapi-runtime and re-used the existing cached tarball for nano@6.3 and then manually compiled them and installed them to the location defined as install_tree: root: in config.yaml.

Let's now return to our spack find command:

spack find output 
$ spack -C ${HOME}/spack-config-templates/0.22.0 find -x -p nano
-- linux-rocky9-x86_64_v4 / gcc@12.2.0 --------------------------
nano@6.3  /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/nano/6.3-m7n4nhp
nano@7.2  /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/nano/7.2-4ew6nde

-- linux-rocky9-x86_64_v4 / oneapi@2024.1.0 ---------------------
nano@6.3  /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/oneapi-2024.1.0/nano/6.3-kwyxa3x
==> 3 installed packages

And we now have a module file available for the version compiled against Intel (note that the module name lists oneapi-2024.1.0 and not gcc-12.2.0):

module avail output 
$ module avail -l nano
- Package/Alias -----------------------.- Versions --------
/data/scratch/abc123/spack/privatemodules/linux-rocky9-x86_64_v4:
nano/6.3-gcc-12.2.0
nano/6.3-oneapi-2024.1.0

/share/apps/rocky9/environmentmodules/apocrita-modules/spack:
nano/7.2-gcc-12.2.0

$ module load nano/6.3-oneapi-2024.1.0
$ which nano
/data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/oneapi-2024.1.0/nano/6.3-kwyxa3x/bin/nano
$ nano --version
 GNU nano, version 6.3
 (C) 1999-2011, 2013-2022 Free Software Foundation, Inc.
 (C) 2014-2022 the contributors to nano
 Compiled options: --disable-libmagic --enable-utf8

Using an alternative dependency

Intel MPI

Open MPI is the most supported MPI implementation

The ITSR Applications Team have compiled almost all centrally installed applications available on Apocrita against bespoke installations of Open MPI integrating advanced features such as Infiniband, and you will receive the best support from us if you also use Open MPI (defined as an external package as advised in the above packages.yaml documentation). Using other MPI versions as detailed below is possible, but not as well-supported.

You'll have noticed that the above examples for Gromacs and LAMMPS all use the ^openmpi dependency. But it is also possible to compile against Intel MPI instead.

Let's take Gromacs as a simple example. The default centrally installed CPU-only variant (the gromacs module) has been compiled against Open MPI, using the following variant:

gromacs ^openmpi ^fftw+openmp

Let's run a spec command using our personal config scope, but this time swapping out ^openmpi with ^intel-oneapi-mpi:

Use the intel-oneapi-* Spack packages

Spack uses the intel-oneapi-* naming scheme for all Intel packages such as intel-oneapi-compilers, intel-oneapi-mpi etc. Search the Spack Packages site for more details.

spack spec output (click to expand) 
$ spack -C ${HOME}/spack-config-templates/0.22.0 spec gromacs ^intel-oneapi-mpi ^fftw+openmp
Input spec
--------------------------------
 -   gromacs
 -       ^fftw+openmp
 -       ^intel-oneapi-mpi

Concretized
--------------------------------
 -   gromacs@2024.1%gcc@12.2.0~cp2k~cuda~cycle_subcounters~double+gmxapi+hwloc~intel_provided_gcc~ipo~mdrun_only+mpi+nblib~nosuffix~opencl+openmp~relaxed_double_precision+shared~sycl build_system=cmake build_type=Release generator=make openmp_max_threads=none arch=linux-rocky9-x86_64_v4
[^]      ^cmake@3.27.9%gcc@12.2.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky9-x86_64_v4
[^]          ^curl@8.7.1%gcc@12.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rocky9-x86_64_v4
[^]              ^nghttp2@1.57.0%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                  ^diffutils@3.10%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^openssl@3.3.0%gcc@12.2.0~docs+shared build_system=generic certs=mozilla arch=linux-rocky9-x86_64_v4
[^]                  ^ca-certificates-mozilla@2023-05-30%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^ncurses@6.5%gcc@11.4.1~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-rocky9-x86_64_v4
[^]          ^zlib-ng@2.1.6%gcc@11.4.1+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky9-x86_64_v4
 -       ^fftw@3.3.10%gcc@12.2.0+mpi+openmp~pfft_patches+shared build_system=autotools precision=double,float arch=linux-rocky9-x86_64_v4
[^]      ^gcc-runtime@12.2.0%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[e]      ^glibc@2.34%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^gmake@4.4.1%gcc@11.4.1~guile build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^gcc-runtime@11.4.1%gcc@11.4.1 build_system=generic arch=linux-rocky9-x86_64_v4
[^]      ^hwloc@2.9.1%gcc@12.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-rocky9-x86_64_v4
[^]          ^libpciaccess@0.17%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^libtool@2.4.7%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                  ^findutils@4.9.0%gcc@11.4.1 build_system=autotools patches=440b954 arch=linux-rocky9-x86_64_v4
[^]                  ^m4@1.4.19%gcc@11.4.1+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-rocky9-x86_64_v4
[^]                      ^libsigsegv@2.14%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^util-macros@1.19.3%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]          ^libxml2@2.10.3%gcc@11.4.1+pic~python+shared build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^xz@5.4.6%gcc@11.4.1~pic build_system=autotools libs=shared,static arch=linux-rocky9-x86_64_v4
[^]      ^intel-oneapi-mpi@2021.12.1%gcc@12.2.0+envmods~external-libfabric~generic-names~ilp64 build_system=generic arch=linux-rocky9-x86_64_v4
[^]      ^openblas@0.3.26%gcc@12.2.0~bignuma~consistent_fpcsr+dynamic_dispatch+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-rocky9-x86_64_v4
[^]          ^perl@5.38.0%gcc@11.4.1+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-rocky9-x86_64_v4
[^]              ^berkeley-db@18.1.40%gcc@11.4.1+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rocky9-x86_64_v4
[^]              ^bzip2@1.0.8%gcc@11.4.1~debug~pic+shared build_system=generic arch=linux-rocky9-x86_64_v4
[^]              ^gdbm@1.23%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                  ^readline@8.2%gcc@11.4.1 build_system=autotools patches=bbf97f1 arch=linux-rocky9-x86_64_v4
[^]      ^pkgconf@2.2.0%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4

So, a new personal variant of ^fftw+openmp needs to be compiled against Intel MPI (which is already in upstream), as well as our new variant of Gromacs. If you follow through and install that personal variant, then, assuming you broadly follow the example modules.yaml template, the module should be named something like:

gromacs-mpi/2024.1-intel-oneapi-mpi-2021.12.1-gcc-12.2.0

And when loading the above module and running mpirun -np $NSLOTS gmx_mpi --version you should see something like:

MPI library:         MPI
MPI library version: Intel(R) MPI Library 2021.12 for Linux* OS

Using an alternative compiler and dependency at the same time

The above Gromacs example has mixed GCC and Intel MPI, but many users might prefer to compile an "all Intel" version, i.e. that compiles against Intel and Intel MPI. Let's spec that:

spack spec output (click to expand) 
$ spack -C ${HOME}/spack-config-templates/0.22.0 spec gromacs ^intel-oneapi-mpi ^fftw+openmp %oneapi@2024.1.0
Input spec
--------------------------------
 -   gromacs
 -       ^fftw%oneapi@2024.1.0+openmp
 -       ^intel-oneapi-mpi

Concretized
--------------------------------
 -   gromacs@2024.1%oneapi@2024.1.0~cp2k~cuda~cycle_subcounters~double+gmxapi+hwloc~intel_provided_gcc~ipo~mdrun_only+mpi+nblib~nosuffix~opencl+openmp~relaxed_double_precision+shared~sycl build_system=cmake build_type=Release generator=make openmp_max_threads=none arch=linux-rocky9-x86_64_v4
[^]      ^cmake@3.27.9%gcc@12.2.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky9-x86_64_v4
[^]          ^curl@8.7.1%gcc@12.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rocky9-x86_64_v4
[^]              ^nghttp2@1.57.0%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^openssl@3.3.0%gcc@12.2.0~docs+shared build_system=generic certs=mozilla arch=linux-rocky9-x86_64_v4
[^]                  ^ca-certificates-mozilla@2023-05-30%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^gcc-runtime@12.2.0%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^ncurses@6.5%gcc@11.4.1~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-rocky9-x86_64_v4
[^]          ^zlib-ng@2.1.6%gcc@11.4.1+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky9-x86_64_v4
 -       ^fftw@3.3.10%oneapi@2024.1.0+mpi+openmp~pfft_patches+shared build_system=autotools precision=double,float arch=linux-rocky9-x86_64_v4
[^]      ^gcc@12.2.0%gcc@11.4.1~binutils+bootstrap~graphite~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages='c,c++,fortran' arch=linux-rocky9-x86_64_v4
[^]          ^diffutils@3.10%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]          ^gawk@5.3.0%gcc@11.4.1~nls build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^libsigsegv@2.14%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^readline@8.2%gcc@11.4.1 build_system=autotools patches=bbf97f1 arch=linux-rocky9-x86_64_v4
[^]          ^gcc-runtime@11.4.1%gcc@11.4.1 build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^gmp@6.2.1%gcc@11.4.1+cxx build_system=autotools libs=shared,static patches=69ad2e2 arch=linux-rocky9-x86_64_v4
[^]              ^autoconf@2.72%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^automake@1.16.5%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^m4@1.4.19%gcc@11.4.1+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-rocky9-x86_64_v4
[^]          ^libtool@2.4.7%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^findutils@4.9.0%gcc@11.4.1 build_system=autotools patches=440b954 arch=linux-rocky9-x86_64_v4
[^]          ^mpc@1.3.1%gcc@11.4.1 build_system=autotools libs=shared,static arch=linux-rocky9-x86_64_v4
[^]          ^mpfr@4.2.1%gcc@11.4.1 build_system=autotools libs=shared,static arch=linux-rocky9-x86_64_v4
[^]              ^autoconf-archive@2023.02.20%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]          ^perl@5.38.0%gcc@11.4.1+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-rocky9-x86_64_v4
[^]              ^berkeley-db@18.1.40%gcc@11.4.1+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rocky9-x86_64_v4
[^]              ^bzip2@1.0.8%gcc@11.4.1~debug~pic+shared build_system=generic arch=linux-rocky9-x86_64_v4
[^]              ^gdbm@1.23%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]          ^texinfo@7.0.3%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^gettext@0.22.5%gcc@11.4.1+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                  ^tar@1.34%gcc@11.4.1 build_system=autotools zip=pigz arch=linux-rocky9-x86_64_v4
[^]                      ^pigz@2.8%gcc@11.4.1 build_system=makefile arch=linux-rocky9-x86_64_v4
[^]          ^zstd@1.5.6%gcc@11.4.1+programs build_system=makefile compression=none libs=shared,static arch=linux-rocky9-x86_64_v4
[e]      ^glibc@2.34%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^gmake@4.4.1%gcc@11.4.1~guile build_system=generic arch=linux-rocky9-x86_64_v4
[^]      ^hwloc@2.9.1%gcc@12.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-rocky9-x86_64_v4
[^]          ^libpciaccess@0.17%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^util-macros@1.19.3%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]          ^libxml2@2.10.3%gcc@11.4.1+pic~python+shared build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^xz@5.4.6%gcc@11.4.1~pic build_system=autotools libs=shared,static arch=linux-rocky9-x86_64_v4
[^]      ^intel-oneapi-mpi@2021.12.1%gcc@12.2.0+envmods~external-libfabric~generic-names~ilp64 build_system=generic arch=linux-rocky9-x86_64_v4
 -       ^intel-oneapi-runtime@2024.1.0%oneapi@2024.1.0 build_system=generic arch=linux-rocky9-x86_64_v4
[^]      ^openblas@0.3.26%gcc@12.2.0~bignuma~consistent_fpcsr+dynamic_dispatch+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-rocky9-x86_64_v4
[^]      ^pkgconf@2.2.0%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4

This time, we need another personal variant of fftw compiled against Intel and Intel MPI, as well as the intel-oneapi-runtime, to be able to compile Gromacs against Intel and Intel MPI. An install of the above spec would output a module named something like:

gromacs-mpi/2024.1-intel-oneapi-mpi-2021.12.1-oneapi-2024.1.0

And when loading the above module and running mpirun -np $NSLOTS gmx_mpi --version you should see something like:

MPI library:         MPI
MPI library version: Intel(R) MPI Library 2021.12 for Linux* OS
...
C compiler:          /share/apps/rocky9/general/apps/spack/0.22.0/lib/spack/env/oneapi/icx IntelLLVM 2024.1.0
C compiler flags:    -xCORE-AVX512 -qopt-zmm-usage=high -Wno-missing-field-initializers -O3 -DNDEBUG
C++ compiler:        /share/apps/rocky9/general/apps/spack/0.22.0/lib/spack/env/oneapi/icpx IntelLLVM 2024.1.0