Installing a personal version of LAMMPS¶

Another very popular application on Apocrita is LAMMPS, but as with Gromacs, we only provide modules for fairly simple compilation variants. Some more advanced users may want to compile additional personal versions with more granular compilation options.

Searching for available variants¶

So, as before, let's see what is on offer by default:

spack find output

$ spack -C ${HOME}/spack-config-templates/0.23.1 find -x -p -v lammps
-- linux-rocky9-x86_64_v4 / gcc@12.2.0 --------------------------
lammps@20230802.3~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell~cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim~kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release fftw_precision=double generator=make gpu_precision=mixed lammps_sizes=smallbig                     /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-b27kssh
lammps@20230802.3~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell+cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim~kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release cuda_arch=70,80,90 fftw_precision=double generator=make gpu_precision=mixed lammps_sizes=smallbig  /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-zqtg4sr
==> 2 installed packages

As with Gromacs, the output of spack find includes an incredibly long list of variants, either marked with ~ (false) or + (true).

Spack Packages website¶

Package information is for the develop branch

The Spack Packages website lists information such as versions available, variants etc. from the develop branch of Spack. Some versions and variants may not be available in the specific Spack release you are using.

The Spack Packages page for LAMMPS can be found here:

https://packages.spack.io/package.html?name=lammps

There is a extremely long list of variants (too long to even screenshot here in full), and the defaults (true) or (false) are marked in brackets after each item in the list. For the versions centrally compiled, we have largely stuck to the defaults.

The `spack info` command¶

spack info output (click to expand)

$ spack -C ${HOME}/spack-config-templates/0.23.1 info lammps
CMakePackage:   lammps

Description:
LAMMPS stands for Large-scale Atomic/Molecular Massively Parallel
Simulator.

Homepage: https://www.lammps.org/

Preferred version:  
20240829.1    https://github.com/lammps/lammps/archive/stable_29Aug2024_update1.tar.gz

Safe versions:  
develop       [git] https://github.com/lammps/lammps.git on branch develop
(etc.)

Deprecated versions:  
20240829      https://github.com/lammps/lammps/archive/stable_29Aug2024.tar.gz
(etc.)

Variants:
asphere [false]                 false, true
    Activate the asphere package
body [false]                    false, true
    Activate the body package
build_system [cmake]            cmake
    Build systems supported by the package
class2 [false]                  false, true
    Activate the class2 package
colloid [false]                 false, true
    Activate the colloid package
compress [false]                false, true
    Activate the compress package
coreshell [false]               false, true
    Activate the coreshell package
cuda [false]                    false, true
    Build with CUDA
cuda_mps [false]                false, true
    (CUDA only) Enable tweaks for running with Nvidia CUDA Multi-process services daemon
dipole [false]                  false, true
    Activate the dipole package
ffmpeg [false]                  false, true
    Build with ffmpeg support
granular [false]                false, true
    Activate the granular package
jpeg [false]                    false, true
    Build with jpeg support
kim [false]                     false, true
    Activate the kim package
kokkos [false]                  false, true
    Activate the kokkos package
kspace [true]                   false, true
    Activate the kspace package
lammps_sizes [smallbig]         bigbig, smallbig, smallsmall
    LAMMPS integer sizes (smallsmall: all 32-bit, smallbig:64-bit #atoms #timesteps, bigbig: also 64-bit imageint, 64-bit atom ids)
lib [true]                      false, true
    Build the liblammps in addition to the executable
manybody [true]                 false, true
    Activate the manybody package
mc [false]                      false, true
    Activate the mc package
misc [false]                    false, true
    Activate the misc package
molecule [true]                 false, true
    Activate the molecule package
mpi [true]                      false, true
    Build with mpi
opencl [false]                  false, true
    Build with OpenCL
openmp [true]                   false, true
    Build with OpenMP
openmp-package [false]          false, true
    Activate the openmp package
opt [false]                     false, true
    Activate the opt package
peri [false]                    false, true
    Activate the peri package
png [false]                     false, true
    Build with png support
poems [false]                   false, true
    Activate the poems package
python [false]                  false, true
    Activate the python package
qeq [false]                     false, true
    Activate the qeq package
replica [false]                 false, true
    Activate the replica package
rigid [true]                    false, true
    Activate the rigid package
rocm [false]                    false, true
    Enable ROCm support
shock [false]                   false, true
    Activate the shock package
srd [false]                     false, true
    Activate the srd package
tools [false]                   false, true
    Build LAMMPS tools (msi2lmp, binary2txt, chain)
voronoi [false]                 false, true
    Activate the voronoi package

when +rocm
  amdgpu_target [none]          none, gfx1010, gfx1011, gfx1012, gfx1013, gfx1030, gfx1031, gfx1032, gfx1033, gfx1034, gfx1035, gfx1036, gfx1100, gfx1101, gfx1102, gfx1103, gfx701, gfx801, gfx802, gfx803,
                gfx900, gfx900:xnack-, gfx902, gfx904, gfx906, gfx906:xnack-, gfx908, gfx908:xnack-, gfx909, gfx90a, gfx90a:xnack+, gfx90a:xnack-, gfx90c, gfx940, gfx941, gfx942
      AMD GPU architecture

when +cuda
  cuda_arch [none]              none, 10, 11, 12, 13, 20, 21, 30, 32, 35, 37, 50, 52, 53, 60, 61, 62, 70, 72, 75, 80, 86, 87, 89, 90, 90a
      CUDA architecture

when build_system=cmake
  build_type [Release]          Debug, MinSizeRel, RelWithDebInfo, Release
      CMake build type
  generator [make]              none
      the build system generator to use

when build_system=cmake ^cmake@3.9:
  ipo [false]                   false, true
      CMake interprocedural optimization

when @20210702:
  adios [false]                 false, true
      Activate the adios package
  atc [false]                   false, true
      Activate the atc package
  awpmd [false]                 false, true
      Activate the awpmd package
  bocs [false]                  false, true
      Activate the bocs package
  brownian [false]              false, true
      Activate the brownian package
  cg-dna [false]                false, true
      Activate the cg-dna package
  colvars [false]               false, true
      Activate the colvars package
  dielectric [false]            false, true
      Activate the dielectric package
  diffraction [false]           false, true
      Activate the diffraction package
  dpd-basic [false]             false, true
      Activate the dpd-basic package
  dpd-meso [false]              false, true
      Activate the dpd-meso package
  dpd-react [false]             false, true
      Activate the dpd-react package
  dpd-smooth [false]            false, true
      Activate the dpd-smooth package
  drude [false]                 false, true
      Activate the drude package
  eff [false]                   false, true
      Activate the eff package
  fep [false]                   false, true
      Activate the fep package
  h5md [false]                  false, true
      Activate the h5md package
  intel [false]                 false, true
      Activate the intel package
  latboltz [false]              false, true
      Activate the latboltz package
  machdyn [false]               false, true
      Activate the machdyn package
  manifold [false]              false, true
      Activate the manifold package
  mesont [false]                false, true
      Activate the mesont package
  mgpt [false]                  false, true
      Activate the mgpt package
  ml-hdnnp [false]              false, true
      Activate the ml-hdnnp package
  ml-iap [false]                false, true
      Activate the ml-iap package
  ml-rann [false]               false, true
      Activate the ml-rann package
  ml-snap [false]               false, true
      Activate the ml-snap package
  mofff [false]                 false, true
      Activate the mofff package
  molfile [false]               false, true
      Activate the molfile package
  netcdf [false]                false, true
      Activate the netcdf package
  phonon [false]                false, true
      Activate the phonon package
  plumed [false]                false, true
      Activate the plumed package
  ptm [false]                   false, true
      Activate the ptm package
  qtb [false]                   false, true
      Activate the qtb package
  reaction [false]              false, true
      Activate the reaction package
  reaxff [false]                false, true
      Activate the reaxff package
  smtbq [false]                 false, true
      Activate the smtbq package
  sph [false]                   false, true
      Activate the sph package
  tally [false]                 false, true
      Activate the tally package
  uef [false]                   false, true
      Activate the uef package
  vtk [false]                   false, true
      Activate the vtk package
  yaff [false]                  false, true
      Activate the yaff package

when @20220803:
  amoeba [false]                false, true
      Activate the amoeba package
  cg-spica [false]              false, true
      Activate the cg-spica package

when @20220504:
  bpm [false]                   false, true
      Activate the bpm package
  electrode [false]             false, true
      Activate the electrode package

when @20210702:20220623
  cg-sdk [false]                false, true
      Activate the cg-sdk package

when @20210728:
  extra-compute [false]         false, true
      Activate the extra-compute package
  extra-dump [false]            false, true
      Activate the extra-dump package
  extra-fix [false]             false, true
      Activate the extra-fix package
  extra-molecule [false]        false, true
      Activate the extra-molecule package
  extra-pair [false]            false, true
      Activate the extra-pair package
  interlayer [false]            false, true
      Activate the interlayer package
  orient [false]                false, true
      Activate the orient package

when @20170922:20230328
  latte [false]                 false, true
      Activate the latte package

when @20230208:
  lepton [false]                false, true
      Activate the lepton package

when @:20181212,20210702:
  meam [false]                  false, true
      Activate the meam package

when @20221222:
  ml-pod [false]                false, true
      Activate the ml-pod package

when @20240627:
  ml-uf3 [false]                false, true
      Activate the ml-uf3 package

when @20200630:20210527
  mliap [false]                 false, true
      Activate the mliap package

when @:20230802.1
  exceptions [false]            false, true
      Build with lammps exceptions
  mpiio [false]                 false, true
      Activate the mpiio package

when @20210408:
  plugin [false]                false, true
      Activate the plugin package

when @:20181212
  reax [false]                  false, true
      Activate the reax package

when @20240829:
  curl [false]                  false, true
      Build with curl support
  rheo [false]                  false, true
      Activate the rheo package

when @:20210527
  snap [false]                  false, true
      Activate the snap package
  user-atc [false]              false, true
      Activate the user-atc package
  user-awpmd [false]            false, true
      Activate the user-awpmd package
  user-colvars [false]          false, true
      Activate the user-colvars package
  user-diffraction [false]      false, true
      Activate the user-diffraction package
  user-dpd [false]              false, true
      Activate the user-dpd package
  user-drude [false]            false, true
      Activate the user-drude package
  user-eff [false]              false, true
      Activate the user-eff package
  user-fep [false]              false, true
      Activate the user-fep package
  user-h5md [false]             false, true
      Activate the user-h5md package
  user-intel [false]            false, true
      Activate the user-intel package
  user-lb [false]               false, true
      Activate the user-lb package
  user-manifold [false]         false, true
      Activate the user-manifold package
  user-mgpt [false]             false, true
      Activate the user-mgpt package
  user-molfile [false]          false, true
      Activate the user-molfile package
  user-omp [false]              false, true
      Activate the user-omp package
  user-phonon [false]           false, true
      Activate the user-phonon package
  user-qtb [false]              false, true
      Activate the user-qtb package
  user-reaxc [false]            false, true
      Activate the user-reaxc package
  user-smd [false]              false, true
      Activate the user-smd package
  user-smtbq [false]            false, true
      Activate the user-smtbq package
  user-sph [false]              false, true
      Activate the user-sph package
  user-tally [false]            false, true
      Activate the user-tally package

when @20180629:
  spin [false]                  false, true
      Activate the spin package

when @20190228:20210527
  user-adios [false]            false, true
      Activate the user-adios package

when @20180511:20210527
  user-bocs [false]             false, true
      Activate the user-bocs package

when @20210514:20210527
  user-brownian [false]         false, true
      Activate the user-brownian package

when @20170504:20210527
  user-cgsdk [false]            false, true
      Activate the user-cgsdk package
  user-netcdf [false]           false, true
      Activate the user-netcdf package

when @20210527
  user-hdnnp [false]            false, true
      Activate the user-hdnnp package
  user-rann [false]             false, true
      Activate the user-rann package
  user-vtk [false]              false, true
      Activate the user-vtk package

when @20170706:20210527
  user-meamc [false]            false, true
      Activate the user-meamc package

when @20200319:20210527
  user-mesodpd [false]          false, true
      Activate the user-mesodpd package
  user-reaction [false]         false, true
      Activate the user-reaction package

when @20200615:20210527
  user-mesont [false]           false, true
      Activate the user-mesont package

when @:20210702
  user-misc [false]             false, true
      Activate the user-misc package

when @20180205:20210527
  user-mofff [false]            false, true
      Activate the user-mofff package

when @20181109:20210527
  user-plumed [false]           false, true
      Activate the user-plumed package
  user-sdpd [false]             false, true
      Activate the user-sdpd package

when @20181010:20210527
  user-ptm [false]              false, true
      Activate the user-ptm package

when @20171023:20210527
  user-uef [false]              false, true
      Activate the user-uef package

when @20190201:20210527
  user-yaff [false]             false, true
      Activate the user-yaff package

when +kspace
  fft [fftw3]                   fftw3, kiss, mkl
      FFT library for KSPACE package
  fftw_precision [double]       double, single
      Select FFTW precision (used by Kspace)

when @20240207:+kspace
  heffte [false]                false, true
      Use heffte as distubuted FFT engine

when @20240417:+kokkos+kspace
  fft_kokkos [fftw3]            cufft, fftw3, hipfft, kiss, mkl
      FFT library for Kokkos-enabled KSPACE package

when ~kokkos
  gpu_precision [mixed]         double, mixed, single
      Select GPU precision (used by GPU package)

Build Dependencies:
adios2  cmake  curl   ffmpeg    gmake  hdf5    hip     hipfft        jpeg     kokkos  latte   llvm-amdgpu  mpi   netcdf-c  opencl  py-build   py-mpi4py  py-pip         py-wheel  python-venv         voropp
blas    cuda   eigen  fftw-api  gsl    heffte  hipcub  hsa-rocr-dev  kim-api  lapack  libpng  mkl          n2p2  ninja     plumed  py-cython  py-numpy   py-setuptools  python    rocm-openmp-extras  vtk

Link Dependencies:
adios2  cuda  eigen   fftw-api  hdf5    hip     hipfft        jpeg     kokkos  latte   llvm-amdgpu  mpi   netcdf-c  plumed  voropp
blas    curl  ffmpeg  gsl       heffte  hipcub  hsa-rocr-dev  kim-api  lapack  libpng  mkl          n2p2  opencl    python  vtk

Run Dependencies:
py-mpi4py  py-numpy  py-setuptools  python  python-venv

Licenses:
GPL-2.0-only

Installing specific variants¶

LAMMPS has an enormous number of different variants and accelerators. If you aren't very familiar with LAMMPS in general, it's highly recommended to read this HPC Carpentry tutorial which goes into a great amount of detail about the most popular variants and accelerators.

We can't possibly cover all possible variants, so we'll focus on just two below:

Activate the KOKKOS LAMMPS package¶

Add a suffix for your private KOKKOS module

By default, the fact that the KOKKOS LAMMPS package has been added won't be reflected in your private module name. We recommend adding:

lammps:
  suffixes:
    +kokkos: kokkos

to the tcl: section of your personal modules.yaml configuration file to help to differentiate. It is already added to the template modules.yaml file in the spack-config-template repository.

Let's look at installing a variant of LAMMPS that activates the KOKKOS package. The default for the kokkos variant is (false) (which appears in the output of Spack commands as ~).

Always run a spec command before install!

You should always check what is about to be installed using the spec command as detailed below and then move onto the install command.

Let's see if we can spec a personal installation of LAMMPS adding in the KOKKOS package:

spack spec output (click to expand)

$ spack -C ${HOME}/spack-config-templates/0.23.1 spec lammps +kokkos ^openmpi
 -   lammps@20240829.1%gcc@14.2.0~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell~cuda~cuda_mps~curl~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~heffte~intel~interlayer~ipo~jpeg~kim+kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~ml-uf3~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica~rheo+rigid~rocm~shock~smtbq~sph~spin~srd~tally~tools~uef~voronoi~vtk~yaff build_system=cmake build_type=Release fft=fftw3 fft_kokkos=fftw3 fftw_precision=double generator=make lammps_sizes=smallbig arch=linux-rocky9-x86_64_v4
[^]      ^cmake@3.27.9%gcc@12.2.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky9-x86_64_v4
[^]          ^curl@8.7.1%gcc@12.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rocky9-x86_64_v4
[^]              ^nghttp2@1.57.0%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                  ^diffutils@3.10%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^openssl@3.3.0%gcc@12.2.0~docs+shared build_system=generic certs=mozilla arch=linux-rocky9-x86_64_v4
[^]                  ^ca-certificates-mozilla@2023-05-30%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[^]              ^perl@5.38.0%gcc@11.4.1+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-rocky9-x86_64_v4
[^]                  ^berkeley-db@18.1.40%gcc@11.4.1+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rocky9-x86_64_v4
[^]                  ^bzip2@1.0.8%gcc@11.4.1~debug~pic+shared build_system=generic arch=linux-rocky9-x86_64_v4
[^]                  ^gdbm@1.23%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                      ^readline@8.2%gcc@11.4.1 build_system=autotools patches=bbf97f1 arch=linux-rocky9-x86_64_v4
[^]              ^pkgconf@2.2.0%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]          ^gcc-runtime@12.2.0%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[e]          ^glibc@2.34%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]          ^ncurses@6.5%gcc@11.4.1~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-rocky9-x86_64_v4
[^]          ^zlib-ng@2.1.6%gcc@11.4.1+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^fftw@3.3.10%gcc@12.2.0~mpi~openmp~pfft_patches+shared build_system=autotools precision=double,float arch=linux-rocky9-x86_64_v4
[^]      ^gcc-runtime@14.2.0%gcc@14.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[e]      ^glibc@2.34%gcc@14.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^gmake@4.4.1%gcc@11.4.1~guile build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^gcc-runtime@11.4.1%gcc@11.4.1 build_system=generic arch=linux-rocky9-x86_64_v4
 -       ^kokkos@4.4.01%gcc@14.2.0~aggressive_vectorization~cmake_lang~compiler_warnings~cuda~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning~wrapper build_system=cmake build_type=Release cxxstd=17 generator=make intel_gpu_arch=none arch=linux-rocky9-x86_64_v4
[e]      ^openmpi@5.0.5%gcc@14.2.0+atomics~cuda~debug+gpfs~internal-hwloc~internal-libevent~internal-pmix~java~lustre~memchecker~openshmem~romio+rsh~static~two_level_namespace+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-rocky9-x86_64_v4

Almost everything required to compile LAMMPS is already available in our central upstream ([^]) The only thing we need to compile is kokkos itself, as well as a new version of LAMMPS integrating KOKKOS. You should notice that the variant list for lammps contains +kokkos as well. Let's go ahead and install this variant:

spack install output (click to expand)

$ spack -C ${HOME}/spack-config-templates/0.23.1 install -j ${NSLOTS} lammps +kokkos ^openmpi
[+] /usr (external glibc-2.34-xri56vcyzs7kkvakhoku3fefc46nw25y)
[+] /usr (external glibc-2.34-vjgsh5eoloariofarhttteo73mj5rgql)
[+] /share/apps/rocky9/general/libs/openmpi/gcc/14.2.0/5.0.5 (external openmpi-5.0.5-ccatyv2dkfmw7f433zsigue7ffe2mrfi)
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/gcc-runtime/12.2.0-w77gg5r
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc-runtime/11.4.1-llid4hw
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-14.2.0/gcc-runtime/14.2.0-4w7sesu
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/fftw/3.3.10-tra2c2j
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/nghttp2/1.57.0-4ntqcoo
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/zlib-ng/2.1.6-g2yruc3
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/ncurses/6.5-4n2uzha
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gmake/4.4.1-xchit5a
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/openssl/3.3.0-filwsx6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/curl/8.7.1-v4za2y6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cmake/3.27.9-o3vtj26
==> Installing kokkos-4.4.01-k6x7lnn2uyev6bovh7jro3e2nz2cldv5 [15/16]
==> No binary for kokkos-4.4.01-k6x7lnn2uyev6bovh7jro3e2nz2cldv5 found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/34/3413f0cb39912128d91424ebd92e8832009e7eeaf6fa8da58e99b0d37860d972.tar.gz
==> No patches needed for kokkos
==> kokkos: Executing phase: 'cmake'
==> kokkos: Executing phase: 'build'
==> kokkos: Executing phase: 'install'
==> kokkos: Successfully installed kokkos-4.4.01-k6x7lnn2uyev6bovh7jro3e2nz2cldv5
  Stage: 1.49s.  Cmake: 3.76s.  Build: 8.65s.  Install: 3.28s.  Post-install: 1.34s.  Total: 19.93s
[+] /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-14.2.0/kokkos/4.4.01-k6x7lnn
==> Installing lammps-20240829.1-fe27hsrxiks7uksp76ooa6b4bbklhafp [16/16]
==> No binary for lammps-20240829.1-fe27hsrxiks7uksp76ooa6b4bbklhafp found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/3a/3aea41869aa2fb8120fc4814cab645686f969e2eb7c66aa5587e500597d482dc.tar.gz
==> No patches needed for lammps
==> lammps: Executing phase: 'cmake'
==> lammps: Executing phase: 'build'
==> lammps: Executing phase: 'install'
==> lammps: Successfully installed lammps-20240829.1-fe27hsrxiks7uksp76ooa6b4bbklhafp
  Stage: 9.40s.  Cmake: 7.26s.  Build: 1m 4.47s.  Install: 2.47s.  Post-install: 1.09s.  Total: 1m 25.96s
[+] /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-14.2.0/lammps/20240829.1-openmpi-5.0.5-fe27hsr

To break down what has happened above:

The existing available upstream dependencies have been used from the central location as opposed to being re-installed personally
Spack has noticed that kokkos and lammps are missing and has thus pulled down source code tarballs to the source_cache directory defined in config.yaml and then manually compiled and installed them to the location defined as install_tree: root: in config.yaml
Spack has then compiled and installed LAMMPS with the KOKKOS LAMMPS package activated

Let's now return to our spack find command:

spack find output

$ spack -C ${HOME}/spack-config-templates/0.23.1 find -x -p -v lammps
-- linux-rocky9-x86_64_v4 / gcc@12.2.0 --------------------------
lammps@20230802.3~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell~cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim~kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release fftw_precision=double generator=make gpu_precision=mixed lammps_sizes=smallbig                     /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-b27kssh
lammps@20230802.3~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell+cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim~kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release cuda_arch=70,80,90 fftw_precision=double generator=make gpu_precision=mixed lammps_sizes=smallbig  /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-zqtg4sr

-- linux-rocky9-x86_64_v4 / gcc@14.2.0 --------------------------
lammps@20240829.1~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell~cuda~cuda_mps~curl~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~heffte~intel~interlayer~ipo~jpeg~kim+kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~ml-uf3~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica~rheo+rigid~rocm~shock~smtbq~sph~spin~srd~tally~tools~uef~voronoi~vtk~yaff build_system=cmake build_type=Release fft=fftw3 fft_kokkos=fftw3 fftw_precision=double generator=make lammps_sizes=smallbig  /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-14.2.0/lammps/20240829.1-openmpi-5.0.5-fe27hsr
==> 3 installed packages

So now we have LAMMPS with KOKKOS activated installed alongside the two existing centrally installed versions.

If we now add our specified private module path using module use as specified in modules.yaml:

module use /data/scratch/${USER}/spack/privatemodules/linux-rocky9-x86_64_v4

Then we should see our personal version available for use:

Output from module avail

$ module avail -l lammps
- Package/Alias -----------------------.- Versions --------.- Last mod. -------
/data/scratch/abc123/spack/privatemodules/linux-rocky9-x86_64_v4:
lammps/20240829.1-openmpi-5.0.5-gcc-14.2.0-kokkos           2025/05/14 12:14:17

/share/apps/rocky9/environmentmodules/apocrita-modules/spack:
lammps-gpu/20230802.3-openmpi-5.0.3-cuda-12.4.0-gcc-12.2.0  2025/05/01 10:40:44
lammps/20230802.3-openmpi-5.0.3-gcc-12.2.0                  2025/05/01 10:40:44

And indeed, we can load and use our personal version:

Output from module load (click to expand)

$ module load lammps/20240829.1-openmpi-5.0.5-gcc-14.2.0-kokkos
Loading lammps/20240829.1-openmpi-5.0.5-gcc-14.2.0-kokkos
  Loading requirement: openmpi/5.0.5-gcc-14.2.0
$ lmp -help

Large-scale Atomic/Molecular Massively Parallel Simulator - 29 Aug 2024 - Update 1

Usage example: lmp -var t 300 -echo screen -in in.alloy

List of command line options supported by this LAMMPS executable:

-echo none/screen/log/both  : echoing of input script (-e)
-help                       : print this help message (-h)
-in none/filename           : read input from file or stdin (default) (-i)
-kokkos on/off ...          : turn KOKKOS mode on or off (-k)
-log none/filename          : where to send log output (-l)
-mdi '<mdi flags>'          : pass flags to the MolSSI Driver Interface
-mpicolor color             : which exe in a multi-exe mpirun cmd (-m)
-cite                       : select citation reminder style (-c)
-nocite                     : disable citation reminder (-nc)
-nonbuf                     : disable screen/logfile buffering (-nb)
-package style ...          : invoke package command (-pk)
-partition size1 size2 ...  : assign partition sizes (-p)
-plog basename              : basename for partition logs (-pl)
-pscreen basename           : basename for partition screens (-ps)
-restart2data rfile dfile ... : convert restart to data file (-r2data)
-restart2dump rfile dgroup dstyle dfile ...
            : convert restart to dump file (-r2dump)
-restart2info rfile         : print info about restart rfile (-r2info)
-reorder topology-specs     : processor reordering (-r)
-screen none/filename       : where to send screen output (-sc)
-skiprun                    : skip loops in run and minimize (-sr)
-suffix gpu/intel/kk/opt/omp: style suffix to apply (-sf)
-var varname value          : set index style variable (-v)

OS: Linux "Rocky Linux 9.4 (Blue Onyx)" 5.14.0-427.22.1.el9_4.x86_64 x86_64

Compiler: GNU C++ 14.2.0 with OpenMP 4.5
C++ standard: C++17
MPI v3.1: Open MPI v5.0.5, package: Open MPI abc123@ddy60 Distribution, ident: 5.0.5, repo rev: v5.0.5, Jul 22, 2024

Accelerator configuration:

KOKKOS package API: Serial
KOKKOS package precision: double
Kokkos library version: 4.4.1

FFT information:

FFT precision  = double
FFT engine  = mpiFFT
FFT library = FFTW3
KOKKOS FFT engine  = mpiFFT
KOKKOS FFT library = KISS

Active compile time flags:

-DLAMMPS_GZIP
-DLAMMPS_SMALLBIG
sizeof(smallint): 32-bit
sizeof(imageint): 32-bit
sizeof(tagint):   32-bit
sizeof(bigint):   64-bit

Available compression formats:

Extension: .gz     Command: gzip
Extension: .bz2    Command: bzip2
Extension: .xz     Command: xz
Extension: .lzma   Command: xz


Installed packages:

KOKKOS KSPACE MANYBODY MOLECULE RIGID
(etc.)

We can see that KOKKOS appears in the listed "Accelerator configuration" and "Installed packages" sections.

Activate the KOKKOS LAMMPS package and CUDA¶

Use a GPU node for compilation and installation

At the time of writing, LAMMPS will not compile with CUDA unless there is a physical GPU (as well as CUDA drivers) present (see this GitHub issue).

Please ensure that you run any personal variant installations enabling CUDA support in an interactive qlogin session on a GPU node, otherwise you will see this error:

/bin/ld: warning: libcuda.so.1, needed by liblammps.so.0, not found (try using -rpath or -rpath-link)

How about if we wanted LAMMPS with the KOKKOS package activated but also GPU support via CUDA? It's possible, but you need to specify the right variant. Remember, Spack will install the default variant unless explicitly told otherwise. If you look again at the variants for LAMMPS, you will see that cuda is set to (false) by default. The centrally available lammps-gpu module loads a version that has explicitly been compiled with CUDA support.

Let's see if we can spec a personal installation of LAMMPS activating the KOKKOS package and CUDA:

Specify your cuda_arch

When installing a CUDA variant of a package from Spack, you must also specify a cuda_arch (sm_70 - Volta V100, sm_80 - Ampere A100, sm_90 - Hopper H100). Some packages, like Gromacs, let you install one variant that supports all three at once, but LAMMPS requires a separate install per cuda_arch required when also activating KOKKOS. Check the output of the spec command for more details.

spack spec output (click to expand)

$ spack -C ${HOME}/spack-config-templates/0.23.1 spec lammps +cuda cuda_arch=80 +kokkos ^openmpi
 -   lammps@20240829.1%gcc@12.2.0~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell+cuda~cuda_mps~curl~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~heffte~intel~interlayer~ipo~jpeg~kim+kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~ml-uf3~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica~rheo+rigid~rocm~shock~smtbq~sph~spin~srd~tally~tools~uef~voronoi~vtk~yaff build_system=cmake build_type=Release cuda_arch=80 fft=fftw3 fft_kokkos=fftw3 fftw_precision=double generator=make lammps_sizes=smallbig arch=linux-rocky9-x86_64_v4
[^]      ^cmake@3.27.9%gcc@12.2.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky9-x86_64_v4
[^]          ^curl@8.7.1%gcc@12.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rocky9-x86_64_v4
[^]              ^nghttp2@1.57.0%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                  ^diffutils@3.10%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^openssl@3.3.0%gcc@12.2.0~docs+shared build_system=generic certs=mozilla arch=linux-rocky9-x86_64_v4
[^]                  ^ca-certificates-mozilla@2023-05-30%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[^]              ^perl@5.38.0%gcc@11.4.1+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-rocky9-x86_64_v4
[^]                  ^berkeley-db@18.1.40%gcc@11.4.1+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rocky9-x86_64_v4
[^]                  ^bzip2@1.0.8%gcc@11.4.1~debug~pic+shared build_system=generic arch=linux-rocky9-x86_64_v4
[^]                  ^gdbm@1.23%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                      ^readline@8.2%gcc@11.4.1 build_system=autotools patches=bbf97f1 arch=linux-rocky9-x86_64_v4
[^]              ^pkgconf@2.2.0%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[e]          ^glibc@2.34%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]          ^ncurses@6.5%gcc@11.4.1~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-rocky9-x86_64_v4
[^]          ^zlib-ng@2.1.6%gcc@11.4.1+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^cuda@12.6.2%gcc@12.2.0~allow-unsupported-compilers~dev build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^libxml2@2.10.3%gcc@11.4.1+pic~python+shared build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^xz@5.4.6%gcc@11.4.1~pic build_system=autotools libs=shared,static arch=linux-rocky9-x86_64_v4
[^]      ^fftw@3.3.10%gcc@12.2.0~mpi~openmp~pfft_patches+shared build_system=autotools precision=double,float arch=linux-rocky9-x86_64_v4
[^]      ^gcc-runtime@12.2.0%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[e]      ^glibc@2.34%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^gmake@4.4.1%gcc@11.4.1~guile build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^gcc-runtime@11.4.1%gcc@11.4.1 build_system=generic arch=linux-rocky9-x86_64_v4
 -       ^kokkos@4.4.01%gcc@12.2.0~aggressive_vectorization~alloc_async~cmake_lang~compiler_warnings+cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning~wrapper build_system=cmake build_type=Release cuda_arch=80 cxxstd=17 generator=make intel_gpu_arch=none arch=linux-rocky9-x86_64_v4
[e]      ^openmpi@5.0.3%gcc@12.2.0~atomics~cuda+gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-rocky9-x86_64_v4

So a similar output to the kokkos variant above, but adding in CUDA (which comes from upstream). Again, almost everything required to compile LAMMPS is already available in our central upstream ([^]). A +cuda variant of kokkos needs to be compiled, as well as a new version of lammps that adds both kokkos and cuda support. You should notice that the variant list for lammps contains +cuda, +kokkos and cuda_arch=80 as well. So, as before, let's go ahead and install our personal variant:

spack install output (click to expand)

$ spack -C ${HOME}/spack-config-templates/0.23.1 install -j ${NSLOTS} lammps +cuda cuda_arch=80 +kokkos ^openmpi
[+] /usr (external glibc-2.34-xri56vcyzs7kkvakhoku3fefc46nw25y)
[+] /usr (external glibc-2.34-q2367nemoxqj6j3fg54cg4mjlkvc7f5n)
[+] /share/apps/rocky9/general/libs/openmpi/gcc/12.2.0/5.0.3 (external openmpi-5.0.3-f6rfqovydeohhtdzs6yl2qiro2hxltpm)
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc-runtime/11.4.1-llid4hw
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/gcc-runtime/12.2.0-w77gg5r
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/xz/5.4.6-rwn7pno
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/ncurses/6.5-4n2uzha
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gmake/4.4.1-xchit5a
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/fftw/3.3.10-tra2c2j
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/zlib-ng/2.1.6-g2yruc3
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/nghttp2/1.57.0-4ntqcoo
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/openssl/3.3.0-filwsx6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/libxml2/2.10.3-q6zmsq6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/curl/8.7.1-v4za2y6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cuda/12.6.2-7ahvfr4
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cmake/3.27.9-o3vtj26
==> Installing kokkos-4.4.01-2pldhyfufreqpapvnkhpbrm3jlmzzn3b [17/18]
==> No binary for kokkos-4.4.01-2pldhyfufreqpapvnkhpbrm3jlmzzn3b found: installing from source
==> Using cached archive: /data/scratch/abc123/spack/cache/_source-cache/archive/34/3413f0cb39912128d91424ebd92e8832009e7eeaf6fa8da58e99b0d37860d972.tar.gz
==> No patches needed for kokkos
==> kokkos: Executing phase: 'cmake'
==> Error: InstallError: Kokkos requires +wrapper when using +cuda without %clang, %cce or +cmake_lang

/share/apps/rocky9/general/apps/spack/0.23.1/var/spack/repos/builtin/packages/kokkos/package.py:419, in cmake_args:
    416        if spec.satisfies("~wrapper+cuda") and not (
    417            spec.satisfies("%clang") or spec.satisfies("%cce") or spec.satisfies("+cmake_lang")
    418        ):
  >>    419            raise InstallError(
    420                "Kokkos requires +wrapper when using +cuda without %clang, %cce or +cmake_lang"
    421            )
    422

See build log for details:
  /tmp/5914926.1.all.q/abc123/spack-stage/spack-stage-kokkos-4.4.01-2pldhyfufreqpapvnkhpbrm3jlmzzn3b/spack-build-out.txt

==> Warning: Skipping build of lammps-20240829.1-qxzpryz7c2sqylpuxumelf6aikb4ngbs since kokkos-4.4.01-2pldhyfufreqpapvnkhpbrm3jlmzzn3b failed
==> Error: lammps-20240829.1-qxzpryz7c2sqylpuxumelf6aikb4ngbs: Package was not installed
==> Error: Installation request failed.  Refer to reported errors for failing package(s).

Oh dear! But Spack has given us a clue. If we look at the Spack Packages page for KOKKOS:

https://packages.spack.io/package.html?name=kokkos

You can see:

So, we need to add ^kokkos+wrapper as well as +kokkos to our spec and install commands.

Let's check that in a new spec command:

spack spec output (click to expand)

$ spack -C ${HOME}/spack-config-templates/0.23.1 spec lammps +cuda cuda_arch=80 +kokkos ^kokkos+wrapper ^openmpi
 -   lammps@20240829.1%gcc@12.2.0~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell+cuda~cuda_mps~curl~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~heffte~intel~interlayer~ipo~jpeg~kim+kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~ml-uf3~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica~rheo+rigid~rocm~shock~smtbq~sph~spin~srd~tally~tools~uef~voronoi~vtk~yaff build_system=cmake build_type=Release cuda_arch=80 fft=fftw3 fft_kokkos=fftw3 fftw_precision=double generator=make lammps_sizes=smallbig arch=linux-rocky9-x86_64_v4
[^]      ^cmake@3.27.9%gcc@12.2.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky9-x86_64_v4
[^]          ^curl@8.7.1%gcc@12.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rocky9-x86_64_v4
[^]              ^nghttp2@1.57.0%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                  ^diffutils@3.10%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^openssl@3.3.0%gcc@12.2.0~docs+shared build_system=generic certs=mozilla arch=linux-rocky9-x86_64_v4
[^]                  ^ca-certificates-mozilla@2023-05-30%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[^]              ^perl@5.38.0%gcc@11.4.1+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-rocky9-x86_64_v4
[^]                  ^berkeley-db@18.1.40%gcc@11.4.1+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rocky9-x86_64_v4
[^]                  ^bzip2@1.0.8%gcc@11.4.1~debug~pic+shared build_system=generic arch=linux-rocky9-x86_64_v4
[^]                  ^gdbm@1.23%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                      ^readline@8.2%gcc@11.4.1 build_system=autotools patches=bbf97f1 arch=linux-rocky9-x86_64_v4
[^]              ^pkgconf@2.2.0%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[e]          ^glibc@2.34%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]          ^ncurses@6.5%gcc@11.4.1~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-rocky9-x86_64_v4
[^]          ^zlib-ng@2.1.6%gcc@11.4.1+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^cuda@12.6.2%gcc@12.2.0~allow-unsupported-compilers~dev build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^libxml2@2.10.3%gcc@11.4.1+pic~python+shared build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^xz@5.4.6%gcc@11.4.1~pic build_system=autotools libs=shared,static arch=linux-rocky9-x86_64_v4
[^]      ^fftw@3.3.10%gcc@12.2.0~mpi~openmp~pfft_patches+shared build_system=autotools precision=double,float arch=linux-rocky9-x86_64_v4
[^]      ^gcc-runtime@12.2.0%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[e]      ^glibc@2.34%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^gmake@4.4.1%gcc@11.4.1~guile build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^gcc-runtime@11.4.1%gcc@11.4.1 build_system=generic arch=linux-rocky9-x86_64_v4
 -       ^kokkos@4.4.01%gcc@12.2.0~aggressive_vectorization~alloc_async~cmake_lang~compiler_warnings+cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning+wrapper build_system=cmake build_type=Release cuda_arch=80 cxxstd=17 generator=make intel_gpu_arch=none arch=linux-rocky9-x86_64_v4
 -           ^kokkos-nvcc-wrapper@4.4.01%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[e]      ^openmpi@5.0.3%gcc@12.2.0~atomics~cuda+gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-rocky9-x86_64_v4

It looks almost the same as our first spec command, but now ^kokkos+wrapper is listed as part of the "Input spec" and we are going to install the required kokkos-nvcc-wrapper dependency as well. Now, onto the install:

spack install output (click to expand)

$ spack -C ${HOME}/spack-config-templates/0.23.1 install -j ${NSLOTS} lammps +cuda cuda_arch=80 +kokkos ^kokkos+wrapper ^openmpi
[+] /usr (external glibc-2.34-xri56vcyzs7kkvakhoku3fefc46nw25y)
[+] /usr (external glibc-2.34-q2367nemoxqj6j3fg54cg4mjlkvc7f5n)
[+] /share/apps/rocky9/general/libs/openmpi/gcc/12.2.0/5.0.3 (external openmpi-5.0.3-f6rfqovydeohhtdzs6yl2qiro2hxltpm)
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/gcc-runtime/12.2.0-w77gg5r
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc-runtime/11.4.1-llid4hw
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/fftw/3.3.10-tra2c2j
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/nghttp2/1.57.0-4ntqcoo
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gmake/4.4.1-xchit5a
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/xz/5.4.6-rwn7pno
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/zlib-ng/2.1.6-g2yruc3
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/ncurses/6.5-4n2uzha
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/libxml2/2.10.3-q6zmsq6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/openssl/3.3.0-filwsx6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cuda/12.6.2-7ahvfr4
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/curl/8.7.1-v4za2y6
==> Installing kokkos-nvcc-wrapper-4.4.01-fclg46uxty4dhivp5huqd4tqevdydeqd [16/19]
==> No binary for kokkos-nvcc-wrapper-4.4.01-fclg46uxty4dhivp5huqd4tqevdydeqd found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/3f/3f7096d17eaaa4004c7497ac082bf1ae3ff47b5104149e54af021a89414c3682.tar.gz
==> No patches needed for kokkos-nvcc-wrapper
==> kokkos-nvcc-wrapper: Executing phase: 'install'
==> kokkos-nvcc-wrapper: Successfully installed kokkos-nvcc-wrapper-4.4.01-fclg46uxty4dhivp5huqd4tqevdydeqd
  Stage: 0.62s.  Install: 0.00s.  Post-install: 0.72s.  Total: 2.48s
[+] /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/kokkos-nvcc-wrapper/4.4.01-fclg46u
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cmake/3.27.9-o3vtj26
==> Installing kokkos-4.4.01-jz4kwbpn4pjc5vv5lm764dqwxeej2u5z [18/19]
==> No binary for kokkos-4.4.01-jz4kwbpn4pjc5vv5lm764dqwxeej2u5z found: installing from source
==> Using cached archive: /data/scratch/abc123/spack/cache/_source-cache/archive/34/3413f0cb39912128d91424ebd92e8832009e7eeaf6fa8da58e99b0d37860d972.tar.gz
==> No patches needed for kokkos
==> kokkos: Executing phase: 'cmake'
==> kokkos: Executing phase: 'build'
==> kokkos: Executing phase: 'install'
==> kokkos: Successfully installed kokkos-4.4.01-jz4kwbpn4pjc5vv5lm764dqwxeej2u5z
  Stage: 0.12s.  Cmake: 12.13s.  Build: 24.24s.  Install: 3.75s.  Post-install: 1.51s.  Total: 43.92s
[+] /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/kokkos/4.4.01-jz4kwbp
==> Installing lammps-20240829.1-xgrqw5mjhv6dky36pmst7ktp2masn5cp [19/19]
==> No binary for lammps-20240829.1-xgrqw5mjhv6dky36pmst7ktp2masn5cp found: installing from source
==> Using cached archive: /data/scratch/abc123/spack/cache/_source-cache/archive/3a/3aea41869aa2fb8120fc4814cab645686f969e2eb7c66aa5587e500597d482dc.tar.gz
==> No patches needed for lammps
==> lammps: Executing phase: 'cmake'
==> Error: ProcessError: Command exited with status 1:
'/share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cmake/3.27.9-o3vtj26/bin/cmake' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:STRING=/data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20240829.1-openmpi-5.0.3-xgrqw5m' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON' '-DCMAKE_INSTALL_RPATH:STRING=/data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20240829.1-openmpi-5.0.3-xgrqw5m/lib;/data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20240829.1-openmpi-5.0.3-xgrqw5m/lib64' '-DCMAKE_PREFIX_PATH:STRING=/data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/kokkos/4.4.01-jz4kwbp;/data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/kokkos-nvcc-wrapper/4.4.01-fclg46u;/share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/fftw/3.3.10-tra2c2j;/share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cuda/12.6.2-7ahvfr4;/share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cuda/12.6.2-7ahvfr4/targets/x86_64-linux/lib/cmake;/share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/libxml2/2.10.3-q6zmsq6;/share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/xz/5.4.6-rwn7pno;/share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cmake/3.27.9-o3vtj26;/share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/zlib-ng/2.1.6-g2yruc3;/share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gmake/4.4.1-xchit5a;/share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc-runtime/11.4.1-llid4hw;/share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/gcc-runtime/12.2.0-w77gg5r;/share/apps/rocky9/general/libs/openmpi/gcc/12.2.0/5.0.3' '-DCMAKE_BUILD_TYPE:STRING=Release' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_POLICY_DEFAULT_CMP0090:STRING=NEW' '-DCMAKE_FIND_USE_PACKAGE_REGISTRY:BOOL=OFF' '-DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON' '-DBUILD_SHARED_LIBS:BOOL=ON' '' '-DBUILD_MPI:BOOL=ON' '-DBUILD_OMP:BOOL=ON' '-DBUILD_TOOLS:BOOL=OFF' '-DENABLE_TESTING:BOOL=OFF' '-DDOWNLOAD_POTENTIALS:BOOL=OFF' '-DEXTERNAL_KOKKOS:BOOL=ON' '-DFFT_KOKKOS:STRING=fftw3' '-DBUILD_LIB:BOOL=ON' '-DCMAKE_TUNE_FLAGS:STRING=-march=x86-64-v4 -mtune=generic' '-DLAMMPS_SIZES:STRING=smallbig' '-DWITH_JPEG:BOOL=OFF' '-DWITH_PNG:BOOL=OFF' '-DWITH_FFMPEG:BOOL=OFF' '-DWITH_CURL:BOOL=OFF' '-DPKG_ADIOS:BOOL=OFF' '-DPKG_AMOEBA:BOOL=OFF' '-DPKG_ASPHERE:BOOL=OFF' '-DPKG_ATC:BOOL=OFF' '-DPKG_AWPMD:BOOL=OFF' '-DPKG_BOCS:BOOL=OFF' '-DPKG_BODY:BOOL=OFF' '-DPKG_BPM:BOOL=OFF' '-DPKG_BROWNIAN:BOOL=OFF' '-DPKG_CG-DNA:BOOL=OFF' '-DPKG_CG-SPICA:BOOL=OFF' '-DPKG_CLASS2:BOOL=OFF' '-DPKG_COLLOID:BOOL=OFF' '-DPKG_COLVARS:BOOL=OFF' '-DPKG_COMPRESS:BOOL=OFF' '-DPKG_CORESHELL:BOOL=OFF' '-DPKG_DIELECTRIC:BOOL=OFF' '-DPKG_DIFFRACTION:BOOL=OFF' '-DPKG_DIPOLE:BOOL=OFF' '-DPKG_DPD-BASIC:BOOL=OFF' '-DPKG_DPD-MESO:BOOL=OFF' '-DPKG_DPD-REACT:BOOL=OFF' '-DPKG_DPD-SMOOTH:BOOL=OFF' '-DPKG_DRUDE:BOOL=OFF' '-DPKG_EFF:BOOL=OFF' '-DPKG_ELECTRODE:BOOL=OFF' '-DPKG_EXTRA-COMPUTE:BOOL=OFF' '-DPKG_EXTRA-DUMP:BOOL=OFF' '-DPKG_EXTRA-FIX:BOOL=OFF' '-DPKG_EXTRA-MOLECULE:BOOL=OFF' '-DPKG_EXTRA-PAIR:BOOL=OFF' '-DPKG_FEP:BOOL=OFF' '-DPKG_GRANULAR:BOOL=OFF' '-DPKG_H5MD:BOOL=OFF' '-DPKG_INTEL:BOOL=OFF' '-DPKG_INTERLAYER:BOOL=OFF' '-DPKG_KIM:BOOL=OFF' '-DPKG_KOKKOS:BOOL=ON' '-DPKG_KSPACE:BOOL=ON' '-DPKG_LATBOLTZ:BOOL=OFF' '-DPKG_LEPTON:BOOL=OFF' '-DPKG_MACHDYN:BOOL=OFF' '-DPKG_MANIFOLD:BOOL=OFF' '-DPKG_MANYBODY:BOOL=ON' '-DPKG_MC:BOOL=OFF' '-DPKG_MEAM:BOOL=OFF' '-DPKG_MESONT:BOOL=OFF' '-DPKG_MGPT:BOOL=OFF' '-DPKG_MISC:BOOL=OFF' '-DPKG_ML-HDNNP:BOOL=OFF' '-DPKG_ML-IAP:BOOL=OFF' '-DPKG_ML-POD:BOOL=OFF' '-DPKG_ML-RANN:BOOL=OFF' '-DPKG_ML-SNAP:BOOL=OFF' '-DPKG_ML-UF3:BOOL=OFF' '-DPKG_MOFFF:BOOL=OFF' '-DPKG_MOLECULE:BOOL=ON' '-DPKG_MOLFILE:BOOL=OFF' '-DPKG_NETCDF:BOOL=OFF' '-DPKG_OPENMP:BOOL=OFF' '-DPKG_OPT:BOOL=OFF' '-DPKG_ORIENT:BOOL=OFF' '-DPKG_PERI:BOOL=OFF' '-DPKG_PHONON:BOOL=OFF' '-DPKG_PLUGIN:BOOL=OFF' '-DPKG_PLUMED:BOOL=OFF' '-DPKG_POEMS:BOOL=OFF' '-DPKG_PTM:BOOL=OFF' '-DPKG_PYTHON:BOOL=OFF' '-DPKG_QEQ:BOOL=OFF' '-DPKG_QTB:BOOL=OFF' '-DPKG_REACTION:BOOL=OFF' '-DPKG_REAXFF:BOOL=OFF' '-DPKG_RHEO:BOOL=OFF' '-DPKG_REPLICA:BOOL=OFF' '-DPKG_RIGID:BOOL=ON' '-DPKG_SHOCK:BOOL=OFF' '-DPKG_SMTBQ:BOOL=OFF' '-DPKG_SPH:BOOL=OFF' '-DPKG_SPIN:BOOL=OFF' '-DPKG_SRD:BOOL=OFF' '-DPKG_TALLY:BOOL=OFF' '-DPKG_UEF:BOOL=OFF' '-DPKG_VORONOI:BOOL=OFF' '-DPKG_VTK:BOOL=OFF' '-DPKG_YAFF:BOOL=OFF' '-DFFT:STRING=fftw3' '-DFFT_USE_HEFFTE:BOOL=OFF' '-DFFT_SINGLE:BOOL=OFF' '/tmp/5919433.1.all.q/abc123/spack-stage/spack-stage-lammps-20240829.1-xgrqw5mjhv6dky36pmst7ktp2masn5cp/spack-src/cmake'

1 error found in build log:
 40    -- Found CUDAToolkit: /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cuda/12.6.2-7ahvfr4/include (found version "12.6.77")
 41    -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
 42    -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
 43    -- Found Threads: TRUE
 44    -- Enabled Kokkos devices: SERIAL;CUDA
 45    -- kokkos_launch_compiler is enabled globally. C++ compiler commands with -DKOKKOS_DEPENDENCE will be redirected to the appropriate compiler for Kokkos
 >> 46    CMake Error at Modules/Packages/KOKKOS.cmake:137 (message):
 47      The CUDA backend of Kokkos requires either KISS FFT or CUFFT.
 48    Call Stack (most recent call first):
 49      CMakeLists.txt:727 (include)
 50
 51
 52    -- Configuring incomplete, errors occurred!

See build log for details:
  /tmp/5919433.1.all.q/abc123/spack-stage/spack-stage-lammps-20240829.1-xgrqw5mjhv6dky36pmst7ktp2masn5cp/spack-build-out.txt

We've now managed to compile kokkos-nvcc-wrapper and kokkos, but we have another error, and this is because the CUDA backend of Kokkos requires either KISS FFT or CUFFT, so we will specify fft_kokkos=kiss as well (the default is fft_kokkos=fftw3, hence why it failed):

spack spec output (click to expand)

$ spack -C ${HOME}/spack-config-templates/0.23.1 spec lammps +cuda cuda_arch=80 +kokkos fft_kokkos=kiss ^kokkos+wrapper ^openmpi
 -   lammps@20240829.1%gcc@12.2.0~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell+cuda~cuda_mps~curl~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~heffte~intel~interlayer~ipo~jpeg~kim+kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~ml-uf3~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica~rheo+rigid~rocm~shock~smtbq~sph~spin~srd~tally~tools~uef~voronoi~vtk~yaff build_system=cmake build_type=Release cuda_arch=80 fft=fftw3 fft_kokkos=kiss fftw_precision=double generator=make lammps_sizes=smallbig arch=linux-rocky9-x86_64_v4
[^]      ^cmake@3.27.9%gcc@12.2.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky9-x86_64_v4
[^]          ^curl@8.7.1%gcc@12.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rocky9-x86_64_v4
[^]              ^nghttp2@1.57.0%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                  ^diffutils@3.10%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^openssl@3.3.0%gcc@12.2.0~docs+shared build_system=generic certs=mozilla arch=linux-rocky9-x86_64_v4
[^]                  ^ca-certificates-mozilla@2023-05-30%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[^]              ^perl@5.38.0%gcc@11.4.1+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-rocky9-x86_64_v4
[^]                  ^berkeley-db@18.1.40%gcc@11.4.1+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rocky9-x86_64_v4
[^]                  ^bzip2@1.0.8%gcc@11.4.1~debug~pic+shared build_system=generic arch=linux-rocky9-x86_64_v4
[^]                  ^gdbm@1.23%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                      ^readline@8.2%gcc@11.4.1 build_system=autotools patches=bbf97f1 arch=linux-rocky9-x86_64_v4
[^]              ^pkgconf@2.2.0%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[e]          ^glibc@2.34%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]          ^ncurses@6.5%gcc@11.4.1~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-rocky9-x86_64_v4
[^]          ^zlib-ng@2.1.6%gcc@11.4.1+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^cuda@12.6.2%gcc@12.2.0~allow-unsupported-compilers~dev build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^libxml2@2.10.3%gcc@11.4.1+pic~python+shared build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^xz@5.4.6%gcc@11.4.1~pic build_system=autotools libs=shared,static arch=linux-rocky9-x86_64_v4
[^]      ^fftw@3.3.10%gcc@12.2.0~mpi~openmp~pfft_patches+shared build_system=autotools precision=double,float arch=linux-rocky9-x86_64_v4
[^]      ^gcc-runtime@12.2.0%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[e]      ^glibc@2.34%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^gmake@4.4.1%gcc@11.4.1~guile build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^gcc-runtime@11.4.1%gcc@11.4.1 build_system=generic arch=linux-rocky9-x86_64_v4
[+]      ^kokkos@4.4.01%gcc@12.2.0~aggressive_vectorization~alloc_async~cmake_lang~compiler_warnings+cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning+wrapper build_system=cmake build_type=Release cuda_arch=80 cxxstd=17 generator=make intel_gpu_arch=none arch=linux-rocky9-x86_64_v4
[+]          ^kokkos-nvcc-wrapper@4.4.01%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[e]      ^openmpi@5.0.3%gcc@12.2.0~atomics~cuda+gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-rocky9-x86_64_v4

With this final amended spec, the installation should complete without any further errors:

spack install output (click to expand)

$ spack -C ${HOME}/spack-config-templates/0.23.1 install -j ${NSLOTS} lammps +cuda cuda_arch=80 +kokkos fft_kokkos=kiss ^kokkos+wrapper ^openmpi
[+] /usr (external glibc-2.34-xri56vcyzs7kkvakhoku3fefc46nw25y)
[+] /usr (external glibc-2.34-q2367nemoxqj6j3fg54cg4mjlkvc7f5n)
[+] /share/apps/rocky9/general/libs/openmpi/gcc/12.2.0/5.0.3 (external openmpi-5.0.3-f6rfqovydeohhtdzs6yl2qiro2hxltpm)
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/gcc-runtime/12.2.0-w77gg5r
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc-runtime/11.4.1-llid4hw
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/nghttp2/1.57.0-4ntqcoo
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/fftw/3.3.10-tra2c2j
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/xz/5.4.6-rwn7pno
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/zlib-ng/2.1.6-g2yruc3
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/ncurses/6.5-4n2uzha
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gmake/4.4.1-xchit5a
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/openssl/3.3.0-filwsx6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/libxml2/2.10.3-q6zmsq6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/curl/8.7.1-v4za2y6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cuda/12.6.2-7ahvfr4
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cmake/3.27.9-o3vtj26
[+] /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/kokkos-nvcc-wrapper/4.4.01-fclg46u
[+] /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/kokkos/4.4.01-jz4kwbp
==> Installing lammps-20240829.1-4whorsa36lbglafbfkqessntmwypnwt4 [19/19]
==> No binary for lammps-20240829.1-4whorsa36lbglafbfkqessntmwypnwt4 found: installing from source
==> Using cached archive: /data/scratch/abc123/spack/cache/_source-cache/archive/3a/3aea41869aa2fb8120fc4814cab645686f969e2eb7c66aa5587e500597d482dc.tar.gz
==> No patches needed for lammps
==> lammps: Executing phase: 'cmake'
==> lammps: Executing phase: 'build'
==> lammps: Executing phase: 'install'
==> lammps: Successfully installed lammps-20240829.1-4whorsa36lbglafbfkqessntmwypnwt4
  Stage: 2.96s.  Cmake: 11.98s.  Build: 11m 47.69s.  Install: 3.21s.  Post-install: 1.84s.  Total: 12m 10.66s
[+] /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20240829.1-openmpi-5.0.3-4whorsa

Let's now return to our spack find command:

spack find output

$ spack -C ${HOME}/spack-config-templates/0.23.1 find -x -p -v lammps
-- linux-rocky9-x86_64_v4 / gcc@12.2.0 --------------------------
lammps@20230802.3~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell~cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim~kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release fftw_precision=double generator=make gpu_precision=mixed lammps_sizes=smallbig                                                   /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-b27kssh
lammps@20230802.3~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell+cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim~kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release cuda_arch=70,80,90 fftw_precision=double generator=make gpu_precision=mixed lammps_sizes=smallbig                                /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-zqtg4sr
lammps@20240829.1~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell+cuda~cuda_mps~curl~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~heffte~intel~interlayer~ipo~jpeg~kim+kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~ml-uf3~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica~rheo+rigid~rocm~shock~smtbq~sph~spin~srd~tally~tools~uef~voronoi~vtk~yaff build_system=cmake build_type=Release cuda_arch=80 fft=fftw3 fft_kokkos=kiss fftw_precision=double generator=make lammps_sizes=smallbig  /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20240829.1-openmpi-5.0.3-4whorsa

-- linux-rocky9-x86_64_v4 / gcc@14.2.0 --------------------------
lammps@20240829.1~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell~cuda~cuda_mps~curl~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~heffte~intel~interlayer~ipo~jpeg~kim+kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~ml-uf3~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica~rheo+rigid~rocm~shock~smtbq~sph~spin~srd~tally~tools~uef~voronoi~vtk~yaff build_system=cmake build_type=Release fft=fftw3 fft_kokkos=fftw3 fftw_precision=double generator=make lammps_sizes=smallbig  /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-14.2.0/lammps/20240829.1-openmpi-5.0.5-fe27hsr
==> 4 installed packages

So now we have LAMMPS with the KOKKOS package activated (+kokkos) and LAMMPS with the KOKKOS package activated (+kokkos) and CUDA (+cuda cuda_arch=80) personally, alongside the two centrally installed non-KOKKOS versions.

As long as we still have our specified private module path loaded using module use as specified in modules.yaml:

module use /data/scratch/${USER}/spack/privatemodules/linux-rocky9-x86_64_v4

Then we should see our personal versions available for use:

Output from module avail

$ module avail -l lammps
- Package/Alias -----------------------.- Versions --------.- Last mod. -------
/data/scratch/abc123/spack/privatemodules/linux-rocky9-x86_64_v4:
lammps-gpu/20240829.1-openmpi-5.0.3-cuda-12.6.2-gcc-12.2.0-kokkos2025/05/16 10:09:50
lammps/20240829.1-openmpi-5.0.5-gcc-14.2.0-kokkos           2025/05/16 09:17:58

/share/apps/rocky9/environmentmodules/apocrita-modules/spack:
lammps-gpu/20230802.3-openmpi-5.0.3-cuda-12.4.0-gcc-12.2.0  2025/05/01 10:40:44
lammps/20230802.3-openmpi-5.0.3-gcc-12.2.0                  2025/05/01 10:40:44

And indeed, we can load and use our personal version:

Use a GPU node for execution

As with compilation and installation. any GPU version of LAMMPS must also be executed on a GPU node, otherwise you will see the following error message as there will be no CUDA libraries present on the node:

lmp: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory

Output from module load (click to expand)

$ module load lammps-gpu/20240829.1-openmpi-5.0.3-cuda-12.6.2-gcc-12.2.0-kokkos
Loading lammps-gpu/20240829.1-openmpi-5.0.3-cuda-12.6.2-gcc-12.2.0-kokkos
  Loading requirement: cuda/12.6.2-gcc-12.2.0 openmpi/5.0.3-gcc-12.2.0

$ lmp -help

Large-scale Atomic/Molecular Massively Parallel Simulator - 29 Aug 2024 - Update 1

Usage example: lmp -var t 300 -echo screen -in in.alloy

List of command line options supported by this LAMMPS executable:

-echo none/screen/log/both  : echoing of input script (-e)
-help                       : print this help message (-h)
-in none/filename           : read input from file or stdin (default) (-i)
-kokkos on/off ...          : turn KOKKOS mode on or off (-k)
-log none/filename          : where to send log output (-l)
-mdi '<mdi flags>'          : pass flags to the MolSSI Driver Interface
-mpicolor color             : which exe in a multi-exe mpirun cmd (-m)
-cite                       : select citation reminder style (-c)
-nocite                     : disable citation reminder (-nc)
-nonbuf                     : disable screen/logfile buffering (-nb)
-package style ...          : invoke package command (-pk)
-partition size1 size2 ...  : assign partition sizes (-p)
-plog basename              : basename for partition logs (-pl)
-pscreen basename           : basename for partition screens (-ps)
-restart2data rfile dfile ... : convert restart to data file (-r2data)
-restart2dump rfile dgroup dstyle dfile ...
            : convert restart to dump file (-r2dump)
-restart2info rfile         : print info about restart rfile (-r2info)
-reorder topology-specs     : processor reordering (-r)
-screen none/filename       : where to send screen output (-sc)
-skiprun                    : skip loops in run and minimize (-sr)
-suffix gpu/intel/kk/opt/omp: style suffix to apply (-sf)
-var varname value          : set index style variable (-v)

OS: Linux "Rocky Linux 9.4 (Blue Onyx)" 5.14.0-427.22.1.el9_4.x86_64 x86_64

Compiler: GNU C++ 12.2.0 with OpenMP 4.5
C++ standard: C++17
MPI v3.1: Open MPI v5.0.3, package: Open MPI abc123@ddy16 Distribution, ident: 5.0.3, repo rev: v5.0.3, Apr 08, 2024

Accelerator configuration:

KOKKOS package API: CUDA Serial
KOKKOS package precision: double
Kokkos library version: 4.4.1

FFT information:

FFT precision  = double
FFT engine  = mpiFFT
FFT library = FFTW3
KOKKOS FFT engine  = mpiFFT
KOKKOS FFT library = KISS

Active compile time flags:

-DLAMMPS_GZIP
-DLAMMPS_SMALLBIG
sizeof(smallint): 32-bit
sizeof(imageint): 32-bit
sizeof(tagint):   32-bit
sizeof(bigint):   64-bit

Available compression formats:

Extension: .gz     Command: gzip
Extension: .bz2    Command: bzip2
Extension: .xz     Command: xz
Extension: .lzma   Command: xz


Installed packages:

KOKKOS KSPACE MANYBODY MOLECULE RIGID
(etc.)

We can see that KOKKOS appears in the listed "Accelerator configuration" and "Installed packages" sections as before, but this time using the "CUDA Serial" KOKKOS package API and "KOKKOS FFT library = KISS".

Summary¶

The above examples show just two of the many possible variants of LAMMPS that you can compile yourself. To compile additional variants, change the required parameters to (true) with + or (false) with ~ in spec and install commands, and remember Spack will always follow the defaults unless explicitly told otherwise.

You could, for example, spec and install a much more complex variant of LAMMPS such as:

lammps +adios +cuda cuda_arch=80 +jpeg +kokkos fft_kokkos=kiss +plumed ^kokkos+wrapper ^openmpi

which would compile with the Adios package activated, GPU support, JPEG support, the KOKKOS package activated with CUDA support and the PLUMED package activated. The more complex the variant, the less is likely to be already in upstream, so bear this in mind, as compilation and installation time may increase as well.

Installing a personal version of LAMMPS¶

Searching for available variants¶

Spack Packages website¶

The spack info command¶

Installing specific variants¶

Activate the KOKKOS LAMMPS package¶

Activate the KOKKOS LAMMPS package and CUDA¶

Summary¶

The `spack info` command¶