Skip to content

Installing a personal version of LAMMPS

Another very popular application on Apocrita is LAMMPS, but as with Gromacs, we only provide modules for fairly simple compilation variants. Some more advanced users may want to compile additional personal versions with more granular compilation options.

Searching for available variants

So, as before, let's see what is on offer by default:

spack find output 
$ spack -C ${HOME}/spack-config-templates/0.22.0 find -x -p -v lammps
-- linux-rocky9-x86_64_v4 / gcc@12.2.0 --------------------------
lammps@20230802.3~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell~cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim~kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release fftw_precision=double generator=make gpu_precision=mixed lammps_sizes=smallbig                     /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-b27kssh
lammps@20230802.3~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell+cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim~kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release cuda_arch=70,80,90 fftw_precision=double generator=make gpu_precision=mixed lammps_sizes=smallbig  /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-zqtg4sr
==> 2 installed packages

$ module avail -l lammps
- Package/Alias -----------------------.- Versions --------
/share/apps/rocky9/environmentmodules/apocrita-modules/spack:
lammps-gpu/20230802.3-openmpi-5.0.3-cuda-12.4.0-gcc-12.2.0
lammps/20230802.3-openmpi-5.0.3-gcc-12.2.0

As with Gromacs, the output of spack find includes an incredibly long list of variants, either marked with ~ (false) or + (true).

Spack Packages website

Package information is for the develop branch

The Spack Packages website lists information such as versions available, variants etc. from the develop branch of Spack. Some versions and variants may not be available in the specific Spack release you are using.

The Spack Packages page for LAMMPS can be found here:

https://packages.spack.io/package.html?name=lammps

There is a extremely long list of variants (too long to even screenshot here in full), and the defaults (true) or (false) are marked in brackets after each item in the list. For the versions centrally compiled, we have largely stuck to the defaults.

The spack info command

spack info output (click to expand) 
$ spack -C ${HOME}/spack-config-templates/0.22.0 info lammps
CMakePackage:   lammps

Description:
    LAMMPS stands for Large-scale Atomic/Molecular Massively Parallel
    Simulator.

Homepage: https://www.lammps.org/

Preferred version:
    20230802.3    https://github.com/lammps/lammps/archive/stable_2Aug2023_update3.tar.gz

Safe versions:
    develop       [git] https://github.com/lammps/lammps.git on branch develop
    20240207.1    https://github.com/lammps/lammps/archive/patch_7Feb2024_update1.tar.gz
(etc.)
Deprecated versions:
    20240207      https://github.com/lammps/lammps/archive/patch_7Feb2024.tar.gz
(etc.)

Variants:
    asphere [false]                 false, true
        Activate the asphere package
    body [false]                    false, true
        Activate the body package
    build_system [cmake]            cmake
        Build systems supported by the package
    class2 [false]                  false, true
        Activate the class2 package
    colloid [false]                 false, true
        Activate the colloid package
    compress [false]                false, true
        Activate the compress package
    coreshell [false]               false, true
        Activate the coreshell package
    cuda [false]                    false, true
        Build with CUDA
    cuda_mps [false]                false, true
        (CUDA only) Enable tweaks for running with Nvidia CUDA Multi-process services daemon
    dipole [false]                  false, true
        Activate the dipole package
    ffmpeg [false]                  false, true
        Build with ffmpeg support
    granular [false]                false, true
        Activate the granular package
    jpeg [false]                    false, true
        Build with jpeg support
    kim [false]                     false, true
        Activate the kim package
    kokkos [false]                  false, true
        Activate the kokkos package
    kspace [true]                   false, true
        Activate the kspace package
    lammps_sizes [smallbig]         bigbig, smallbig, smallsmall
        LAMMPS integer sizes (smallsmall: all 32-bit, smallbig:64-bit #atoms #timesteps, bigbig: also 64-bit imageint, 64-bit atom ids)
    lib [true]                      false, true
        Build the liblammps in addition to the executable
    manybody [true]                 false, true
        Activate the manybody package
    mc [false]                      false, true
        Activate the mc package
    misc [false]                    false, true
        Activate the misc package
    molecule [true]                 false, true
        Activate the molecule package
    mpi [true]                      false, true
        Build with mpi
    opencl [false]                  false, true
        Build with OpenCL
    openmp [true]                   false, true
        Build with OpenMP
    openmp-package [false]          false, true
        Activate the openmp package
    opt [false]                     false, true
        Activate the opt package
    peri [false]                    false, true
        Activate the peri package
    png [false]                     false, true
        Build with png support
    poems [false]                   false, true
        Activate the poems package
    python [false]                  false, true
        Activate the python package
    qeq [false]                     false, true
        Activate the qeq package
    replica [false]                 false, true
        Activate the replica package
    rigid [true]                    false, true
        Activate the rigid package
    rocm [false]                    false, true
        Enable ROCm support
    shock [false]                   false, true
        Activate the shock package
    srd [false]                     false, true
        Activate the srd package
    voronoi [false]                 false, true
        Activate the voronoi package

    when build_system=cmake
      build_type [Release]          Debug, MinSizeRel, RelWithDebInfo, Release
          CMake build type
      generator [make]              none
          the build system generator to use

    when build_system=cmake ^cmake@3.9:
      ipo [false]                   false, true
          CMake interprocedural optimization

    when +cuda
      cuda_arch [none]              none, 10, 11, 12, 13, 20, 21, 30, 32, 35, 37, 50, 52, 53, 60, 61, 62, 70, 72, 75, 80, 86, 87, 89, 90
          CUDA architecture

    when ~kokkos
      gpu_precision [mixed]         double, mixed, single
          Select GPU precision (used by GPU package)

    when +kspace
      fftw_precision [double]       double, single
          Select FFTW precision (used by Kspace)

    when +rocm
      amdgpu_target [none]          none, gfx1010, gfx1011, gfx1012, gfx1013, gfx1030, gfx1031, gfx1032, gfx1033, gfx1034, gfx1035, gfx1036, gfx1100, gfx1101, gfx1102, gfx1103, gfx701, gfx801, gfx802, gfx803,
                                    gfx900, gfx900:xnack-, gfx902, gfx904, gfx906, gfx906:xnack-, gfx908, gfx908:xnack-, gfx909, gfx90a, gfx90a:xnack+, gfx90a:xnack-, gfx90c, gfx940, gfx941, gfx942
          AMD GPU architecture

    when @:20181212
      reax [false]                  false, true
          Activate the reax package

    when @:20181212,20210702:
      meam [false]                  false, true
          Activate the meam package

    when @:20210527
      snap [false]                  false, true
          Activate the snap package
      user-atc [false]              false, true
          Activate the user-atc package
      user-awpmd [false]            false, true
          Activate the user-awpmd package
      user-colvars [false]          false, true
          Activate the user-colvars package
      user-diffraction [false]      false, true
          Activate the user-diffraction package
      user-dpd [false]              false, true
          Activate the user-dpd package
      user-drude [false]            false, true
          Activate the user-drude package
      user-eff [false]              false, true
          Activate the user-eff package
      user-fep [false]              false, true
          Activate the user-fep package
      user-h5md [false]             false, true
          Activate the user-h5md package
      user-intel [false]            false, true
          Activate the user-intel package
      user-lb [false]               false, true
          Activate the user-lb package
      user-manifold [false]         false, true
          Activate the user-manifold package
      user-mgpt [false]             false, true
          Activate the user-mgpt package
      user-molfile [false]          false, true
          Activate the user-molfile package
      user-omp [false]              false, true
          Activate the user-omp package
      user-phonon [false]           false, true
          Activate the user-phonon package
      user-qtb [false]              false, true
          Activate the user-qtb package
      user-reaxc [false]            false, true
          Activate the user-reaxc package
      user-smd [false]              false, true
          Activate the user-smd package
      user-smtbq [false]            false, true
          Activate the user-smtbq package
      user-sph [false]              false, true
          Activate the user-sph package
      user-tally [false]            false, true
          Activate the user-tally package

    when @:20210702
      user-misc [false]             false, true
          Activate the user-misc package

    when @:20230802.1
      exceptions [false]            false, true
          Build with lammps exceptions
      mpiio [false]                 false, true
          Activate the mpiio package

    when @20170504:20210527
      user-cgsdk [false]            false, true
          Activate the user-cgsdk package
      user-netcdf [false]           false, true
          Activate the user-netcdf package

    when @20170706:20210527
      user-meamc [false]            false, true
          Activate the user-meamc package

    when @20170922:20230328
      latte [false]                 false, true
          Activate the latte package

    when @20171023:20210527
      user-uef [false]              false, true
          Activate the user-uef package

    when @20180205:20210527
      user-mofff [false]            false, true
          Activate the user-mofff package

    when @20180511:20210527
      user-bocs [false]             false, true
          Activate the user-bocs package

    when @20180629:
      spin [false]                  false, true
          Activate the spin package

    when @20181010:20210527
      user-ptm [false]              false, true
          Activate the user-ptm package

    when @20181109:20210527
      user-plumed [false]           false, true
          Activate the user-plumed package
      user-sdpd [false]             false, true
          Activate the user-sdpd package

    when @20190201:20210527
      user-yaff [false]             false, true
          Activate the user-yaff package

    when @20190228:20210527
      user-adios [false]            false, true
          Activate the user-adios package

    when @20200319:20210527
      user-mesodpd [false]          false, true
          Activate the user-mesodpd package
      user-reaction [false]         false, true
          Activate the user-reaction package

    when @20200615:20210527
      user-mesont [false]           false, true
          Activate the user-mesont package

    when @20200630:20210527
      mliap [false]                 false, true
          Activate the mliap package

    when @20210408:
      plugin [false]                false, true
          Activate the plugin package

    when @20210514:20210527
      user-brownian [false]         false, true
          Activate the user-brownian package

    when @20210527
      user-hdnnp [false]            false, true
          Activate the user-hdnnp package
      user-rann [false]             false, true
          Activate the user-rann package
      user-vtk [false]              false, true
          Activate the user-vtk package

    when @20210702:20220623
      cg-sdk [false]                false, true
          Activate the cg-sdk package

    when @20210702:
      adios [false]                 false, true
          Activate the adios package
      atc [false]                   false, true
          Activate the atc package
      awpmd [false]                 false, true
          Activate the awpmd package
      bocs [false]                  false, true
          Activate the bocs package
      brownian [false]              false, true
          Activate the brownian package
      cg-dna [false]                false, true
          Activate the cg-dna package
      colvars [false]               false, true
          Activate the colvars package
      dielectric [false]            false, true
          Activate the dielectric package
      diffraction [false]           false, true
          Activate the diffraction package
      dpd-basic [false]             false, true
          Activate the dpd-basic package
      dpd-meso [false]              false, true
          Activate the dpd-meso package
      dpd-react [false]             false, true
          Activate the dpd-react package
      dpd-smooth [false]            false, true
          Activate the dpd-smooth package
      drude [false]                 false, true
          Activate the drude package
      eff [false]                   false, true
          Activate the eff package
      fep [false]                   false, true
          Activate the fep package
      h5md [false]                  false, true
          Activate the h5md package
      intel [false]                 false, true
          Activate the intel package
      latboltz [false]              false, true
          Activate the latboltz package
      machdyn [false]               false, true
          Activate the machdyn package
      manifold [false]              false, true
          Activate the manifold package
      mesont [false]                false, true
          Activate the mesont package
      mgpt [false]                  false, true
          Activate the mgpt package
      ml-hdnnp [false]              false, true
          Activate the ml-hdnnp package
      ml-iap [false]                false, true
          Activate the ml-iap package
      ml-rann [false]               false, true
          Activate the ml-rann package
      ml-snap [false]               false, true
          Activate the ml-snap package
      mofff [false]                 false, true
          Activate the mofff package
      molfile [false]               false, true
          Activate the molfile package
      netcdf [false]                false, true
          Activate the netcdf package
      phonon [false]                false, true
          Activate the phonon package
      plumed [false]                false, true
          Activate the plumed package
      ptm [false]                   false, true
          Activate the ptm package
      qtb [false]                   false, true
          Activate the qtb package
      reaction [false]              false, true
          Activate the reaction package
      reaxff [false]                false, true
          Activate the reaxff package
      smtbq [false]                 false, true
          Activate the smtbq package
      sph [false]                   false, true
          Activate the sph package
      tally [false]                 false, true
          Activate the tally package
      uef [false]                   false, true
          Activate the uef package
      vtk [false]                   false, true
          Activate the vtk package
      yaff [false]                  false, true
          Activate the yaff package

    when @20210728:
      extra-compute [false]         false, true
          Activate the extra-compute package
      extra-dump [false]            false, true
          Activate the extra-dump package
      extra-fix [false]             false, true
          Activate the extra-fix package
      extra-molecule [false]        false, true
          Activate the extra-molecule package
      extra-pair [false]            false, true
          Activate the extra-pair package
      interlayer [false]            false, true
          Activate the interlayer package
      orient [false]                false, true
          Activate the orient package

    when @20220504:
      bpm [false]                   false, true
          Activate the bpm package
      electrode [false]             false, true
          Activate the electrode package

    when @20220803:
      amoeba [false]                false, true
          Activate the amoeba package
      cg-spica [false]              false, true
          Activate the cg-spica package

    when @20221222:
      ml-pod [false]                false, true
          Activate the ml-pod package

    when @20230208:
      lepton [false]                false, true
          Activate the lepton package

Build Dependencies:
    adios2  cmake  curl   ffmpeg    gmake  hip     hipfft        jpeg     kokkos  latte   llvm-amdgpu  n2p2      ninja   plumed    py-cython  py-numpy  py-setuptools  python       rocm-openmp-extras  vtk
    blas    cuda   eigen  fftw-api  hdf5   hipcub  hsa-rocr-dev  kim-api  lapack  libpng  mpi          netcdf-c  opencl  py-build  py-mpi4py  py-pip    py-wheel       python-venv  voropp

Link Dependencies:
    adios2  cmake  curl   ffmpeg    hdf5  hipcub  hsa-rocr-dev  kim-api  lapack  libpng       mpi   netcdf-c  plumed  voropp
    blas    cuda   eigen  fftw-api  hip   hipfft  jpeg          kokkos   latte   llvm-amdgpu  n2p2  opencl    python  vtk

Run Dependencies:
    py-mpi4py  py-numpy  py-setuptools  python  python-venv

Licenses:
    GPL-2.0-only

Installing specific variants

LAMMPS has an enormous number of different variants and accelerators. If you aren't very familiar with LAMMPS in general, it's highly recommended to read this HPC Carpentry tutorial which goes into a great amount of detail about the most popular variants and accelerators.

We can't possibly cover all possible variants, so we'll focus on just two below:

Activate the KOKKOS LAMMPS package

Add a suffix for your private KOKKOS module

By default, the fact that the KOKKOS LAMMPS package has been added won't be reflected in your private module name. We recommend adding:

lammps:
  suffixes:
    +kokkos: kokkos

to the tcl: section of your personal modules.yaml configuration file to help to differentiate. It is already added to the template modules.yaml file in the spack-config-template repository.

Let's look at installing a variant of LAMMPS that activates the KOKKOS package. The default for the kokkos variant is (false) (which appears in the output of Spack commands as ~).

Always run a spec command before install!

You should always check what is about to be installed using the spec command as detailed below and then move onto the install command.

Let's see if we can spec a personal installation of LAMMPS adding in the KOKKOS package:

spack spec output (click to expand) 
$ spack -C ${HOME}/spack-config-templates/0.22.0 spec lammps +kokkos ^openmpi
Input spec
--------------------------------
 -   lammps+kokkos
 -       ^openmpi

Concretized
--------------------------------
 -   lammps@20230802.3%gcc@12.2.0~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell~cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim+kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release fftw_precision=double generator=make lammps_sizes=smallbig arch=linux-rocky9-x86_64_v4
[^]      ^cmake@3.27.9%gcc@12.2.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky9-x86_64_v4
[^]          ^curl@8.7.1%gcc@12.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rocky9-x86_64_v4
[^]              ^nghttp2@1.57.0%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                  ^diffutils@3.10%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^openssl@3.3.0%gcc@12.2.0~docs+shared build_system=generic certs=mozilla arch=linux-rocky9-x86_64_v4
[^]                  ^ca-certificates-mozilla@2023-05-30%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[^]              ^perl@5.38.0%gcc@11.4.1+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-rocky9-x86_64_v4
[^]                  ^berkeley-db@18.1.40%gcc@11.4.1+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rocky9-x86_64_v4
[^]                  ^bzip2@1.0.8%gcc@11.4.1~debug~pic+shared build_system=generic arch=linux-rocky9-x86_64_v4
[^]                  ^gdbm@1.23%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                      ^readline@8.2%gcc@11.4.1 build_system=autotools patches=bbf97f1 arch=linux-rocky9-x86_64_v4
[^]              ^pkgconf@2.2.0%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]          ^ncurses@6.5%gcc@11.4.1~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-rocky9-x86_64_v4
[^]          ^zlib-ng@2.1.6%gcc@11.4.1+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^fftw@3.3.10%gcc@12.2.0~mpi~openmp~pfft_patches+shared build_system=autotools precision=double,float arch=linux-rocky9-x86_64_v4
[^]      ^gcc-runtime@12.2.0%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[e]      ^glibc@2.34%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^gmake@4.4.1%gcc@11.4.1~guile build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^gcc-runtime@11.4.1%gcc@11.4.1 build_system=generic arch=linux-rocky9-x86_64_v4
 -       ^kokkos@4.3.00%gcc@12.2.0~aggressive_vectorization~compiler_warnings~cuda~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning~wrapper build_system=cmake build_type=Release cxxstd=17 generator=make intel_gpu_arch=none arch=linux-rocky9-x86_64_v4
[e]      ^openmpi@5.0.3%gcc@12.2.0~atomics~cuda+gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-rocky9-x86_64_v4

Almost everything required to compile LAMMPS is already available in our central upstream ([^]) The only thing we need to compile is kokkos itself, as well as a new version of LAMMPS integrating KOKKOS. You should notice that the variant list for lammps contains +kokkos as well. Let's go ahead and install this variant:

spack install output (click to expand) 
$ spack -C ${HOME}/spack-config-templates/0.22.0 install -j ${NSLOTS} lammps +kokkos ^openmpi
[+] /usr (external glibc-2.34-xri56vcyzs7kkvakhoku3fefc46nw25y)
[+] /share/apps/rocky9/general/libs/openmpi/gcc/12.2.0/5.0.3 (external openmpi-5.0.3-f6rfqovydeohhtdzs6yl2qiro2hxltpm)
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/gcc-runtime/12.2.0-w77gg5r
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc-runtime/11.4.1-llid4hw
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/fftw/3.3.10-tra2c2j
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/nghttp2/1.57.0-4ntqcoo
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/ncurses/6.5-4n2uzha
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/zlib-ng/2.1.6-g2yruc3
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gmake/4.4.1-xchit5a
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/openssl/3.3.0-filwsx6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/curl/8.7.1-v4za2y6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cmake/3.27.9-o3vtj26
==> Installing kokkos-4.3.00-6dnhfd2htczo3lzn5sy6zfhq67lpq6wx [13/14]
==> No binary for kokkos-4.3.00-6dnhfd2htczo3lzn5sy6zfhq67lpq6wx found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/53/53cf30d3b44dade51d48efefdaee7a6cf109a091b702a443a2eda63992e5fe0d.tar.gz
==> No patches needed for kokkos
==> kokkos: Executing phase: 'cmake'
==> kokkos: Executing phase: 'build'
==> kokkos: Executing phase: 'install'
==> kokkos: Successfully installed kokkos-4.3.00-6dnhfd2htczo3lzn5sy6zfhq67lpq6wx
  Stage: 1.67s.  Cmake: 2.08s.  Build: 7.58s.  Install: 2.24s.  Post-install: 1.01s.  Total: 15.55s
[+] /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/kokkos/4.3.00-6dnhfd2
==> Installing lammps-20230802.3-6leubbszycwxf4tnuuo2he4cvzi2ckle [14/14]
==> No binary for lammps-20230802.3-6leubbszycwxf4tnuuo2he4cvzi2ckle found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/66/6666e28cb90d3ff01cbbda6c81bdb85cf436bbb41604a87f2ab2fa559caa8510.tar.gz
==> No patches needed for lammps
==> lammps: Executing phase: 'cmake'
==> lammps: Executing phase: 'build'
==> lammps: Executing phase: 'install'
==> lammps: Successfully installed lammps-20230802.3-6leubbszycwxf4tnuuo2he4cvzi2ckle
  Stage: 8.31s.  Cmake: 6.14s.  Build: 1m 33.26s.  Install: 2.08s.  Post-install: 0.85s.  Total: 1m 51.31s
[+] /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-6leubbs

To break down what has happened above:

  • The existing available upstream dependencies have been used from the central location as opposed to being re-installed personally
  • Spack has noticed that kokkos and lammps are missing and has thus pulled down source code tarballs to the source_cache directory defined in config.yaml and then manually compiled and installed them to the location defined as install_tree: root: in config.yaml
  • Spack has then compiled and installed LAMMPS with the KOKKOS LAMMPS package activated

Let's now return to our spack find command:

spack find output 
$ spack -C ${HOME}/spack-config-templates/0.22.0 find -x -p -v lammps
-- linux-rocky9-x86_64_v4 / gcc@12.2.0 --------------------------
lammps@20230802.3~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell~cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim~kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release fftw_precision=double generator=make gpu_precision=mixed lammps_sizes=smallbig                     /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-b27kssh
lammps@20230802.3~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell~cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim+kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release fftw_precision=double generator=make lammps_sizes=smallbig                                         /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-6leubbs
lammps@20230802.3~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell+cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim~kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release cuda_arch=70,80,90 fftw_precision=double generator=make gpu_precision=mixed lammps_sizes=smallbig  /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-zqtg4sr
==> 3 installed packages

So now we have LAMMPS with KOKKOS activated installed alongside the two existing centrally installed versions.

If we now add our specified private module path using module use as specified in modules.yaml:

module use /data/scratch/${USER}/spack/privatemodules/linux-rocky9-x86_64_v4

Then we should see our personal version available for use:

Output from module avail 
$ module avail -l lammps
- Package/Alias -----------------------.- Versions --------
/data/scratch/abc123/spack/privatemodules/linux-rocky9-x86_64_v4:
lammps/20230802.3-openmpi-5.0.3-gcc-12.2.0-kokkos

/share/apps/rocky9/environmentmodules/apocrita-modules/spack:
lammps-gpu/20230802.3-openmpi-5.0.3-cuda-12.4.0-gcc-12.2.0
lammps/20230802.3-openmpi-5.0.3-gcc-12.2.0

And indeed, we can load and use our personal version:

Output from module load (click to expand) 
$ module load lammps/20230802.3-openmpi-5.0.3-gcc-12.2.0-kokkos
Loading lammps/20230802.3-openmpi-5.0.3-gcc-12.2.0-kokkos
  Loading requirement: openmpi/5.0.3-gcc-12.2.0
$ lmp -help

Large-scale Atomic/Molecular Massively Parallel Simulator - 2 Aug 2023 - Update 3

Usage example: lmp -var t 300 -echo screen -in in.alloy

List of command line options supported by this LAMMPS executable:

-echo none/screen/log/both  : echoing of input script (-e)
-help                       : print this help message (-h)
-in none/filename           : read input from file or stdin (default) (-i)
-kokkos on/off ...          : turn KOKKOS mode on or off (-k)
-log none/filename          : where to send log output (-l)
-mdi '<mdi flags>'          : pass flags to the MolSSI Driver Interface
-mpicolor color             : which exe in a multi-exe mpirun cmd (-m)
-cite                       : select citation reminder style (-c)
-nocite                     : disable citation reminder (-nc)
-nonbuf                     : disable screen/logfile buffering (-nb)
-package style ...          : invoke package command (-pk)
-partition size1 size2 ...  : assign partition sizes (-p)
-plog basename              : basename for partition logs (-pl)
-pscreen basename           : basename for partition screens (-ps)
-restart2data rfile dfile ... : convert restart to data file (-r2data)
-restart2dump rfile dgroup dstyle dfile ...
                            : convert restart to dump file (-r2dump)
-reorder topology-specs     : processor reordering (-r)
-screen none/filename       : where to send screen output (-sc)
-skiprun                    : skip loops in run and minimize (-sr)
-suffix gpu/intel/opt/omp   : style suffix to apply (-sf)
-var varname value          : set index style variable (-v)

OS: Linux "Rocky Linux 9.4 (Blue Onyx)" 5.14.0-427.22.1.el9_4.x86_64 x86_64

Compiler: GNU C++ 12.2.0 with OpenMP 4.5
C++ standard: C++17
MPI v3.1: Open MPI v5.0.3, package: Open MPI abc123@ddy16 Distribution, ident: 5.0.3, repo rev: v5.0.3, Apr 08, 2024

Accelerator configuration:

KOKKOS package API: Serial
KOKKOS package precision: double
Kokkos library version: 4.3.0

Active compile time flags:

-DLAMMPS_GZIP
-DLAMMPS_SMALLBIG
sizeof(smallint): 32-bit
sizeof(imageint): 32-bit
sizeof(tagint):   32-bit
sizeof(bigint):   64-bit

Available compression formats:

Extension: .gz     Command: gzip
Extension: .bz2    Command: bzip2
Extension: .xz     Command: xz
Extension: .lzma   Command: xz
Extension: .lz4    Command: lz4


Installed packages:

KOKKOS KSPACE MANYBODY MOLECULE RIGID
(etc.)

We can see that KOKKOS appears in the listed "Accelerator configuration" and "Installed packages" sections.

Activate the KOKKOS LAMMPS package and CUDA

Use a GPU node for compilation and installation

At the time of writing, LAMMPS will not compile with CUDA unless there is a physical GPU (as well as CUDA drivers) present (see this GitHub issue).

Please ensure that you run any personal variant installations enabling CUDA support in an interactive qlogin session on a GPU node, otherwise you will see this error:

/bin/ld: warning: libcuda.so.1, needed by liblammps.so.0, not found (try using -rpath or -rpath-link)

How about if we wanted LAMMPS with the KOKKOS package activated but also GPU support via CUDA? It's possible, but you need to specify the right variant. Remember, Spack will install the default variant unless explicitly told otherwise. If you look again at the variants for LAMMPS, you will see that cuda is set to (false) by default. The centrally available lammps-gpu module loads a version that has explicitly been compiled with CUDA support.

Let's see if we can spec a personal installation of LAMMPS activating the KOKKOS package and CUDA:

Specify your cuda_arch

When installing a CUDA variant of a package from Spack, you must also specify a cuda_arch (sm_70 - Volta V100, sm_80 - Ampere A100, sm_90 - Hopper H100). Some packages, like Gromacs, let you install one variant that supports all three at once, but LAMMPS requires a separate install per cuda_arch required when also activating KOKKOS. Check the output of the spec command for more details.

spack spec output (click to expand) 
$ spack -C ${HOME}/spack-config-templates/0.22.0 spec lammps +cuda cuda_arch=80 +kokkos ^openmpi
Input spec
--------------------------------
 -   lammps+cuda+kokkos cuda_arch=80
 -       ^openmpi

Concretized
--------------------------------
 -   lammps@20230802.3%gcc@12.2.0~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell+cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim+kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release cuda_arch=80 fftw_precision=double generator=make lammps_sizes=smallbig arch=linux-rocky9-x86_64_v4
[^]      ^cmake@3.27.9%gcc@12.2.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky9-x86_64_v4
[^]          ^curl@8.7.1%gcc@12.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rocky9-x86_64_v4
[^]              ^nghttp2@1.57.0%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                  ^diffutils@3.10%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^openssl@3.3.0%gcc@12.2.0~docs+shared build_system=generic certs=mozilla arch=linux-rocky9-x86_64_v4
[^]                  ^ca-certificates-mozilla@2023-05-30%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[^]              ^perl@5.38.0%gcc@11.4.1+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-rocky9-x86_64_v4
[^]                  ^berkeley-db@18.1.40%gcc@11.4.1+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rocky9-x86_64_v4
[^]                  ^bzip2@1.0.8%gcc@11.4.1~debug~pic+shared build_system=generic arch=linux-rocky9-x86_64_v4
[^]                  ^gdbm@1.23%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                      ^readline@8.2%gcc@11.4.1 build_system=autotools patches=bbf97f1 arch=linux-rocky9-x86_64_v4
[^]              ^pkgconf@2.2.0%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]          ^ncurses@6.5%gcc@11.4.1~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-rocky9-x86_64_v4
[^]          ^zlib-ng@2.1.6%gcc@11.4.1+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^cuda@12.4.0%gcc@12.2.0~allow-unsupported-compilers~dev build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^libxml2@2.10.3%gcc@11.4.1+pic~python+shared build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^xz@5.4.6%gcc@11.4.1~pic build_system=autotools libs=shared,static arch=linux-rocky9-x86_64_v4
[^]      ^fftw@3.3.10%gcc@12.2.0~mpi~openmp~pfft_patches+shared build_system=autotools precision=double,float arch=linux-rocky9-x86_64_v4
[^]      ^gcc-runtime@12.2.0%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[e]      ^glibc@2.34%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^gmake@4.4.1%gcc@11.4.1~guile build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^gcc-runtime@11.4.1%gcc@11.4.1 build_system=generic arch=linux-rocky9-x86_64_v4
 -       ^kokkos@4.3.00%gcc@12.2.0~aggressive_vectorization~compiler_warnings+cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning~wrapper build_system=cmake build_type=Release cuda_arch=80 cxxstd=17 generator=make intel_gpu_arch=none arch=linux-rocky9-x86_64_v4
[e]      ^openmpi@5.0.3%gcc@12.2.0~atomics~cuda+gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-rocky9-x86_64_v4

So a similar output to the kokkos variant above, but adding in CUDA (which comes from upstream). Again, almost everything required to compile LAMMPS is already available in our central upstream ([^]). A +cuda variant of kokkos needs to be compiled, as well as a new version of lammps that adds both kokkos and cuda support. You should notice that the variant list for lammps contains +cuda, +kokkos and cuda_arch=80 as well. So, as before, let's go ahead and install our personal variant:

spack install output (click to expand) 
$ spack -C ${HOME}/spack-config-templates/0.22.0 install -j ${NSLOTS} lammps +cuda cuda_arch=80 +kokkos ^openmpi
[+] /usr (external glibc-2.34-xri56vcyzs7kkvakhoku3fefc46nw25y)
[+] /share/apps/rocky9/general/libs/openmpi/gcc/12.2.0/5.0.3 (external openmpi-5.0.3-f6rfqovydeohhtdzs6yl2qiro2hxltpm)
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/gcc-runtime/12.2.0-w77gg5r
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc-runtime/11.4.1-llid4hw
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/fftw/3.3.10-tra2c2j
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/nghttp2/1.57.0-4ntqcoo
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gmake/4.4.1-xchit5a
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/xz/5.4.6-rwn7pno
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/zlib-ng/2.1.6-g2yruc3
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/ncurses/6.5-4n2uzha
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/openssl/3.3.0-filwsx6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/libxml2/2.10.3-q6zmsq6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/curl/8.7.1-v4za2y6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cuda/12.4.0-b2w3zo5
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cmake/3.27.9-o3vtj26
==> Installing kokkos-4.3.00-lz47vrsl22clnzamkuxqeyk6iwqweshp [16/17]
==> No binary for kokkos-4.3.00-lz47vrsl22clnzamkuxqeyk6iwqweshp found: installing from source
==> Using cached archive: /data/scratch/abc123/spack/cache/_source-cache/archive/53/53cf30d3b44dade51d48efefdaee7a6cf109a091b702a443a2eda63992e5fe0d.tar.gz
==> No patches needed for kokkos
==> kokkos: Executing phase: 'cmake'
==> Error: InstallError: Kokkos requires +wrapper when using +cudawithout clang

/share/apps/rocky9/general/apps/spack/0.22.0/var/spack/repos/builtin/packages/kokkos/package.py:300, in cmake_args:
        297        if spec.satisfies("~wrapper+cuda") and not (
        298            spec.satisfies("%clang") or spec.satisfies("%cce")
        299        ):
  >>    300            raise InstallError("Kokkos requires +wrapper when using +cuda" "without clang")
        301
        302        options = [
        303            from_variant("CMAKE_POSITION_INDEPENDENT_CODE", "pic"),

See build log for details:
  /tmp/1234567.1.all.q/abc123/spack-stage/spack-stage-kokkos-4.3.00-lz47vrsl22clnzamkuxqeyk6iwqweshp/spack-build-out.txt

==> Warning: Skipping build of lammps-20230802.3-vq63svce7rp4dvt6n2mdvji5mr4jym35 since kokkos-4.3.00-lz47vrsl22clnzamkuxqeyk6iwqweshp failed
==> Error: lammps-20230802.3-vq63svce7rp4dvt6n2mdvji5mr4jym35: Package was not installed
==> Error: Installation request failed.  Refer to reported errors for failing package(s).

Oh dear! But Spack has given us a clue. If we look at the Spack Packages page for KOKKOS:

https://packages.spack.io/package.html?name=kokkos

You can see:

KOKKOS nvcc-wrapper

So, we need to add ^kokkos+wrapper as well as +kokkos to our spec and install commands. Let's check that in a new spec command:

spack spec output (click to expand) 
$ spack -C ${HOME}/spack-config-templates/0.22.0 spec lammps +cuda cuda_arch=80 +kokkos ^kokkos+wrapper ^openmpi
Input spec
--------------------------------
 -   lammps+cuda+kokkos cuda_arch=80
 -       ^kokkos+wrapper
 -       ^openmpi

Concretized
--------------------------------
 -   lammps@20230802.3%gcc@12.2.0~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell+cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim+kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release cuda_arch=80 fftw_precision=double generator=make lammps_sizes=smallbig arch=linux-rocky9-x86_64_v4
[^]      ^cmake@3.27.9%gcc@12.2.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky9-x86_64_v4
[^]          ^curl@8.7.1%gcc@12.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rocky9-x86_64_v4
[^]              ^nghttp2@1.57.0%gcc@12.2.0 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                  ^diffutils@3.10%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^openssl@3.3.0%gcc@12.2.0~docs+shared build_system=generic certs=mozilla arch=linux-rocky9-x86_64_v4
[^]                  ^ca-certificates-mozilla@2023-05-30%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[^]              ^perl@5.38.0%gcc@11.4.1+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-rocky9-x86_64_v4
[^]                  ^berkeley-db@18.1.40%gcc@11.4.1+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rocky9-x86_64_v4
[^]                  ^bzip2@1.0.8%gcc@11.4.1~debug~pic+shared build_system=generic arch=linux-rocky9-x86_64_v4
[^]                  ^gdbm@1.23%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]                      ^readline@8.2%gcc@11.4.1 build_system=autotools patches=bbf97f1 arch=linux-rocky9-x86_64_v4
[^]              ^pkgconf@2.2.0%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]          ^ncurses@6.5%gcc@11.4.1~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-rocky9-x86_64_v4
[^]          ^zlib-ng@2.1.6%gcc@11.4.1+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^cuda@12.4.0%gcc@12.2.0~allow-unsupported-compilers~dev build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^libxml2@2.10.3%gcc@11.4.1+pic~python+shared build_system=autotools arch=linux-rocky9-x86_64_v4
[^]              ^xz@5.4.6%gcc@11.4.1~pic build_system=autotools libs=shared,static arch=linux-rocky9-x86_64_v4
[^]      ^fftw@3.3.10%gcc@12.2.0~mpi~openmp~pfft_patches+shared build_system=autotools precision=double,float arch=linux-rocky9-x86_64_v4
[^]      ^gcc-runtime@12.2.0%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[e]      ^glibc@2.34%gcc@11.4.1 build_system=autotools arch=linux-rocky9-x86_64_v4
[^]      ^gmake@4.4.1%gcc@11.4.1~guile build_system=generic arch=linux-rocky9-x86_64_v4
[^]          ^gcc-runtime@11.4.1%gcc@11.4.1 build_system=generic arch=linux-rocky9-x86_64_v4
 -       ^kokkos@4.3.00%gcc@12.2.0~aggressive_vectorization~compiler_warnings+cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning+wrapper build_system=cmake build_type=Release cuda_arch=80 cxxstd=17 generator=make intel_gpu_arch=none arch=linux-rocky9-x86_64_v4
 -           ^kokkos-nvcc-wrapper@4.1.00%gcc@12.2.0 build_system=generic arch=linux-rocky9-x86_64_v4
[e]      ^openmpi@5.0.3%gcc@12.2.0~atomics~cuda+gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-rocky9-x86_64_v4

It looks almost the same as our first spec command, but now ^kokkos+wrapper is listed as part of the "Input spec" and we are going to install the required kokkos-nvcc-wrapper dependency as well. Now, onto the install:

spack install output (click to expand) 
$ spack -C ${HOME}/spack-config-templates/0.22.0 install -j ${NSLOTS} lammps +cuda cuda_arch=80 +kokkos ^kokkos+wrapper ^openmpi
[+] /usr (external glibc-2.34-xri56vcyzs7kkvakhoku3fefc46nw25y)
[+] /share/apps/rocky9/general/libs/openmpi/gcc/12.2.0/5.0.3 (external openmpi-5.0.3-f6rfqovydeohhtdzs6yl2qiro2hxltpm)
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc-runtime/11.4.1-llid4hw
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/gcc-runtime/12.2.0-w77gg5r
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/ncurses/6.5-4n2uzha
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/xz/5.4.6-rwn7pno
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gmake/4.4.1-xchit5a
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/nghttp2/1.57.0-4ntqcoo
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/fftw/3.3.10-tra2c2j
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/zlib-ng/2.1.6-g2yruc3
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/openssl/3.3.0-filwsx6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/libxml2/2.10.3-q6zmsq6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/curl/8.7.1-v4za2y6
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cuda/12.4.0-b2w3zo5
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/cmake/3.27.9-o3vtj26
==> Installing kokkos-nvcc-wrapper-4.1.00-cjcrn7ia7zbxie6aqcsedfigcg6knuwh [16/18]
==> No binary for kokkos-nvcc-wrapper-4.1.00-cjcrn7ia7zbxie6aqcsedfigcg6knuwh found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/cf/cf725ea34ba766fdaf29c884cfe2daacfdc6dc2d6af84042d1c78d0f16866275.tar.gz
==> No patches needed for kokkos-nvcc-wrapper
==> kokkos-nvcc-wrapper: Executing phase: 'install'
==> kokkos-nvcc-wrapper: Successfully installed kokkos-nvcc-wrapper-4.1.00-cjcrn7ia7zbxie6aqcsedfigcg6knuwh
  Stage: 0.17s.  Install: 0.00s.  Post-install: 0.33s.  Total: 0.82s
[+] /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/kokkos-nvcc-wrapper/4.1.00-cjcrn7i
==> Installing kokkos-4.3.00-2hk2xsxfnbaxit3q3pctj6fxa2abr4as [17/18]
==> No binary for kokkos-4.3.00-2hk2xsxfnbaxit3q3pctj6fxa2abr4as found: installing from source
==> Using cached archive: /data/scratch/abc123/spack/cache/_source-cache/archive/53/53cf30d3b44dade51d48efefdaee7a6cf109a091b702a443a2eda63992e5fe0d.tar.gz
==> No patches needed for kokkos
==> kokkos: Executing phase: 'cmake'
==> kokkos: Executing phase: 'build'
==> kokkos: Executing phase: 'install'
==> kokkos: Successfully installed kokkos-4.3.00-2hk2xsxfnbaxit3q3pctj6fxa2abr4as
  Stage: 0.07s.  Cmake: 5.37s.  Build: 14.77s.  Install: 1.14s.  Post-install: 0.35s.  Total: 22.09s
[+] /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/kokkos/4.3.00-2hk2xsx
==> Installing lammps-20230802.3-mavmhn4dcnfgbbwk4x2r4whq2z3uvyjp [18/18]
==> No binary for lammps-20230802.3-mavmhn4dcnfgbbwk4x2r4whq2z3uvyjp found: installing from source
==> Using cached archive: /data/scratch/abc123/spack/cache/_source-cache/archive/66/6666e28cb90d3ff01cbbda6c81bdb85cf436bbb41604a87f2ab2fa559caa8510.tar.gz
==> No patches needed for lammps
==> lammps: Executing phase: 'cmake'
==> lammps: Executing phase: 'build'
==> lammps: Executing phase: 'install'
==> lammps: Successfully installed lammps-20230802.3-mavmhn4dcnfgbbwk4x2r4whq2z3uvyjp
  Stage: 2.23s.  Cmake: 3.44s.  Build: 5m 33.99s.  Install: 2.38s.  Post-install: 0.78s.  Total: 5m 43.53s
[+] /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-mavmhn4

This time the installation should complete without any further errors, compiling kokkos-nvcc-wrapper, kokkos and then lammps.

Let's now return to our spack find command:

spack find output 
$ spack -C ${HOME}/spack-config-templates/0.22.0 find -x -p -v lammps
-- linux-rocky9-x86_64_v4 / gcc@12.2.0 --------------------------
lammps@20230802.3~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell~cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim~kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release fftw_precision=double generator=make gpu_precision=mixed lammps_sizes=smallbig                     /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-b27kssh
lammps@20230802.3~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell~cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim+kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release fftw_precision=double generator=make lammps_sizes=smallbig                                         /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-6leubbs
lammps@20230802.3~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell+cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim~kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release cuda_arch=70,80,90 fftw_precision=double generator=make gpu_precision=mixed lammps_sizes=smallbig  /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-zqtg4sr
lammps@20230802.3~adios~amoeba~asphere~atc~awpmd~bocs~body~bpm~brownian~cg-dna~cg-spica~class2~colloid~colvars~compress~coreshell+cuda~cuda_mps~dielectric~diffraction~dipole~dpd-basic~dpd-meso~dpd-react~dpd-smooth~drude~eff~electrode~extra-compute~extra-dump~extra-fix~extra-molecule~extra-pair~fep~ffmpeg~granular~h5md~intel~interlayer~ipo~jpeg~kim+kokkos+kspace~latboltz~lepton+lib~machdyn~manifold+manybody~mc~meam~mesont~mgpt~misc~ml-hdnnp~ml-iap~ml-pod~ml-rann~ml-snap~mofff+molecule~molfile+mpi~netcdf~opencl+openmp~openmp-package~opt~orient~peri~phonon~plugin~plumed~png~poems~ptm~python~qeq~qtb~reaction~reaxff~replica+rigid~rocm~shock~smtbq~sph~spin~srd~tally~uef~voronoi~vtk~yaff build_system=cmake build_type=Release cuda_arch=80 fftw_precision=double generator=make lammps_sizes=smallbig                            /data/scratch/abc123/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/lammps/20230802.3-openmpi-5.0.3-mavmhn4
==> 4 installed packages

So now we have LAMMPS with the KOKKOS package activated (+kokkos) and LAMMPS with the KOKKOS package activated (+kokkos) and CUDA (+cuda cuda_arch=80) personally, alongside the two centrally installed non-KOKKOS versions.

As long as we still have our specified private module path loaded using module use as specified in modules.yaml:

module use /data/scratch/${USER}/spack/privatemodules/linux-rocky9-x86_64_v4

Then we should see our personal versions available for use:

Output from module avail 
$ module avail -l lammps
- Package/Alias -----------------------.- Versions --------
/data/scratch/abc123/spack/privatemodules/linux-rocky9-x86_64_v4:
lammps-gpu/20230802.3-openmpi-5.0.3-cuda-12.4.0-gcc-12.2.0-kokkos
lammps/20230802.3-openmpi-5.0.3-gcc-12.2.0-kokkos

/share/apps/rocky9/environmentmodules/apocrita-modules/spack:
lammps-gpu/20230802.3-openmpi-5.0.3-cuda-12.4.0-gcc-12.2.0
lammps/20230802.3-openmpi-5.0.3-gcc-12.2.0

And indeed, we can load and use our personal version:

Use a GPU node for execution

As with compilation and installation. any GPU version of LAMMPS must also be executed on a GPU node, otherwise you will see the following error message as there will be no CUDA libraries present on the node:

lmp: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory
Output from module load (click to expand) 
$ module load lammps-gpu/20230802.3-openmpi-5.0.3-cuda-12.4.0-gcc-12.2.0-kokkos
Loading lammps-gpu/20230802.3-openmpi-5.0.3-cuda-12.4.0-gcc-12.2.0-kokkos
  Loading requirement: cuda/12.4.0-gcc-12.2.0 openmpi/5.0.3-gcc-12.2.0
$ lmp -help

Large-scale Atomic/Molecular Massively Parallel Simulator - 2 Aug 2023 - Update 3

Usage example: lmp -var t 300 -echo screen -in in.alloy

List of command line options supported by this LAMMPS executable:

-echo none/screen/log/both  : echoing of input script (-e)
-help                       : print this help message (-h)
-in none/filename           : read input from file or stdin (default) (-i)
-kokkos on/off ...          : turn KOKKOS mode on or off (-k)
-log none/filename          : where to send log output (-l)
-mdi '<mdi flags>'          : pass flags to the MolSSI Driver Interface
-mpicolor color             : which exe in a multi-exe mpirun cmd (-m)
-cite                       : select citation reminder style (-c)
-nocite                     : disable citation reminder (-nc)
-nonbuf                     : disable screen/logfile buffering (-nb)
-package style ...          : invoke package command (-pk)
-partition size1 size2 ...  : assign partition sizes (-p)
-plog basename              : basename for partition logs (-pl)
-pscreen basename           : basename for partition screens (-ps)
-restart2data rfile dfile ... : convert restart to data file (-r2data)
-restart2dump rfile dgroup dstyle dfile ...
                            : convert restart to dump file (-r2dump)
-reorder topology-specs     : processor reordering (-r)
-screen none/filename       : where to send screen output (-sc)
-skiprun                    : skip loops in run and minimize (-sr)
-suffix gpu/intel/opt/omp   : style suffix to apply (-sf)
-var varname value          : set index style variable (-v)

OS: Linux "Rocky Linux 9.4 (Blue Onyx)" 5.14.0-427.22.1.el9_4.x86_64 x86_64

Compiler: GNU C++ 12.2.0 with OpenMP 4.5
C++ standard: C++17
MPI v3.1: Open MPI v5.0.3, package: Open MPI abc123@ddy16 Distribution, ident: 5.0.3, repo rev: v5.0.3, Apr 08, 2024

Accelerator configuration:

KOKKOS package API: CUDA Serial
KOKKOS package precision: double
Kokkos library version: 4.3.0

Active compile time flags:

-DLAMMPS_GZIP
-DLAMMPS_SMALLBIG
sizeof(smallint): 32-bit
sizeof(imageint): 32-bit
sizeof(tagint):   32-bit
sizeof(bigint):   64-bit

Available compression formats:

Extension: .gz     Command: gzip
Extension: .bz2    Command: bzip2
Extension: .xz     Command: xz
Extension: .lzma   Command: xz


Installed packages:

KOKKOS KSPACE MANYBODY MOLECULE RIGID
(etc.)

We can see that KOKKOS appears in the listed "Accelerator configuration" and "Installed packages" sections as before, but this time using the "CUDA Serial" KOKKOS package API.

Summary

The above examples show just two of the many possible variants of LAMMPS that you can compile yourself. To compile additional variants, change the required parameters to (true) with + or (false) with ~ in spec and install commands, and remember Spack will always follow the defaults unless explicitly told otherwise.

You could, for example, spec and install a much more complex variant of LAMMPS such as:

lammps +adios +cuda cuda_arch=80 +jpeg +kokkos +plumed ^kokkos+wrapper ^openmpi

which would compile with the Adios package activated, GPU support, JPEG support, the KOKKOS package activated with CUDA support and the PLUMED package activated. The more complex the variant, the less is likely to be already in upstream, so bear this in mind, as compilation and installation time may increase as well.