Skip to content

Python example

Be wary of Spack py-* packages

Whilst Spack does package a lot of Python packages (for example, numpy), they are really designed to be used as part of a dependency chain for Spack-installed applications. They are also limited to the versions available in the specific release of Spack you are using. We recommend using pip install commands in a Spack environment (or alternatively not using Spack at all, but using Python virtualenvs or Conda environments) instead of installing py-* Spack packages directly.

Using Python packages in Spack environments is only applicable to very specific use cases. The pros and cons of different methods are covered in the official Spack documentation.

Let's take a look at a fairly simple example for providing the libsndfile dependency for the sndfile PyPi Python package.

The is no module for libsndfile on Apocrita, but even if we install a personal version of libsndfile using our custom configuration scope and generate a private module, we get stuck:

sndfile Private Module example 
$ module load gcc/12.2.0 python/3.12.1-gcc-12.2.0 libsndfile/1.0.28-gcc-12.2.0
Loading python/3.12.1-gcc-12.2.0
  Loading requirement: openssl/3.3.0-gcc-12.2.0 sqlite/3.43.2-gcc-12.2.0
$ virtualenv venv
created virtual environment CPython3.12.1.final.0-64 in 2231ms
  creator CPython3Posix(dest=/tmp/4748513.1.all.q/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/data/home/abc123/.local/share/virtualenv)
    added seed packages: pip==23.2.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
$ source venv/bin/activate
(venv) $ pip install sndfile
Collecting sndfile
  Downloading sndfile-0.2.0.tar.gz (4.3 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting cffi>=1.0.0 (from sndfile)
  Obtaining dependency information for cffi>=1.0.0 from https://files.pythonhosted.org/packages/b2/d5/da47df7004cb17e4955df6a43d14b3b4ae77737dff8bf7f8f333196717bf/cffi-1.17.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Using cached cffi-1.17.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting pycparser (from cffi>=1.0.0->sndfile)
  Obtaining dependency information for pycparser from https://files.pythonhosted.org/packages/13/a3/a812df4e2dd5696d1f351d58b8fe16a405b234ad2886a0dab9183fb78109/pycparser-2.22-py3-none-any.whl.metadata
  Using cached pycparser-2.22-py3-none-any.whl.metadata (943 bytes)
Using cached cffi-1.17.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (479 kB)
Using cached pycparser-2.22-py3-none-any.whl (117 kB)
Building wheels for collected packages: sndfile
  Building wheel for sndfile (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for sndfile (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-312/sndfile
      copying sndfile/__init__.py -> build/lib.linux-x86_64-cpython-312/sndfile
      copying sndfile/build.py -> build/lib.linux-x86_64-cpython-312/sndfile
      copying sndfile/formats.py -> build/lib.linux-x86_64-cpython-312/sndfile
      copying sndfile/io.py -> build/lib.linux-x86_64-cpython-312/sndfile
      copying sndfile/vio.py -> build/lib.linux-x86_64-cpython-312/sndfile
      running build_ext
      generating cffi module 'build/temp.linux-x86_64-cpython-312/sndfile._sndfile.c'
      creating build/temp.linux-x86_64-cpython-312
      building 'sndfile._sndfile' extension
      creating build/temp.linux-x86_64-cpython-312/build/temp.linux-x86_64-cpython-312
      /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc/12.2.0-6frskzg/bin/gcc -fno-strict-overflow -DNDEBUG -g -O3 -Wall -fPIC -fPIC -I/tmp/4748513.1.all.q/venv/include -I/share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/python/3.12.1-sd2mpei/include/python3.12 -c build/temp.linux-x86_64-cpython-312/sndfile._sndfile.c -o build/temp.linux-x86_64-cpython-312/build/temp.linux-x86_64-cpython-312/sndfile._sndfile.o
      build/temp.linux-x86_64-cpython-312/sndfile._sndfile.c:575:10: fatal error: sndfile.h: No such file or directory
        575 | #include <sndfile.h>
            |          ^~~~~~~~~~~
      compilation terminated.
      error: command '/share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc/12.2.0-6frskzg/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for sndfile
Failed to build sndfile
ERROR: Could not build wheels for sndfile, which is required to install pyproject.toml-based projects

This is because Spack doesn't support traditional environment exports such as ACLOCAL_PATH, LD_LIBRARY_PATH, LIBRARY_PATH, C_INCLUDE_PATH and CPLUS_INCLUDE_PATH in its modules by design. Instead it compiles dependency paths during Spack installations via rpath. The University of Cambridge has a good explanatory post.

Whilst we could try and go down the route of manually exporting more traditional environment exports either by adding them into custom modules or just literally running export commands every time we need to compile something outside of Spack, it's much easier to use a Spack environment.