Skip to content

Python example

Be wary of Spack py-* packages

Whilst Spack does package a lot of Python packages (for example, numpy), they are really designed to be used as part of a dependency chain for Spack-installed applications. They are also limited to the versions available in the specific release of Spack you are using. We recommend using pip install commands in a Spack environment (or alternatively not using Spack at all, but using Python virtualenvs or Conda environments) instead of installing py-* Spack packages directly.

Using Python packages in Spack environments is only applicable to very specific use cases. The pros and cons of different methods are covered in the official Spack documentation.

Let's take a look at a fairly simple example for providing the libsndfile dependency for the sndfile PyPi Python package.

The is no module for libsndfile on Apocrita, but even if we install a personal version of libsndfile using our custom configuration scope and generate a private module, we get stuck:

sndfile Private Module example (click to expand) 
$ module load gcc/12.2.0 python/3.12.1-gcc-12.2.0 libsndfile/1.2.2-gcc-12.2.0
Loading python/3.12.1-gcc-12.2.0
  Loading requirement: openssl/3.3.0-gcc-12.2.0 sqlite/3.43.2-gcc-12.2.0
$ virtualenv venv
created virtual environment CPython3.12.1.final.0-64 in 2905ms
  creator CPython3Posix(dest=/tmp/5928758.1.all.q/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/data/home/abc123/.local/share/virtualenv)
added seed packages: pip==25.0.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
$ source venv/bin/activate
(venv) $ pip install sndfile
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting sndfile
  Downloading sndfile-0.2.0.tar.gz (4.3 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting cffi>=1.0.0 (from sndfile)
  Downloading cffi-1.17.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting pycparser (from cffi>=1.0.0->sndfile)
  Downloading pycparser-2.22-py3-none-any.whl.metadata (943 bytes)
Downloading cffi-1.17.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (479 kB)
Downloading pycparser-2.22-py3-none-any.whl (117 kB)
Building wheels for collected packages: sndfile
  Building wheel for sndfile (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for sndfile (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [33 lines of output]
  /tmp/5928758.1.all.q/pip-build-env-9_lmxclj/overlay/lib/python3.12/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
  !!

      ********************************************************************************
      Please consider removing the following classifiers in favor of a SPDX license expression:

      License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)

      See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
      ********************************************************************************

  !!
    self._finalize_license_expression()
  running bdist_wheel
  running build
  running build_py
  creating build/lib.linux-x86_64-cpython-312/sndfile
  copying sndfile/__init__.py -> build/lib.linux-x86_64-cpython-312/sndfile
  copying sndfile/build.py -> build/lib.linux-x86_64-cpython-312/sndfile
  copying sndfile/formats.py -> build/lib.linux-x86_64-cpython-312/sndfile
  copying sndfile/io.py -> build/lib.linux-x86_64-cpython-312/sndfile
  copying sndfile/vio.py -> build/lib.linux-x86_64-cpython-312/sndfile
  running build_ext
  generating cffi module 'build/temp.linux-x86_64-cpython-312/sndfile._sndfile.c'
  creating build/temp.linux-x86_64-cpython-312
  building 'sndfile._sndfile' extension
  creating build/temp.linux-x86_64-cpython-312/build/temp.linux-x86_64-cpython-312
  /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc/12.2.0-6frskzg/bin/gcc -fno-strict-overflow -DNDEBUG -g -O3 -Wall -fPIC -fPIC -I/tmp/5928758.1.all.q/venv/include -I/share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/python/3.12.1-sd2mpei/include/python3.12 -c build/temp.linux-x86_64-cpython-312/sndfile._sndfile.c -o build/temp.linux-x86_64-cpython-312/build/temp.linux-x86_64-cpython-312/sndfile._sndfile.o
  build/temp.linux-x86_64-cpython-312/sndfile._sndfile.c:575:10: fatal error: sndfile.h: No such file or directory
    575 | #include <sndfile.h>
    |          ^~~~~~~~~~~
  compilation terminated.
  error: command '/share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc/12.2.0-6frskzg/bin/gcc' failed with exit code 1
  [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for sndfile
Failed to build sndfile

[notice] A new release of pip is available: 25.0.1 -> 25.1.1
[notice] To update, run: pip install --upgrade pip
ERROR: Failed to build installable wheels for some pyproject.toml based projects (sndfile)

This is because Spack doesn't support traditional environment exports such as ACLOCAL_PATH, LD_LIBRARY_PATH, LIBRARY_PATH, C_INCLUDE_PATH and CPLUS_INCLUDE_PATH in its modules by design. Instead it compiles dependency paths during Spack installations via rpath. The University of Cambridge has a good explanatory post.

Whilst we could try and go down the route of manually exporting more traditional environment exports either by adding them into custom modules or just literally running export commands every time we need to compile something outside of Spack, it's much easier to use a Spack environment.