Support loading env settings from environment modules

[tiran]

Linux distributions are using environment modules to provide dynamic environment variables. For example Fedora supports multiple implementations of MPI. It uses environment modules to configure a build environment for a specific implementation, e.g. mpi/openmpi-x86_64 for OpenMPI on arch x86_64.

Fromager should gain an option to set extra environment variables from a module providers. It's going to be a bit annoying. The module provider is designed for shells.

• modulecmd is provided by package environment-modules on Fedora and CentOS Stream
• The command needs env var MODULESHOME and MODULEPATH. On c9s, the settings are

  MODULESHOME=/usr/share/Modules
  MODULEPATH=/etc/scl/modulefiles:/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles

• The output of modulecmd is designed to be sources by a shell. cmd output looks easiest to parse:

  $ modulecmd cmd load mpi/openmpi-$(arch)
  set MPI_COMPILER=openmpi-x86_64
  set LD_LIBRARY_PATH=/usr/lib64/openmpi/lib
  set MANPATH=/usr/share/man/openmpi-x86_64:/usr/share/man:
  set MPI_INCLUDE=/usr/include/openmpi-x86_64
  set MPI_MAN=/usr/share/man/openmpi-x86_64
  set _LMFILES_=/usr/share/modulefiles/mpi/openmpi-x86_64
  set LOADEDMODULES=mpi/openmpi-x86_64
  set PKG_CONFIG_PATH=/usr/lib64/openmpi/lib/pkgconfig
  set MPI_PYTHON3_SITEARCH=/usr/lib64/python3.9/site-packages/openmpi
  set MPI_HOME=/usr/lib64/openmpi
  set MPI_FORTRAN_MOD_DIR=/usr/lib64/gfortran/modules/openmpi
  set __MODULES_LMALTNAME=mpi/openmpi-x86_64&as|mpi/default&as|mpi/latest
  set __MODULES_LMCONFLICT=mpi/openmpi-x86_64&mpi
  set __MODULES_SHARE_MANPATH=:2
  set MPI_SUFFIX=_openmpi
  set MPI_SYSCONFIG=/etc/openmpi-x86_64
  set MPI_LIB=/usr/lib64/openmpi/lib
  set PATH=/usr/lib64/openmpi/bin:/home/cheimes/.local/bin:/home/cheimes/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
  set MPI_BIN=/usr/lib64/openmpi/bin
  set errorlevel=0
  

• The module names are arch-specific. We need to account for that.

[dhellmann]

Why should fromager add direct support for this, instead of asking the user to activate the module(s) before invoking fromager?

[tiran]

Packages may need different env modules -- or no env module vars at all. The environment module system exists to configure a (development) environment for an implementation. For example Fedora has multiple implementations of MPI like OpenMPI and MPICH. Environment modules can be arch specific, too. Or they can differ between distributions and package versions.

For downstream, we could hard-code the env vars for OpenMPI in the container files. Over time, the settings are going to become more complicated and diverse.

[dhellmann]

So 2 different packages might need different env modules, and those modules might conflict with each other? So fromager needs to be able to activate them (or whatever) on a per-package case?

[tiran]

Yes, that's sums it up well.