Singularity, apptainer regression with envfile and shell evaluation

[antoinetran]

Issue:
Currently, when running an Argo POD to Slurm, which adds the whole Kubernetes YAML inside one environment variable named ARGO_TEMPLATE, the singularity cannot run the container because it tries to evaluate the environment variable as if in a shell.

Fix:
Singularity >= 3.10.0 (https://github.com/sylabs/singularity/blob/main/CHANGELOG.md#3100-2022-05-17) and Apptainer >= 1.1.0 (https://github.com/apptainer/apptainer/releases/tag/v1.1.0) brings an --no-eval . The purpose is:

Added a new action flag --no-eval which:

    Prevents shell evaluation of APPTAINERENV_ / --env / --env-file environment variables as they are injected in the container, to match OCI behavior. Applies to all containers.
    Prevents shell evaluation of the values of CMD / ENTRYPOINT and command line arguments for containers run or built directly from an OCI/Docker source. Applies to newly built containers only, use apptainer inspect to check version that container was built with.

More detail about this in sylabs/singularity#487

I tested adding --no-eval and now the singularity is running fine with the complex env. This is an example I cut (too long):

ARGO_TEMPLATE='{"name":"registerdcc","inputs":{},"outputs":{},"affinity":{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":1,"preference":{"matchExpressions":[{"key":"kubernetes.io/hostname","operator":"In","values":["interlink-slurm    -node"]}]}}]}},"metadata":{"annotations":{"slurm-job.vk.io/image-root":"docker://"}},"script":{"name":"","image":"python:3.8.18-bullseye","command":["bash"],"env":[{"name":"http_proxy","valueFrom":{"secretKeyRef":{"name":"XXXXproxy","key":"http_proxy"}}},    {"name":"https_proxy","valueFrom":{"secretKeyRef":{"name":"XXXXproxy","key":"https_proxy"}}},{"name":"no_proxy","valueFrom":{"secretKeyRef":{"name":"XXXXproxy","key":"no_proxy"}}},{"name":"PM_API_URL_ROOT","value":"wss://XXXX"}],"resources":{}

[dciangot]

@antoinetran is this something that you see like a time to time annotation to pass to the pod, or would you use no eval as the default? can this hurt anything?

[antoinetran]

I think we should put that as default. We want the env values specified in YAML to be copied to singularity exactly as they were in YAML. Because the example of ARGO_TEMPLATE is probably the most complex one, I am confident any other env var will also works with the flag.

There can be some impact in CMD / ENTRYPOINT, but the way I see it, I think the no eval is more clean. If this is an issue, we can revert and go back to adding this flag manually, or exclude the flag thanks to an annotations later.

[antoinetran]

I would like to add in documentation at https://github.com/interTwin-eu/interlink-slurm-plugin?tab=readme-ov-file#warning-pods-requirements-warning : another section like

COTS requirements
Singularity >= 3.10.0 or Apptainer >= 1.1.0

Also, I would like to remove

Docker entrypoints are not supported by Singularity. This means you have to manually specify a command to be executed. If you don't, /bin/sh is assumed as the default one.

This is simply not true anymore, thanks to #32

[dciangot]

+1
thx!