phalanx w/ CUDA 12: KokkosScalarFunctionsFad.cpp(251): error: An extended __host__ __device__ lambda cannot first-capture variable in constexpr-if context

[eugeneswalker]

Bug Report

@rppawlo @crtrott @ndellingwood

Description

I'm hitting an error while trying to do a Spack development build of Trilinos master branch on commit 5aaae1a

Here using:

• Ubuntu 20.04 based Docker container
• GCC 11.4.0
• CUDA 12.0.1
• MPICH 4.1.2
• Kokkos NVCC Wrapper 4.1.00

Here is my Spack environment:

spack:
  view: false

  concretizer:
    reuse: false
    unify: false

  compilers:
  - compiler:
      spec: gcc@=11.4.0
      paths:
        cc: /usr/bin/gcc
        cxx: /usr/bin/g++
        f77: /usr/bin/gfortran
        fc: /usr/bin/gfortran
      flags: {}
      operating_system: ubuntu20.04
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []

  packages:
    all:
      require: '%gcc'
      providers:
        blas: [openblas]
        mpi: [mpich]
      target: [x86_64]
      variants: +mpi
    mpi:
      require: mpich
    mpich:
      require: '@4.1.2 ~wrapperrpath ~hwloc'
      buildable: false
      externals:
      - spec: [email protected] ~argobots~cuda+fortran~hwloc+hydra+libxml2+pci~rocm+romio~slurm~vci~verbs~wrapperrpath build_system=autotools datatype-engine=auto device=ch4 netmod=ofi pmi=pmi
        prefix: /usr/local/mpich/install/mpich
    boost:
      require: cxxstd=17
    libffi:
      require: "@3.4.4"
    cuda:
      require: "@12.0.1"

  specs:
  - trilinos@develop +testing +amesos +amesos2 +anasazi +aztec +belos +boost +epetra +epetraext +ifpack +ifpack2 +intrepid +intrepid2 +isorropia +kokkos +ml +minitensor +muelu +nox +piro +phalanx +rol +rythmos +sacado +stk +shards +shylu +stokhos +stratimikos +teko +tempus +tpetra +trilinoscouplings +zoltan +zoltan2 +superlu-dist gotype=long_long +wrapper +cuda cuda_arch=80

Concretization:

 -   a7y6cf3  trilinos@develop%[email protected]~adelus~adios2+amesos+amesos2+anasazi+aztec~basker+belos+boost~chaco~complex+cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus+explicit_template_instantiation~float+fortran~gtest~hdf5~hypre+ifpack+ifpack2+intrepid+intrepid2~ipo+isorropia+kokkos~mesquite+minitensor+ml+mpi+muelu~mumps+nox~openmp~panzer+phalanx+piro~python~rocm~rocm_rdc+rol+rythmos+sacado~scorec+shards+shared+shylu+stk+stokhos+stratimikos~strumpack~suite-sparse~superlu+superlu-dist+teko+tempus+testing+thyra+tpetra+trilinoscouplings~uvm+wrapper~x11+zoltan+zoltan2 build_system=cmake build_type=Release cuda_arch=80 cxxstd=17 generator=make gotype=long_long arch=linux-ubuntu20.04-x86_64
[+]  ia5ufsp      ^[email protected]%[email protected]~atomic~chrono~clanglibcpp~container~context~contract~coroutine~date_time~debug+exception~fiber~filesystem+graph~graph_parallel~icu~iostreams~json~locale~log+math+mpi+multithreaded~nowide~numpy~pic~program_options~python~random~regex~serialization+shared~signals~singlethreaded+stacktrace~system~taggedlayout~test~thread~timer~type_erasure~versionedlayout~wave build_system=generic cxxstd=17 patches=a440f96 visibility=hidden arch=linux-ubuntu20.04-x86_64
[+]  j56nveb      ^[email protected]%[email protected]~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-ubuntu20.04-x86_64
[+]  krhswgy          ^[email protected]%[email protected]~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-ubuntu20.04-x86_64
[+]  y72nyuk              ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  m2nqw34              ^[email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=linux-ubuntu20.04-x86_64
[+]  thgudgh                  ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=linux-ubuntu20.04-x86_64
[+]  4ahtcrh          ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  vkxpza5      ^[email protected]%[email protected]~allow-unsupported-compilers~dev build_system=generic arch=linux-ubuntu20.04-x86_64
[+]  w3n2rgb          ^[email protected]%[email protected]+pic~python+shared build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  bb6pvt7              ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=linux-ubuntu20.04-x86_64
[+]  7gqmxvf              ^[email protected]%[email protected]~pic build_system=autotools libs=shared,static arch=linux-ubuntu20.04-x86_64
[+]  axbtmvn      ^[email protected]%[email protected]~guile build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  tkc6zfe      ^[email protected]%[email protected]~cairo+cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools cuda_arch=none libs=shared,static arch=linux-ubuntu20.04-x86_64
[+]  wjzjxgg          ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  3cmrrml              ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  dije4dh                  ^[email protected]%[email protected]+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-ubuntu20.04-x86_64
[+]  uxiki2z                      ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  pv4s6pa              ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  hjpgnti          ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  f4abxog      ^[email protected]%[email protected] build_system=generic arch=linux-ubuntu20.04-x86_64
[+]  l6dookb      ^[email protected]%[email protected]~gdb~int64~ipo~real64+shared build_system=cmake build_type=Release generator=make patches=4991da9,93a7903,b1225da arch=linux-ubuntu20.04-x86_64
[e]  4vf2w2d      ^[email protected]%[email protected]~argobots~cuda+fortran~hwloc+hydra+libxml2+pci~rocm+romio~slurm~vci~verbs~wrapperrpath build_system=autotools datatype-engine=auto device=ch4 netmod=ofi pmi=pmi arch=linux-ubuntu20.04-x86_64
[+]  2n67kk5      ^[email protected]%[email protected]~bignuma~consistent_fpcsr+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-ubuntu20.04-x86_64
[+]  ymeouv6          ^[email protected]%[email protected]+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-ubuntu20.04-x86_64
[+]  z5d3lru              ^[email protected]%[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu20.04-x86_64
[+]  mxcjdbv              ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=linux-ubuntu20.04-x86_64
[+]  zafnh7n                  ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  adx2gsg              ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  bhdepb5                  ^[email protected]%[email protected] build_system=autotools patches=bbf97f1 arch=linux-ubuntu20.04-x86_64
[+]  wcatis4      ^[email protected]%[email protected]~gdb~int64~ipo+shared build_system=cmake build_type=Release generator=make patches=4f89253,50ed208,704b84f arch=linux-ubuntu20.04-x86_64
[+]  ur6zd25      ^superlu-dist@develop%[email protected]~cuda~int64~ipo~openmp~rocm+shared build_system=cmake build_type=Release generator=make arch=linux-ubuntu20.04-x86_64
[+]  hpmj5kc      ^[email protected]%[email protected]+compat+opt build_system=autotools patches=299b958,ae9077a,b692621 arch=linux-ubuntu20.04-x86_64

Error:

...
  >> 43151    /e4s-develop/collab/trilinos/ctest-build/Trilinos/packages/phalanx/test/Kokkos/tKokkosScalarFunctionsFad.cpp(251): error: An extended __host__ __device__ lambda cannot first-capture var
              iable in constexpr-if context
     43152
     43153    1 error detected in the compilation of "/e4s-develop/collab/trilinos/ctest-build/Trilinos/packages/phalanx/test/Kokkos/tKokkosScalarFunctionsFad.cpp".
  >> 43154    make[2]: *** [packages/phalanx/test/Kokkos/CMakeFiles/Phalanx_tKokkosScalarFunctionsFad.dir/build.make:79: packages/phalanx/test/Kokkos/CMakeFiles/Phalanx_tKokkosScalarFunctionsFad.dir/
              tKokkosScalarFunctionsFad.cpp.o] Error 2
     43155    make[2]: Leaving directory '/e4s-develop/collab/trilinos/ctest-build/Trilinos/spack-build-ir446o2'
  >> 43156    make[1]: *** [CMakeFiles/Makefile2:103499: packages/phalanx/test/Kokkos/CMakeFiles/Phalanx_tKokkosScalarFunctionsFad.dir/all] Error 2
     43157    make[1]: *** Waiting for unfinished jobs....

Steps to Reproduce

$> docker run -it --name trilinos ecpe4s/ubuntu20.04-runner-amd64-gcc-11.4-mpi-base:2023.08.20

root@265338f8846a:/# git clone https://github.com/eugeneswalker/spack /spack
root@265338f8846a:/# git -C /spack checkout 97e40a4bdc8529bf9ae542cc69007796d77b9665

root@265338f8846a:/# . /spack/share/spack/setup-env.sh
root@265338f8846a:/# cat <<EOF >spack.yaml
# copy the contents of the spack.yaml pasted earlier in this issue here
EOF

root@265338f8846a:/# spack env activate -d .
root@265338f8846a:/# spack concretize -f | tee concretize.log
...
root@265338f8846a:/# spack install --only dependencies --include-build-deps
... will install dependencies

root@265338f8846a:/# git clone https://github.com/trilinos/Trilinos.git
root@265338f8846a:/# cd Trilinos
root@265338f8846a:/# git checkout 5aaae1ada6fe1ce777e671a0ff84fdc4f0779406
root@265338f8846a:/# nohup bash -c "time spack dev-build -j24 trilinos@develop +testing +amesos +amesos2 +anasazi +aztec +belos +boost +epetra +epetraext +ifpack +ifpack2 +intrepid +intrepid2 +isorropia +kokkos +ml +minitensor +muelu +nox +piro +phalanx +rol +rythmos +sacado +stk +shards +shylu +stokhos +stratimikos +teko +tempus +tpetra +trilinoscouplings +zoltan +zoltan2 +superlu-dist gotype=long_long +wrapper +cuda cuda_arch=80" &

root@265338f8846a:/# tail -f nohup.out # paths will differ from what is shown below
...
  >> 43151    /e4s-develop/collab/trilinos/ctest-build/Trilinos/packages/phalanx/test/Kokkos/tKokkosScalarFunctionsFad.cpp(251): error: An extended __host__ __device__ lambda cannot first-capture var
              iable in constexpr-if context
     43152
     43153    1 error detected in the compilation of "/e4s-develop/collab/trilinos/ctest-build/Trilinos/packages/phalanx/test/Kokkos/tKokkosScalarFunctionsFad.cpp".
  >> 43154    make[2]: *** [packages/phalanx/test/Kokkos/CMakeFiles/Phalanx_tKokkosScalarFunctionsFad.dir/build.make:79: packages/phalanx/test/Kokkos/CMakeFiles/Phalanx_tKokkosScalarFunctionsFad.dir/
              tKokkosScalarFunctionsFad.cpp.o] Error 2
     43155    make[2]: Leaving directory '/e4s-develop/collab/trilinos/ctest-build/Trilinos/spack-build-ir446o2'
  >> 43156    make[1]: *** [CMakeFiles/Makefile2:103499: packages/phalanx/test/Kokkos/CMakeFiles/Phalanx_tKokkosScalarFunctionsFad.dir/all] Error 2
     43157    make[1]: *** Waiting for unfinished jobs....
...
real	13m11.927s
user	196m20.774s
sys	28m20.253s

[jwillenbring]

@sebrowne @fryeguy52 Have either of you run into something like this before?

[rppawlo]

This issue doesn't show up in any of our cuda testing. I'm guessing this check only happens in cuda 12. I see the issue in the cuda programming guide, just don't have a machine where I can test a fix. I'll put up a fix shortly and try to find a machine to reproduce.

[eugeneswalker]

This issue doesn't show up in any of our cuda testing. I'm guessing this check only happens in cuda 12. I see the issue in the cuda programming guide, just don't have a machine where I can test a fix. I'll put up a fix shortly and try to find a machine to reproduce.

Do you want access to our machines at University of Oregon? We have multiple machines with A100. We could get your access set up within a day or two, if not sooner, if you submit this simple form and mention you are on the Trilinos team working on GPU builds:

• https://systems.nic.uoregon.edu/account/

[rppawlo]

PR with fix: #12359

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

[sebrowne]

Closed with #12359