Tacho: Tacho doesn't declare dependency on rocblas properly

[masterleinad]

Bug Report

Configuring Trilinos with HIP support but without TPL_ROCBLAS compiles but at runtime, I'm seeing

symbol lookup error: /tmp/trilinos-install/lib64/libtacho.so.16: undefined symbol: rocblas_create_handle

which indicates that Tacho is using rocblas without declaring a dependency on it. E.g.,

Trilinos/packages/shylu/shylu_node/tacho/src/impl/Tacho_NumericTools_LevelSet.hpp

Lines 558 to 565 in 8bded3a

	#if defined(KOKKOS_ENABLE_HIP)
	if (!_is_rocblas_created) {
	_status = rocblas_create_handle(&_handle_blas);
	checkDeviceBlasStatus("rocblasCreate");
	_is_rocblas_created = true;
	_handle_lapack = _handle_blas;
	}
	#endif

looks suspicious.

[masterleinad]

Note that even after fixing the link problem by manually finding and linking rocblas, I am observing runtime failures (possibly related to the fact that some places in the package check for the ROC* TPLS and others just for KOKKOS_ENABLE_HIP). It seems that Tacho should really require rocblas, rocthrust and rocasolve when the HIP backend is enabled.

[iyamazaki]

Thank you, @masterleinad. Just to make sure I understand, when you enable ROCBLAS, ROCSPARSE, and ROCSOLVER, the issue with running Tacho goes away? And,it sounds like these are available for including and linking for compiling and building, but the symbols cannot be resolved at runtime with the dynamically-linked libraries?

[masterleinad]

Just to make sure I understand, when you enable ROCBLAS, ROCSPARSE, and ROCSOLVER, the issue with running Tacho goes away?

Yes, the issue goes away when those TPLs are enabled (since that implicitly also enables ShyLU_NodeTacho_ENABLE_ROC* and the respective preprocessor variables get defined).

And,it sounds like these are available for including and linking for compiling and building, but the symbols cannot be resolved at runtime with the dynamically-linked libraries?

Yes, the API interface for rocblas seems to be available without enabling the corresponding TPL. It seems the include directories coming with the compiler are sufficient. The symbols are missing when not explicitly linking against the rocblas library.
Again, I tried just linking against it in the downstream application but ran into memory access issues on the GPU that only got resolved after also enabling the TPLs in Trilinos (which makes manually linking against those TPLs downstream obsolete).

[iyamazaki]

Thank you @masterleinad ! We created a PR, and hope it looks okay.

[iyamazaki]

@masterleinad. I'll close this issue. Please let me know if you encounter additional issues!!