Introduce global setup/cleanup in APIs

[yasahi-hpc]

Fixes #217

This PR aims at adding KokkosFFT::initialize and KokkosFFT::finalize in order to correctly handle setup functions which cannot be called more than once.

• call setup functions of fftw and rocfft in KokkosFFT::initialize
• call cleanup functions of fftw and rocfft in KokkosFFT::finalize
• Add tests to ensure that KokkosFFT::initialize and KokkosFFT::finalize can only be called when Kokkos is available
• Add KokkosFFT::initialize and KokkosFFT::finalize in examples

Edited 15/Jan, after discussions we have changed the implementation.

• call fftw and rocfft setup function in static local lambda function which is supposed to be called at the first time of API call
• register fftw and rocfft cleanup function in static local lambda function with Kokkos::push_finalize_hook()

[dalg24]

Did you consider leveraging Kokkos::push_finalize_hook() instead?

[yasahi-hpc]

Did you consider leveraging Kokkos::push_finalize_hook() instead?

Thank you, that is interesting. At least, I can use Kokkos::push_finalize_hook() inside KokkosFFT::finalize() to ensure that the cleanup is made only once inside Kokkos::finalize().

However, we also need to call fftw_init_threads() and rocfft_setup() before using FFT APIs. If I am not wrong, a hook for Kokkos::initialize() is not available in Kokkos

[dalg24]

I overlooked that you actually acquire resources as well. You are correct, Kokkos::push_finalize_hook() would not accommodate for this.
That said we could look into an approach where we initialize the first time around that these resources are needed.

I would encourage seriously looking at alternatives. I see the introduction of KokkosFFT::{initialize,finalize} as a last resort measure.

[tpadioleau]

LGTM

[tpadioleau]

Suggested change

	add_library(fft STATIC KokkosFFT_Core.cpp)
	add_library(fft KokkosFFT_Core.cpp)

I would not force it to be STATIC to allow people to use BUILD_SHARED_LIBS, https://cmake.org/cmake/help/latest/variable/BUILD_SHARED_LIBS.html.

[tpadioleau]

Suggested change

	if (!(Kokkos::is_initialized() || Kokkos::is_finalized())) {
	if (!Kokkos::is_initialized()) {

Do we need to test is_finalized ?

[yasahi-hpc]

I would like to exclude the situation after finalization, where Kokkos::is_initialized() returns false and Kokkos::is_finalized() returns true. Accordingly, if we just check Kokkos::is_initialized(), we do not know whether Kokkos is before initialization or after finalization.

[tpadioleau]

I would encourage seriously looking at alternatives. I see the introduction of KokkosFFT::{initialize,finalize} as a last resort measure.

Oh really why ? Are both functions a problem or only Kokkos::finalize ?

[yasahi-hpc]

I overlooked that you actually acquire resources as well. You are correct, Kokkos::push_finalize_hook() would not accommodate for this. That said we could look into an approach where we initialize the first time around that these resources are needed.

I would encourage seriously looking at alternatives. I see the introduction of KokkosFFT::{initialize,finalize} as a last resort measure.

Makes sense. The alternative solution I have in mind is to rely on static local object which is called at the first time when KokkosFFT API is called (see 154c12d). Unfortunately, we decide not to employ this approach because it may be unsafe.

[yasahi-hpc]

@tpadioleau @dalg24
I have update the PR based on our discussion
May I have your review please