Throwing a nice error if CUDA-aware MPI is not configured

[glwagner]

Revisiting this briefly --- we have found that MPI.has_cuda() only tests Open MPI, so it is not a general solution for determining whether CUDA-aware MPI is available.

However, there are other possible solutions. For example we can write a test like

using MPI
using CUDA
MPI.Init()

function sendrecv_works(grid)
    arch = architecture(grid)
    comm = arch.communicator
    rank = arch.local_rank
    size = MPI.Comm_size(comm)
    dst = mod(rank+1, size)
    src = mod(rank-1, size)
    N = 4
    FT = eltype(grid)
    send_mesg = CuArray{FT}(undef, N)
    recv_mesg = CuArray{FT}(undef, N)
    fill!(send_mesg, FT(rank))
    CUDA.synchronize()
    try
        MPI.Sendrecv!(send_mesg, dst, 0, recv_mesg, src, 0, comm)
        return true
    catch err
        @warn "MPI.Sendrecv test failed." exception=(err, catch_backtrace())
        return false
    end
end

adapted from https://gist.github.com/luraess/0063e90cb08eb2208b7fe204bbd90ed2

Originally posted by @glwagner in #3883 (comment)

[simone-silvestri]

Sounds good. We should also test that the message is received correctly, in that way, this test cannot pass if MPI is configured incorrectly (for example when erroneously size == 1 and dst == src as in the case of @francispoulin #3981 which would make this pass even without CUDA-aware MPI).