Nested parallelism: different behaviour between multisession and multicore

[Carambarbe]

I'm running 20 simulations of a model in R, on a UNIX computing cluster with 60 cores.
Each simulation is divided in 3 "sub-simulations" evaluated in parallel: I thus wanted to configure a nested parallel future.

I discovered a different behaviour between these two plan() settings:
plan(list(tweak(multisession, workers = 20, tweak(multicore, workers = 3)))
plan(list(tweak(multicore, workers = 20, tweak(multicore, workers = 3)))
First one managed to launch 60 parallel (sub-)simulations, while the second one launched only 22 simulations (only the 3 sub-simulations of the first simulation, and the first sub-simulations of the 19 simulations remaining).

Hope that's clear...
Did I miss something about multicore ?

Thank you !

[HenrikBengtsson]

Both of your setups will run 20 first-level workers, which each run 3 second-level workers.

The multisession workers are persistent, i.e. they are launched once, and then serve multiple futures (parallel tasks) over their life spans. These are easy to spot when using tools such as ps, pstree, top, ...

The multicore workers are transient, i.e. a new one is launched for each future, and when the future is completed, it's ended. Because of the limited lifespan of each multicore worker, it's probably that you don't catch them all at the right time, but they're there.

You can adapt the 'Example: A Remote Compute Cluster' (https://future.futureverse.org/articles/future-3-topologies.html#example-a-remote-compute-cluster) to convince yourself that you indeed get 20*3 parallel workers at any time.