Ballooning parallel times

[MTCam]

Ballooning setup times vs. number of elements have been observed on Quartz. Looks like these are associated with the mesh distribution (i.e. mesh_dist.send_mesh_parts ) NOT partitioning with metis.

Serial I/O rates seem ok - I'm checking (with LC) on what to expect from NFS on this particular FS

Parallel I/O rates do not scale as expected. Our viz files write in roughly constant time at (IO BW HERE). I/O also does not seem to be a parallel bottleneck (yet). Here are initial parallel timings from Quartz (CPU only), gotten by strong scaling the Y0 case (running only 5 steps).

5 steps may not be enough to see the actual scaling of advancement, due to (code generation?) on the first call to constructs using kernels.

[inducer]

• It'd be interesting to see whether this is influenced by what @matthiasdiener found in https://github.com/illinois-ceesd/planning/issues/27. See also Workaround for file locking issues with large numbers of ranks #31.

[inducer]

• As discussed in the meeting, could you try raw write throughput? (from Python or C?)

[MTCam]

Here's the raw write speeds gathered with LukeO's code here running on Quartz on the shared file space where the previous tests were run.
rates.pdf

And here is the one from running the same benchmark on the parallel filesystem, but from Lassen this time due to Quartz DAT. I'll grab the same timing data for that filesystem and start running exclusively there from the parallel filesystem from now on.

lassen_io_rates_scratch.pdf