Using the parallel engine to visualize large datasets

[Jwsembower]

Hi VisIt team,

I am trying to use VisIt in parallel to visualize a very large 3D dataset (~2.4 Billion cells). When I first tried to do this, the engine crashes immediately with no helpful error message. I then shrank the problem down to ~1.2 Billion cells and was able to visualize that data on 128 cores on Perlmutter (NERSC's HPC). I then tried the 2.4 billion cell dataset again with 2 nodes and 256 cores total and still the engine crashes. How can I use more cores/memory to visualize these large datasets? I am using h5 files with the Vs flavor.

[JustinPrivitera]

How many domains are in your dataset? You don't want to have more processors than domains.

Here are some tips from @markcmiller86 for helping another user with a similar problem:

1. Can he run serial and before plotting anything use SIL selection to reduce domain count to say 6 or 10 (assuming he knows which domains to select so that boundaries are actually present)? From that, watch top or something to get idea of memory being used by engine_ser and viewer processes (should put in Scalable Rendering mode BTW). Extrapolate from that and see if the numbers jive.
2. Run with -debug 1 (1 not 5)…oom messages should appear in debug logs if in fact that is happening. Grep for ‘out of memory’.
3. Set material options to simplify heavily mixed zones to 2 (or maybe 3)
4. Increase node code but reduce processor/node count
5. Do same as 1 except in small parallel starting with few domains and keep enabling double the number until it ooms there. Extrapolate from that experience.

[jrobcary]

We have parallel I/O with one big file. Does VisIt divide this up into domains? Or do we need to do that somehow?

[markcmiller86]

What file format (database plugin)?

We really ought to support some automagic parallel decomposition of various ascii file formats (e.g. .xyz or .csv format...in fact, I dunno if Blueprint plugin might already handle a single, very large .csv file by automagically decomposing...it might) but we don't in general. There may be one ascii formath that does. A handful of plugins support automagic decomposition for regular/structured grids.

So, likely answer, you will need to decompose yourself. VisIt's parallel decomposition approach is to piggy back off the data producer. Depending on file format in use, you may need a file per mpi rank.

[JustinPrivitera]

To my knowledge, Blueprint does not automatically decompose CSV datasets. But you can use the Blueprint plugin to repartition data.

[jrobcary]

This is the Vs plugin, which we have modified in our fork, but we have not put in a pull request for this. Our thought is to rebase our fork from the project, then proceed to fix.

Does your answer mean that it is possible, in principle, for the data reading plugin to provide the decomp'd data to VisIt?

[markcmiller86]

Does your answer mean that it is possible, in principle, for the data reading plugin to provide the decomp'd data to VisIt?

Yes. its possible. For examples, look for plugins that set md->SetFormatCanDoDomainDecomposition(true); which it looks like Vs does somewhere. The plugin winds up lying to VisIt and says there is only one block available regardless of MPI task count. Then, all MPI ranks make queries for data and plugin decides which to serve up on each rank.

[markcmiller86]

BTW, @jrobcary you say "one big file", but does that mean it is only a single domain dataset? For example, its possible to have a Silo file that is huge but also has many domains in that one file. It is the domain count that controls the maximum number of MPI ranks you can use...without resorting to doing on-the-fly decomposition in the plugin itself.

[jrobcary]

The code runs on many processors, so has many domains in the domain decomp sense, but they are all put together into one file using parallel IO.

[markcmiller86]

When you say "put together", what do you mean? Are all the little pieces knitted together to form one giant, single coherent piece in the file? And, if so, upon read back do you have the automagic ability to essentially reverse that? If so, are you using MPI-IO or similar to do that? Is the file residing on a parallel file system? If all are true and Vs plugin is designed to operate in this scenario, then VisIt should scale in performance on an MPI rank count up to the specified domain count. Beyond that will offer no improvement in performance.

If, instead, the file just kinda sorta represents a container of a bunch of pieces (which are not knitted together), then that too should still work ok if the Vs plugin is designed to handle that scenario. In that case, the "decomposition" algorithm is kinda sorta decided for you by VisIt's load balance algorithm. Performance can still suffer if the file is not stored on a bonfied parallel file system though.

Do you know what scenario, if any, is in play here?

[jrobcary]

Yes, all the pieces form one giant, single, coherent HDF5 dataset. And yes, we use parallel IO (HDF5 on top of MPI-IO) to read the data back in. You mention "specified domain count". Is that just what one gets from MPI_Comm_size(MPI_COMM_WORLD, ..)?
Or does VisIt have a separate way of specifying this?

[markcmiller86]

@jrobcary, my apologies for being confusing. If the Vs plugin already has the ability do H5Dread() calls down into the one HDF5 file with one (massive) monolithic piece and each MPI rank just sets the HDF5 data selection that that rank is expected to read, then I think you are good.

But, I just briefly grep'd the Vs plugin code. I am not seeing any HDF5 calls that I would associate with the above modality of scalable, parallel interaction with a single, massive object in an HDF5 file. For example, I would expect to see calls there to set the file access properties to the MPI-IO driver. I don't see that though I also didn't look real hard either.

[jrobcary]

@markcmiller86 , how does one set the domain count? Eg, I'd like it to simply be MPI_comm_size.

[markcmiller86]

Right. So, when a format advertises that it can do domain decomp itself (via md->SetFormatCanDoDomainDecomposition(true); then you don't really specify a domain count for your meshes. Instead, in PopulateDatabaseMetadata, you tell VisIt that any mesh you want the plugin to handle the decomposition for that it has just one block. The combination of md->SetFormatCanDoDomainDecomposition(true); and one block means that VisIt will issue GetMesh() and GetVar() calls on every MPI rank and the domain id passed in on MPI Rank K is K and the plugin uses the MPI rank to decide which portion of the mesh (or variable) to respond with on that MPI rank.

[marcdurant]

Hi @markcmiller86 -

I'm helping the Tech-X folks get this figured out.

I have a question about src/avt/DBAtts/MetaData/avtMeshMetaData.h - there is a field in this class named NumberCells that is declared as a "const int". Looking at the Vs plugin, it appears that we are sending the total number of cells in the mesh to this variable. In our case, we have a mesh with more than 2,147,483,647 cells (4136750750) so presumably this integer is overflowing to be negative.

Luckily there's code in avtMeshMetaData::SetNumberCells() that ignores negative values, but do you know any possible downstream effects of leaving NumberCells unset?

Thanks!

[markcmiller86]

@marcdurant this is NEWS to me. I've never seen that before. Doesn't really make sense to have tha kind of fine-grained metadata there either because, as you can imagine, a number of different operations can change that value.

I went poking around and see no hits for it anywhere except in some very rudimentary bounds checking when outputting to debug logs.

I think you can safely ignore this field.

[Jwsembower]

We believe that we have built VisIt properly, but when we go to launch VisIt in parallel, we find that it is actually launching the serial engine. In the VisIt documentation under the parallel launch option it explains that we need to add a -par option or any other parallel run option to launch the parallel engine. When we go to our build of VisIt and launch using the command

./visit -config ~/.visit/hosts/host_nersc_parlmutter.xml -par

I see this output:

sembower@nid200036:/global/cfs/cdirs/txsupp/cary/perlmutter/dev2/volatile-gcc1230/visit4ctk-stable.r17549-parsh/bin> ./visit -config ~/.visit/hosts/host_nersc_parlmutter.xml -par
Running: gui3.2.1 -config /global/homes/s/sembower/.visit/hosts/host_nersc_parlmutter.xml -engineargs -l;mpirun -launchengine localhost
Running: viewer3.2.1 -config /global/homes/s/sembower/.visit/hosts/host_nersc_parlmutter.xml -engineargs -l;mpirun -geometry 2164x1369+396+0 -borders 45,10,10,10 -shift 0,0 -preshift 0,37 -defer -host 127.0.0.1 -port 5600 -launchengine localhost
Running: mdserver3.2.1 -host 127.0.0.1 -port 5601
Running: engine_ser3.2.1 -host 127.0.0.1 -port 5600

Where it shows that it is running the serial engine. I verified that the parallel engine is indeed there. How can I get it to launch the parallel engine? The node that I am running interactively on has 128 cores on it.

[JustinPrivitera]

perhaps some of these resources may help:

parallel run:
https://visit-sphinx-github-user-manual.readthedocs.io/en/develop/python_scripting/visit_utils.html
https://visit-sphinx-github-user-manual.readthedocs.io/en/develop/python_scripting/quickrecipes.html#opening-a-compute-engine

launch parallel visit with existing allocation:
from cmd line:
visit -l srun -np 36

from python:
import visit_utils
visit_utils.engine.open(nprocs=36)

visit itself has lots of tools to help:
go to options -> host profiles -> launch profiles -> parallel to see all the options and things