export parallel LATA to sequential vtk

[edbutaye]

Hello,
I performed a CFD computation of more than 320 millions nodes on 2304 CPUs. I would like to export lata files written in parallel to a single VTK file (as it is already the case for a sequential computation).
Is it possible with Visit?

Thank you in advance.

Edouard Butaye

[markcmiller86]

Hmmm...the conditions of parallel and single VTK file are likely in conflict at multiple levels.

Is your goal merely to have just a single file with all the data? Or, do you also want that single file to represent a coherent single mesh or is it ok for it to be a bunch of different meshes that all just kinda sorta meet at their exterior nodes to form an aggregate larger mesh? Although VTK format may support the latter (I am not 100% sure if it does), our current VTK exporter will not do that. In any event, even if VTK does support a bunch of individual mesh pieces in one file, the next issue you have is writing all those pieces to the file. VTK is really a serialization format. So, it does not lend itself well to writing multiple pieces of a single file concurrently from different processors. You almost certainly would have to serialize that step to satisfy VTK format/API. Finally, even if you could write the VTK concurrently from multiple processors, you next need a file system that actually supports that too (e.g. Lustre or Spectrum Scale)?

If your goal is really to merge all the pieces into a single, coherent mesh, thats even more problematic given what you have described as the scale of the input (320M nodes across 2304 MPI ranks). You could use VisIt's Merge operator but you will likely run out of memory because it will attempt to build up the final result in memory before it ever attempts to write anything to disk.

[edbutaye]

Thank you for your answer @markcmiller86 !
To be more specific, I need to export lata files to a VTK format to load them with python (PyVista module) and compute some statistics. Pyvista can manage to load parallel VTK files (extension VTU) with a PVTU file that sum up the data. For what I noticed, Visit don't export on a VTU/PVTU format but on VTR/VTM format (I would have to check the later). I think I can manage to deal with both format with my post-processing code.
However, each VTK (or VTU/VTR) file contains the mesh of the "real" computational domain AND some "virtual" meshes belonging to other CPUs (required for discretization schemes). So even if I manage to concatenate all the data of each individual file in a DataFrame for example, I would have to remove all the duplicated cells. This can be a time-consuming operation with such large datasets.
To overcome this issue, one can add the "reconnect=1e-7" option to the lata file before importing it with Visit. Only the "real" mesh will be read and the export in VTK format will only include a single file. However, with such large datasets, the Visit version I used cannot manage to open the data with this option and I also have to use a GPU.

My question is the following: is it possible to "reconnect" the mesh (remove duplicated cells due to parallel computation) after loading the data on Visit ? Or can this be specified before export?

Sincerely,
Edouard Butaye

[markcmiller86]

@edbutaye apologies for failing to respond to your follow-up remarks. I did just try something in VisIt using, of all things, a Save Window operation instead of an Export Database operation which sometimes have a very similar meaning. But, when I was running a 3 processor engine (on a small test dataset) and I selected VTK as the format and checked Force parallel merge check box and then did a Save Window, the VTK file I got looked merged and correct.

I do not know how this will fare with really, really large data but you might wanna give it a try.