Cannot read openPMD file

[eschnett]

I want to open an ADIOS-based openPMD file. I am using VisIt 3.1.4 on macOS 11.4, downloaded as a binary.

I have written the file with our application (the Einstein Toolkit), and I can read it there as well. Both bpls and openpmd-ls display the information in the file correctly, and they don't indicate any errors.

There are two issues:

(1) I cannot get VisIt to even open the file. The file selection box only lets me open files, but ADIOS bp file are stored as a directory. I tried pointing VisIt to one of the files in the directory, but this didn't work either, not even when I explicitly choose OpenPMD as file format.

(2) I created a *.visit file that points to the openPMD file (i.e. directory). I cannot open this file either; VisIt reports the error

VisIt could not read from the file "/Users/eschnett/simulations/brill-lindquist-write/output-0000/brill-lindquist-write/brill-lindquist-write.openpmd.visit".

The generated error message was:

There was an error opening /Users/eschnett/simulations/brill-lindquist-write/output-0000/brill-lindquist-write/brill-lindquist-write.openpmd.visit. It may be an invalid file.  VisIt tried using the following file format readers to open the file: OpenPMD

The following error(s) may be helpful in identifying the problem:
The selected database type was not correct for the given file.  The message from the database was Not an openPMD file: Error opening the file

I again explicitly chose OpenPMD as file format.

[biagas]

@ax3l Would you have any insight on this issue?

[eschnett]

I just ran into a similar issue with ParaView, which had better error messages. There, the problem was that the binary's ADIOS library was built without support for Blosc. Could there be a similar issue here?

[biagas]

Our OpenPMD reader has no dependencies on ADIOS, only HDF5. Not sure if that is important, I am not familiar with that reader's internals at this moment. Did you try opening as ADIOS (or ADIOS2) instead of OpenPMD?

Would you be willing to provide a minimal set of data for testing purposes?

[ax3l]

Hi and thanks for the report & ping!

The status of the VisIt reader is a direct implementation of the openPMD-standard 1.* against the HDF5 C libraries, first developed as a standalone plugin a while back (org repo) and recently updated for mainlining by us (cc @dpugmire).

The VisIt plugin was developed a while back, and since then we created a new C++/Python middleware library called openPMD-api that can be used for rapid implementations and supports ADIOS1/ADIOS2/HDF5/JSON data files with a unified interface.

In the last ParaView release (5.9) we mainlined a first plugin based on this library, which in about ~450 lines lines of parallel Python at the moment. We include this in the ParaView binary distribution and lack of Blosc is a good issue to fix in it, thanks for raising this! (Note that this is only a problem if you wrote compressed files though.) Raised now in a ticket and PRs opened.

I would love for us to "modernize" the VisIt plugin for ADIOS2 support as well. For VisIt builds, is there a way to either also write Python plugins in the same way and include an additional library build in the VisIt distributions?

openPMD-api is designed to work well with CMake super-builds, adds no external dependencies besides the ones you already have (MPI/HDF5/ADIOS2/Python, all individually controllable), can be used directly from C++ or from Python, and would reduce the lines of logic and file format binding code that we have to maintain in VisIt. We have a C++11 version, but generally moved forward to C++14 as a requirement now (release 0.13.0+). Does this look compatible with the VisIt toolchain requirements?

[eschnett]

@biagas I cannot open the *.bp directory directly. However, I can point a *.visit file to it, and I can then explicitly open it as ADIOS2 file. This recognizes the metadata correctly. Since openPMD uses a particular way to lay out datasets in the file, a lot of internals are exposed that shouldn't be, but I was able to find a dataset to display.

Drawing that dataset crashed the engine. This is using VisIt 3.1.4, both the viewer and the engine running locally on macOS.

I am happy to provide the file. I reduced the size to 1.3 MByte; is that sufficiently small, or should it be even smaller?

[brugger1]

Yes, 1.3MB is fine. If you would like to attach it to the issue, you will need to put it in a "zip" or "tar.gz" file.

[eschnett]

Here is the sample file: brill-lindquist-write.it00000000.bp.tar.gz

[aowen87]

@eschnett which variables are you trying to plot?

You can open your directory as a database by launching visit from the command line:
/path/to/VisIt -o /path/to/brill-lindquist-write.it00000000.bp

We should really allow that through the GUI as well...

I don't see many variables available once opened. There is a mesh, which successfully plots, and there's a single variable that starts off as a blank name. Trying to plot that causes a crash. The blank name may have something to do with that.