Invalid size for the current dataset

[wlin8633]

Dear developer team,
I have recently tried to use Visit of version 3.3.3 to visualize some data from WarpX 24.04 using serial engine and got the following error. I've checked that the amount of memory required to load the data is sufficient. It's about 125 MB per file and saved in OpenPMD format using HDF5 backend. If I visualize everything using parallel engine, everything just works fine. However, it's quite unconvenient to always use parallel engine when dealing with small datasets. The attachment is one of the dump file. Thank you so much for your great work.

ERROR: Pseudocolor: ()
viewer.exe: There was an error opening /data/0/fields//E/x: Invalid size for the current dataset (3032064 25165824)

openpmd_000000.zip

[markcmiller86]

@wlin8633 sorry for delay in responding.

Hmm...I see same on my mac in VisIt 3.3.1. In looking at the OpenPMD reader code, there are some serial and parallel code paths that set the mesh size. I suspect the serial code path is not setting the mesh size properly and getting bogus (e.g. uninitialized mem) sizes.

By any chance has the OpenPMD "format" (I know its an HDF5 file) changed at all recently? I mean, is mesh size information being handled differently than before? In our test suite data, I see a softwareVersion attribute with value "4" whereas in your file I see that with value "unknown"

[markcmiller86]

Ok, I think I found the issue. Your file is using CHUNKED storage for HDF5 datasets. That is certainly fine to do. But, the plugin code has a size-based smoke check here...

visit/src/databases/OpenPMD/OpenPMDClasses/PMDFile.C

Line 552 in c039f36

if (numValues == int(datasetStorageSize/dataSize))

which is comparing numValues (obtained from multiplying dataset dimensions) with the dividend of storage size of the dataset (which includes overheads for chunked storage) and data type size (8 for double in this case).

That logic works only for CONTIGUOUS storage.

The right answer is to update the logic in the plugin to use logical size of the datasset instead of storage size or use only CONTIGUOUS storage in the data producer.

Does this make sense?

[markcmiller86]

FYI...#19530

[wlin8633]

@markcmiller86 Thank you so much for your kind reply and bug fixes. Awfully sorry for delay in responding.

I just tested against VisIt of version 3.3.3 with your great fixes applied, and it works fine as shown in the below figure. However, when I tested 3.4.1 with the fixes applied using the same OpenPMD file as previously uploaded, "Pseudocolor" and "Volume" plots crashed the application with the following error as shown in the following. The same error also appears when drawing the sliced 3D Pseudocolor plot. Interestingly, drawing "Contour" and "Histogram" plots seem to work without any problem.

I reproduced the error of VisIt 3.4.1 with debug flag on and attached the logs in the following.

VisIt 3.3.3 OpenPMD Fix applied

Running: gui3.3.3
Running: viewer3.3.3 -geometry 1415x928+505+54 -borders 26,4,4,4 -shift 0,0 -preshift 4,26 -defer -host 127.0.0.1 -port 5600
Running: mdserver3.3.3 -host 127.0.0.1 -port 5601
Error opening plugin file: /home/u3942590/.local/opt/visit/3.3.3/linux-x86_64_gcc-8.5/3.3.3/linux-x86_64/plugins/databases/libMIDXDatabase.so (/home/u3942590/.local/opt/visit/3.3.3/linux-x86_64_gcc-8.5/3.3.3/linux-x86_64/plugins/databases/libMIDXDatabase.so: undefined symbol: _ZN3MPI8Datatype4FreeEv)
Running: engine_ser3.3.3 -idle-timeout 480 -threads 14 -host 127.0.0.1 -port 5600
 

Visit 3.4.1 Plot Errors:

Running: gui3.4.1 -debug
Warning: debug level not specified, assuming 1
Running: viewer3.4.1 -debug -geometry 1121x766+415+27 -borders 26,4,4,4 -shift 0,0 -preshift 4,26 -defer -host 127.0.0.1 -port 5600
Warning: debug level not specified, assuming 1
Warning: debug level not specified, assuming 1
Running: mdserver3.4.1 -debug 1 1121x766+415+27 -host 127.0.0.1 -port 5601
Error opening plugin file: /opt/pkgs/VisIt/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/plugins/databases/libMIDXDatabase.so (/opt/pkgs/VisIt/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/plugins/databases/libMIDXDatabase.so: undefined symbol: _ZN3MPI8Datatype4FreeEv)
Running: engine_ser3.4.1 -idle-timeout 480 -threads 8 -debug 1 1121x766+415+27 -host 127.0.0.1 -port 5600
2024-05-22 16:28:18.813 (  38.126s) [        F441E000]      vtkDataReader.cxx:3367   ERR| vtkRectilinearGridReader (0x5ec9310): Cannot read field header! for file: 
VisIt's viewer exited abnormally! Aborting the Graphical User Interface. VisIt's developers may be reached via our GitHub discussions, https://github.com/visit-dav/visit/discussions

Visit 3.4.1 Logs:
Log.zip

Best regards,
Wei Lin

[markcmiller86]

@wlin8633 I am still working on cause. That said, there appears to be a work-around and that is to set scalable rendering mode to Always