Upgrade BoxLib to AMReX #3242
Description
BoxLib has been superseded by AMReX. We should upgrade our builds and plugin to use AMReX. What happens to our ability to read/vis old boxlib data files?
-----------------------REDMINE MIGRATION-----------------------
This ticket was migrated from Redmine. As such, not all
information was able to be captured in the transition. Below is
a complete record of the original redmine ticket.
Ticket number: 3257
Status: Pending
Project: VisIt
Tracker: Bug
Priority: Normal
Subject: Upgrade BoxLib to AMReX
Assigned to: -
Category: -
Target version: -
Author: Mark Miller
Start: 10/30/2018
Due date:
% Done: 0%
Estimated time:
Created: 10/30/2018 12:19 am
Updated: 10/30/2018 06:47 pm
Likelihood: 3 - Occasional
Severity: 2 - Minor Irritation
Found in version: 2.12.3
Impact:
Expected Use:
OS: All
Support Group: Any
Description:
BoxLib has been superseded by AMReX. We should upgrade our builds and plugin to use AMReX. What happens to our ability to read/vis old boxlib data files?
Comments:
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Development
Participants
Activity
biagas commentedon Jun 2, 2023
I tried a build of AMReX 23.05 on Windows.
I turned off AMReX_MPI, because I wanted a serial version to use with our Boxlib reader (I was hardcoding changes in the Boxlib reader just to see if it would work).
However, there are MPI related AMRex headers that still get included when you try to include AMReX in a project, and hardcoded MPI logic in other Headers (like AMReX_Box.H) that isn't wrapped with an #ifdef AMReX_MPI.
Makes me conclude we can only build a parallel version.
Won't that cause problems with our DB plugins (MDServer lib portion and ENGINE_SER lib portion)?
markcmiller86 commentedon Jun 2, 2023
We've run into this kind of thing before and you've seen my hacks. I tried my best to create non-mpi and mpi installs of moab and hdf5...hdf5_mpi and moab_mpi vs. hdf5 and moab.
But, @cyrus might have a better strategy...that is to let the mpi stuff be there in a
_ser.soshared lib and just know it will (should) never be called...maybe should never actually make through a compile to a_ser.sotarget...maybe even add an assertion somewhere that ensures that.biagas commentedon Jun 2, 2023
The hdf5 vs hdf5_mpi works because hdf5 can actually build a serial version of itself. I don't think having the two separate libs for serial and parallel is necessarily a hack, just a necessity.
I think AMReX is different in that it cannot be built as a serial library (at least not so far as I have discovered).
Doesn't ADIOS2 also require MPI? Guess I should check how that is handled in the plugin.
BenWibking commentedon Jul 6, 2023
As an AMReX and VisIt user, I want to note that AMReX has added support for particles (see, e.g.: https://amrex-codes.github.io/amrex/docs_html/Particle.html#particle-io).
In order to read outputs with particles, we have to use either yt or Paraview at the moment, but we'd be very happy to use VisIt for this as well :)
It should be possible to read them using
vtkAMReXParticlesReader: https://gitlab.kitware.com/vtk/vtk/-/blob/master/IO/AMR/vtkAMReXParticlesReader.cxxmarkcmiller86 commentedon Jul 6, 2023
@BenWibking it looks like VTK has two AMReX related readers...
Any understanding how these relate? Can they be used together on the same AMReX file(s)?
BenWibking commentedon Jul 6, 2023
vtkAMReXGridReaderreads the grid/field data and the latter reads the particles. For backward compatibility reasons (I think), the particle data is stored in separate files (but there should be a correspondence for outputs at the same time).@atmyers @WeiqunZhang should be able to help more with this.
BenWibking commentedon Jul 6, 2023
My understanding is that they should be usable on the same AMReX files (which are actually directories), with the particles stored in a subdirectory of the main AMReX file/directory.
The particle format is less well documented, so hopefully the AMReX developers can fill in the gaps here.
atmyers commentedon Jul 6, 2023
It should be possible to build a serial version of AMReX by setting
AMReX_MPItoOff. Some things may betypedefed out in that case; for example,MPI_DatatypebecomesintifAMReX_MPIisFALSE. But no actual code from the mpi headers will be used, and if it is, we'd consider that a bug in AMReX.atmyers commentedon Jul 6, 2023
As @BenWibking said, the particle data is stored in subdirectories inside the main AMReX plotfile directory. Particles are optional in AMReX, and there may be multiple subdirectories corresponding to different particle types (e.g.
electrons,positrons, etc... ). The way that the particles are split into separate files is described in theHeaderfile for the particle subdirectory and in general works the same way as the mesh data.I've never used the vtk readers before, but here is a python script that parses the particle data in AMReX plotfiles as numpy arrays: https://github.com/AMReX-Codes/amrex/blob/development/Tools/Py_util/amrex_particles_to_vtp/amrex_binary_particles_to_vtp.py
BenWibking commentedon Jul 6, 2023
@atmyers If there is a subdirectory of a
plt*directory that is not prefixed withLevel_(such aselectronsorpositronsin your example), can it always be assumed that this subdirectory corresponds to a set of particles of a given type?atmyers commentedon Jul 7, 2023
Probably, but better would be to check for the presence of a valid
Headerfile inside that directory. The first entry will be a string that saysVersion_*, for exampleVersion_Two_Dot_Zero_double.markcmiller86 commentedon Jul 7, 2023
Ok, thanks @atmyers and @BenWibking for info. This will help in deciding how best to move forward.
On a related note, can either of you provide reliable test data that manifests various features of interest (e.g. particles, amrexvec_nu, etc.)?
BenWibking commentedon Jul 8, 2023
There are many such datasets available here (listed under 'boxlib frontend'). Some of them are old datasets that used Boxlib, but most are newer datasets from AMReX.
For example, the
GaussianBeamdataset from WarpX has two AMR levels and two particle species: https://yt-project.org/data/GaussianBeam.tar.gz. If I recall correctly, I think @atmyers contributed the WarpX ones, so he can provide guidance on what's the best one to use.I'm not actually familiar with the amrexvec_nu feature. Can you point to where this is documented/used?
BenWibking commentedon Jul 8, 2023
I should add that some outputs have node- and face-centered fields. I don't think the VTK reader can read those. I think yt is currently the only code that can read those, but @atmyers is the expert on this.
atmyers commentedon Jul 12, 2023
Probably the one here would be best (two species, cell-centered fields only): https://yt-project.org/data/PlasmaAcceleration_v2.tar.gz