Repackage libsim examples @ visitusers.org #VisIt-tutorial-in-situ

[markcmiller86]

Is your feature request related to a problem?

There are some great libsim examples here. Each has a standalone tarball to run. But, they are all on NERSC. We should re-package to GitHub and employ as part of our regular testing too...I recommend taking some of those simulation tests and instead of running them via embedding in our test suite, run them instead stand-alone like a user running these examples would.

We probably don't need to handle all the examples on visitusers.org but a handful would be good.

[markcmiller86]

In fact, we might want a new target as part of a release that packages up these tarballs (like we do with some example data sets).

[markcmiller86]

The mandelbrot example at NERSC works with current develop. Here are some more things to be aware of...

• The tarball for that example had an object file from a previous build that can get in the way (patch.o).
• No matter what I tried (I tried a lot) I could not get VisIt to dlopen() the libsimV2runtime_ser.dylib with an installed binary download of VisIt. See more on that below on the section on dlopen searching.
• I did get it to work with my own develop build but only after using the -dir /path/to/visit/home option to mandelbrot (that option doesn't fix things when using the binary download of VisIt though)
  • I did run into some other (benign as far as the example goes) dlopen errors...

  dlerror() = dlopen(libospray_module_ispc.dylib, 10): image not found
  dlerror() = dlopen(libospray_module_ispc_native_float.dylib, 10): image not found
  dlerror() = dlopen(libospray_module_ispc_avx2_float.dylib, 10): image not found
  dlerror() = dlopen(libospray_module_ispc_avx_float.dylib, 10): image not found
  dlerror() = dlopen(libospray_module_ispc_sse4_float.dylib, 10): image not found
  #osp: INITIALIZATION ERROR --> could not open module lib ospray_module_ispc
  

• Another way it worked was to copy libsimV2runtime_ser.dylib to $cwd so dlopen could find it.
• If one types run at mandelbrot command prompt, it starts spewing messages for each simulation timestep. But, that prevents a user from seeing/knowing that s/he can type halt in the terminal to pause it. I would be better to use a \r form of printf for these messages.
• help at mandelbrot command produces a useful list of commands. However, it would be better if alphabetized and step command is missing from the help output.

dlopen search potential gatcha with code-sign

See below a portion of man page on dlopen from macOS. In particular, pay attention to the remarks regarding code-signed applications...

SEARCHING
     dlopen() searches for a compatible Mach-O file in the directories specified by a set of environment variables and
     the process's current working directory.  When set, the environment variables contain a colon-separated list of
     directory paths, which can be absolute or relative to the current working directory.

     When path does not contain a slash character (i.e. it is just a leaf name), dlopen() searches the following until
     it finds a compatible Mach-O file: $LD_LIBRARY_PATH, $DYLD_LIBRARY_PATH, current working directory, $DYLD_FALL-
     BACK_LIBRARY_PATH.

     When path looks like a framework path (e.g. /stuff/foo.framework/foo), dlopen() searches the following until it
     finds a compatible Mach-O file: $DYLD_FRAMEWORK_PATH (with framework partial path from path ), then the supplied
     path (using current working directory for relative paths), then $DYLD_FALLBACK_FRAMEWORK_PATH (with framework
     partial path from path ).

     When path contains a slash but is not a framework path (i.e. a full path or a partial path to a dylib), dlopen()
     searches the following until it finds a compatible Mach-O file: $DYLD_LIBRARY_PATH (with leaf name from path ),
     then the supplied path (using current working directory for relative paths), then $DYLD_FALLBACK_LIBRARY_PATH
     (with leaf name from path ).

     Note: If DYLD_FALLBACK_LIBRARY_PATH is not set, dlopen operates as if DYLD_FALLBACK_LIBRARY_PATH was set to
     $HOME/lib:/usr/local/lib:/usr/lib.

     Note: If DYLD_FALLBACK_FRAMEWORK_PATH is not set, dlopen operates as if DYLD_FALLBACK_FRAMEWORK_PATH was set to
     $HOME/Library/Frameworks:/Library/Frameworks:/Network/Library/Frameworks:/System/Library/Frameworks.

     Note: There are no configuration files to control dlopen searching.

     Note: If the main executable is a set[ug]id binary or codesigned with entitlements, then all environment vari-
     ables are ignored, and only a full path can be used.

[markcmiller86]

When I try same operations with VisIt 3.2.1 on surface, it doesn't work. I built mandelbrot using /usr/tce/packages/gcc/gcc-6.1.0/bin/g++ which is the same compiler used to build VisIt. The one error message called out from the others, below, is the correct path to libsimV2runtime_ser.so and it doesn't load it.

/usr/gapps/visit/3.2.1/bin/visit &
./mandelbrot -trace trace.txt -dir /usr/gapps/visit/3.2.1/linux-x86_64
.
.
.
Attempting to preload osmesa as libGL -- calling dlopen(/usr/gapps/visit/3.1.4/linux-x86_64/lib/osmesa/libGL.so.1)
dlopen error: libLLVM-5.0.so: cannot open shared object file: No such file or directory
failed to preload osmesa from VISIT_MESA_LIB: /usr/gapps/visit/3.1.4/linux-x86_64/lib/osmesa/libGL.so.1
Calling dlopen(libsimV2runtime_ser.so)
Failed to open the VisIt library: libLLVM-6.0.so: cannot open shared object file: No such file or directory

Calling getenv(LD_LIBRARY_PATH)
getenv returned: /usr/gapps/visit/3.1.4/linux-x86_64/lib/osmesa:/usr/gapps/visit/3.1.4/linux-x86_64/lib/mesagl:/usr/gapps/visit/3.1.4/linux-x86_64/lib:/usr/gapps/visit/bin/../current/linux-x86_64/lib:/usr/gapps/visit/3.1.4/linux-x86_64/lib/osmesa:/usr/gapps/visit/3.1.4/linux-x86_64/lib/mesagl:/usr/gapps/visit/3.1.4/linux-x86_64/lib:/usr/gapps/visit/bin/../current/linux-x86_64/lib:/usr/gapps/visit/3.2.1/linux-x86_64/lib
Calling dlopen(/usr/gapps/visit/3.1.4/linux-x86_64/lib/osmesa/libsimV2runtime_ser.so)
dlopen error: /usr/gapps/visit/3.1.4/linux-x86_64/lib/osmesa/libsimV2runtime_ser.so: cannot open shared object file: No such file or directory
Calling dlopen(/usr/gapps/visit/3.1.4/linux-x86_64/lib/mesagl/libsimV2runtime_ser.so)
dlopen error: /usr/gapps/visit/3.1.4/linux-x86_64/lib/mesagl/libsimV2runtime_ser.so: cannot open shared object file: No such file or directory

Calling dlopen(/usr/gapps/visit/3.1.4/linux-x86_64/lib/libsimV2runtime_ser.so)

dlopen error: libLLVM-5.0.so: cannot open shared object file: No such file or directory
Calling dlopen(/usr/gapps/visit/bin/../current/linux-x86_64/lib/libsimV2runtime_ser.so)
dlopen error: libLLVM-5.0.so: cannot open shared object file: No such file or directory
Calling dlopen(/usr/gapps/visit/3.1.4/linux-x86_64/lib/osmesa/libsimV2runtime_ser.so)
dlopen error: /usr/gapps/visit/3.1.4/linux-x86_64/lib/osmesa/libsimV2runtime_ser.so: cannot open shared object file: No such file or directory
Calling dlopen(/usr/gapps/visit/3.1.4/linux-x86_64/lib/mesagl/libsimV2runtime_ser.so)
dlopen error: /usr/gapps/visit/3.1.4/linux-x86_64/lib/mesagl/libsimV2runtime_ser.so: cannot open shared object file: No such file or directory
Calling dlopen(/usr/gapps/visit/3.1.4/linux-x86_64/lib/libsimV2runtime_ser.so)
dlopen error: libLLVM-5.0.so: cannot open shared object file: No such file or directory
Calling dlopen(/usr/gapps/visit/bin/../current/linux-x86_64/lib/libsimV2runtime_ser.so)
dlopen error: libLLVM-5.0.so: cannot open shared object file: No such file or directory
Calling dlopen(/usr/gapps/visit/3.2.1/linux-x86_64/lib/libsimV2runtime_ser.so)
dlopen error: libLLVM-6.0.so: cannot open shared object file: No such file or directory
LoadVisItLibrary Failed to open the VisIt library: (null)

[markcmiller86]

I just noticed another potential issue here. Even though I built mandlebrot with 3.2.1 and pass it a -dir option pointing to 3.2.1, all the error messages are for 3.1.4 paths.

[markcmiller86]

By switching to VisIt version 3.1.4 (because I don't know why all those paths are specific to that version but maybe using that specific version will help) and setting LD_LIBRARY_PATH to dir containing libLLVM-5.0.so, I get quite a bit further but it still says VisIt did not connect:

trace.txt