Unable to start VisIt from command line on MacOS Monterey

[doncip]

Describe the bug

Unable to start VisIt from command line on MacOS Monterey.
Used the command in the terminal: visit -cli -nowin -s /path/to/script/script.py

Helpful additional information

• Did VisIt crash: It crashed when launching the compute engine
• Did you get wrong results: **

To Reproduce

Steps to reproduce the behavior. For example:

1. Launch from command line with the command: visit -cli -nowin -s /path/to/script/script.py (script attached)
2. Viewer is started but the following messages appear:

[cli_0]: PMIU_parse_keyvals: unexpected key delimiter at character 57 in cmd
[cli_0]: parse_kevals failed -1
[cli_1]: PMIU_parse_keyvals: unexpected key delimiter at character 57 in cmd
[cli_1]: parse_kevals failed -1
Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(572):
MPID_Init(175).......: channel initialization failed
MPID_Init(510).......: PMI_Get_id returned -1
Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(572):
MPID_Init(175).......: channel initialization failed
MPID_Init(510).......: PMI_Get_id returned -1

Expected behavior

Start computation. The script has been already successfully tested on a Windows machine

Attachments

• Python script attached

Desktop

• OS and version: MacOS Monterey 12.6
• VisIt Version: 3.3.1 and 3.3.0

PlotSingle.py.zip

[markcmiller86]

Can you run visit -cli -nowin -debug 5 -s /path/to/script/script.py and attach here another zip file (or gzip'd tar file) with all level 5 debug logs.

[doncip]

I attach the output of the command here
VisitLogs.zip

[markcmiller86]

Thanks. Unfortunately, those do not provide any information.

What happens if you run...

visit -np 2 -cli -nowin -o /path/to/visit/test/data/multi_ucd3d.silo

Also, can you please paste the entire command line(s) VisIt echos when launching as in...

/Applications/VisIt.app/Contents/Resources/bin/visit -np 2 -o /Users/miller86/visit/visit/data/silo_pdb_test_data/multi_ucd3d.silo
Running: gui3.3.1 -o /Users/miller86/visit/visit/data/silo_pdb_test_data/multi_ucd3d.silo -engineargs -l;mpiexec;-np;2 -launchengine localhost
Running: viewer3.3.1 -engineargs -l;mpiexec;-np;2 -geometry 1567x1177+353+0 -borders 20,0,0,1 -shift 0,0 -preshift 0,0 -defer -host 127.0.0.1 -port 5600 -launchengine localhost
Running: mdserver3.3.1 -host 127.0.0.1 -port 5601
Running: /Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64/bin/mpiexec -n 2 /Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64/bin/engine_par -plugindir /Users/miller86/.visit/3.3.1/darwin-x86_64/plugins:/Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64/plugins -visithome /Applications/VisIt.app/Contents/Resources/3.3.1 -visitarchhome /Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64 -host 127.0.0.1 -port 5600

[doncip]

I ran the command you indicated, and here is the output:

Running: cli3.3.1 -nowin -o /Applications/VisIt.app/Contents/Resources/data/multi_ucd3d.silo -engineargs -l;mpiexec;-np;2 -launchengine localhost
Running: viewer3.3.1 -nowin -noint -engineargs -l;mpiexec;-np;2 -host 127.0.0.1 -port 5600 -launchengine localhost
Running: /Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64/bin/mpiexec -n 2 /Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64/bin/engine_par -plugindir /Users/Mattia/.visit/3.3.1/darwin-x86_64/plugins:/Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64/plugins -visithome /Applications/VisIt.app/Contents/Resources/3.3.1 -visitarchhome /Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64 -host 127.0.0.1 -port 5600
[cli_0]: PMIU_parse_keyvals: unexpected key delimiter at character 56 in cmd
[cli_0]: parse_kevals failed -1
[cli_1]: PMIU_parse_keyvals: unexpected key delimiter at character 56 in cmd
[cli_1]: parse_kevals failed -1
Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(572): 
MPID_Init(175).......: channel initialization failed
MPID_Init(510).......: PMI_Get_id returned -1
Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(572): 
MPID_Init(175).......: channel initialization failed
MPID_Init(510).......: PMI_Get_id returned -1

[markcmiller86]

Hmm...this is very strange. Ok, let me ask if you could please try the following

#include <stdio.h>
#include <mpi.h>

int main(int argc, char **argv)
{
    int rank = 0;
    int size = 1;

   MPI_Init(&argc, &argv);

   MPI_Comm_rank(MPI_COMM_WORLD, &rank);
   MPI_Comm_size(MPI_COMM_WORLD, &size);

   printf("rank=%02d, size=%02d\n", rank, size);

   MPI_Finalize();
}

1. take the above program and make bar.c with it.
2. compile it with...
   /Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64/bin/mpicc bar.c -o bar
3. Run it with...
   /Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64/bin/mpiexec -n 5 ./bar
4. cut-n-paste the output (and/or errors) from all of the above operations here

You should get something like...

rank=00, size=05
rank=03, size=05
rank=01, size=05
rank=04, size=05
rank=02, size=05

[markcmiller86]

I am attaching here the bar executable I get on my mac (catalina). If you are willing and trust me, try uncompressing (gunzip bar.gz) and running it on your machine with the same run line (step 3 above).

bar.gz (MD5 (bar.gz) = e9f95cd129070d0e538cdb745004f949

And see what you get.

This could be pointing to some kind of incompatability with mpich between catalina and monterey.

One final thing, paste here the output from running otool -l on bar here.

[doncip]

Hi, I started with making the bar executable, but when compiling the script you wrote I got this:

bar.c:2:10: fatal error: 'mpi.h' file not found
#include <mpi.h>
         ^~~~~~~
1 error generated.

If I understood correctly, MPI should be embedded with Visit. It is not clear to me why it is not pointing to the correct header file.

Executing your file returned this:

dyld[12603]: Library not loaded: '/Users/miller86/visit/visit/33rc/release/build-mb-3.3.1-darwin-10.15-x86_64-release/thirdparty_shared/third_party/mpich/3.3.1/darwin-x86_64/lib/libmpi.12.dylib'
  Referenced from: '/Users/Mattia/Desktop/bar'
  Reason: tried: '/Users/miller86/visit/visit/33rc/release/build-mb-3.3.1-darwin-10.15-x86_64-release/thirdparty_shared/third_party/mpich/3.3.1/darwin-x86_64/lib/libmpi.12.dylib' (no such file), '/usr/local/lib/libmpi.12.dylib' (no such file), '/usr/lib/libmpi.12.dylib' (no such file)
dyld[12604]: Library not loaded: '/Users/miller86/visit/visit/33rc/release/build-mb-3.3.1-darwin-10.15-x86_64-release/thirdparty_shared/third_party/mpich/3.3.1/darwin-x86_64/lib/libmpi.12.dylib'
  Referenced from: '/Users/Mattia/Desktop/bar'
  Reason: tried: '/Users/miller86/visit/visit/33rc/release/build-mb-3.3.1-darwin-10.15-x86_64-release/thirdparty_shared/third_party/mpich/3.3.1/darwin-x86_64/lib/libmpi.12.dylib' (no such file), '/usr/local/lib/libmpi.12.dylib' (no such file), '/usr/lib/libmpi.12.dylib' (no such file)
dyld[12605]: Library not loaded: '/Users/miller86/visit/visit/33rc/release/build-mb-3.3.1-darwin-10.15-x86_64-release/thirdparty_shared/third_party/mpich/3.3.1/darwin-x86_64/lib/libmpi.12.dylib'
  Referenced from: '/Users/Mattia/Desktop/bar'
  Reason: tried: '/Users/miller86/visit/visit/33rc/release/build-mb-3.3.1-darwin-10.15-x86_64-release/thirdparty_shared/third_party/mpich/3.3.1/darwin-x86_64/lib/libmpi.12.dylib' (no such file), '/usr/local/lib/libmpi.12.dylib' (no such file), '/usr/lib/libmpi.12.dylib' (no such file)
dyld[12606]: Library not loaded: '/Users/miller86/visit/visit/33rc/release/build-mb-3.3.1-darwin-10.15-x86_64-release/thirdparty_shared/third_party/mpich/3.3.1/darwin-x86_64/lib/libmpi.12.dylib'
  Referenced from: '/Users/Mattia/Desktop/bar'
  Reason: tried: '/Users/miller86/visit/visit/33rc/release/build-mb-3.3.1-darwin-10.15-x86_64-release/thirdparty_shared/third_party/mpich/3.3.1/darwin-x86_64/lib/libmpi.12.dylib' (no such file), '/usr/local/lib/libmpi.12.dylib' (no such file), '/usr/lib/libmpi.12.dylib' (no such file)
dyld[12607]: Library not loaded: '/Users/miller86/visit/visit/33rc/release/build-mb-3.3.1-darwin-10.15-x86_64-release/thirdparty_shared/third_party/mpich/3.3.1/darwin-x86_64/lib/libmpi.12.dylib'
  Referenced from: '/Users/Mattia/Desktop/bar'
  Reason: tried: '/Users/miller86/visit/visit/33rc/release/build-mb-3.3.1-darwin-10.15-x86_64-release/thirdparty_shared/third_party/mpich/3.3.1/darwin-x86_64/lib/libmpi.12.dylib' (no such file), '/usr/local/lib/libmpi.12.dylib' (no such file), '/usr/lib/libmpi.12.dylib' (no such file)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 12603 RUNNING AT MacBook Air di Mattia
=   EXIT CODE: 6
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions

I guess this is just because you precompiled the executable in your environment.

[markcmiller86]

Ok, I am learning something about what might be going wrong.

I adjusted some things on my end and am attaching two things here for you to try (still working with this hello-mpi-world, bar.c example).

First, here is an updated bar mach-O executable... bar.gz where I've fixed some rpath stuff. Can you please try running it again (step 3 above) and see if it works.

Next, I am going to ask if you can replace the mpicc you have in /Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64/bin with this one...mpicc.gz (maybe save the old one by renaming it to mpicc.old first).

Then, try steps to build bar from bar.c above and see if it builds and if so, if it runs too.

[doncip]

I ran you executable, but still getting the following:

(base) MacBook Air di Mattia:Downloads Mattia$ /Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64/bin/mpiexec -n 2 ./bar
[cli_1]: PMIU_parse_keyvals: unexpected key delimiter at character 56 in cmd
[cli_1]: parse_kevals failed -1
[cli_0]: PMIU_parse_keyvals: unexpected key delimiter at character 56 in cmd
[cli_0]: parse_kevals failed -1
Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(572): 
MPID_Init(175).......: channel initialization failed
MPID_Init(510).......: PMI_Get_id returned -1
Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(572): 
MPID_Init(175).......: channel initialization failed
MPID_Init(510).......: PMI_Get_id returned -1

I compiled the bar.c file with the mpicc you sent and this is the output:

(base) MacBook Air di Mattia:TestVisIt Mattia$ /Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64/bin/mpiexec -n 2 ./bar
dyld[16620]: Library not loaded: '@rpath/lib/libmpi.dylib'
  Referenced from: '/Users/Mattia/Desktop/TestVisIt/bar'
  Reason: tried: '/usr/local/lib/libmpi.dylib' (no such file), '/usr/lib/libmpi.dylib' (no such file)
dyld[16621]: Library not loaded: '@rpath/lib/libmpi.dylib'
  Referenced from: '/Users/Mattia/Desktop/TestVisIt/bar'
  Reason: tried: '/usr/local/lib/libmpi.dylib' (no such file), '/usr/lib/libmpi.dylib' (no such file)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 16620 RUNNING AT MacBook Air di Mattia
=   EXIT CODE: 6
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions

Probably the reference to libmpi was not correct in your mpicc but at least the output changed...

[markcmiller86]

Ok, I am still stumped by what is going on here. Do you have any MPICH environment variables defined in your default shell. To check myself on macOS, I do this...

env | grep -i mpi

If you get something output, can you paste here?

[markcmiller86]

So, examining your most recent attempt to run the bar executable I gave you...

dyld[16620]: Library not loaded: '@rpath/lib/libmpi.dylib'

means the executable bar was unable to load libmpi. What bothers me is that @rpath in bar should resolve to /Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64. It does on my (catalina) system. I can confirm by using otool -l and looking at the LC_RPATH entry...

% otool -l bar
bar:
Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777223          3  0x80           2    19       1552 0x00200085
Load command 0
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __PAGEZERO
   vmaddr 0x0000000000000000
   vmsize 0x0000000100000000
.
. (truncated)
.
Load command 18
          cmd LC_RPATH
      cmdsize 80
         path /Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64 (offset 12)

Can you please confirm you get same LC_RPATH entry using otool -l command as I do?

Next, can you confirm that there is indeed a libmpi.dylib in /Applications/VisIt.app/Contents/Resources/3.3.1/darwin-x86_64/lib directory AND that you do have permission to read it (maybe by trying to run otool -l on it also)?