parallel engine cores during test-suite run

[biagas]

Describe the bug

I run the test suite on pascal with --lessverbose and capture the output to a file.
I see this error a lot when I run in parallel mode.

srun: error: pascal3: tasks 0-1: Aborted (core dumped)

[markcmiller86]

In parallel mode, how does the script limit the number of "jobs" it has running simultaneously and does changing that number help? I think its the -n # option or --num-processes=#. Default is 8. At 2 task/job, that fits on one 18 core node. At 3 tasks/job, it does not. Not sure if that is relevant or not as I often run a single MPI job oversubscribed (more tasks than cores) without issue on my OSX system. I dunno how it would handle multiple smaller MPI jobs that in the aggregate exceed the number of actual cores.

[cyrush]

We are changing -n to default to 1 in #4094

There is some stream crossing going on with the multi processing case.

Just to be clear, is this an issue with -n > 1 or just an issue with --lessverbose and an mpi launch?

[biagas]

I see the engine cores when I run with '--lessverbose -m parallel'. I do not use the -n option.

[brugger1]

If you don't specify -n 1, I believe it will run multiple tests at once (n > 1). Can you run it with -n 1 and see if it still gives the errors.

[biagas]

My bad, I looked again at my command line and I do pass '-n 1'.