VisIt client-server not working on Stampede3

[BenWibking]

I am attempting to use VisIt in client/server mode on a new cluster. It does not allow batch job names to have the ":" character, which VisIt attempts to do:

Running: sbatch --export=HOME=/home1/02661/bwibking,LIBPATH=/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1/linux-x86_64/lib,LD_LIBRARY_PATH=/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1/linux-x86_64/lib/osmesa:/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1/linux-x86_64/lib/mesagl:/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1/linux-x86_64/lib:/home1/02661/bwibking/visit3_4_1.linux-x86_64/bin/../3.4.1/linux-x86_64/lib:/opt/apps/intel24/hdf5/1.14.3/x86_64/lib:/opt/intel/oneapi/ccl/2021.11/lib:/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.11/lib:/opt/intel/oneapi/mkl/2024.0/lib:/opt/intel/oneapi/ipp/2021.10/lib:/opt/intel/oneapi/ippcp/2021.9/lib:/opt/intel/oneapi/dpl/2022.3/lib:/scratch/projects/compilers/intel24.0/oneapi/dnnl/2024.0/lib:/opt/intel/oneapi/debugger/2024.0/opt/debugger/lib:/scratch/projects/compilers/intel24.0/oneapi/dal/2024.0/lib:/opt/intel/oneapi/compiler/2024.0/include/sycl/ext/intel/esimd/emu:/opt/intel/oneapi/compiler/2024.0/opt/compiler/lib:/opt/intel/oneapi/compiler/2024.0/lib:/opt/intel/oneapi/tbb/2021.11/lib:/opt/intel/oneapi/compiler/2024.0/lib/clang/17/lib/x86_64-unknown-linux-gnu:/opt/apps/gcc/13.2.0/lib64:/opt/apps/gcc/13.2.0/lib:/usr/lib64:/scratch/projects/compilers/intel24.0/oneapi/intelpython/python3.9/lib,VISITHOME=/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1,VISITARCHHOME=/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1/linux-x86_64,VISITPLUGINDIR=/home1/02661/bwibking/.visit/3.4.1/linux-x86_64/plugins:/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1/linux-x86_64/plugins,PYTHONHOME=/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1/linux-x86_64/lib/python --partition=skx-dev --time=01:00:00 --ntasks=16 --nodes=1 --tasks-per-node=16 /tmp/visit.bwibking.12:43:34

-----------------------------------------------------------------
          Welcome to the Stampede3 Supercomputer
-----------------------------------------------------------------

No reservation for this job
--> Verifying valid submit host (login4)...OK
--> Verifying valid jobname...FAILED


Job names are not allowed to contain the ":" character.
--> submitted jobname = visit.bwibking.12:43:34

Please remove and resubmit the job.

Is there a workaround for this?

[biagas]

In internallauncher, there are functions called CreateFilename for each type of launcher (eg sbatch).
You could modify the appropriate launcher to replace : with a different separator when retrieving the time.

I see this in the qsub submitter which could serve as an example:
tdate = time.asctime().replace(" ", "-")

[BenWibking]

Thanks, that worked.

Now the batch job fails with this message:

VisIt: Message - Running: sbatch --export=HOME=/home1/02661/bwibking,LIBPATH=/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1/linux-x86_64/lib,LD_LIBRARY_PATH=/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1/linux-x86_64/lib/osmesa:/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1/linux-x86_64/lib/mesagl:/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1/linux-x86_64/lib:/home1/02661/bwibking/visit3_4_1.linux-x86_64/bin/../3.4.1/linux-x86_64/lib:/opt/apps/intel24/hdf5/1.14.3/x86_64/lib:/opt/intel/oneapi/ccl/2021.11/lib:/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.11/lib:/opt/intel/oneapi/mkl/2024.0/lib:/opt/intel/oneapi/ipp/2021.10/lib:/opt/intel/oneapi/ippcp/2021.9/lib:/opt/intel/oneapi/dpl/2022.3/lib:/scratch/projects/compilers/intel24.0/oneapi/dnnl/2024.0/lib:/opt/intel/oneapi/debugger/2024.0/opt/debugger/lib:/scratch/projects/compilers/intel24.0/oneapi/dal/2024.0/lib:/opt/intel/oneapi/compiler/2024.0/include/sycl/ext/intel/esimd/emu:/opt/intel/oneapi/compiler/2024.0/opt/compiler/
VisIt: Message - lib:/opt/intel/oneapi/compiler/2024.0/lib:/opt/intel/oneapi/tbb/2021.11/lib:/opt/intel/oneapi/compiler/2024.0/lib/clang/17/lib/x86_64-unknown-linux-gnu:/opt/apps/gcc/13.2.0/lib64:/opt/apps/gcc/13.2.0/lib:/usr/lib64:/scratch/projects/compilers/intel24.0/oneapi/intelpython/python3.9/lib,VISITHOME=/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1,VISITARCHHOME=/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1/linux-x86_64,VISITPLUGINDIR=/home1/02661/bwibking/.visit/3.4.1/linux-x86_64/plugins:/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1/linux-x86_64/plugins,PYTHONHOME=/home1/02661/bwibking/visit3_4_1.linux-x86_64/3.4.1/linux-x86_64/lib/python --partition=skx-dev --time=01:00:00 --ntasks=16 --nodes=1 --tasks-per-node=16 /tmp/visit.bwibking.13-20-31
VisIt: Message - 
-----------------------------------------------------------------
          Welcome to the Stampede3 Supercomputer                 
-----------------------------------------------------------------
VisIt: Message - No reservation for this job
--> Verifying valid submit host (login4)...OK
--> Verifying valid jobname...OK
VisIt: Message - --> Verifying valid ssh keys...OK
--> Verifying access to desired queue (skx-dev)...OK
--> Checking available allocation 
VisIt: Message - (TG-AST090040)...OK
VisIt: Message - --> Quotas are not currently enabled for filesystem /home1/02661/bwibking...OK
VisIt: Message - --> Verifying that quota for filesystem /work2/02661/bwibking/stampede3 is at  0.00% allocated...OK
VisIt: Message - Submitted batch job 551617
VisIt: Error - Communication with the compute engine on host "login4.stampede3.tacc.utexas.edu" has been lost.

Is VisIt getting confused by the extra output from sbatch, or did something go wrong elsewhere?

[biagas]

I am not sure about this. Have you tried running with '-debug 5' and taking a look at the generated log files? (generated on both client and server).

I found a very old note related to needing to use VNC instead of client-server to stampede. Was that still true with stampede2?

I'm wondering if stampede3 needs a custom launcher or host profile?

[BenWibking]

Here are the log files: visit_logs.tar.gz.

I took a look at them and the relevant part seemed to be (although I'm not sure how to interpret it):

ParentProcess::GetClientSocketDescriptor: Connected socket
ParentProcess::GetClientSocketDescriptor: Set up using port 23849
ParentProcess::GetClientSocketDescriptor: Creating a socket
ParentProcess::GetClientSocketDescriptor: Setting socket options
ParentProcess::GetClientSocketDescriptor: Calling connect
(If you see no messages after this one, VisIt was not
able to connect to the client machine.  Nine times out
of ten, this is a firewall issue on the client machine.
It could also mean that VisIt was unable to resolve the
IP address for the client machine.  You may need to verify the contents of /etc/hosts.)
ParentProcess::GetClientSocketDescriptor: Connected socket
ParentProcess::Connect: Ordering the connections.
Exception: (LostConnectionException) /home/visit/visit3.4.1/src/common/comm/SocketConnection.C, line 786: <The reason for the exception was not described>
signalhandler_core: SIGABRT! (Aborted)
signalhandler_exit: SIGTERM!

I never used stampede2, so I don't know.

This is the host profile I created:

[BenWibking]

I opened a ticket with TACC and their reply was that they don't support client-server mode and said to use VNC instead. That is what I'll do for now, but I'll ask if there's a technical reason why it doesn't work on Stampede3.

[BenWibking]

Ok, TACC says:

TACC has never supported client server mode on the hpc systems. The main reason is client server mode is difficult to maintain.  Remote desktop use greatly simplifies support in that all that is required is for the user to be able to connect via vnc or dcv. We avoid the possibility of client server version mismatch problems as well as network issues that we have no control over. 

If you have any suggests for unofficially getting client-server working based on the logs I provided, let me know.