[BUG] pyscenic grn errors: Resource temporarily unavailable & empty intersection

[radiasso]

Hello,
I'm trying to run pyscenic grn as detailed in the tutorial.

I have my anndata object adata, used to create the required loom object output_SCENIC.loom:

row_attrs = {
    "Gene": np.array(adata.var_names) ,
}
col_attrs = {
    "CellID": np.array(adata.obs_names) ,
    "nGene": np.array( np.sum(adata.X.transpose()>0 , axis=0)).flatten() ,
    "nUMI": np.array( np.sum(adata.X.transpose() , axis=0)).flatten() ,
}
lp.create( f_loom_path_scenic, adata.X.transpose(), row_attrs, col_attrs)

I downloaded the TF file list for mouse from here.

And then ran the command for pyscenic grn as shown in the tutorial:
pyscenic grn /mypath/Loom/output_SCENIC.loom /mypath/Loom/mouse_TFs.txt -o adj.csv --num_workers 20

Errors encountered:

2025-03-18 14:05:05,407 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.
Traceback (most recent call last):
  File "/home/giuliacarangelo/.local/bin/pyscenic", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/giuliacarangelo/.local/share/pipx/venvs/pyscenic/lib/python3.12/site-packages/pyscenic/cli/pyscenic.py", line 713, in main
    args.func(args)
  File "/home/giuliacarangelo/.local/share/pipx/venvs/pyscenic/lib/python3.12/site-packages/pyscenic/cli/pyscenic.py", line 59, in find_adjacencies_command
    ex_mtx = load_exp_matrix(
             ^^^^^^^^^^^^^^^^
  File "/home/giuliacarangelo/.local/share/pipx/venvs/pyscenic/lib/python3.12/site-packages/pyscenic/cli/utils.py", line 134, in load_exp_matrix
    return load_exp_matrix_as_loom(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/giuliacarangelo/.local/share/pipx/venvs/pyscenic/lib/python3.12/site-packages/pyscenic/cli/utils.py", line 82, in load_exp_matrix_as_loom
    with lp.connect(fname, mode="r", validate=False) as ds:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/giuliacarangelo/.local/share/pipx/venvs/pyscenic/lib/python3.12/site-packages/loompy/loompy.py", line 1634, in connect
    return LoomConnection(filename, mode, validate=validate)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/giuliacarangelo/.local/share/pipx/venvs/pyscenic/lib/python3.12/site-packages/loompy/loompy.py", line 88, in __init__
    self._file = h5py.File(filename, mode)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/giuliacarangelo/.local/share/pipx/venvs/pyscenic/lib/python3.12/site-packages/h5py/_hl/files.py", line 564, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/giuliacarangelo/.local/share/pipx/venvs/pyscenic/lib/python3.12/site-packages/h5py/_hl/files.py", line 238, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 102, in h5py.h5f.open
BlockingIOError: [Errno 11] Unable to synchronously open file (unable to lock file, errno = 11, error message = 'Resource temporarily unavailable')

Libraries and versions:

• pySCENIC version: 0.12.1
• Installation method: pip
• Run environment: jupyter notebook (for file preparation) and CLI (for tool usage)
• OS: Ubuntu 24.04
• Python version: 3.12.3

At that point, I tried different approaches, each one raising an error. The approach that got me the farthest was creating a csv file for the expression matrix, instead of the loom file:

expression_matrix = adata.X.T
if isinstance(expression_matrix, csr_matrix):
    c = expression_matrix.toarray()  # Convert sparse to dense matrix

gene_names = adata.var_names
cell_ids = adata.obs_names
df = pd.DataFrame(expression_matrix, index=gene_names, columns=cell_ids)
df.to_csv("Loom/expression_matrix.csv")

Which then behaved as follows:

2025-03-18 13:45:11,896 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.

2025-03-18 13:46:17,083 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks.
/home/giuliacarangelo/.local/share/pipx/venvs/pyscenic/lib/python3.12/site-packages/distributed/node.py:187: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 41263 instead
  warnings.warn(
preparing dask client
parsing input
not shutting down client, client was created externally
finished
Traceback (most recent call last):
  File "/home/giuliacarangelo/.local/bin/pyscenic", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/giuliacarangelo/.local/share/pipx/venvs/pyscenic/lib/python3.12/site-packages/pyscenic/cli/pyscenic.py", line 713, in main
    args.func(args)
  File "/home/giuliacarangelo/.local/share/pipx/venvs/pyscenic/lib/python3.12/site-packages/pyscenic/cli/pyscenic.py", line 106, in find_adjacencies_command
    network = method(
              ^^^^^^^
  File "/home/giuliacarangelo/.local/share/pipx/venvs/pyscenic/lib/python3.12/site-packages/arboreto/algo.py", line 39, in grnboost2
    return diy(expression_data=expression_data, regressor_type='GBM', regressor_kwargs=SGBM_KWARGS,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/giuliacarangelo/.local/share/pipx/venvs/pyscenic/lib/python3.12/site-packages/arboreto/algo.py", line 115, in diy
    expression_matrix, gene_names, tf_names = _prepare_input(expression_data, gene_names, tf_names)
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/giuliacarangelo/.local/share/pipx/venvs/pyscenic/lib/python3.12/site-packages/arboreto/algo.py", line 229, in _prepare_input
    raise ValueError('Intersection of gene_names and tf_names is empty.')
ValueError: Intersection of gene_names and tf_names is empty.

When I know for a fact that there are several genes (over 1000) as intersection, I checked personally.

I can't seem to figure this out, since neither the tutorial nor my other attempts seem to work. I even tried to change --num_workers 1 to see if this whole thing was related to parallel working, and added --method grnboost2 just to be sure.
Thank you for all your help in advance!