Indexing gets stuck on some files

[nielstron]

Describe the bug

When running a re-index (using php occ memories:index --force --folder path/to/external_storage) on some (sub-)folder of an external storage, I see progress ... until at some point the script just gets stuck on one file (i.e. /niels/files/storagebox/Backup/Pictures/2019BlackberryMissing/IMG_20191214_1044121.jpgfor a complete scan of my user --user niels , but different subfolders feature different files for getting stuck and also not specifying a user gets stuck).

I would expect that after some time, the operation just times out and the next file is processed. However, I see that progress stalls, usually even hours after the script was started.

Steps To Reproduce

1. php occ memories:index --force --folder path/to/external_storage
2. wait

Platform

• OS:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

• Browser: N/A
• Memories Version: Version 7.3.1
• Nextcloud Version: Nextcloud Hub 8 (29.0.6)
• PHP Version:

$ php --version
PHP 8.3.6 (cli) (built: Jun 13 2024 15:23:20) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.3.6, Copyright (c) Zend Technologies
    with Zend OPcache v8.3.6, Copyright (c), by Zend Technologies

Screenshots

No response

Additional context

No errors are being reported. The issue remains quite consistently, albeit sometimes the file works. I therefore assume its some locking or downloading-from-external-storage issue. However I would assume that this should simply time out after 30 seconds or a user-defined timeout rather than block the whole process.

The external storage in question is connected via SFTP using the user-settings.

[AdmiralNemo]

I've encountered this issue as well, but in my case, the files the indexer gets "stuck" on are local to the Nextcloud server rather than external storage folders.

[pulsejet]

Is there a particular file (e.g. a sample you can provide) that it gets stuck on? The reader does have a 30s timeout so it ideally should never get stuck.

[pulsejet]

Please try the patch a4b8960

[AdmiralNemo]

Ok, in my case, the problem ended up being a deadlock of some sort on the database server. There was an INSERT INTO oc_memories ... query stuck in "active" state that was preventing any further changes to the table. After terminating that client, occ memories:index was able to run successfully without getting stuck.