Redis::hMset() SSL error: 0A00010F:SSL routines::bad length

[gbalduzzi]

Horizon Version

5.31.1

Laravel Version

12.7.2

PHP Version

8.2.22

Redis Driver

PhpRedis

Redis Version

6.0

Database Driver & Version

No response

Description

My project is running on digital ocean clusters with a Digital ocean managed redis database.

Lately, the traffic on the server increased and a bunch of this error started to pop up, right now it affects around 1 every 5 scheduled jobs. This is the stacktrace:

Redis::hMset(): SSL operation failed with code 1. OpenSSL Error messages:
error:0A00010F:SSL routines::bad length {"exception":"[object] (ErrorException(code: 0): Redis::hMset(): SSL operation failed with code 1. OpenSSL Error messages:
error:0A00010F:SSL routines::bad length at /var/www/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php:116)
[stacktrace]
#0 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php(256): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->handleError(2, 'Redis::hMset():...', '/var/www/vendor...', 116)
#1 [internal function]: Illuminate\\Foundation\\Bootstrap\\HandleExceptions->Illuminate\\Foundation\\Bootstrap\\{closure}(2, 'Redis::hMset():...', '/var/www/vendor...', 116)
#2 /var/www/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(116): Redis->hMset('524397ee-1948-4...', Array)
#3 /var/www/vendor/laravel/framework/src/Illuminate/Redis/Connections/PhpRedisConnection.php(531): Illuminate\\Redis\\Connections\\Connection->command('hmset', Array)
#4 /var/www/vendor/laravel/framework/src/Illuminate/Redis/Connections/PhpRedisConnection.php(136): Illuminate\\Redis\\Connections\\PhpRedisConnection->command('hmset', Array)
#5 /var/www/vendor/laravel/horizon/src/Repositories/RedisJobRepository.php(373): Illuminate\\Redis\\Connections\\PhpRedisConnection->hmset('524397ee-1948-4...', Array)
#6 /var/www/vendor/laravel/horizon/src/Listeners/MarkJobAsReserved.php(36): Laravel\\Horizon\\Repositories\\RedisJobRepository->reserved('redis', 'default', Object(Laravel\\Horizon\\JobPayload))
#7 /var/www/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(479): Laravel\\Horizon\\Listeners\\MarkJobAsReserved->handle(Object(Laravel\\Horizon\\Events\\JobReserved))
#8 /var/www/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(286): Illuminate\\Events\\Dispatcher->Illuminate\\Events\\{closure}('Laravel\\\\Horizon...', Array)
#9 /var/www/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(266): Illuminate\\Events\\Dispatcher->invokeListeners('Laravel\\\\Horizon...', Array, false)
#10 /var/www/vendor/laravel/horizon/src/RedisQueue.php(204): Illuminate\\Events\\Dispatcher->dispatch('Laravel\\\\Horizon...')
#11 /var/www/vendor/laravel/horizon/src/RedisQueue.php(141): Laravel\\Horizon\\RedisQueue->event('default', Object(Laravel\\Horizon\\Events\\JobReserved))
#12 /var/www/vendor/laravel/framework/src/Illuminate/Support/helpers.php(399): Laravel\\Horizon\\RedisQueue->Laravel\\Horizon\\{closure}(Object(Illuminate\\Queue\\Jobs\\RedisJob))
#13 /var/www/vendor/laravel/horizon/src/RedisQueue.php(139): tap(Object(Illuminate\\Queue\\Jobs\\RedisJob), Object(Closure))
#14 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(350): Laravel\\Horizon\\RedisQueue->pop('default', 0)
#15 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(365): Illuminate\\Queue\\Worker->Illuminate\\Queue\\{closure}('default', 0)
#16 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(163): Illuminate\\Queue\\Worker->getNextJob(Object(Laravel\\Horizon\\RedisQueue), 'default')
#17 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(148): Illuminate\\Queue\\Worker->daemon('redis', 'default', Object(Illuminate\\Queue\\WorkerOptions))
#18 /var/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(131): Illuminate\\Queue\\Console\\WorkCommand->runWorker('redis', 'default')
#19 /var/www/vendor/laravel/horizon/src/Console/WorkCommand.php(51): Illuminate\\Queue\\Console\\WorkCommand->handle()
#20 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Laravel\\Horizon\\Console\\WorkCommand->handle()
#21 /var/www/vendor/laravel/framework/src/Illuminate/Container/Util.php(43): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#22 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(96): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))
#23 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#24 /var/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(754): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#25 /var/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(211): Illuminate\\Container\\Container->call(Array)
#26 /var/www/vendor/symfony/console/Command/Command.php(279): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#27 /var/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(180): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#28 /var/www/vendor/symfony/console/Application.php(1094): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#29 /var/www/vendor/symfony/console/Application.php(342): Symfony\\Component\\Console\\Application->doRunCommand(Object(Laravel\\Horizon\\Console\\WorkCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#30 /var/www/vendor/symfony/console/Application.php(193): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#31 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(197): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#32 /var/www/artisan(33): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))

I'm posting here because redis is used for cache, session and to store a bunch of other data with pretty heavy traffic and I'm only receiving this specific error on the Redis::hMset by horizon.

My config:

database.php:

'default' => [
            'scheme' => env('REDIS_SCHEME', 'tls'),
            'url' => env('REDIS_URL'),
            'host' => env('REDIS_HOST', '127.0.0.1'),
            'password' => env('REDIS_PASSWORD', null),
            'port' => env('REDIS_PORT', 6379),
            'database' => env('REDIS_DB', 0),
            'timeout' => 2,
            'read_timeout' => 3,
            'persistent' => true,
        ],

queue.php

'redis' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue' => 'default',
            'retry_after' => 65,
            'block_for' => 2,
        ],

horizon.php

'supervisor-prod' => [
                'connection' => 'redis',
                'queue' => ['default'],
                'balance' => 'auto',
                'maxProcesses' => env('HORIZON_MAX_PROCESSES', 10),
                'minProcesses' => env('HORIZON_MIN_PROCESSES', 3),
                'memory' => 128,
                'tries' => 3,
                'nice' => 0,
            ],

Steps To Reproduce

Unsure, hard to reproduce locally because of the tsl connection required

[github-actions]

Thank you for reporting this issue!

As Laravel is an open source project, we rely on the community to help us diagnose and fix issues as it is not possible to research and fix every issue reported to us via GitHub.

If possible, please make a pull request fixing the issue you have described, along with corresponding tests. All pull requests are promptly reviewed by the Laravel team.

Thank you!

[gbalduzzi]

I added:

'ssl_options' => [
   'verify_peer' => false,
],

To the default redis connection in database.php and I don't see the exceptions anymore.
I feel it's more a workaround than an actual fix but it works

EDIT: this didn't work (it only appeared so after the deployment), see next comment

[gbalduzzi]

Update for anyone facing a similar issue, I found a solution thanks to rechl answer here on the phpredis repository: phpredis/phpredis#2437 (comment)

Basically:

• Upgrade phpredis to >= 6.1.0
• use persistent in combination with this snippet to lock the persistent connection id to the active process:

if (app()->runningInConsole()) {
    config([
        'database.redis.queue.persistent_id' => 'redis-pid-'.getmypid().'-queue',
    ]);
}