NVENC encoding fails for 10-bit HEVC sources on Pascal GPUs (GTX 1080 Ti) due to missing pixel format conversion

[Tusam3]

Describe the bug

When attempting to transcode 10-bit HEVC (H.265) video files using NVIDIA NVENC acceleration on a Pascal-generation GPU (specifically, a GeForce GTX 1080 Ti), the go-vod transcoder consistently fails with the error Provided device doesn't support required NVENC features.

Steps To Reproduce

1. Upload a 10-bit HEVC (H.265) video file to Nextcloud.
2. Ensure NVIDIA drivers are installed and functional (nvidia-smi shows healthy output).
3. In Nextcloud Memories Administration settings:
   • Enable "Video Streaming" and "Live Transcoding".
   • Under "HW Acceleration", enable "Enable acceleration with NVENC".
   • Keep "Enable NVENC Temporal AQ" checked and "CUDA scaler" selected.
   • Save settings.
4. Attempt to play the 10-bit HEVC video in Memories, triggering transcoding.
5. Observe go-vod logs (e.g., in data/nextcloud.log or systemd journal) for the FFmpeg command execution and error messages.

Platform

- Nextcloud Version: 31.0.5
- Memories App Version: 7.5.2
- go-vod Version: 0.2.6
- Operating System: Manjaro Linux
- NVIDIA Driver Version: 570.144
- CUDA Version: 12.8
- FFmpeg Version: n7.1.1 (built with --enable-cuda-llvm and --enable-nvenc)
- GPU: NVIDIA GeForce GTX 1080 Ti (Pascal architecture)

Screenshots

No response

Additional context

Expected Behavior:

The video should transcode successfully using NVENC hardware acceleration, converting the 10-bit HEVC input to 8-bit H.264 output.

Actual Behavior:

The go-vod transcoder fails with the following errors in FFmpeg:

[h264_nvenc @ 0x...] 10 bit encode not supported
[h264_nvenc @ 0x...] Provided device doesn't support required NVENC features
[vost#0:0/h264_nvenc @ 0x...] Error while opening encoder - maybe incorrect parameters such as bit_rate, rate, width or height.

Root Cause and Proposed Solution (based on debugging):

The issue stems from go-vod attempting to feed a 10-bit video stream directly to the h264_nvenc encoder. Pascal-generation NVENC hardware (like the GTX 1080 Ti) does not support 10-bit H.264 encoding (it only supports 8-bit H.264). While it can decode 10-bit HEVC and encode 10-bit HEVC, it cannot perform 10-bit H.264 encoding.

Manual FFmpeg testing confirms that inserting hwdownload,format=nv12 into the video filter (-vf) chain resolves this. This forces the 10-bit GPU-decoded frames to be downloaded to CPU memory, converted to 8-bit nv12 format, and then potentially re-uploaded for NVENC encoding.

Working FFmpeg command (example):

Bash

ffmpeg -hwaccel cuda -i input.mp4 -vf "format=nv12|cuda,hwupload,scale_cuda=force_original_aspect_ratio=decrease:passthrough=0:w=854:h=854,hwdownload,format=nv12" -c:v h264_nvenc -preset p6 -y output.mp4

Request:

Please consider implementing logic within go-vod to detect 10-bit HEVC input when h264_nvenc is selected for output on hardware that doesn't support 10-bit H.264 encoding, and automatically insert the hwdownload,format=nv12 filter into the FFmpeg command.