OpenCV video decoding too slow for vidgear - ffmpeg might be better suited for decoding

[golyalpha]

Detailed Description

I've ran some encode and decode benchmarks, and it's becoming fairly obvious, that VidGear (CamGear API) is currently unable to decode 1080p60 video on computers where, given certain settings, ffmpeg can encode 1080p60 in real time or slightly faster without hardware acceleration.

Context

Currently, it's impossible to pass 1080p60 H264 (only tested codec) video through VidGear on computers that should be able to decode 1080p60 video just fine. The idea would be to replace the OpenCV VideoStream API with something more performant, like ffmpeg, since ffmpeg is capable of outputting raw video into stdout.

The CamGear API should not need to change from the developer's standpoint.

Your Environment

• VidGear version: 0.1.8
• Branch: PyPI
• Python version: 3.8.2
• pip version: 19.2.3
• Operating System and version: Win10Pro 1909

Any Other Important Information

Encode/Decode benchmarks for VidGear (encode is compressed, so ffmpeg):

> poetry run sp-benchmark
Results:
        Encode:
                1080p: 71.34738617215714
                900p: 94.22808977047293
                720p: 137.51681644444432
                480p: 388.3952044196786
                360p: 506.7212349134308
                240p: 1020.0560010744543
                144p: 1860.4607896260777
        Decode:
                1080p: 36.054442749368185
                900p: 44.78923780306475
                720p: 55.349642074620796
                480p: 76.08067848749076
                360p: 81.93545752827764
                240p: 90.02970867849261
                144p: 99.95882945711747

ffmpeg decode benchmark (4k60):

> ffmpeg -i .\streampipe\benchmark\resources\bbb_sunflower_2160p_60fps_normal.mp4 -f null -
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '.\streampipe\benchmark\resources\bbb_sunflower_2160p_60fps_normal.mp4':
  Duration: 00:10:34.53, start: 0.000000, bitrate: 8487 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 3840x2160 [SAR 1:1 DAR 16:9], 8002 kb/s, 60 fps, 60 tbr, 60k tbn, 120 tbc (default)
    Stream #0:1(und): Audio: mp3 (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 160 kb/s (default)
    Stream #0:2(und): Audio: ac3 (ac-3 / 0x332D6361), 48000 Hz, 5.1(side), fltp, 320 kb/s (default)
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
  Stream #0:2 -> #0:1 (ac3 (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
    Stream #0:0(und): Video: wrapped_avframe, yuv420p(progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc (default)
    Stream #0:1(und): Audio: pcm_s16le, 48000 Hz, 5.1(side), s16, 4608 kb/s (default)
frame=38072 fps=152 q=-0.0 Lsize=N/A time=00:10:34.56 bitrate=N/A speed=2.53x
video:19928kB audio:356706kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

[abhiTronix]

The idea would be to replace the OpenCV VideoStream API with something more performant, like ffmpeg, since ffmpeg is capable of outputting raw video into stdout.

@golyalpha I Don't think that replacing OpenCV with FFmpeg is a good idea. Since the only leverage of FFmpeg over OpenCV is performance, While OpenCV is more advantageous as:

• OpenCV seamlessly support multiple backends, which include powerful GStreamer, libav and FFmpeg itself too.
• OpenCV is available on pretty much every Linux distribution, while FFmpeg might not be (for legal reasons).
• OpenCV itself is available under flexible 3-clause BSD license while FFmpeg you have to make sure that no GPL components are enabled (some notable examples are x264 (H264 encoder) and libac3 (Dolby AC3 audio codec)). See https://www.ffmpeg.org/legal.html for details.
• Plus FFmpeg is still buggy to be completely adopted reliably in my experience.

This is some of the reason why it's not a good idea. Being said that, we can still implement another Videocapture gear which works purely on FFmpeg. Thanks for this idea anyways.

[golyalpha]

Hmm, it is quite possible that merely using a different OpenCV backend - FFmpeg in this case - could potentially result in the lost performance to be gained, but I am quite skeptical to be honest. Gonna run some more benchmarks, and see.

As for FFmpeg not being available on some linux distributions - afaik, most offer system or user packages using which you can install ffmpeg, that being said, certain features (like hwaccel support) might be missing.

[golyalpha]

Alright, it seems that OpenCV is the part slowing down the decode process, because I just ran the same benchmark with ffmpeg as OpenCV backend, and the results are pretty much within the margin of error compared to the first one.
Implementing a new gear specifically for ffmpeg decoding does seem to be a direction we could both agree on.

[Thomasedv]

I ended up using another python module(PyAv) to decode frames because opencv (and vidgear as well because of it) just gives me less frames, despite reporting a higher total frame count. FFmpeg both reports and gives me all frames.

While less ideal, i used that for decoding and Writegear for encoding because it's so well made and easy to use.

I should probably investigate if the ffmpeg backend for opencv works better though.

[golyalpha]

@Thomasedv That frame count mismatch sounds like a bug, though I haven't had that issue.

[Thomasedv]

It likely is, probably related to the file format I was working with. But it's what pushed me to just swap to something other than opencv.

[golyalpha]

@abhiTronix, how open are you to using PyAV instead of FFMpeg in a subprocess?

[abhiTronix]

@abhiTronix, how open are you to using PyAV instead of FFMpeg in a subprocess?

@golyalpha PyAV directly provides Pythonic binding for the FFmpeg libraries, so there's no need of subprocess, thanks for bring this up @Thomasedv .

[golyalpha]

Yeah, no, those were two separate things (PyAV OR FFMpeg in a subprocess), I didn't mean using PyAV in a subprocess.

[abhiTronix]

I should probably investigate if the ffmpeg backend for opencv works better though.

@Thomasedv Nope. It works worse according to my experience.

[abhiTronix]

@golyalpha Can you benchmark PyAV too?

[golyalpha]

I can try, sure.