Add support for closed captions

[programmerjake]

Adds support for ASTC A/53 Part 4 Closed Captions (essentially EIA-608 and CTA-708).

This adds a extensible side-data mixer (side-data naming is based on FFmpeg's AVFrameSideDataType), which for now just handles closed captions. For closed captions, for each layer, it chooses which input layer to take the closed captions from based on a priority associated with each layer. That priority can be any positive floating-point number. Zero, NaN, or negative numbers mean that there is no priority, so that layer won't be used for closed captions, allowing you to turn off closed captions.

The side-data mixer currently doesn't properly support switching the closed captions source in the middle of an input video, currently it'll just stop the previous video's side-data and start the next one's side data, rather than e.g. building a proper EIA-608 transition sequence. It does warn when you change the source like that though.

The side-data mixer also has functionality to use side-data from dropped frames in the input -- the code allows different side-data types to handle that differently since some types of side-data may only want to use the side-data from the frames that are actually sent to the output (e.g. AV_FRAME_DATA_DYNAMIC_HDR_PLUS I guess), and some may not want to miss any frames (closed captions, where missing a frame of data can cause the captions to be garbled).

So far, I've implemented closed captions support in:

• FFmpeg producer: it supports getting closed captions from either the video stream's side data, or from a separate subtitles stream (e.g. in .mxf files). It also has partial untested support for reading a subtitles-only file (e.g. .scc).
• FFmpeg consumer: it supports putting closed captions into the video stream's side data. It doesn't yet support generating the VANC stream for .mxf files, or other file types that need the closed captions to be a separate subtitles stream.

I added a ACMP command (MIXER CLOSED_CAPTIONS_PRIORITY) for changing a layer's closed-captions priority.

Since information on which inputs to use for closed captions or not is stored in the frame transforms, and those are all cleared by the MIXER CLEAR command, I decided to add a per-producer closed-captions priority setting that is associated with the producer rather than only with frame transforms. to set that, you have to pass the new CLOSED_CAPTIONS_PRIORITY option to whatever LOAD/LOADBG/PLAY command you use to create that producer. That has the benefit of also following the transition between different producers in a layer, rather than you having to be quick and sending MIXER CLOSED_CAPTIONS_PRIORITY at the right time. An additional benefit of having the new CLOSED_CAPTIONS_PRIORITY argument for PLAY is that you can enable closed captions directly from the config file if you didn't want to start up an ACMP client or type any commands into stdin.

I'm planning on adding closed captions support for the decklink producer, but I'm waiting to rebase this PR on top of @niklaspandersson's work #92 (comment)

[programmerjake]

I'm not sure this PR will fix everything wanted in #92, so I'm not marking this as fixing that issue.

[programmerjake]

rebased on #1642

[programmerjake]

rebased on #1642 again, I also added EIA-708 VANC generation code for the decklink consumer based on libklvanc and the FFmpeg decklink code, afaict it generates VANC packets that are sent to @niklaspandersson's code successfully, but the SDI monitors I have hooked up don't show any closed captions -- either I'm not using them correctly or something is broken or both. (edit: it now works in progressive video modes, interlaced modes are still buggy)

stuff from the log:

[2025-06-13 19:10:13.669] [trace]   decklink consumer: got A53_CC side data: fc 97 23 fd 80 80 fa 00 00 fa 00 00 fa 00 00 fa and 44 bytes more
...
[2025-06-13 19:10:13.669] [trace]   decklink consumer: generated VANC packet from A53_CC side data: 96 69 49 4f 43 00 70 72 f4 fc 97 23 fd 80 80 fa 00 00 fa 00 00 fa 00 00 fa 00 00 fa 00 00 fa 00 and 41 bytes more

[programmerjake]

turns out the bugs in interlaced modes were actually partially ffmpeg bugs, fixed in FFmpeg/FFmpeg@61a9f3c

so if we want this to work we'll need a newer version of ffmpeg than is available from apt on the Ubuntu versions we use.

[dimitry-ishenko]

turns out the bugs in interlaced modes were actually partially ffmpeg bugs, fixed in FFmpeg/FFmpeg@61a9f3c

so if we want this to work we'll need a newer version of ffmpeg than is available from apt on the Ubuntu versions we use.

@programmerjake that patch was merged into ffmpeg 6.1 and Ubuntu 22.04 (Jammy) is the only distro that doesn't have it. Which Ubuntu versions are you referring to?

[programmerjake]

@programmerjake that patch was merged into ffmpeg 6.1 and Ubuntu 22.04 (Jammy) is the only distro that doesn't have it. Which Ubuntu versions are you referring to?

ah, I thought it wasn't available on noble last I checked. regardless we'll need to change it to build ffmpeg rather than use the system ffmpeg when the system ffmpeg is too old.

[Julusian]

regardless we'll need to change it to build ffmpeg rather than use the system ffmpeg when the system ffmpeg is too old.

we used to do this, but it was a pain to maintain, and there was push from users to use the system version.
I am open to having a cmake config field so that it can be built against a non-system version, but I don't think we will ship builds which do that

[programmerjake]

we used to do this, but it was a pain to maintain, and there was push from users to use the system version.

then we can't use ffmpeg for framerate conversion (also used whenever the channel is set to interlaced mode) when there are closed captions, since it messes up closed captions by duplicating and/or deleting frames with their closed captions side data, which was fixed in that commit.

[Julusian]

then we can't use ffmpeg for framerate conversion (also used whenever the channel is set to interlaced mode) when there are closed captions, since it messes up closed captions by duplicating and/or deleting frames with their closed captions side data, which was fixed in that commit.

As long as it works with the version in ubuntu 24.04, then I don't mind it not working 100% in 22.04.
Especially as that will leave 2 solutions for anyone who needs it, either update your ubuntu or rebuild against a custom ffmpeg.

[dimitry-ishenko]

regardless we'll need to change it to build ffmpeg rather than use the system ffmpeg when the system ffmpeg is too old.

@programmerjake sorry I don't follow. Like I said, Ubuntu Jammy is the only version that has FFmpeg older than 6.1. Are you on Debian Stable maybe? I believe they still use FFmpeg 5.1

[programmerjake]

I thought the point of supporting Ubuntu Jammy is that it's supposed to just work, requiring you to compile your own ffmpeg and not providing one with the released binary packages doesn't sound like being supported to me.

I'm using Debian 12 for development, but that's not what I'm complaining about since you never claimed to support it, though if it can use the same mechanism I want to have for supporting newer ffmpeg on jammy, that'd be nice.

[dimitry-ishenko]

I thought the point of supporting Ubuntu Jammy is that it's supposed to just work, requiring you to compile your own ffmpeg and not providing one with the released binary packages doesn't sound like being supported to me.

But, it does just work... Heck, I even have a version for Focal in my repo:

https://launchpad.net/~ppa-verse/+archive/ubuntu/casparcg

(which I will probably get rid of now that Focal is no longer officially supported.)

Now, if we want support for the Closed Captions for Jammy, then we have two options:

1. Try to back-port FFmpeg 6.1.1 from noble to jammy.
2. Try to back-port the patch you've linked to above into ffmpeg 4.4.2 which is used in jammy.

Both of these options will need to be hosted somewhere. I don't mind throwing it in my repo, since I already need a patched version of FFmpeg with DeckLink support (for another project).

I'm using Debian 12 for development, but that's not what I'm complaining about since you never claimed to support it, though if it can use the same mechanism I want to have for supporting newer ffmpeg on jammy, that'd be nice.

I just confirmed that noble's version of FFmpeg 6.1.1 package does build on Debian stable. If you are interested, I can set up auto-build on github and you can grab .deb packages from there.

Let me see if 6.1.1 compiles for jammy.

[programmerjake]

Now, if we want support for the Closed Captions for Jammy, then we have two options:

1. Try to back-port FFmpeg 6.1.1 from noble to jammy.
2. Try to back-port the patch you've linked to above into ffmpeg 4.4.2 which is used in jammy.

The commit I linked to isn't the whole fix, it's just the latest in a sequence of commits that fix the bug, since they apparently had a partial fix before.

I was thinking that since we're using custom ffmpeg, we'd compile the same version of ffmpeg used on windows (iirc 7.0.2) and just statically link it to casparcg.

Both of these options will need to be hosted somewhere. I don't mind throwing it in my repo, since I already need a patched version of FFmpeg with DeckLink support (for another project).

Oh, FFmpeg with DeckLink can't be statically linked to CasparCG since they both include the DeckLink loader.

I'm using Debian 12 for development, but that's not what I'm complaining about since you never claimed to support it, though if it can use the same mechanism I want to have for supporting newer ffmpeg on jammy, that'd be nice.

I just confirmed that noble's version of FFmpeg 6.1.1 package does build on Debian stable. If you are interested, I can set up auto-build on github and you can grab .deb packages from there.

I've just been compiling them myself which isn't a problem for me, i'm compiling everything from source anyway since I want to enable asserts and debug info and stuff. I'm not using Debian 12 for running CasparCG in production.

[dedicatedbroadcastsolutions]

Oh, FFmpeg with DeckLink can't be statically linked to CasparCG since they both include the DeckLink loader.

Why are we using ffmpeg with decklink enabled? I thought casparCG has its own decklink implementation...

[dimitry-ishenko]

The commit I linked to isn't the whole fix, it's just the latest in a sequence of commits that fix the bug, since they apparently had a partial fix before.

OK I got it.

I was thinking that since we're using custom ffmpeg, we'd compile the same version of ffmpeg used on windows (iirc 7.0.2) and just statically link it to casparcg.

Static linking is an infernal abomination, a heretical curse spat in the face of divine coding practices. For large projects, it bloats binaries to grotesque, unholy sizes, rips open vulnerabilities like a gaping wound in the digital abyss, squanders memory with the ravenous greed of a demon, and chains developers to an endless, torturous grind of recompilation for every trivial shift in a subproject, dragging their souls through a screaming void of build cycles.

Let's not do static linking.

[dimitry-ishenko]

Oh, FFmpeg with DeckLink can't be statically linked to CasparCG since they both include the DeckLink loader.

Why are we using ffmpeg with decklink enabled? I thought casparCG has its own decklink implementation...

"We" are not. "I" am. For another project. I just offered to backport 6.1.1 to jammy since I already had to patch it for another project. But, I take my words back. Backporting FFmpeg 6.1.1 to jammy is no trivial task. There are at least 10 other packages that will need to be backported as well.

So, never mind. I agree with @Julusian: if someone wants this feature, they will either have to upgrade to noble or compile ffmpeg on their own.

[dimitry-ishenko]

if it isn't in jammy's ffmpeg why go to a lot of effort to enable all the new codecs and stuff in a backport to jammy? imo just disable the ffmpeg configure features that need stuff not in jammy.

That's why I've opted to back-port noble's version of ffmpeg (6.1.1) into jammy only. That way noble and oracular are not impacted.

Backporting only to jammy was exactly what I meant, I see no reason to have our own ffmpeg for noble or oracular or any other distro if their ffmpeg already has everything we need.

@programmerjake I've back-ported ffmpeg into jammy in my PPA and recompiled the server with it. Not that it matters right now I guess, but when server 2.5 is ready for release (ahem @Julusian) it should work with your patches.

https://launchpad.net/~ppa-verse/+archive/ubuntu/casparcg

[programmerjake]

So as long as it is communicated well that this is known to be broken in some scenarios, I am ok with that

ok, in that case I'll add compile-time detection and log an error at runtime whenever it would be broken, and then just disable closed captions support in the ffmpeg producer.

Added in the latest commit. I decided to use runtime version detection whenever the libavfilter major version is <= 9 (the first fixed version is libavfilter 9.8.101), since the user might be using a fixed version of the dynamic library even if they compiled with a broken version.