Very slow/poor network conditions now fire critical TIMEOUT errors when watching a VOD stream

[loicraux]

Have you read the FAQ and checked for duplicate open issues?

Yes

If the problem is related to FairPlay, have you read the tutorial?

N/A

What version of Shaka Player are you using?

Latest version

Can you reproduce the issue with our latest release version?

Yes

Can you reproduce the issue with the latest code from main?

Yes

Are you using the demo app or your own custom app?

Own custom app, but reproducible with the demo app

If custom app, can you reproduce the issue using our demo app?

Yes, reproducible using the demo app.

What browser and OS are you using?

Chrome or FF (not browser-related)

For embedded devices (smart TVs, etc.), what model and firmware version are you using?

N/A

What are the manifest and license server URIs?

Any manifest and/or license server

What configuration are you using? What is the output of player.getNonDefaultConfiguration()?

No particular configuration. The default retryParameters are used.

What did you do?

1. Load some VOD content using shaka player and start playback
2. Open DevTools and select a very poor network throttling profile, like 20 kbit/s
3. Wait for a while

What did you expect to happen?

A RECOVERABLE TIMEOUT (1003) network error is thrown, for a RequestType.SEGMENT.

What actually happened?

A CRITICAL TIMEOUT (1003) error is thrown, for a RequestType.SEGMENT.

We can agree on what is meant by "critical", but since we already have the RECOVERABLE type, I consider that a CRITICAL error is not recoverable and prevents the content from being played, is that correct?

From this point on, I display an overlay in the UI to alert the user that a critical error has occurred, so that they can copy the diagnostic information and contact support.

However, in this scenario, the error is recoverable, since all you have to do is disable network throttling in DevTools for VOD content playback to resume. And as long as the error is recoverable, it can't be critical in my opinion... What do you think?

This change in categorization RECOVERABLE->CRITICAL appeared with the 4.11.7 release, because of this commit, part of this Pull Request, to fix this issue.

Prior to this change, recoverable timeout network errors would be thrown, which imho better reflects the severity of these errors.

Are you planning to send a PR to fix it?

Perhaps, if it is indeed a bug, and if there is agreement on how to correct it :)

[avelad]

@joeyparrish since you include the related change, can you review it? Thanks!

[joeyparrish]

A CRITICAL TIMEOUT (1003) error is thrown, for a RequestType.SEGMENT.

We can agree on what is meant by "critical", but since we already have the RECOVERABLE type, I consider that a CRITICAL error is not recoverable and prevents the content from being played, is that correct?

That's correct.

From this point on, I display an overlay in the UI to alert the user that a critical error has occurred, so that they can copy the diagnostic information and contact support.

However, in this scenario, the error is recoverable, since all you have to do is disable network throttling in DevTools for VOD content playback to resume. And as long as the error is recoverable, it can't be critical in my opinion... What do you think?

I disagree. There is no way at all for the player to know that this error is recoverable. How should the player know that the bandwidth can be magically increased by dev tools? In the general case, the default logic is correct, in my opinion.

But if you want more control over this for your application, you should look into the configuration for streaming.failureCallback. (https://shaka-player-demo.appspot.com/docs/api/shaka.extern.html#.StreamingConfiguration) From that callback you can call player.retryStreaming() under any conditions you choose, including "always".

The default callback has the following logic:

1. Is it VOD? Assume that because VOD doesn't change, this failure can't be fixed by trying again, so just fail.
2. Is it an HTTP error on live? Retry after a second.
3. Is it a timeout on live? Retry right away.

Source for the default callback:

shaka-player/lib/player.js

Line 7467 in 6a9ee08

defaultStreamingFailureCallback_(error) {

You can find a tutorial on error handling here: https://shaka-player-demo.appspot.com/docs/api/tutorial-errors.html

Does this help?

[loicraux]

I have to admit that it's a bit strange and not very clear to me...

For the moment I've added this piece of code to my application so as not to show the user the critical TIMEOUT (1003) code errors, for requests of type RequestType.SEGMENT :

(note that I don't use the shaka player UI)

public handleShakaPlayerCriticalError({ code, category, data }: util.Error): CriticalErrorEventParams | boolean {
    logger.error(`The critical error occurred while loading/playing the manifest ${this.stream?.getManifestFileUrl()}`);

    if (
        code === shaka.util.Error.Code.TIMEOUT &&
        Array.isArray(data) &&
        data[1] === shaka.net.NetworkingEngine.RequestType.SEGMENT
    ) {
        logger.warn(`Timeout while loading segment ${data[0]}`);
        // Let's not show this error to the user, as it is indeed not critical:
        return false;
    }

   // Process other critical errors ...
}

Now, with this code, if I lower the network quality from NO THROTTLING to a custom quality of 20 kbits/s in DevTools :

1. I don't display the error in an overlay because of the code above,
2. and I receive the 'buffering' event, which allows me to display an overlay with a buffering UI:

If I now increase the network quality in DevTools, playback resumes on its own (and the buffering overlay disappears), I don't have to do anything more for this to happen... (And this is the behavior I am looking for)

Now, if I simply add this streaming.failureCallback to my configuration (it does nothing but log something) :

this.shakaPlayer.configure({
    streaming: {
        failureCallback: ({ code, category, data }: util.Error) => {
            logger.error(`[Streaming Failure Callback] code: ${code}, category: ${category}, data: ${data}`);
        }
    },

... Then nothing works if I lower the network quality from NO THROTTLING to a custom quality of 20 kbits/s in DevTools :

• The 'buffering' event is no longer sent
• Playback does not resume when I increase the network quality again in DevTools...

However, this callback does the same thing as the default callback (i.e. nothing), since it is a VOD stream:

  defaultStreamingFailureCallback_(error) {
    // For live streams, we retry streaming automatically for certain errors.
    // For VOD streams, all streaming failures are fatal.
    if (!this.isLive()) {
      return;
    }

    // ...
  }

This is what confuses me about this streaming.failureCallback ...

[loicraux]

Also I found that the 'buffering' event is only sent once, after the first drop in network quality.... It is not sent after other subsequent drops in network quality ... ?

What I'd simply like to achieve is buffering events always sent when the buffer is empty, and the player to always re-try to fetch segments on failures or timeouts. Should I play with the streaming.retryParameters ?

[loicraux]

I see I have two options to make sure the playback of the VOD stream continues when the network quality gets better :

1. Either I programmatically call shakaPlayer.retryStreaming() in my custom streaming.failureCallback
2. Or I play with the streaming.retryParameters

The problem is that both APIs/structures are poorly documented in my opinion, so that I don't know which is the best option... (Note that our .dash chunks last 3 seconds)

Programmatically call shakaPlayer.retryStreaming() in my custom streaming.failureCallback

It works, but the problem is I don't know what retryStreaming exactly does from the documentation :

Retry streaming after a streaming failure has occurred.
When the player has not loaded content or is loading content, this will be a no-op and will return false.
If the player has loaded content, and streaming has not seen an error, this will return false.
If the player has loaded content, and streaming seen an error, but the could not resume streaming, this will return false.

The cases where it is a no-op and it returns false are documented, but not the nominal case : what does "retry streaming" means ? Are licenses requests fetched again ? Is the manifest also fetched again ? Does it skip the failed chunks to proceed with the next ones ? I find this not clear

Option 2.: Change the retryParameters

I can set maxAttempts to 100_000 and it looks like it does the job. But with a default backoffFactor of 2, it can quickly take a while before a new attempt is made, right ?

When does Shaka Player considers a simple window.fetch or XHR to time out ? Is this controlled by streaming.retryParameters.timeout, streaming.retryParameters.stallTimeout or streaming.retryParameters.connectionTimeout ? This is not clear.

In the documentation of the retryParameters, it is mentioned that these timeouts params are delays "before we abort"... What is aborted ? The current attempt ? Or the whole networking request (that is made of N attempts) ? This is also not clear...

Also, what happens if a chunk is simply missing (error 404) and I've set streaming.retryParameters.maxAttempts to 100_000 : will it make 100_000 attempts before it eventually skips it and proceed with the next chunk ?

As you can see, many options (there is also streaming.maxDisabledTime .... :) but it is a bit hard to determine which one is the appropriate just from reading the documentation...

Your help is welcome ! Thanks !

[avelad]

@joeyparrish ping!