[exporter/tinybird] Limit EventsAPI requests to 10MB

[jordivilaseca]

Component(s)

exporter/tinybird

Is your feature request related to a problem? Please describe.

The Tinybird EventsAPI has a limit of 10MB for each request. In case of surpassing this limit, the request returns a 413 error code.

The OpenTelemetry collector can be configured to batch data together (it's recommended to do that). There are two ways of achieving this:

• Using the batching processor. It only supports batching by time or number of signals.
• Configuring the Tinybird Exporter batch size following the exporterhelper pattern. It supports batching by requests, items and bytes (from most performant to least performant). In case of batching by bytes, the batcher uses the protobuf representation of the data to compute the size (really different from the generated JSON output).

In both ways, the problem is the same. There is no way to ensure that a request to the EventsAPI is smaller than 10MB. The user could try to overpass this limitation configuring a batch size of at most N elements, but this number depends on the type of signal and the data it contains, making it difficult to configure.

Describe the solution you'd like

When the EventAPI request body is being built in the Tinybird Exporter, we have to detect when the requests body is bigger than 10MB and perform different requests of at max 10MB each. This solution works out-of-the-box and requires 0 config from the user point of view.

The idea is not to reimplement batching here, just detecting this specific case and act accordingly to avoid getting an error from the EventsAPI

Describe alternatives you've considered

No response

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• exporter/tinybird: @mx-psi @jordivilaseca @MoreraAlejandro

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[VihasMakwana]

@jordivilaseca Hello!

A question:

Let's say we decide to split the request body into multiple parts, 10 MB each. We would then send it to the backend. What if one request succeeds and the other fails? This would be a partial failure. If we retry sending entire batch, this would lead to data duplication.

[axw]

FWIW, the Elasticsearch exporter had the same issue and addressed it in the way you described @jordivilaseca: #36396

(@VihasMakwana makes a good point, but I expect this problem already exists in the event of connection loss and client retries.)