Allow to enable/disable (automatic) OpenTelemetry tracing dynamically

[sergeykad]

Is your feature request related to a problem? Please describe.
We are worried about the potential performance impact of using OpenTelemetry and would like to be able to enable/disable it as needed without application restarts.

Describe the solution you'd like
An option to replace Tracer used by GlobalOpenTelemetry from the regular tracer to a no-op tracer should do the trick. For example something like GlobalOpenTelemetry.get().disable().

Describe alternatives you've considered
It probably can be implemented as an extension, but I am not sure how, and it feels that it should be part of the standard API. There is no easy way to replace Tracer, so maybe it can be done with a custom Sampler implementation. However, I am not sure how I can enable/disable it at runtime.

Additional context
https://github.com/open-telemetry/community/discussions/1048
#5972
#3389

[trask]

maybe it can be done with a custom Sampler implementation. However, I am not sure how I can enable/disable it at runtime.

this is doable, e.g. Jaeger Sampler has remote controlled sampling percentage.

[sergeykad]

@trask Are you referring to the Jaeger SDKs? It is deprecated, so it will not be a good solution for us.
https://www.jaegertracing.io/docs/1.48/sampling/

[mateuszrzeszutek]

Hey @sergeykad ,

No, we mean the remote Jaeger sampler implemented in the OTel SDK extensions package: https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/jaeger-remote-sampler
It does not use any of the deprecated Jaeger SDKs, but is a re-implementation of the Jaeger's remote sampler, but for OpenTelemetry.

[trask]

we mean the remote Jaeger sampler implemented in the OTel SDK extensions package: https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/jaeger-remote-sampler

and to clarify, I just meant this as an example of what's possible, not necessarily that you should use this particular sampler

[sergeykad]

Thanks. I'll try implementing it, but it would be great if it could be part of open-telemetry.

[trask]

this could be considered once there is a standard remote configuration mechanism for opentelemetry, e.g. https://github.com/open-telemetry/oteps/blob/main/text/0225-configuration.md#opamp

[IOOOTARobby]

Extremely desirable feature, looking forward to it! 👍

[breedx-splk]

@sergeykad @IOOOTARobby as users, what are the conditions under which you would expect to leverage this type of feature?

Sometimes folks think it's a good idea to turn off observability when things are running "too hot" or to build some kind of throttle that increases sampling when CPU or memory is above a threshold or whatever. That seems fine on paper, but what ultimately ends up happening is that you lose visibility during the most crucially important times! Things start trending toward a problem and then you can see it on a graph and then >POP< you just lose data and the dashboards go dark. That's not great either.

I definitely understand wanting this control, and it still seems like a useful thing to have, I'd just like to better understand when it might be used. That knowledge could help inform design decisions around it.

Thanks!

[sergeykad]

@breedx-splk Our management is very focused on maintaining production performance. We're working to ensure that any potential performance impact from OpenTelemetry instrumentation is minimal and ability to switch it off as needed will significantly mitigate any concerns.

I agree with you that it's better to keep it enabled all the time, but we want to use it mostly on development environments and enable it only occasionally on production at least until we build more confidence with it.