Description
Is your feature request related to a problem? Please describe.
Currently, the OpenTelemetry Java Agent does not support exporting telemetry data directly to Kafka. Users who want to send telemetry to Kafka must rely on an intermediate OpenTelemetry Collector to achieve this. This adds complexity to the observability setup, requiring additional configuration and infrastructure, which could be avoided with native Kafka support in the Java Agent.
Describe the solution you'd like
Implement a native Kafka exporter in the OpenTelemetry Java Agent to allow telemetry data (traces, metrics, and logs) to be sent directly to Kafka topics without requiring an intermediate OpenTelemetry Collector. The exporter should support configurable Kafka brokers, topics, and optional authentication mechanisms to integrate seamlessly with existing Kafka setups. This would simplify the observability pipeline and reduce operational overhead.
Describe alternatives you've considered
The primary alternative is using the OpenTelemetry Collector as an intermediary. The Java Agent sends telemetry data to the Collector using OTLP, and the Collector forwards it to Kafka. While this approach works, it introduces additional complexity and potential points of failure. Another alternative is to develop custom instrumentation or a wrapper around the Java Agent to achieve direct Kafka support, but this requires significant effort and maintenance from end users. A native Kafka exporter in the Java Agent would eliminate these complexities.
Additional context
Direct Kafka support in the OpenTelemetry Java Agent would be particularly useful for organizations that use Kafka as their central telemetry processing pipeline. It aligns with the trend of reducing intermediate components in observability stacks to enhance performance and reliability. Additionally, native support would make the agent more versatile for diverse use cases, such as real-time stream processing and analytics with Kafka-based platforms like Apache Flink or ksqlDB. This feature would also be consistent with the growing adoption of Kafka in modern cloud-native architectures.