Provide solution for testing custom in app instrumentations with built-in agent instrumentations using agent-for-testing

[SKozak]

Is your feature request related to a problem? Please describe.

I have a Kafka Streams application where I'm linking spans through stream processing.
I'm heavily using kafka-clients and kafka-streams built-in instrumentations, @WithSpan and OpenTelemetry API.

I know how to do this properly and it's working in runtime, but I need some official solutions to test this custom instrumentation tailored for my app.

I found that there is a package testing/agent-for-testing in official repo, and want to use it, but this agent is very simple and doesn't have additional instrumentations I needed.

Looking at repo and documentations i found otel.javaagent.extensions feature which in theory should allow adding additional instrumentations like this:

configurations {
 ...
    testOtelAgent
    kafka
    kafkaStreams
}


...
kafkaStreams 'io.opentelemetry.javaagent.instrumentation:opentelemetry-javaagent-kafka-streams-0.11:2.12.0-alpha'
kafka 'io.opentelemetry.javaagent.instrumentation:opentelemetry-javaagent-kafka-clients-0.11:2.12.0-alpha'

def allInstrumentationFiles = [
        configurations.kafka.files,
        configurations.kafkaCommon.files,
].flatten().collect { it.absolutePath }.join(',')

test {
...
  jvmArgs "-Dotel.javaagent.extensions=$allInstrumentationFiles"
  jvmArgs "-javaagent:${configurations.testOtelAgent.singleFile}"
}

But it doesn't work, because shaded classes aren't published to Maven Central.
When trying to use kafka-clients instrumentation (like above), we get NoClassDefFoundError for classes like io/opentelemetry/javaagent/shaded/instrumentation/kafka/internal/KafkaProducerRequest because they're not in the published artifact.

I struggled for 4-5 days and finally managed to build my own test-agent modifying testing/agent-for-testing/build.gradle.kts

// Create separate configurations for each Kafka module
val kafkaClientsLib by configurations.creating {
  isCanBeResolved = true
  isCanBeConsumed = false
}

val kafkaStreamsLib by configurations.creating {
  isCanBeResolved = true
  isCanBeConsumed = false
}

...
  // Split Kafka dependencies into separate configurations
  kafkaClientsLib(project(":instrumentation:kafka:kafka-clients:kafka-clients-0.11:javaagent", configuration = "shadowRuntimeElements"))
  kafkaStreamsLib(project(":instrumentation:kafka:kafka-streams-0.11:javaagent", configuration = "shadowRuntimeElements"))

...

tasks {
  jar {
   ...

    // Add Kafka clients with renamed jar
    from(kafkaClientsLib) {
      into("extensions")
      rename { "kafka-clients-$it" }
    }

    // Add Kafka streams with renamed jar
    from(kafkaStreamsLib) {
      into("extensions")
      rename { "kafka-streams-$it" }
    }

...

}

It works, but this isn't a solution I'm satisfied with. I think it would be helpful in general if there was an official solution to test own custom instrumentations in app.

Describe the solution you'd like

Publish the shaded jars that are currently built during the local build process. When building locally with ./gradlew :instrumentation:kafka:kafka-clients:kafka-clients-0.11:javaagent:build, a shaded agent-testing.jar is created containing needed classes, but this isn't published to Maven Central.

Then in application tests, I would be able to use:

configurations {
  ...
    testOtelAgent
    kafka
}

...
 testOtelAgent 'io.opentelemetry.javaagent:opentelemetry-agent-for-testing:2.12.0-alpha'
//shaded artifact
 kafka 'io.opentelemetry.javaagent.instrumentation:opentelemetry-javaagent-kafka-clients-0.11-shaded:2.12.0-alpha' 

...
test {
...
    jvmArgs "-Dotel.javaagent.extensions=${configurations.kafka.files,}"
    jvmArgs "-javaagent:${configurations.testOtelAgent.singleFile}"
...

Describe alternatives you've considered

Provide an enhanced version of agent-for-testing that includes all instrumentations but has them disabled by default. Users could then enable only needed instrumentations via system properties like:

-Dotel.instrumentation.kafka-clients.enabled=true
-Dotel.instrumentation.kafka-streams.enabled=true

Additional context

Example desired usage:

Option 1) Using published shaded artifacts

dependencies {
    testOtelAgent 'io.opentelemetry.javaagent:opentelemetry-agent-for-testing:2.12.0-alpha'
    kafka 'io.opentelemetry.javaagent.instrumentation:opentelemetry-javaagent-kafka-clients-0.11-shaded:2.12.0-alpha' // shaded artifact
}

test {
    jvmArgs "-javaagent:${configurations.testOtelAgent.singleFile}"
    jvmArgs "-Dotel.javaagent.extensions=${configurations.kafka.files.collect { it.absolutePath }.join(',')}"
}

Option 2) Using full agent with disabled instrumentations by default

dependencies {
    testOtelAgent 'io.opentelemetry.javaagent:opentelemetry-agent-for-testing-full:2.12.0-alpha' // full agent with disabled instrumentations 
}

test {
    systemProperty "otel.instrumentation.kafka-clients.enabled", "true" //enable wanted instrumentations 
    systemProperty "otel.instrumentation.spring-kafka.enabled", "true"
    jvmArgs "-javaagent:${configurations.testOtelAgent.singleFile}"
    jvmArgs "-Dotel.javaagent.extensions=${configurations.kafka.files.collect { it.absolutePath }.join(',')}"
}

I'm willing to contribute and provide a PR once we agree on the preferred approach (either publishing shaded artifacts or enhancing agent-for-testing with all instrumentations disabled by default).

[SKozak]

Hi everyone, I just wanted to follow up on this issue. It's been more than a month with no response, and I'm wondering if there's any update or if more information is needed from my side.

I’d appreciate any insights or guidance on this. Thanks!

[jameyg42]

otel.instrumentation.common.default-enabled config property already exists to disable all instrumentations (your Option #2), but the classloaderOptimization checks in most instrumentations would disable them anyway for most unit tests.

I imagine at least a handful of instrumentations need to run in the agent classloader and won't work as extensions, but I wonder if your Option #1 ask would be to not only publish the instrumentations as separate artifacts, but to provide a way to load instrumentations into the agent classloader (e.g. otel.javaagent.instrumentations=<jar list> in addition to otel.javaagent.extensions to signal whether an instrumentation should get an isolated classloader)