Fix metadata propagation from scrape to otel components

[ptodev]

PR Description

Fixes propagation of metric metadata, but only for pipelines which start with a prometheus.scrape and transition into otelcol components. The metrics need to be sent by an otelcol exporter.

I can see the metadata in live debugging. Before:

After:

Which issue(s) this PR fixes

Related to #547

Notes to the Reviewer

I need to assess the performance impact of this feature before writing the docs and before opening the PR for review.
For now it'd be good to keep it as an experimental opt-in feature until we're sure it's stable and without consuming much more resources.

PR Checklist

• CHANGELOG.md updated
• Documentation added
• Tests updated
• Config converters updated

[kgeckhart]

Looks like the otel prometheus receiver requires it just like it requires a metadata store to be present

alloy/internal/component/otelcol/receiver/prometheus/internal/transaction.go

Lines 459 to 466 in 7a5d8b9

	target, ok := scrape.TargetFromContext(t.ctx)
	if !ok {
	return nil, errors.New("unable to find target in context")
	}
	t.mc, ok = scrape.MetricMetadataStoreFromContext(t.ctx)
	if !ok {
	return nil, errors.New("unable to find MetricMetadataStore in context")
	}

[kgeckhart]

Using

prometheus.remote_write "k3d" {
	endpoint {
		url = "http://mimir.k3d.localhost:9999/api/prom/push"

		headers = {
			"X-Scope-OrgID" = "001",
		}
	}
}

prometheus.write.queue "k3d" {
	endpoint "default" {
		url = "http://mimir.k3d.localhost:9999/api/prom/push"

		headers = {
			"X-Scope-OrgID" = "001",
		}
	}
}

prometheus.exporter.unix "integrations_node_exporter" { }

prometheus.scrape "integrations_node_exporter" {
	targets    = prometheus.exporter.unix.integrations_node_exporter.targets
        honor_metadata = true

        // forward_to = [prometheus.relabel.remotewrite_relabel.receiver]
        forward_to = [prometheus.relabel.writequeue_relabel.receiver]
}

prometheus.relabel "remotewrite_relabel" {
	forward_to = [prometheus.remote_write.k3d.receiver]

	rule {
		action = "replace"
		replacement = "remotewrite"
		target_label = "job"
	}
}

prometheus.relabel "writequeue_relabel" {
	forward_to = [prometheus.write.queue.k3d.receiver]

	rule {
		action = "replace"
		replacement = "writequeue"
		target_label = "job"
	}
}

I was able to verify mimir had metadata when using prometheus.write.queue but has no metadata for prometheus.remote_write.

I'll see if I can test out the other components we talked about, anything else using Fanout, and update the metadata ticket.