4755 – Add otel collector (#456)

This adds otel-collector. It scrapes the Prometheus endpoint for our metrics and sends it to Honeycomb.

I added the bare minimum configuration for it to work. There are features that might be interesting, but I think we could add those as we add the metrics and understand what makes sense for us.

- Adding a batch processor might be interesting: https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md
- Maybe activating internal telemetry? https://opentelemetry.io/docs/collector/internal-telemetry#activate-internal-telemetry-in-the-collector

Resources
 - https://docs.honeycomb.io/integrations/metrics/prometheus/
 - https://docs.honeycomb.io/send-data/opentelemetry/collector/
 - https://github.com/open-telemetry/opentelemetry-collector
 - https://opentelemetry.io/docs/collector/quick-start/
 - https://opentelemetry.io/docs/collector/installation/

References: 4755, 4396
PR: 456

Author: vasconsaurus

.gitignore

@@ -26,4 +26,5 @@ config/cookies.txt
 coverage
 .idea
 .byebug_history
-*.env
+*.env
+otel-collector-config.yaml

config/otel-collector-config.yaml.example

@@ -0,0 +1,29 @@
+receivers:
+  prometheus:
+    config:
+      scrape_configs:
+        - job_name: "prometheus"
+          scrape_interval: 15s
+          static_configs:
+            - targets: ["pender:3200"] 
+
+processors:
+  batch:
+
+exporters:
+  otlp/metrics:
+    endpoint: "api.honeycomb.io:443" # US instance
+    #endpoint: "api.eu1.honeycomb.io:443" # EU instance
+    headers:
+      "x-honeycomb-team": "<Honeycomb API KEY>"
+      "x-honeycomb-dataset": "pender"
+
+service:
+  # telemetry:
+  #   logs:
+  #     level: "debug"
+  pipelines:
+    metrics:
+      receivers: [prometheus]
+      processors: []
+      exporters: [otlp/metrics]

docker-compose.yml

@@ -57,3 +57,9 @@ services:
     environment:
       RAILS_ENV: development
       SERVER_PORT: 3200
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib
+    volumes:
+      - ./config/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
+    depends_on:
+      - pender