OpenTelemetry and M3DB Integration

Input and output integration overview

This plugin receives traces, metrics, and logs from OpenTelemetry clients and agents via gRPC, enabling comprehensive observability of applications.

This plugin allows Telegraf to stream metrics to M3DB using the Prometheus Remote Write protocol, enabling scalable ingestion through the M3 Coordinator.

Integration details

OpenTelemetry

The OpenTelemetry plugin is designed to receive telemetry data such as traces, metrics, and logs from clients and agents implementing OpenTelemetry via gRPC. This plugin initiates a gRPC service that listens for incoming telemetry data, making it distinct from standard plugins that collect metrics at defined intervals. The OpenTelemetry ecosystem aids developers in observing and understanding their applications’ performance by providing a vendor-neutral way to instrument, generate, collect, and export telemetry data. Key features of this plugin include customizable connection timeouts, adjustable maximum message sizes for incoming data, and options for specifying span, log, and profile dimensions to tag the incoming metrics. With this flexibility, organizations can tailor their telemetry collection to meet precise observability requirements and ensure seamless data integration into systems like InfluxDB.

M3DB

This configuration uses Telegraf’s HTTP output plugin with prometheusremotewrite format to send metrics directly to M3DB through the M3 Coordinator. M3DB is a distributed time series database designed for scalable, high-throughput metric storage. It supports ingestion of Prometheus remote write data via its Coordinator component, which manages translation and routing into the M3DB cluster. This approach enables organizations to collect metrics from systems that aren’t natively instrumented for Prometheus (e.g., Windows, SNMP, legacy systems) and ingest them efficiently into M3’s long-term, high-performance storage engine. The setup is ideal for high-scale observability stacks with Prometheus compatibility requirements.

Configuration

OpenTelemetry

[[inputs.opentelemetry]]
  ## Override the default (0.0.0.0:4317) destination OpenTelemetry gRPC service
  ## address:port
  # service_address = "0.0.0.0:4317"

  ## Override the default (5s) new connection timeout
  # timeout = "5s"

  ## gRPC Maximum Message Size
  # max_msg_size = "4MB"

  ## Override the default span attributes to be used as line protocol tags.
  ## These are always included as tags:
  ## - trace ID
  ## - span ID
  ## Common attributes can be found here:
  ## - https://github.com/open-telemetry/opentelemetry-collector/tree/main/semconv
  # span_dimensions = ["service.name", "span.name"]

  ## Override the default log record attributes to be used as line protocol tags.
  ## These are always included as tags, if available:
  ## - trace ID
  ## - span ID
  ## Common attributes can be found here:
  ## - https://github.com/open-telemetry/opentelemetry-collector/tree/main/semconv
  ## When using InfluxDB for both logs and traces, be certain that log_record_dimensions
  ## matches the span_dimensions value.
  # log_record_dimensions = ["service.name"]

  ## Override the default profile attributes to be used as line protocol tags.
  ## These are always included as tags, if available:
  ## - profile_id
  ## - address
  ## - sample
  ## - sample_name
  ## - sample_unit
  ## - sample_type
  ## - sample_type_unit
  ## Common attributes can be found here:
  ## - https://github.com/open-telemetry/opentelemetry-collector/tree/main/semconv
  # profile_dimensions = []

  ## Override the default (prometheus-v1) metrics schema.
  ## Supports: "prometheus-v1", "prometheus-v2"
  ## For more information about the alternatives, read the Prometheus input
  ## plugin notes.
  # metrics_schema = "prometheus-v1"

  ## Optional TLS Config.
  ## For advanced options: https://github.com/influxdata/telegraf/blob/v1.18.3/docs/TLS.md
  ##
  ## Set one or more allowed client CA certificate file names to
  ## enable mutually authenticated TLS connections.
  # tls_allowed_cacerts = ["/etc/telegraf/clientca.pem"]
  ## Add service certificate and key.
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"

M3DB

# Configuration for sending metrics to M3
[outputs.http]
  ## URL is the address to send metrics to
  url = "https://M3_HOST:M3_PORT/api/v1/prom/remote/write"

  ## HTTP Basic Auth credentials
  username = "admin"
  password = "password"

  ## Data format to output.
  data_format = "prometheusremotewrite"

  ## Outgoing HTTP headers
  [outputs.http.headers]
    Content-Type = "application/x-protobuf"
    Content-Encoding = "snappy"
    X-Prometheus-Remote-Write-Version = "0.1.0"

Input and output integration examples

OpenTelemetry

1. Unified Monitoring Across Services: Use the OpenTelemetry plugin to collect and consolidate telemetry data from various microservices within a Kubernetes environment. By instrumenting each service with OpenTelemetry, you can utilize this plugin to gather a holistic view of application performance and dependencies in real-time, enabling faster troubleshooting and improved reliability of complex systems.

2. Enhanced Debugging with Traces: Implement this plugin to capture end-to-end traces of requests flowing through multiple services. For instance, when a user initiates a transaction that triggers several backend services, the OpenTelemetry plugin can record detailed traces that highlight performance bottlenecks, giving developers the necessary insights to debug issues and optimize their code.

3. Dynamic Load Testing and Performance Monitoring: Leverage the capabilities of this plugin during load testing phases by collecting live metrics and traces under simulated higher loads. This approach helps to evaluate the resilience of the application components and identify potential performance degradations preemptively, ensuring a smooth user experience in production.

4. Integrated Logging and Metrics for Real-Time Monitoring: Combine the OpenTelemetry plugin with logging frameworks to gather real-time logs alongside metric data, creating a powerful observability platform. For example, integrate it within a CI/CD pipeline to monitor builds and deployments, while collecting logs that help diagnose failures or performance issues in real-time.

M3DB

1. Large-Scale Cloud Infrastructure Monitoring: Deploy Telegraf agents across thousands of virtual machines and containers to collect metrics and stream them into M3DB through the M3 Coordinator. This provides reliable, long-term visibility with minimal storage overhead and high availability.

2. Legacy System Metrics Ingestion: Use Telegraf to gather metrics from older systems that lack native Prometheus exporters (e.g., Windows servers, SNMP devices) and forward them to M3DB via remote write. This bridges modern observability workflows with legacy infrastructure.

3. Centralized App Telemetry Aggregation: Collect application-specific telemetry using Telegraf’s plugin ecosystem (e.g., exec, http, jolokia) and push it into M3DB for centralized storage and query via PromQL. This enables unified analytics across diverse data sources.

4. Hybrid Cloud Observability: Install Telegraf agents on-prem and in the cloud to collect and remote-write metrics into a centralized M3DB cluster. This ensures consistent visibility across environments while avoiding the complexity of running Prometheus federation layers.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.