Default Configuration - Standalone EDOT Collector

The standalone EDOT Collector comes with a default configuration that covers pipelines for the collection of logs, host metrics and data from OTel SDKs. The following sampling files are available:

Use Cases	Direct ingestion into Elasticsearch	Managed OTLP Endpoint
Platform logs	📄 Logs - ES	📄 Logs - OTLP
Platform logs & host metrics	📄 Logs | Metrics - ES	📄 Logs | Metrics - OTLP
Platform logs, host metrics, application telemetry	📄 Logs | Metrics | App - ES (default)	📄 Logs | Metrics | App - OTLP (default)

Use the above example configurations as a reference when configuring your upstream collector or customizing your EDOT Collector configuration.

The following sections describe the the pipelines of the above configurations split down by single use cases.

• Direct Ingestion into Elasticsearch
  • Logs Collection Pipeline
  • Application / Traces Collection Pipeline
  • Host Metrics Collection Pipeline
• Ingestion through the Managed OTLP Endpoint

Direct Ingestion into Elasticsearch

For self-managed and Elastic Cloud Hosted Stack deployment use cases (as of Elastic Stack version 8.17.3), we recommend ingesting OpenTelemetry data from the EDOT Collector directly into Elasticsearch using the elasticsearch exporter.

Learn more about the configuration options for the elasticsearch exporter in the corresponding documentation or learn about common configuration use cases for the elasticsearch exporter.

The elasticsearch exporter comes with two relevant data ingestion modes:

• ecs: Writes data in backwards compatible ECS format. Original attribute names and semantics may get lost in translation.
• otel: OTel attribute names and semantics are preserved.

Overall, the goal of EDOT is to preserve OTel data formats and semantics as much as possible, hence, otel is the default mode for the EDOT Collector. However, some use cases might require data to be exported in ECS format for backwards compatibility reasons.

Logs Collection Pipeline

flowchart LR
    filelog@{ shape: proc, label: "filelog
    fa:fa-right-to-bracket"} -->|M| resourcedetection@{ shape: proc, label: "resourcedetection
    fa:fa-gears"}
    resourcedetection -->|M| es_exporter@{ shape: proc, label: "elasticsearch
    fa:fa-right-from-bracket"}
    es_exporter -->|otel| otel@{ shape: framed-circle, label: "otel" }

For log collection, the default configuration utilizes the filelog receiver to read log entries from files.

In addition, the resourcedetection processor enriches the log entries with meta information about the corresponding host and operating system.

Data is written out directly into Elasticsearch with the elasticsearch exporter using the OTel-native data mode.

Application / Traces Collection Pipeline

flowchart LR
    otlp@{ shape: proc, label: "OTLP
    fa:fa-right-to-bracket"} -->|T| elastictrace@{ shape: proc, label: "elastictrace
    fa:fa-gears"}
    elastictrace -->|T| es_exporter@{ shape: proc, label: "elasticsearch
    fa:fa-right-from-bracket"}
    elastictrace -->|T| elasticapm@{ shape: hex, label: "elasticapm
    fa:fa-link"}
    elasticapm -->|M| es_exporter
    otlp -->|L/M| es_exporter
    otlp -->|L| elasticapm
    es_exporter -->|otel| otel@{ shape: framed-circle, label: "otel" }

The application pipeline in the EDOT Collector receives data from OTel SDKs through the OTLP receiver. While logs and metrics are exported as is into Elasticsearch, traces involve two additional processors / connectors.

The elastictrace processor enriches trace data with additional attributes that improve the user experience in the Elastic Observability UIs. In addition, the elasticapm connector generates pre-aggregated APM metrics from tracing data.

Both components, elastictrace and elasticapm are required for Elastic APM UIs to work properly. However, both components are not (yet) included in the OpenTelemetry Collector Contrib repository. Thus, to use OpenTelemetry with Elastic for APM use cases one of the following options is available:

• use the EDOT Collector with that configuration to ingest data into Elasticsearch
• or build a custom, EDOT-like for ingesting data into Elasticsearch
• or use Elastic’s managed OTLP endpoint that would do the enrichment for you

Application-related OTel data is ingested into Elasticsearch in OTel-native format using the elasticsearch exporter.

Host Metrics Collection Pipeline

flowchart LR
    hostmetrics@{ shape: proc, label: "hostmetrics
    fa:fa-right-to-bracket"} -->|M| elasticinframetrics@{ shape: proc, label: "elasticinframetrics
    fa:fa-gears"}
    elasticinframetrics --> |M| resourcedetection@{ shape: proc, label: "resourcedetection
    fa:fa-gears"}
    resourcedetection --> |M| attributes@{ shape: proc, label: "attributes
    fa:fa-gears"}
    attributes --> |M| resource@{ shape: proc, label: "resource
    fa:fa-gears"}
    resource --> |M| es_exporter_ecs@{ shape: proc, label: "elasticsearch
    fa:fa-right-from-bracket"}
    es_exporter_ecs -->|ecs| ecs@{ shape: framed-circle, label: "ecs" }

This pipeline utilizes the hostmetrics receiver to collect disk, filesystem, cpu, memory, process and network metrics for the corresponding host.

For backwards compatibility host metrics are translated into ECS-compatible system metrics using the elasticinframetrics processor and, finally, are ingested in ecs format through the elasticsearch exporter.

The resourcedetection processor enriches the metrics with meta information about the corresponding host and operating system. The attributes and resource processor are used to set some fields for proper routing of the ECS-based system metrics data into corresponding Elasticsearch data streams.

Ingestion through the Managed OTLP Endpoint

When ingesting OTel data through Elastics Managed OTLP endpoint, all the enrichment that is required for an optimal experience in the Elastic solutions is happening behind the managed OTLP endpoint and, thus, is transparent to the users.

Accordingly, the collector configuration for all the use cases is relatively simple and is only about local data collection and context enrichment:

flowchart LR
    hostmetrics@{ shape: proc, label: "hostmetrics
    fa:fa-right-to-bracket"} -->|M| resourcedetection@{ shape: proc, label: "resourcedetection
    fa:fa-gears"}

    filelog@{ shape: proc, label: "filelog
    fa:fa-right-to-bracket"} -->|L| resourcedetection

    otlp@{ shape: proc, label: "OTLP
    fa:fa-right-to-bracket"} -->|L/M/T| otlp_exporter@{ shape: proc, label: "OTLP
    fa:fa-right-from-bracket"}

    resourcedetection --> |L/M| otlp_exporter

Platform logs are scraped with the filelog receiver, host metrics are collected through the hostmetrics receiver and both signals are enriched with meta information through the resourcedetection processor.

Data from OTel SDKs is piped through the OTLP receiver directly to the OTLP exporter that sends data for all the signals to the managed OTLP endpoint.

As you can see, with the managed OTLP Endpoint there is no need for configuring any Elastic-specific components, such as elasticinframetrics, elastictrace processors, elasticapm connector or the elasticsearch exporter. Hence, with this deployment model the edge setup and configuration can be 100% vendor agnostic.