Default Configuration - EDOT Collectors on Kubernetes

The Kubernetes setup utilizes the OpenTelemetry Operator to automate orchestration of EDOT Collectors:

• EDOT Collector Cluster: Collection of cluster metrics.
• EDOT Collector Daemon: Collection of node metrics, logs and application telemetry.
• EDOT Collector Gateway: Pre-processing, aggregation and ingestion of data into Elastic.

Direct ingestion into Elasticsearch	Managed OTLP Endpoint
flowchart LR cluster@{ shape: proc, label: "Cluster Collector fa:fa-microchip"} -->|otlp| gateway@{ shape: procs, label: "Gateway Collectors fa:fa-microchip"} daemon@{ shape: procs, label: "Daemonset Collectors fa:fa-microchip"} -->|otlp| gateway gateway ==>|_bulk| es@{ shape: db, label: "Elasticsearch" } style es stroke:#33f,stroke-width:2px,color:#000;	flowchart LR cluster@{ shape: proc, label: "Cluster Collector fa:fa-microchip"} -->|otlp| gateway@{ shape: procs, label: "Gateway Collectors fa:fa-microchip"} daemon@{ shape: procs, label: "Daemonset Collectors fa:fa-microchip"} -->|otlp| gateway gateway ==>|otlp| otlp@{ shape: display, label: "Managed OTLP endpoint" } style otlp stroke:#33f,stroke-width:2px,color:#000;

The following sections describe the default pipelines for the different roles of EDOT collectors in a Kubernetes setup.

• Pipeline - Cluster Collector
• Pipeline - Daemonset Collectors
• Pipeline - Gateway Collectors
  • Direct ingestion into Elasticsearch
  • Managed OTLP Endpoint

Pipeline - Cluster Collector

flowchart LR
    k8s_cluster@{ shape: proc, label: "k8s_cluster
    fa:fa-right-to-bracket"} -->|M| k8sattributes@{ shape: proc, label: "k8sattributes
    fa:fa-gears"}

    k8sattributes -->|M| resourcedetection@{ shape: procs, label: "resourcedetection
    fa:fa-gears"}

    k8sobjects@{ shape: proc, label: "k8sobjects
    fa:fa-right-to-bracket"} -->|L| resourcedetection

    resourcedetection -->|L/M| resource@{ shape: procs, label: "resource
    fa:fa-gears"}

    resource -->|L/M| otlp_exporter@{ shape: proc, label: "OTLP
    fa:fa-right-from-bracket"}

The main purpose of the Cluster Collector is to collect Kubernetes cluster-level metrics (using the k8s_cluster receiver) and cluster events (k8sobjects receiver) and forward them to the Gateway Collector through OTLP. The resource and resourcedetection processors enrich the cluster-level data with corresponding meta information.

Pipeline - Daemonset Collectors

flowchart LR
    otlp@{ shape: proc, label: "OTLP
    fa:fa-right-to-bracket"} -->|T/L/M| batch@{ shape: proc, label: "batch
    fa:fa-gears"}

    batch -->|T/L/M| resource@{ shape: proc, label: "resource
    fa:fa-right-from-bracket"}

    resource -->|T/L/M| otlp_exporter@{ shape: proc, label: "OTLP
    fa:fa-right-from-bracket"}

%% logs pipeline
    filelog@{ shape: proc, label: "filelog
    fa:fa-right-to-bracket"} -->|L| batch_lm@{ shape: proc, label: "batch
    fa:fa-gears"}

    batch_lm -->|L/M| k8sattributes@{ shape: proc, label: "k8sattributes
    fa:fa-gears"}

    k8sattributes -->|L/M| resourcedetection@{ shape: procs, label: "resourcedetection
    fa:fa-gears"}

    resourcedetection -->|L/M| resource@{ shape: procs, label: "resource
    fa:fa-gears"}

    resource -->|L/M| otlp_exporter

%% system metrics pipeline
    kubletstats@{ shape: proc, label: "kubletstats
    fa:fa-right-to-bracket"} -->|M| batch_lm
    hostmetrics@{ shape: proc, label: "hostmetrics
    fa:fa-right-to-bracket"} -->|M| batch_lm

The Daemonset Collectors gather telemetry associated with corresponding, individual Kubernetes nodes:

1. Host metrics and container logs

   filelog and hostmetrics receivers are used to gather container logs and host metrics, respectively. The kubletstats receiver collects additional Kubernetes Node, Pod and Container metrics. The logs and metrics are batched for better performance (batch processor) and then enriched with meta information using the k8sattributes, resourcedetection and resource processors.

2. Application teleemtry through OTLP from OTel SDKs

   The Daemonset Collectors also receive the application telemetry from OTel SDKs that instrument services / pods running on corresponding Kubernetes nodes. The Daemonset Collectors receive that data through OTLP, batch the data (batch processor) and pass it on to the Gateway Collector through the OTLP exporter.

Pipeline - Gateway Collectors

The Gateway Collectors pipelines differ fundamentally between the two different deployment use cases ‘Direct ingestion into Elasticsearch’ and using Elastic’s ‘Managed OTLP Endpoint’.

Direct ingestion into Elasticsearch

In self-managed and Elastic Cloud Hosted Stack deployment use cases the main purpose of the Gateway Collector is the central enrichment of data before the OpenTelemetry data is being ingested directly into Elasticsearch using the elasticsearch exporter.

flowchart LR
    otlp@{ shape: proc, label: "OTLP
    fa:fa-right-to-bracket"} -->|T/L| batch@{ shape: proc, label: "batch
    fa:fa-gears"}
    batch -->|T| elastictrace@{ shape: proc, label: "elastictrace
    fa:fa-gears"}
    elastictrace -->|T| es_exporter@{ shape: proc, label: "elasticsearch
    fa:fa-right-from-bracket"}
    es_exporter -->|otel| otel@{ shape: framed-circle, label: "otel" }
    elastictrace -->|T| elasticapm@{ shape: hex, label: "elasticapm
    fa:fa-link"}
    elasticapm -->|M| es_exporter

    batch -->|L| elasticapm

    otlp -->|M| routing@{ shape: hex, label: "routing
    fa:fa-link"}
    routing -->|M| batch
    batch -->|L/M| es_exporter

    routing -->|"M (infra)"| elasticinframetrics@{ shape: proc, label: "elasticinframetrics
    fa:fa-gears"}
    elasticinframetrics -->|M| attributes@{ shape: proc, label: "attributes
    fa:fa-gears"}
    attributes -->|M| resource@{ shape: proc, label: "resource
    fa:fa-gears"}
    resource -->|M| batch_ecs@{ shape: proc, label: "batch
    fa:fa-gears"}
    batch_ecs -->|M| es_exporter_ecs@{ shape: proc, label: "elasticsearch
    fa:fa-right-from-bracket"}
    es_exporter_ecs -->|ecs| ecs@{ shape: framed-circle, label: "ecs" }

Hence, this Gateway Collector configuration comprises the pipelines for data enrichment of application telemetry and host metrics (for details, refer to the linked descriptions of the corresponding standalone use cases).

The routing connector separates the infrastructure metrics from other metrics and routes them into the ECS-based pipeline, with ECS-compatibility exporter mode. Other metrics are exported in OTel-native format to Elasticsearch.

Managed OTLP Endpoint

With the managed OTLP Endpoint, the Gateway Collector configuration simply pipes all the data from the OTLP receiver through a batch processor before the data is being exported through OTLP to the managed endpoint.

flowchart LR
    otlp@{ shape: proc, label: "OTLP
    fa:fa-right-to-bracket"} -->|T/L/M| batch@{ shape: proc, label: "batch
    fa:fa-gears"}
    batch -->|T/L/M| otlp_exporter@{ shape: proc, label: "OTLP
    fa:fa-right-from-bracket"}

With this scenario there’s no need to do any Elastic-specific enrichment in your Kubernetes cluster, as all of that happens behind the managed OTLP endpoint.