Data streams

Data streams are optimized for append-only time-series data. They handle index rollover and lifecycle automatically, using the naming convention {type}-{dataset}-{namespace}.

Full name: logs-myapp-production

[ElasticsearchMappingContext]
[Entity<LogEntry>(
    Target = EntityTarget.DataStream,
    DataStreamType = "logs",
    DataStreamDataset = "myapp",
    DataStreamNamespace = "production"
)]
public static partial class LoggingContext;
		

Data stream bootstrap creates:

1. Component templates via ComponentTemplateStep:
   • logs-myapp-settings: index settings
   • logs-myapp-mappings: field mappings
2. Data stream template via DataStreamTemplateStep:
   • Index pattern: logs-myapp-*
   • References component templates
   • Includes "data_stream": {} to enable data stream behavior
   • Infers built-in component templates based on type (for example, logs-settings, logs-mappings for logs type)

var options = new IngestChannelOptions<LogEntry>(transport, LoggingContext.LogEntry.Context);
using var channel = new IngestChannel<LogEntry>(options);

await channel.BootstrapElasticsearchAsync(BootstrapMethod.Failure);
		

Data streams use create operations (append-only). Documents cannot be updated or deleted through the bulk API -- they can only be appended.

The DataStreamIngestStrategy is automatically selected for EntityTarget.DataStream.

Data streams support two lifecycle approaches:

• Data stream lifecycle (recommended): use IngestStrategies.DataStream(context, "30d") for automatic retention. See data stream lifecycle.
• ILM (self-managed only): use BootstrapStrategies.DataStreamWithIlm(...) for multi-phase lifecycle. See ILM managed.

• Time-series: end-to-end guide for time-series data
• TSDB: time-series database mode for metrics
• LogsDB: LogsDB mode for log ingestion