AI index
An AI index is an index or data stream that follows the Elasticsearch base schema for AI agent context (ai-index@mappings, Elasticsearch 9.6 and later). Setting AiIndex = true on [Index<T>] or [DataStream<T>] makes the generated mapping and template follow that schema while keeping your own fields.
Derive from AiIndexDocument, add your own fields, and set AiIndex = true:
public class Article : AiIndexDocument
{
[Keyword]
[JsonPropertyName("author")]
public string? Author { get; set; }
}
[ElasticsearchMappingContext]
[Index<Article>(Name = "ai-index-idx-articles", AiIndex = true)]
public static partial class ArticleContext;
var article = new Article
{
Id = "article-1",
Title = "A title",
Content = "The text of the article",
Author = "someone"
};
AiIndexDocument has a property for each base field, with the JSON name and field type of ai-index@mappings: Timestamp, Id, Type, Title, Description, Content, Tags, Attributes, UpdatedAt, ExpiresAt, References and Governance. Empty fields are not written. Timestamp defaults to the time the instance is created.
The base class declares no roles except [Id] on Id. Add the roles your channel needs in the derived type. A data stream needs [Timestamp], which you put on an override of Timestamp. A property of the derived type with the same name wins, so an override also changes the mapping. Repeat [JsonPropertyName] on an override: only the attributes of the most derived property count.
public class Article : AiIndexDocument
{
[Text(Analyzer = "standard")]
[JsonPropertyName("title")]
public override string? Title { get; set; }
[Timestamp]
[JsonPropertyName("@timestamp")]
public override DateTimeOffset Timestamp { get; set; }
}
A derived type with its own [Id] property, for example a URL, uses that property for _id instead of Id.
A type that does not derive from AiIndexDocument can still set AiIndex = true. The base fields it does not map are added to the mapping, with the same rules as below. Fields that the type defines itself keep their definition, so a type with a text tags field keeps it.
Mapping. The base fields are added next to your own: @timestamp, id, type, title, description and content (text with a .semantic semantic_text multi-field), tags, attributes, references, updated_at, expires_at and governance. The merge is additive: a field you already map keeps its definition, and a partly mapped field only gains what it lacks. In the example Title stays text with its .keyword multi-field and gains .semantic.
The library fills in only the provenance, see Provenance. Set the other properties of AiIndexDocument yourself, or, for a type that does not derive from it, add properties with the JSON names you want, for example [JsonPropertyName("content")]. To fill a base field from a field of your own, use copy_to in your mapping.
Template. The index template composes ai-index@mappings before the generated mappings and ai-index@custom after the generated settings. Both are optional: a cluster without them (before 9.6, or without ai-index@custom) uses the copy of the base mapping that is already in the generated mapping. The template priority is 501, because Elasticsearch's own ai-index-idx-* and ai-index-ds-* templates use 500 and only the highest priority template that matches an index applies.
governance.provenance records which mechanism wrote a document. Pass the mechanism to a sync orchestrator and it stamps every AiIndexDocument it writes:
var orchestrator = new IncrementalSyncOrchestrator<Article>(transport, primary, secondary)
{
Provenance = new AiIndexActor
{
Uri = "workflow://docs-sync",
Metadata = new() { ["version"] = "3", ["run_id"] = runId }
}
};
IncrementalSyncOrchestrator and DeltaSyncOrchestrator both have the Provenance option. For each document they set governance.provenance.updated_by and updated_at to the batch timestamp, and created_by when it is empty. Documents that do not derive from AiIndexDocument are not touched. Without an orchestrator, call AiIndexDocument.StampProvenance(actor, at) yourself.
- A document whose content hash matches the stored one is not replaced, so it keeps its provenance.
ForceReindexreplaces it, and then it gets the new stamp. - A replaced document carries what the incoming document has. The orchestrator does not read the stored one, so
created_byis only the original creator if you carry it over from the stored document; otherwise it is the mechanism that wrote this version. - The actor instance is shared by all documents of a run. Do not change it while the orchestrator runs.
- The base mapping is an embedded copy of registry version 4. A later change to
ai-index@mappingson the cluster is composed in, but only reaches a generated mapping when the library is updated. - The data stream defaults of AI indices (90 day retention,
@timestampdescending sort) are not applied. Configure retention through the bootstrap strategy. - Legacy channels (
IndexChannel,DataStreamChannel) do not read the option. UseIngestChannelor the orchestrators.