ecs

ECS Tooling Usage

In addition to the published schema and artifacts, the ECS repo contains tools to generate artifacts based on ECS schemas and your custom field definitions.

Why Use ECS Tooling?

For detailed developer documentation, see scripts/docs/README.md.

NOTE - These tools and their functionality are considered experimental.

Table of Contents

Quick Start Example

Here’s a complete example that generates artifacts with:

python scripts/generator.py \
  --ref v9.1.0 \
  --semconv-version v1.38.0 \
  --subset ../my-project/fields/subset.yml \
  --include ../my-project/fields/custom/ \
  --out ../my-project/

This generates:

Note: The --semconv-version flag is required. Use the version from the otel-semconv-version file or a specific version like v1.38.0.

Setup and Install

Requirements: Python 3.8+, git

Clone and setup:

git clone https://github.com/elastic/ecs
cd ecs
git checkout v9.1.0  # Optional: target specific version
pip install -r scripts/requirements.txt  # virtualenv recommended

Basic Usage

Generate artifacts from the current ECS schema:

make generate
# or
python scripts/generator.py --semconv-version v1.38.0

Key points:

For complete documentation on how the generator works, see:

Key Generator Options

Include Custom Fields

Add custom fields to ECS schemas:

python scripts/generator.py \
  --semconv-version v1.38.0 \
  --include ../myproject/custom-fields/ \
  --out ../myproject/out/

Custom field format - Use the same YAML format as ECS schemas:

---
- name: widgets
  title: Widgets
  group: 2
  short: Fields describing widgets
  description: Widget-related fields
  type: group
  fields:
    - name: id
      level: extended
      type: keyword
      short: Unique identifier of the widget
      description: Unique identifier of the widget.

Supports: Directories, multiple paths, wildcards (*.yml), combining with --ref

See also: Schema format documentation

Subset - Use Only Needed Fields

Generate artifacts with only the fields you need (reduces mapping size):

python scripts/generator.py \
  --semconv-version v1.38.0 \
  --subset ../myproject/subset.yml

Example subset file:

---
name: web_logs
fields:
  base:
    fields:
      "@timestamp": {}
  http:
    fields: "*"        # All http fields
  url:
    fields: "*"        # All url fields
  user_agent:
    fields:
      original: {}     # Specific fields only

Subset format:

Tips:

For detailed subset documentation with examples, see scripts/docs/schema-pipeline.md

Ref - Target Specific ECS Version

Generate artifacts from a specific ECS version:

python scripts/generator.py \
  --semconv-version v1.38.0 \
  --ref v9.0.0

Combines with other options:

# Generate from ECS v9.0.0 + custom fields
python scripts/generator.py \
  --semconv-version v1.38.0 \
  --ref v9.0.0 \
  --include ../myproject/fields/custom

Loads schemas from git history (tags, branches, commits). Requires git.

Other Options

--out <directory> - Output to custom directory

python scripts/generator.py --semconv-version v1.38.0 --out ../myproject/

--exclude <files> - Remove specific fields (for testing deprecation impact)

python scripts/generator.py --semconv-version v1.38.0 --exclude deprecated-fields.yml

--strict - Enable strict validation (required for CI/CD)

python scripts/generator.py --semconv-version v1.38.0 --strict

Strict mode requires the following conditions, else the script exits on an exception:

Example error when running with --strict:

$ python scripts/generator.py --ref v1.4.0 --semconv-version v1.38.0 --strict
Loading schemas from git ref v1.4.0
Running generator. ECS version 1.4.0
...
ValueError: Short descriptions must be single line, and under 120 characters (current length: 134).
Offending field or field set: number
Short description:
  Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.

Without --strict, the same issue produces a warning and the script continues:

$ python scripts/generator.py --ref v1.4.0 --semconv-version v1.38.0
Loading schemas from git ref v1.4.0
Running generator. ECS version 1.4.0
~/dev/ecs/scripts/generators/ecs_helpers.py:176: UserWarning: Short descriptions must be single line, and under 120 characters (current length: 134).
Offending field or field set: number
Short description:
  Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.

This will cause an exception when running in strict mode.

--template-settings / --mapping-settings - Custom Elasticsearch template settings

python scripts/generator.py \
  --semconv-version v1.38.0 \
  --template-settings ../myproject/template.json \
  --mapping-settings ../myproject/mappings.json

This is an example template.json to be passed with --template-settings-legacy:

{
  "index_patterns": ["mylog-*"],
  "order": 1,
  "settings": {
    "index": {
      "mapping": {
        "total_fields": {
          "limit": 10000
        }
      },
      "refresh_interval": "1s"
    }
  },
  "template": {
    "mappings": {}
  }
}

This is an example mapping.json to be passed with --mapping-settings:

{
  "_meta": {
    "version": "1.5.0"
  },
    "date_detection": false,
    "dynamic_templates": [
      {
        "strings_as_keyword": {
          "mapping": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "match_mapping_type": "string"
        }
      }
    ],
    "properties": {}
}

The mappings object in template.json and the properties object in mapping.json are left empty — they will be filled in automatically by the script.

--intermediate-only - Generate only intermediate files (for debugging)

--force-docs - Generate docs even with --subset/--include/--exclude

Additional Resources

Complete Documentation

Module-Specific Guides

Contributing

This site is open source. Improve this page.