In an ever-evolving threat landscape, security operations are reaching a tipping point. As the velocity and complexity of threats increase, teams expand and managed environments multiply. Commonly, manual approaches to rule management become a bottleneck. This is where Detections as Code (DaC) steps in, not just as a tool, but as a methodology.

DaC as a methodology applies software development practices to the creation, management, and deployment of security detection rules. By treating detection rules as code, it enables version control, automated testing, and deployment processes, enhancing collaboration, consistency, and agility in response to threats. DaC streamlines the detection rule lifecycle, ensuring high-quality detections through peer reviews and automated tests. This methodology also supports compliance with change management requirements and fosters a mature security posture.

That's why we’re excited to share the latest updates to Elastic's detection-rules, our open repository for writing, testing, and managing security detection rules in Elastic, that also allows you to create your own Detections as Code (DaC) framework. Continue reading for highlighted implementation examples using extended functionality, and the announcement of Elastic's free Detections as Code Workshop.

Elastic Security DaC: The journey from alpha to general availability

With the functionality now provided in detection-rules repository, users can manage all their detection rules as code, review rule tunings, automatically test and validate rules, and automate rules deployment across their environments.

Pre-2024: Elastic’s internal use of DaC

Elastic threat research and detection engineering team created and used the detection-rules repository to develop, test, manage and release prebuilt rules, following DaC principles - reviewing rules as a team, automating their tests and release. The repository also has an interactive CLI to create rules, so engineers could start working on the rules right there.

As the security community's interest in as-code principles grew, and the available Elastic Security APIs already allowed users to implement their custom Detections as code solutions, Elastic decided to extend the detection-rules repository functionality to enable our users to benefit from our tooling and aid them in creating their DaC processes.

Here are the key milestones of Elastic’s user-focused DaC development from alpha to general availability.

May 2024: Alpha release of new "roll your own” features

Our detection-rules repository is adjusted for customer use, allowing for managing custom rules, adapting the test suite for user needs, and allowing for management of actions and exceptions alongside the rules.

Key additions:

• Custom rules directory support
• Select which test to run based on your requirements
• Exceptions and Actions support

We also published an extensive guidance for Detections as Code with examples of implementation with Elastic Security using detection-rules repository.

August 2024: "Roll your own” features now beta

The functionality is extended to allow import and export of custom rules between Elastic Security and repository, more configuration options and versioning functionality extended to custom rules.

New features added:

• Bulk import/export of custom rules (based on Elastic Security APIs)
• Fully configurable unit test, validation, and schemas
• Version lock for custom rules

March - August 2025: are generally available and supported

Using DaC with Elastic Security 8.18 and up:

• Supports prebuilt rules management. You can export all prebuilt rules from Elastic Security and store them alongside your custom rules.
• Support for rules filtering for export added.

Adjacent to DaC efforts, we also released new Terraform resources (V0.12.0 and V0.13.0) in October-December 2025, allowing Terraform users to manage detection rules and exceptions.

With this foundation spelled out, let's explore the powerful features that are available to streamline your detection engineering process.

Detection-rules DaC functionality highlights

There are a few worthwhile additions since our last DaC publication, which we’ll expand on below.

Additional filters

The filter functionality available when exporting rules from Kibana has been extended to allow you to precisely define which rules to sync in DaC. Here are the new flags:

Let’s take an example of when you wish to organize rules by data source, and want to export the AWS rules to a specific folder. In this case, let’s use filtering on tags for data sources and export all rules with the Data Source AWS tag:

python -m detection_rules kibana export-rules -d dac_test/rules #add rules to the dac_test/rules folder
-sv #strip the version fields from all rules
-cro #export only custom rules
-eq "alert.attributes.tags: "Data Source: AWS"" # export only rules with "Data Source: AWS" tag

See Kibana documentation for query string filtering for the underlying API call used here and the list all detection rules API call for example available fields to construct the query filter.

Custom folder structure

In the detection-rules repo, we use a folder structure based on platform, integration, and MITRE ATT&CK information. This helps us with our organization and rule development. This is by no means the only method of organization. You may want to organize your rules by customer, date, or source as examples. This will vary greatly depending on your use case.

Whether you use this export process or manual organization, once you have your rules in a location or folder structure that you like, you can now keep this local structure even when re-exporting rules. It is important to note that the new rules need to be placed in their desired location manually. The local rule-loading mechanism detects where the rules are placed in order to know where to put them. If the rule is not there, it will then use the specified output directory to place the new rule(s). To use the local rule loading for updating existing rules use the --load-rule-loading / -lr flag for the kibana export-rules and import-rules-to-repo commands. These flags enable you to make use of the local folders specified in your config.yaml.

Let’s look at example with the rules organised in folders the following way:

rule_dirs:
- rules
my_test_rule.toml
- another_rules_dir
high_number_of_process_and_or_service_terminations.toml

We’ll specify the following in the config.yaml file:

rule_dirs:
- rules
- another_rules_dir

With the new -lr option, rule updates from Kibana will now use these additional paths instead of exporting directly to the specified directory.

Running python -m detection_rules kibana --space test_local export-rules -d dac_test/rules/ -sv -ac -e -lr,will export rules from test_local space, my_test_rule.toml will be written to dac_test/rules/ as it was already on disk there and high_number_of_process_and_or_service_terminations.toml will be written to dac_test/another_rules_dir/.

This can be particularly useful if you have the same rules in different sub-folder configurations for different customers. For example, let’s say you have your rules broken down by platform and integration similar to Elastic’s prebuilt rule folder structure. For your customers, SOCs, or threat-hunting teams, having the rules organized underneath these platform/integration folders may be the most useful mechanism for them to manage the rules. However, your information security team or primary detection engineering team may want to manage the rules by initiative or rule author instead so that all the rules a particular individual or team is responsible for are organized in one place. Now with the local rule-loading flags, you can simply have two configuration files and the duplicated rules in each structure. When you are exporting updates for the rules, you would then use the environment variable to select the appropriate configuration file and export the rule updates. These updates will then be applied to the rules in place, maintaining the directory structure.

Miscellaneous local loading updates

In addition to the above, we have added two smaller new features designed to help users who are adding local information in the detection rules TOML files and schema. These are as follows:

1. Local date support from the local files where the local date will be maintained from the original file
2. Upgrades to the auto gen feature to inherit known types from existing schema.

The local date component can be useful when one wants more manual control over the date field in the file. Without using the override, the date will be based on when the Kibana rule contents were exported. Using the --local-creation-date flag, the date will not be updated when the file contents are re-exported.

The automatic schema generation has been updated to inherit the types from other indices/integrations if they are present. This provides a potentially more accurate schema, as well as reducing the need for manual updates after the fact. For example, you have a rule that uses the index “new-integration*” with the following fields:

• host.os.type.new_field
• dll.Ext.relative_file_creation_time
• process.name.okta.thread

Instead of each of these fields being added to the schema with a default type, their types are inherited from existing schemas. In this case, the types for dll.Ext.relative_file_creation_time and process.name.okta.thread are inherited.

{
  "new-integration*": {
    "dll.Ext.relative_file_creation_time": "double",
    "host.os.type.new_field": "keyword",
    "process.name.okta.thread": "keyword"
  }
}

To see how to use this with your custom data types, see the Custom schemas usage section within the Implementation examples part of this blog.

Expanding on usage examples

Below you will find more examples of DaC implementations, these are not focused on new functionality additions, but go deeper on the topics we see discussed in the community.

It’s worth noting that Detections as Code features are provided as components that can be used to build a custom implementation for your chosen process and architecture. When implementing DaC in your production environment, treat it as an engineering process and follow the best practices.

DaC implementation with Gitlab

When we look at implementations of DaC typically this revolves around using some form of CI/CD product to automatically perform rule management based on a given trigger. These triggers vary considerably based on the desired setup, specifically the authoritative source of rules and the desired state of your version control system (VCS). For a much more in-depth exploration of some of these considerations, see our DaC Reference Material. Below is a simple example using Gitlab as VCS provider and using its in-built CI/CD via Gitlab Actions.

stages:                # Define the pipeline stages
  - sync               # Add a 'sync' stage

sync-to-production:    # Define a job named 'sync-to-production'
  stage: sync          # Assign this job to the 'sync' stage
  image: python:3.12   # Use the Python 3.12 Docker image
  variables:
    CUSTOM_RULES_DIR: $CUSTOM_RULES_DIR    # Set custom rules env var
  script:                                  # List of commands to run 
    - python -m pip install --upgrade pip  # Upgrade pip
    - pip cache purge                      # Clear pip cache
    - pip install .[dev]                   # Install package w/ dev deps
    - |  # Multi-line command to import rules                                        
      FLAGS="-d ${CUSTOM_RULES_DIR}/rules/ --overwrite -e -ac"
      python -m detection_rules kibana --space production import-rules $FLAGS
  environment:
    name: production   # Specify deployment environment as 'production'
  only:
    refs:
      - main           # Run this job only on the 'main' branch
    changes:
      - '**/*.toml'    # Run this job only if .toml files have changed

This is very similar to other inbuilt CI/CD from other Git-based VCS like Gitlab and Gitea. The main difference being in the syntax determining the triggering event. The DaC commands such as kibana import-rules would be the same regardless of VCS. In this example, we are syncing rules from our fork of the detection-rules repo to our Kibana Production Space. This is based on a number of prior decisions being made, for instance requiring unit tests to pass before merging rule updates and that rules on main being ready for prod. For a Github-based walkthrough of these considerations for this particular approach, please take a look at our demo video.

Custom Unit Testing tips and examples

When considering DaC as a capability to add to your detection toolkit, setting up the CI/CD and base infrastructure should be considered as the first step in an ongoing process to improve the quality and usefulness of your rules. One of the key purposes in having “as code” tooling is adding the ability to further customize tooling to your needs and environment.

One example of this is unit testing for rules. Beyond base functionality testing, some other key existing unit tests enforce Elastic-specific considerations around rule performance and optimization, as well as organization of metadata and tagging. This helps detection engineers and threat researchers remain consistent in their rule development. Building on this example, one may want to consider adding custom unit tests based on your specific needs.

To illustrate this, take a Security Operations Center (SOC) environment where there are a number of analysts responsible for various different domains and tasks. When an alert is raised in the SIEM, it may not be immediately obvious who should handle remediation, or what team(s) need to be informed of the incident. Tagging the rules with a team tag: e.g. Team: Windows Servers similarly to how Elastic uses tags for data sources, can provide the SOC with a point of contact directly in the alert for who can help with remediation.

In our DaC environment, we can quickly create a new testing module to enforce this on all of the custom rules (or pre-built too). For this test, we are going to enforce having a Team: <some name> tag on all production rules that are not authored by Elastic. In the detection-rules repo, our testing is handled through the Python test suite called pytest and as such unit tests are organized into python modules (files) and subsequent classes and functions in these files under the tests/ folder. To add tests simply either add classes or functions to the existing files or create a new one. In general, we recommend creating new test files so that you can receive updates to the existing tests from Elastic without having to merge the differences.

We will start by creating a new python file called test_custom_rules.py in the tests/ directory with the following contents:

# test_custom_rules.py

"""Unit Tests for Custom Rules."""

from .base import BaseRuleTest


class TestCustomRules(BaseRuleTest):
    """Test custom rules for given criteria."""

    def test_custom_rule_team_tag(self):
        """Unit test that all custom rules have a Team: <team_name> tag."""
        tag_format = "Team: <team_name>"
        for rule in self.all_rules:
            if "Elastic" not in rule.contents.data.author:
                tags = rule.contents.data.tags
                if tags:
                    self.assertTrue(
                        any(tag.startswith("Team: ") for tag in tags),
                        f"Custom rule {rule.contents.data.rule_id} does not have a {tag_format} tag",
                    )
                else:
                    raise AssertionError(
                        f"Custom rule {rule.contents.data.rule_id} does not have any tags, include a {tag_format} tag"
                    )

Now each non-Elastic rule will be required to have a tag in the specified pattern for a team responsible for remediation. E.g. Team: Team A.

Custom schemas usage

Elastic’s ability to bring your own data types also extends to our DaC capabilities. For example, let’s take a look at some custom schemas for network protocols. Diverse data you have in your stack can of course be queried by your rules, and we will also want to leverage the applicable validation and testing for any custom rules on these data types too. This is where Custom schemas come in handy.

When we are validating queries, the query is parsed into the respective fields and the types of these fields are compared against what is provided in a given schema (e.g. ECS schema, the AWS Integration for AWS data, etc.). For custom data types, this follows the same validation path, with the ability to pull from locally defined custom schemas. These schema files can be built by hand as one or more json files; however, if you have some sample data already in your stack, you can take advantage of this and use it as validation and generate your schemas automatically.

Assuming you already have a custom rules folder configured (if not see instructions), you can turn on automatic schema generation by adding auto_gen_schema_file: <path_to_your_json_file> to your config file. This will generate a schema file in the specified location that will be used to add entries for each field and index combination. The file will be updated during any command where rule contents are validated against a schema, including import-rules-to-repo, kibana export-rules, view-rule, and others. This will also automatically add it to your stack-schema-map.yaml file when using a custom rules directory and config.

With this power comes an increased responsibility on rule reviewers as any field used in the query is immediately assumed to be valid and added to the schema. One way to mitigate risk is to utilize a development space that has access to the data. In the PR, one can then link to a successful execution of the query with stack level validation on its data types. Once this is approved, one can remove the auto_gen_schema_file addition to the config and you now have a known valid schema based on your custom data. This provides a baseline for other rule authors to build upon as needed and maintains the type checking validation.

Learn more about DaC and try it yourself

You can experience Elastic Security's Detections as Code (DaC) functionality firsthand with our interactive Instruqt training. This training provides a straightforward way to explore core DaC features in a pre-configured test environment, eliminating the need for manual setup. Give it a try!

If you are implementing DaC, share your experience, ask your questions and help others on the community slack DaC channel.

Trial Elastic Security

To experience the full benefits of what Elastic has to offer for detection engineers, start your Elastic Security free trial. Visit elastic.co/security to learn more.

Die Entscheidung über die Veröffentlichung der in diesem Blogeintrag beschriebenen Leistungsmerkmale und Features sowie deren Zeitpunkt liegt allein bei Elastic. Es ist möglich, dass noch nicht verfügbare Leistungsmerkmale oder Features nicht rechtzeitig oder überhaupt nicht veröffentlicht werden.