Define include/exclude filters

You can define multiple filters for inputs to include or exclude events from data ingestion.

inputs:
  - type: "s3-sqs"
    id: "arn:aws:sqs:%REGION%:%ACCOUNT%:%QUEUENAME%"
    include:
      - "[a-zA-Z]"
    exclude:
      - "skip this"
      - "skip also this"
    outputs:
      - type: "elasticsearch"
        args:
          elasticsearch_url: "arn:aws:secretsmanager:eu-central-1:123456789:secret:es_url"
          username: "arn:aws:secretsmanager:eu-west-1:123456789:secret:es_secrets:username"
          password: "arn:aws:secretsmanager:eu-west-1:123456789:secret:es_secrets:password"
          es_datastream_name: "logs-generic-default"

You can define a list of regular expressions within inputs.[].include. If this list is populated, only messages matching any of the defined regular expressions will be forwarded to the outputs.

You can define a list of regular expressions within inputs.[].exclude. If this list is populated, only messages not matching any of the defined regular expressions will be forwarded to the outputs i.e. every message will be forwarded to the outputs unless it matches any of the defined regular expressions.

Both config parameters are optional, and can be set independently of each other. In terms of rule precedence, the exclude filter is applied first and then the include filter, so exclude takes precedence if both are specified.

All regular expressions are case-sensitive and should follow Python’s 3.9 regular expression syntax.

Messages are scanned for terms that match the defined filters. Use the ^ (caret) special character to explicitly anchor the regex to the position before the first character of the string, and use $ to anchor at the end.

No flags are used when the regular expression is compiled. Please refer to inline flag documentation for alternative options for multiline, case-insensitive, and other matching behaviors.

« Use custom tags JSON content discovery »