Add Kubernetes metadata

The add_kubernetes_metadata processor annotates each event with relevant metadata based on which Kubernetes pod the event originated from. This processor only adds metadata to the events that do not have it yet present.

At startup, it detects an in_cluster environment and caches the Kubernetes-related metadata. Events are only annotated if a valid configuration is detected. If it’s not able to detect a valid Kubernetes configuration, the events are not annotated with Kubernetes-related metadata.

Each event is annotated with:

In addition, the node and namespace metadata are added to the pod metadata.

The add_kubernetes_metadata processor has two basic building blocks:

Indexers use pod metadata to create unique identifiers for each one of the pods. These identifiers help to correlate the metadata of the observed pods with actual events. For example, the ip_port indexer can take a Kubernetes pod and create identifiers for it based on all its pod_ip:container_port combinations.

Matchers use information in events to construct lookup keys that match the identifiers created by the indexers. For example, when the fields matcher takes ["metricset.host"] as a lookup field, it would construct a lookup key with the value of the field metricset.host. When one of these lookup keys matches with one of the identifiers, the event is enriched with the metadata of the identified pod.

Each Beat can define its own default indexers and matchers which are enabled by default. For example, Filebeat enables the container indexer, which identifies pod metadata based on all container IDs, and a logs_path matcher, which takes the log.file.path field, extracts the container ID, and uses it to retrieve metadata.

You can find more information about the available indexers and matchers, and some examples in Indexers and matchers.

The configuration below enables the processor when heartbeat is run as a pod in Kubernetes.

processors:
  - add_kubernetes_metadata:
      # Defining indexers and matchers manually is required for heartbeat, for instance:
      #indexers:
      #  - ip_port:
      #matchers:
      #  - fields:
      #      lookup_fields: ["metricset.host"]
      #labels.dedot: true
      #annotations.dedot: true

The configuration below enables the processor on a Beat running as a process on the Kubernetes node.

processors:
  - add_kubernetes_metadata:
      host: <hostname>
      # If kube_config is not set, KUBECONFIG environment variable will be checked
      # and if not present it will fall back to InCluster
      kube_config: $Heartbeat Reference [master]/.kube/config
      # Defining indexers and matchers manually is required for heartbeat, for instance:
      #indexers:
      #  - ip_port:
      #matchers:
      #  - fields:
      #      lookup_fields: ["metricset.host"]
      #labels.dedot: true
      #annotations.dedot: true

The configuration below has the default indexers and matchers disabled and enables ones that the user is interested in.

processors:
  - add_kubernetes_metadata:
      host: <hostname>
      # If kube_config is not set, KUBECONFIG environment variable will be checked
      # and if not present it will fall back to InCluster
      kube_config: ~/.kube/config
      default_indexers.enabled: false
      default_matchers.enabled: false
      indexers:
        - ip_port:
      matchers:
        - fields:
            lookup_fields: ["metricset.host"]
      #labels.dedot: true
      #annotations.dedot: true

The add_kubernetes_metadata processor has the following configuration settings:

      add_resource_metadata:
        namespace:
          include_labels: ["namespacelabel1"]
          #labels.dedot: true
          #annotations.dedot: true
        node:
          include_labels: ["nodelabel2"]
          include_annotations: ["nodeannotation1"]
          #labels.dedot: true
          #annotations.dedot: true
        deployment: false
        cronjob: false

      kube_client_options:
        qps: 5
        burst: 10

processors:
- add_kubernetes_metadata:
    ...
    default_indexers.enabled: false
    default_matchers.enabled: false
    indexers:
      - ip_port:
    matchers:
      - field_format:
          format: '%{[destination.ip]}:%{[destination.port]}'

processors:
- add_kubernetes_metadata:
    ...
    default_indexers.enabled: false
    default_matchers.enabled: false
    indexers:
      - ip_port:
    matchers:
      - fields:
          lookup_fields: ['destination.ip', 'server.ip']

processors:
  - add_kubernetes_metadata:
      indexers:
        - container:
      matchers:
        - fields:
            lookup_fields: ['system.process.cgroup.id']
            regex_pattern: 'cri-containerd-(?P<key>[0-9a-z]+)\.scope'