IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Configure logging for standalone Elastic Agents Configure monitoring for standalone Elastic Agents »

› ›

Kubernetes autodiscovery with Elastic Agent

edit

Kubernetes autodiscovery with Elastic Agent

edit

Conditions based autodiscover

edit

You can define autodiscover conditions to allow Elastic Agent to automatically identify Pods and start monitoring them using predefined integrations. For example, to automatically identify a Redis Pod and monitor it with the Redis integration, add the following input configuration to the DaemonSet manifest:

- name: redis
  type: redis/metrics
  use_output: default
  meta:
    package:
      name: redis
      version: 0.3.6
  data_stream:
    namespace: default
  streams:
    - data_stream:
        dataset: redis.info
        type: metrics
      metricsets:
        - info
      hosts:
        - '${kubernetes.pod.ip}:6379'
      idle_timeout: 20s
      maxconn: 10
      network: tcp
      period: 10s
      condition: ${kubernetes.labels.app} == 'redis'

To set the target host dynamically only for a targeted Pod based on its labels, use a variable in the Elastic Agent policy to return path information from the provider:

- data_stream:
      dataset: kubernetes.scheduler
      type: metrics
  metricsets:
    - scheduler
  hosts:
    - '${kubernetes.pod.ip}:10251'
  period: 10s
  condition: ${kubernetes.labels.component} == 'kube-scheduler'

The policy generated by this configuration looks like:

- hosts:
  - 172.18.0.4:10251
  metricsets:
  - scheduler
  module: kubernetes
  period: 10s
  processors:
  - add_fields:
    fields:
      namespace: kube-system
      labels:
        component: kube-scheduler
        tier: control-plane
      pod:
        ip: 172.18.0.4
        name: kube-scheduler-kind-control-plane
        uid: 6da04645-04b4-4cb2-b203-2ad58abc6cdf
    target: kubernetes

To set the log path of Pods dynamically in the configuration, use a variable in the Elastic Agent policy to return path information from the provider:

streams:
  - data_stream:
      dataset: generic
    symlinks: true
    paths:
      - /var/log/containers/*${kubernetes.container.id}.log

The policy generated by this configuration looks like:

- paths:
  - /var/log/containers/*c957652eca53594ce496c7b237d19f05be339ebfe281b99ce1c0a0401e48ce3a.log
  processors:
    - add_fields:
        fields:
          container:
            name: kube-scheduler
          labels:
            component: kube-scheduler
            tier: control-plane
          namespace:
            labels:
              kubernetes_io/metadata_name: kube-system
            name: kube-system
            uid: 436369c1-cc50-4fdd-8212-d0215bf66ffa
          node:
            hostname: kind-control-plane
            labels:
              beta_kubernetes_io/os: linux
              kubernetes_io/arch: amd64
            name: kind-control-plane
            uid: d93dc62a-b103-4b81-b7cd-9eaf7957d6d2
          pod:
            ip: 172.18.0.2
            name: kube-scheduler-kind-control-plane
            uid: c052b569-c772-43e8-89cc-149ed6f5c69a
        target: kubernetes
    - add_fields:
        fields:
          id: 7b9754b983ec4e8b9bda39dfbe949a1b6b06c0316dc599031381707bb4ce23b6
          image:
            name: k8s.gcr.io/kube-scheduler:v1.21.1
          runtime: containerd
        target: container
    - add_fields:
        fields:
          cluster:
            name: kind-kind
            url: https://127.0.0.1:52500
        target: orchestrator

Hints (annotations) based autodiscover

edit

This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

Standalone Elastic Agent supports autodiscover based on hints from the provider. The hints mechanism looks for hints in Kubernetes Pod annotations that have the prefix co.elastic.hints. As soon as the container starts, Elastic Agent checks it for hints and launches the proper configuration for the container. Hints tell Elastic Agent how to monitor the container by using the proper integration. This is the full list of supported hints:

Required hints:

edit

co.elastic.hints/packageedit

The package to use for monitoring.

co.elastic.hints/hostedit

The host to use for metrics retrieval.

Optional hints available:

edit

co.elastic.hints/data_streamedit

The list of data_streams to enable. If not specified, the integration’s default data_streams are used. To find the defaults, refer to the Elastic integrations documentation.

If data_streams are specified, hints can be defined for each data_stream, for example, co.elastic.hints/key.period: 10m. Otherwise the data_stream will use the top level hints (co.elastic.hints/period: 10m) in its configuration.

co.elastic.hints/metrics_pathedit

The path to retrieve the metrics from.

co.elastic.hints/periodedit

The time interval for metrics retrieval, for example: 10s

co.elastic.hints/timeoutedit

Metrics retrieval timeout, for example: 3s

co.elastic.hints/usernameedit

The username to use for authentication

co.elastic.hints/passwordedit

The password to use for authentication. It is recommended to retrieve this sensitive information from an ENV variable and avoid placing passwords in plain text.

co.elastic.hints/streamedit

The stream to use for logs collection, for example stdout/stderr.

If the specified package has no logs support, a generic container’s logs input will be used as a fallback.

Available packages that support hints autodiscovery

edit

The available packages that are supported through hints can be found at https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-standalone/templates.d

Configure hints autodiscovery

edit

To enable hints, you must add hints.enabled: true to the provider’s configuration, for example:

providers:
  kubernetes:
    hints.enabled: true

Then ensure that the proper volumes and volumeMounts are specified by uncommenting the appropriate sections in the kubernetes manifest.

volumeMounts:
- name: external-inputs
  mountPath: /etc/elastic-agent/inputs.d
...
volumes:
- name: external-inputs
  emptyDir: {}
...

An init container is also required to download the hints templates. The init container is already defined, so uncomment the respective section:

initContainers:
- name: k8s-templates-downloader
  image: busybox:1.28
  command: ['sh']
  args:
    - -c
    - >-
      mkdir -p /etc/elastic-agent/inputs.d &&
      wget -O - https://github.com/elastic/elastic-agent/archive/8.5.tar.gz | tar xz -C /etc/elastic-agent/inputs.d --strip=5 "elastic-agent-main/deploy/kubernetes/elastic-agent-standalone/templates.d"
  volumeMounts:
    - name: external-inputs
      mountPath: /etc/elastic-agent/inputs.d

Hints autodiscovery: Example

edit

Enabling hints allows users deploying Pods on the cluster to automatically turn on Elastic monitoring at Pod deployment time. For example, to deploy a Redis Pod on the cluster and automatically enable Elastic monitoring, add the proper hints as annotations on the Pod:

apiVersion: v1
kind: Pod
metadata:
  name: redis
  annotations:
    co.elastic.hints/package: redis
    co.elastic.hints/data_streams: info, key, keyspace
    co.elastic.hints/host: '${kubernetes.pod.ip}:6379'
    co.elastic.hints/info.period: 5s
  labels:
    k8s-app: redis
    app: redis
spec:
  containers:
  - image: redis
    imagePullPolicy: IfNotPresent
    name: redis
    ports:
    - name: redis
      containerPort: 6379
      protocol: TCP

After deploying this Pod, the data will start flowing in automatically. Note that at this point all the assets (dashboards, ingest pipelines, and so on) related to the Redis integration are not installed. You need to explicitly install them through Kibana.

Hints autodiscovery: Troubleshooting

edit

When things do not work as expected, you may need to troubleshoot your setup. Here we provide some directions to speed up your investigation:

Exec inside an Agent’s Pod and run the inspect command to verify how inputs are constructed dynamically:
```
./elastic-agent inspect -c /etc/elastic-agent/agent.yml output -o default -v -d "*"
```
Specifically examine how the inputs are being populated.
View the Elastic Agent logs:
```
tail -f /etc/elastic-agent/data/logs/elastic-agent-*.ndjson
```
Verify that the hints feature is enabled in the config and look for hints-related logs like: "Generated hints mappings are …" In these logs, you can find the mappings that are extracted out of the annotations and determine if the values are able to populate a specific input.

View the Metricbeat and Filebeat logs:

tail -f /etc/elastic-agent/data/logs/default/metricbeat-*.ndjson

View the target input template, for example:

cat -f /etc/elastic-agent/inputs.d/redis.yaml

« Configure logging for standalone Elastic Agents Configure monitoring for standalone Elastic Agents »