« Quickstart Configuration Examples »
Elastic Docs ›Elastic Cloud on Kubernetes [2.14] ›Orchestrating Elastic Stack applications ›Run Fleet-managed Elastic Agent on ECK

Configuration

edit

Configuration

edit

Fleet-managed Elastic Agents must connect to Fleet Server to receive their configurations. You can deploy Fleet Server instances using ECKs Agent CRD with the appropriate configuration, as shown in Fleet mode and Fleet Server.

To know more about Fleet architecture and related components, check the Fleet documentation.

Fleet mode and Fleet Server

edit

To run both Fleet Server and Elastic Agent in Fleet-managed mode, set the mode configuration element to fleet.

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent-sample
spec:
  mode: fleet

To run Fleet Server, set the fleetServerEnabled configuration element to true, as shown in this example:

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server-sample
spec:
  mode: fleet
  fleetServerEnabled: true

You can leave the default value false for any other case.

Configure Kibana

edit

To have Fleet running properly, the following settings must be correctly set in the Kibana configuration:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-sample
spec:
  config:
    xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-sample-es-http.default.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-sample-agent-http.default.svc:8220"]
    xpack.fleet.packages:
      - name: system
        version: latest
      - name: elastic_agent
        version: latest
      - name: fleet_server
        version: latest
    xpack.fleet.agentPolicies:
      - name: Fleet Server on ECK policy
        id: eck-fleet-server
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        package_policies:
        - name: fleet_server-1
          id: fleet_server-1
          package:
            name: fleet_server
      - name: Elastic Agent on ECK policy
        id: eck-agent
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        is_default: true
        package_policies:
          - name: system-1
            id: system-1
            package:
              name: system
  • xpack.fleet.agents.elasticsearch.hosts must point to the Elasticsearch cluster where Elastic Agents should send data. For ECK-managed Elasticsearch clusters ECK creates a Service accessible through https://ES_RESOURCE_NAME-es-http.ES_RESOURCE_NAMESPACE.svc:9200 URL, where ES_RESOURCE_NAME is the name of Elasticsearch resource and ES_RESOURCE_NAMESPACE is the namespace it was deployed within. See Storing local state in host path volume for details on adjusting this field when running agent as non-root as it becomes required.
  • xpack.fleet.agents.fleet_server.hosts must point to Fleet Server that Elastic Agents should connect to. For ECK-managed Fleet Server instances, ECK creates a Service accessible through https://FS_RESOURCE_NAME-agent-http.FS_RESOURCE_NAMESPACE.svc:8220 URL, where FS_RESOURCE_NAME is the name of Elastic Agent resource with Fleet Server enabled and FS_RESOURCE_NAMESPACE is the namespace it was deployed in.
  • xpack.fleet.packages are required packages to enable Fleet Server and Elastic Agents to enroll.
  • xpack.fleet.agentPolicies policies are needed for Fleet Server and Elastic Agents to enroll to, check https://www.elastic.co/guide/en/fleet/current/agent-policy.html for more information.

Set referenced resources

edit

Both Fleet Server and Elastic Agent in Fleet mode can be automatically set up with Fleet by ECK. The ECK operator can set up Fleet in Kibana (which otherwise requires manual steps) and enroll Fleet Server in the default Fleet Server policy. Elastic Agent can be automatically enrolled in the default Elastic Agent policy. To allow ECK to set this up, provide a reference to a ECK-managed Kibana through the kibanaRef configuration element.

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server-sample
spec:
  kibanaRef:
    name: kibana

ECK can also facilitate the connection between Elastic Agents and a ECK-managed Fleet Server. To allow ECK to set this up, provide a reference to Fleet Server through the fleetServerRef configuration element.

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent-sample
spec:
  fleetServerRef:
    name: fleet-server-sample

Set the elasticsearchRefs element in your Fleet Server to point to the Elasticsearch cluster that will manage Fleet. Leave elasticsearchRefs empty or unset it for any Elastic Agent running in Fleet mode as the Elasticsearch cluster to target will come from Kibana’s xpack.fleet.agents.elasticsearch.hosts configuration element.

Currently, Elastic Agent in Fleet mode supports only a single output, so only a single Elasticsearch cluster can be referenced.

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server-sample
spec:
  elasticsearchRefs:
  - name: elasticsearch-sample

By default, every reference targets all instances in your Elasticsearch, Kibana and Fleet Server deployments, respectively. If you want to direct traffic to specific instances, refer to Traffic Splitting for more information and examples.

Customize Elastic Agent configuration

edit

In contrast to Elastic Agents in standalone mode, the configuration is managed through Fleet, and it cannot be defined through config or configRef elements.

Upgrade the Elastic Agent specification

edit

You can upgrade the Elastic Agent version or change settings by editing the YAML specification file. ECK applies the changes by performing a rolling restart of the Agent’s Pods. Depending on the settings that you used, ECK will set up Fleet in Kibana, enrolls the agent in Fleet, or restarts Elastic Agent on certificate rollover.

Choose the deployment model

edit

Depending on the use case, Elastic Agent may need to be deployed as a Deployment, a DaemonSet, or a StatefulSet. To choose how to deploy your Elastic Agents, provide a podTemplate element under the deployment or the daemonSet element in the specification. If you choose the deployment option, you can additionally specify the strategy used to replace old Pods with new ones.

Similarly, you can set the update strategy when deploying as a DaemonSet. This allows you to control the rollout speed for new configuration by modifying the maxUnavailable setting:

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent-sample
spec:
  version: 8.15.3
  daemonSet:
    strategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: 3
...

Refer to Set compute resources for Beats and Elastic Agent for more information on how to use the Pod template to adjust the resources given to Elastic Agent.

Role Based Access Control for Elastic Agent

edit

Some Elastic Agent features, such as the Kubernetes integration, require that Agent Pods interact with Kubernetes APIs. This functionality requires specific permissions. Standard Kubernetes RBAC rules apply. For example, to allow API interactions:

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent-sample
spec:
  version: 8.15.3
  elasticsearchRefs:
  - name: elasticsearch-sample
  daemonSet:
    podTemplate:
      spec:
        automountServiceAccountToken: true
        serviceAccountName: elastic-agent
...
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-agent
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  - nodes
  - nodes/metrics
  - nodes/proxy
  - nodes/stats
  - events
  verbs:
  - get
  - watch
  - list
- nonResourceURLs:
  - /metrics
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-agent
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-agent
subjects:
- kind: ServiceAccount
  name: elastic-agent
  namespace: default
roleRef:
  kind: ClusterRole
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io

Deploy Elastic Agent in secured clusters

edit

To deploy Elastic Agent in clusters with the Pod Security Policy admission controller enabled, or in OpenShift clusters, you might need to grant additional permissions to the Service Account used by the Elastic Agent Pods. Those Service Accounts must be bound to a Role or ClusterRole that has use permission for the required Pod Security Policy or Security Context Constraints. Different Elastic Agent integrations might require different settings set in their PSP/SCC.

Customize Fleet Server Service

edit

By default, ECK creates a Service for Fleet Server that Elastic Agents can connect through. You can customize it using the http configuration element. Check more information on how to make changes to the Service and customize the TLS configuration.

Control Fleet policy selection

edit

ECK uses the default policy to enroll Elastic Agents in Fleet and the default Fleet Server policy to enroll Fleet Server. A different policy can be chosen by using the policyID attribute in the Elastic Agent resource:

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server-sample
spec:
  policyID: my-custom-policy
...

Please note that the environment variables related to policy selection mentioned in the Elastic Agent docs like FLEET_SERVER_POLICY_ID will be managed by the ECK operator.

Running as a non-root user

edit

In order to run Elastic Agent as a non-root user you must choose how you want to persist data to the Agent’s volume.

  1. Run Elastic Agent with an emptyDir volume. This has the downside of not persisting data between restarts of the Elastic Agent which can duplicate work done by the previous running Agent.
  2. Run Elastic Agent with a hostPath volume in addition to a DaemonSet running as root that sets up permissions for the agent user.

In addition to these decisions, if you are running Elastic Agent in Fleet mode as a non-root user, you must configure certificate_authorities.ssl in each xpack.fleet.outputs to trust the CA of the Elasticsearch Cluster.

To run Elastic Agent with an emptyDir volume.

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server
spec:
  deployment:
    podTemplate:
      spec:
        securityContext: 
          fsGroup: 1000
        volumes:
        - name: agent-data
          emptyDir: {}
...

Gid 1000 is the default group at which the Agent container runs. Adjust as necessary if runAsGroup has been modified.

To run Elastic Agent with a hostPath volume and a DaemonSet to maintain permissions.

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server-sample
  namespace: elastic-apps
spec:
  mode: fleet
  fleetServerEnabled: true
  deployment: {}
...
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent-sample
  namespace: elastic-apps
spec:
  daemonSet: {}
...
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: manage-agent-hostpath-permissions
  namespace: elastic-apps
spec:
  selector:
    matchLabels:
      name: manage-agent-hostpath-permissions
  template:
    metadata:
      labels:
        name: manage-agent-hostpath-permissions
    spec:
      # serviceAccountName: elastic-agent 
      volumes:
        - hostPath:
            path: /var/lib/elastic-agent
            type: DirectoryOrCreate
          name: "agent-data"
      initContainers:
        - name: manage-agent-hostpath-permissions
          # image: registry.access.redhat.com/ubi9/ubi-minimal:latest 
          image: docker.io/bash:5.2.15
          resources:
            limits:
              cpu: 100m
              memory: 32Mi
          securityContext:
            # privileged: true 
            runAsUser: 0
          volumeMounts:
            - mountPath: /var/lib/elastic-agent
              name: agent-data
          command:
          - 'bash'
          - '-e'
          - '-c'
          - |-
            # Adjust this with /var/lib/elastic-agent/YOUR-NAMESPACE/YOUR-AGENT-NAME/state
            # Multiple directories are supported for the fleet-server + agent use case.
            dirs=(
              "/var/lib/elastic-agent/default/elastic-agent/state"
              "/var/lib/elastic-agent/default/fleet-server/state"
              )
            for dir in ${dirs[@]}; do
              mkdir -p "${dir}"
              # chcon is only required when running an an SELinux-enabled/OpenShift environment.
              # chcon -Rt svirt_sandbox_file_t "${dir}"
              chmod g+rw "${dir}"
              # Gid 1000 is the default group at which the Agent container runs. Adjust as necessary if `runAsGroup` has been modified.
              chgrp 1000 "${dir}"
              if [ -n "$(ls -A ${dir} 2>/dev/null)" ]
              then
                # Gid 1000 is the default group at which the Agent container runs. Adjust as necessary if `runAsGroup` has been modified.
                chgrp 1000 "${dir}"/*
                chmod g+rw "${dir}"/*
              fi
            done
      containers:
        - name: sleep
          image: gcr.io/google-containers/pause-amd64:3.2

This is only required when running in an SElinux-enabled/OpenShift environment. Ensure this user has been added to the privileged security context constraints (SCC) in the correct namespace. oc adm policy add-scc-to-user privileged -z elastic-agent -n elastic-apps

UBI is only required when needing the chcon binary when running in an SELinux-enabled/OpenShift environment. If that is not required then the following smaller image can be used instead: docker.io/bash:5.2.15

Privileged is only required when running in an SElinux-enabled/OpenShift environment.

When running Agent in fleet mode as a non-root user Kibana must be configured in order to properly accept the CA of the Elasticsearch cluster.

---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-sample
spec:
  config:
    # xpack.fleet.agents.elasticsearch.hosts: 
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-sample-agent-http.default.svc:8220"]
    xpack.fleet.outputs:
    - id: eck-fleet-agent-output-elasticsearch
      is_default: true
      name: eck-elasticsearch
      type: elasticsearch
      hosts:
      - "https://elasticsearch-sample-es-http.default.svc:9200" 
      ssl:
        certificate_authorities: ["/mnt/elastic-internal/elasticsearch-association/default/elasticsearch-sample/certs/ca.crt"]

This entry must not exist when running agent in fleet mode as a non-root user.

Note that the correct URL for Elasticsearch is https://ELASTICSEARCH_NAME-es-http.YOUR-NAMESPACE.svc:9200

Note that the correct path for Elasticsearch certificate_authorities is /mnt/elastic-internal/elasticsearch-association/YOUR-NAMESPACE/ELASTICSEARCH-NAME/certs/ca.crt
« Quickstart Configuration Examples »