« Add tags to filter the Agents list Create an agent policy without using the UI »
Elastic Docs ›Fleet and Elastic Agent Guide [8.16] ›Centrally manage Elastic Agents in Fleet

Elastic Agent policies

edit

Elastic Agent policies

edit

A policy is a collection of inputs and settings that defines the data to be collected by an Elastic Agent. Each Elastic Agent can only be enrolled in a single policy.

Within an Elastic Agent policy is a set of individual integration policies. These integration policies define the settings for each input type. The available settings in an integration depend on the version of the integration in use.

Fleet uses Elastic Agent policies in two ways:

  • Policies are stored in a plain-text YAML file and sent to each Elastic Agent to configure its inputs.
  • Policies provide a visual representation of an Elastic Agents configuration in the Fleet UI.

Policy benefits

edit

Elastic Agent policies have many benefits that allow you to:

  • Apply a logical grouping of inputs aimed for a particular set of hosts.
  • Maintain flexibility in large-scale deployments by quickly testing changes before rolling them out.
  • Provide a way to group and manage larger swaths of your infrastructure landscape.

For example, it might make sense to create a policy per operating system type: Windows, macOS, and Linux hosts. Or, organize policies by functional groupings of how the hosts are used: IT email servers, Linux servers, user work-stations, etc. Or perhaps by user categories: engineering department, marketing department, etc.

Policy types

edit

In most use cases, Fleet provides complete central management of Elastic Agents. However some use cases, like running in Kubernetes or using our hosted Elasticsearch Service on Elastic Cloud, require Elastic Agent infrastructure management outside of Fleet. With this in mind, there are two types of Elastic Agent policies:

  • regular policy: The default use case, where Fleet provides full central management for Elastic Agents. Users can manage Elastic Agent infrastructure by adding, removing, or upgrading Elastic Agents. Users can also manage Elastic Agent configuration by updating the Elastic Agent policy.

  • hosted policy: A policy where something else provides central management for Elastic Agents. For example, in Kubernetes, adding, removing, and upgrading Elastic Agents should be configured directly in Kubernetes. Allowing Fleet users to manage Elastic Agents would conflict with any Kubernetes configuration.

    Hosted policies also apply when using our hosted Elasticsearch Service on Elastic Cloud. Elastic Cloud is responsible for hosting Elastic Agents and assigning them to a policy. Platform operators, who create and manage Elastic deployments can add, upgrade, and remove Elastic Agents through the Elastic Cloud console.

Hosted policies display a lock icon in the Fleet UI, and actions are restricted. The following table illustrates the Fleet user actions available to different policy types:

Fleet user action Regular policy Hosted policy

Create a policy

yes

no

Add an integration

yes

no

Apply a policy

yes

no

Edit or delete an integration

yes

no

Copy a policy

yes

no

Edit or delete a policy

yes

no

Add custom fields

yes

no

Configure agent monitoring

yes

no

Change the output of a policy

yes

no

Add a Fleet Server to a policy

yes

no

Configure secret values in a policy

yes

no

Set the maximum CPU usage

yes

no

Set the Elastic Agent log level

yes

no

Change the Elastic Agent binary download location

yes

no

Set the Elastic Agent host name format

yes

no

Set an unenrollment timeout for inactive agents

yes

no

See also the recommended scaling options for an Elastic Agent policy.

Create a policy

edit

To manage your Elastic Agents and the data they collect, create a new policy:

In Fleet, open the Agent policies tab and click Create agent policy.

  1. Name your policy. All other fields are optional and can be modified later. By default, each policy enables the system integration, which collects system information and metrics.

  2. Create the agent policy:

    • To use the UI, click Create agent policy.
    • To use the Fleet API, click Preview API request and run the request.

Also see Create an agent policy without using the UI.

Add an integration to a policy

edit

An Elastic Agent policy consists of one or more integrations that are applied to the agents enrolled in that policy. When you add an integration, the policy created for that integration can be shared with multiple Elastic Agent policies. This reduces the number of integrations policies that you need to actively manage.

To add a new integration to one or more Elastic Agent policies:

  1. In Fleet, click Agent policies. Click the name of a policy you want to add an integration to.
  2. Click Add <integration>.
  3. The Integrations page shows Elastic Agent integrations along with other types, such as Beats. Scroll down and select Elastic Agent only to view only integrations that work with Elastic Agent.
  4. You can opt to install an Elastic Agent if you haven’t already, or choose Add integration only to proceed.
  5. In Step 1 on the Add <integration> page, you can select the configuration settings specific to the integration.

  6. In Step 2 on the page, you have two options:

    1. If you’d like to create a new policy for your Elastic Agents, on the New hosts tab specify a name for the new agent policy and choose whether or not to collect system logs and metrics. Collecting logs and metrics will add the System integration to the new agent policy.
    2. If you already have an Elastic Agent policy created, on the Existing hosts tab use the drop-down menu to specify one or more agent policies that you’d like to add the integration to.
  7. Click Save and continue to confirm your settings.

This action installs the integration and adds it to the Elastic Agent policies that you specified. Fleet distributes the new integration policy to all Elastic Agents that are enrolled in the agent policies.

You can update the settings for an installed integration at any time:

  1. In Kibana, go to the Integrations page.
  2. On the Integration policies tab, for the integration that you like to update open the Actions menu and select Edit integration.

  3. On the Edit <integration> page you can update any configuration settings and also update the list of Elastic Agent polices to which the integration is added.

    If you clear the Agent policies field, the integration will be removed from any Elastic Agent policies to which it had been added.

    To identify any integrations that have been "orphaned", that is, not associated with any Elastic Agent policies, check the Agent polices column on the Integration policies tab. Any integrations that are installed but not associated with an Elastic Agent policy are as labeled as No agent policies.

Apply a policy

edit

You can apply policies to one or more Elastic Agents. To apply a policy:

  1. In Fleet, click Agents.

  2. Select the Elastic Agents you want to assign to the new policy.

    After selecting one or more Elastic Agents, click Assign to new policy under the Actions menu.

    Assign to new policy dropdown

    Unable to select multiple agents? Confirm that your subscription level supports selective agent policy reassignment in Fleet. For more information, refer to Elastic Stack subscriptions.

  3. Select the Elastic Agent policy from the dropdown list, and click Assign policy.

The Elastic Agent status indicator and Elastic Agent logs indicate that the policy is being applied. It may take a few minutes for the policy change to complete before the Elastic Agent status updates to "Healthy".

Edit or delete an integration policy

edit

Integrations can easily be reconfigured or deleted. To edit or delete an integration policy:

  1. In Fleet, click Agent policies. Click the name of the policy you want to edit or delete.

  2. Search or scroll to a specific integration. Open the Actions menu and select Edit integration or Delete integration.

    Editing or deleting an integration is permanent and cannot be undone. If you make a mistake, you can always re-configure or re-add an integration.

Any saved changes are immediately distributed and applied to all Elastic Agents enrolled in the given policy.

To update any secret values in an integration policy, refer to Configure secret values in a policy.

Copy a policy

edit

Policy definitions are stored in a plain-text YAML file that can be downloaded or copied to another policy:

  1. In Fleet, click Agent policies. Click the name of the policy you want to copy or download.

  2. To copy a policy, click Actions → Copy policy. Name the new policy, and provide a description. The exact policy definition is copied to the new policy.

    Alternatively, view and download the policy definition by clicking Actions → View policy.

Edit or delete a policy

edit

You can change high-level configurations like a policy’s name, description, default namespace, and agent monitoring status as necessary:

  1. In Fleet, click Agent policies. Click the name of the policy you want to edit or delete.

  2. Click the Settings tab, make changes, and click Save changes

    Alternatively, click Delete policy to delete the policy. Existing data is not deleted. Any agents assigned to a policy must be unenrolled or assigned to a different policy before a policy can be deleted.

Add custom fields

edit

Use this setting to add a custom field and value set to all data collected from the Elastic Agents enrolled in an Elastic Agent policy. Custom fields are useful when you want to identify or visualize all of the data from a group of agents, and possibly manipulate the data downstream.

To add a custom field:

  1. In Fleet, click Agent policies. Select the name of the policy you want to edit.
  2. Click the Settings tab and scroll to Custom fields.
  3. Click Add field.

  4. Specify a field name and value.

    Sceen capture showing the UI to add a custom field and value
  5. Click Add another field for additional fields. Click Save changes when you’re done.

To edit a custom field:

  1. In Fleet, click Agent policies. Select the name of the policy you want to edit.
  2. Click the Settings tab and scroll to Custom fields. Any custom fields that have been configured are shown.
  3. Click the edit icon to update a field or click the delete icon to remove it.

Note that adding custom tags is not supported for a small set of inputs:

  • apm
  • cloudbeat and all cloudbeat/* inputs
  • cloud-defend
  • fleet-server
  • pf-elastic-collector, pf-elastic-symbolizer, and pf-host-agent

Configure agent monitoring

edit

Use these settings to collect monitoring logs and metrics from Elastic Agent. All monitoring data will be written to the specified Default namespace.

  1. In Fleet, click Agent policies. Select the name of the policy you want to edit.
  2. Click the Settings tab and scroll to Agent monitoring.

  3. Select whether to collect agent logs, agent metrics, or both, from the Elastic Agents that use the policy.

    When this setting is enabled an Elastic Agent integration is created automatically.

  4. Expand the Advanced monitoring options section to access advanced settings.
  5. Save your changes for the updated monitoring settings to take effect.
Advanced agent monitoring settings
edit

HTTP monitoring endpoint

Enabling this setting exposes a /liveness API endpoint that you can use to monitor Elastic Agent health according to the following HTTP codes:

  • 200: Elastic Agent is healthy. The endpoint returns a 200 OK status as long as Elastic Agent is responsive and can process configuration changes.
  • 500: A component or unit is in a failed state.
  • 503: The agent coordinator is unresponsive.

You can pass a failon parameter to the /liveness endpoint to determine what component state will result in a 500 status. For example, curl 'localhost:6792/liveness?failon=degraded' will return 500 if a component is in a degraded state.

The possible values for failon are:

  • degraded: Return an error if a component is in a degraded state or failed state, or if the agent coordinator is unresponsive.
  • failed: Return an error if a unit is in a failed state, or if the agent coordinator is unresponsive.
  • heartbeat: Return an error only if the agent coordinator is unresponsive.

If no failon parameter is provided, the default failon behavior is heartbeat.

The HTTP monitoring endpoint can also be used with Kubernetes, to restart the container for example.

When you enable this setting, you need to provide the host URL and port where the endpoint can be accessed. Using the default localhost is recommended.

When the HTTP monitoring endpoint is enabled you can also select to Enable profiling at /debug/pprof. This controls whether the Elastic Agent exposes the /debug/pprof/ endpoints together with the monitoring endpoints.

The heap profiles available from /debug/pprof/ are included in Elastic Agent diagnostics by default. CPU profiles are also included when the --cpu-profile option is included. For full details about the profiles exposed by /debug/pprof/ refer to the pprof package documentation.

Profiling at /debug/pprof is disabled by default. Data produced by these endpoints can be useful for debugging but present a security risk. It’s recommended to leave this option disabled if the monitoring endpoint is accessible over a network.

Diagnostics rate limiting

You can set a rate limit for the action handler for diagnostics requests coming from Fleet. The setting affects only Fleet-managed Elastic Agents. By default, requests are limited to an interval of 1m and a burst value of 1. This setting does not affect diagnostics collected through the CLI.

Diagnostics file upload

This setting configures retries for the file upload client handling diagnostics requests coming from Fleet. The setting affects only Fleet-managed Elastic Agents. By default, a maximum of 10 retries are allowed with an initial duration of 1s and a backoff duration of 1m. The client may retry failed requests with exponential backoff.

Change the output of a policy

edit

Assuming your Elastic Stack subscription level supports per-policy outputs, you can change the output of a policy to send data to a different output.

  1. In Fleet, click Settings and view the list of available outputs. If necessary, click Add output to add a new output with the settings you require. For more information, refer to Output settings.
  2. Click Agent policies. Click the name of the policy you want to change, then click Settings.

  3. Set Output for integrations and (optionally) Output for agent monitoring to use a different output, for example, Logstash. You might need to scroll down to see these options.

    Unable to select a different output? Confirm that your subscription level supports per-policy outputs in Fleet.

    Screen capture showing the Logstash output policy selected in an agent policy
  4. Save your changes.

Any Elastic Agents enrolled in the agent policy will begin sending data to the specified outputs.

Add a Fleet Server to a policy

edit

If you want to connect multiple agents to a specific on-premises Fleet Server, you can add that Fleet Server to a policy.

Screen capture showing how to add a Fleet Server to a policy when creating or updating the policy.

When the policy is saved, all agents assigned to the policy are configured to use the new Fleet Server as the controller.

Make sure that the Elastic Agents assigned to this policy all have connectivity to the Fleet Server that you added. Lack of connectivity will prevent the Elastic Agent from checking in with the Fleet Server and receiving policy updates, but the agents will still forward data to the cluster.

Configure secret values in a policy

edit

When you create an integration policy you often need to provide sensitive information such as an API key or a password. To help ensure that data can’t be accessed inappropriately, any secret values used in an integration policy are stored separately from other policy details.

As well, after you’ve saved a secret value in Fleet, the value is hidden in both the Fleet UI and in the agent policy definition. When you view the agent policy (Actions → View policy), an environment variable is displayed in place of any secret values, for example ${SECRET_0}.

In order for sensitive values to be stored secretly in Fleet, all configured Fleet Servers must be on version 8.10.0 or higher.

Though secret values stored in Fleet are hidden, they can be updated. To update a secret value in an integration policy:

  1. In Fleet, click Agent policies. Select the name of the policy you want to edit.
  2. Search or scroll to a specific integration. Open the Actions menu and select Edit integration. Any secret information is marked as being hidden.

  3. Click the link to replace the secret value with a new one.

    Screen capture showing a hidden secret value as part of an integration policy
  4. Click Save integration. The original secret value is overwritten in the policy.

Set the maximum CPU usage

edit

You can limit the amount of CPU consumed by Elastic Agent. This parameter limits the number of operating system threads that can be executing Go code simultaneously in each Go process. You can specify an integer value not less than 0, which is the default value that stands for "all available CPUs".

This limit applies independently to the agent and each underlying Go process that it supervises. For example, if Elastic Agent is configured to supervise two Beats with a CPU usage limit of 2 set in the policy, then the total CPU limit is six, where each of the three processes (one Elastic Agent and two Beats) may execute independently on two CPUs.

This setting is similar to the Beats max_procs setting. For more detail, refer to the GOMAXPROCS function in the Go runtime documentation.

  1. In Fleet, click Agent policies. Select the name of the policy you want to edit.
  2. Click the Settings tab and scroll to Advanced settings.
  3. Set Limit CPU usage as needed. For example, to limit Go processes supervised by Elastic Agent to two operating system threads each, set this value to 2.

Set the Elastic Agent log level

edit

You can set the minimum log level that Elastic Agents using the selected policy will send to the configured output. The default setting is info.

  1. In Fleet, click Agent policies. Select the name of the policy you want to edit.
  2. Click the Settings tab and scroll to Advanced settings.
  3. Set the Agent logging level.
  4. Save your changes.

You can also set the log level for an individual agent:

  1. In Fleet, click Agents. Under the Host header, select the Elastic Agent you want to edit.
  2. On the Logs tab, set the Agent logging level and apply your changes. Or, you can choose to reset the agent to use the logging level specified in the agent policy.

Change the Elastic Agent binary download location

edit

Elastic Agents must be able to access the Elastic Artifact Registry to download binaries during upgrades. By default Elastic Agents download artifacts from the artifact registry at https://artifacts.elastic.co/downloads/.

For Elastic Agents that cannot access the internet, you can specify agent binary download settings, and then configure agents to download their artifacts from the alternate location. For more information about running Elastic Agents in a restricted environment, refer to Air-gapped environments.

To change the binary download location:

  1. In Fleet, click Agent policies. Select the name of the policy you want to edit.
  2. Click the Settings tab and scroll to Agent binary download.
  3. Specify the address where you are hosting the artifacts repository or select the default to use the location specified in the Fleet agent binary download settings.

Set the Elastic Agent host name format

edit

The Host name format setting controls the format of information provided about the current host through the host.name key, in events produced by Elastic Agent.

  1. In Fleet, click Agent policies. Select the name of the policy you want to edit.
  2. Click the Settings tab and scroll to Host name format.

  3. Select one of the following:

    • Hostname: Information about the current host is in a non-fully-qualified format (somehost, rather than somehost.example.com). This is the default reporting format.
    • Fully Qualified Domain Name (FQDN): Information about the current host is in FQDN format (somehost.example.com rather than somehost). This helps you to distinguish between hosts on different domains that have similar names. The fully qualified hostname allows each host to be more easily identified when viewed in Kibana, for example.
  4. Save your changes.

FQDN reporting is not currently supported in APM.

For FQDN reporting to work as expected, the hostname of the current host must either:

  • Have a CNAME entry defined in DNS.
  • Have one of its corresponding IP addresses respond successfully to a reverse DNS lookup.

If neither pre-requisite is satisfied, host.name continues to report the hostname of the current host in a non-fully-qualified format.

Set an unenrollment timeout for inactive agents

edit

You can configure a length of time after which any inactive Elastic Agents are automatically unenrolled and their API keys invalidated. This setting is useful when you have agents running in an ephemeral environment, such as Docker or Kubernetes, and you want to prevent inactive agents from consuming unused API keys.

To configure an unenrollment timeout for inactive agents:

  1. In Fleet, click Agent policies. Select the name of the policy you want to edit.
  2. Click the Settings tab and scroll to Inactive agent unenrollment timeout.
  3. Specify an unenrollment timeout period in seconds.
  4. Save your changes.

After you set an unenrollment timeout, any inactive agents are unenrolled automatically after the specified period of time. The unenroll task runs every ten minutes, and it unenrolls a maximum of one thousand agents at a time.

Policy scaling recommendations

edit

A single instance of Fleet supports a maximum of 500 Elastic Agent policies. If more policies are configured, UI performance might be impacted.

« Add tags to filter the Agents list Create an agent policy without using the UI »