- Elastic Cloud Enterprise - Elastic Cloud on your Infrastructure: other versions:
- Introducing Elastic Cloud Enterprise
- Preparing your installation
- Installing Elastic Cloud Enterprise
- Identify the deployment scenario
- Install ECE on a public cloud
- Install ECE on your own premises
- Alternative: Install ECE with Ansible
- Log into the Cloud UI
- Install ECE on additional hosts
- Migrate ECE to Podman hosts
- Post-installation steps
- Configuring your installation
- System deployments configuration
- Configure deployment templates
- Tag your allocators
- Edit instance configurations
- Create instance configurations
- Create deployment templates
- Configure system deployment templates
- Configure index management for templates
- Updating custom templates to support
node_roles
and autoscaling - Updating custom templates to support Integrations Server
- Default instance configurations
- Include additional Kibana plugins
- Manage snapshot repositories
- Manage licenses
- Change the ECE API URL
- Change endpoint URLs
- Enable custom endpoint aliases
- Configure allocator affinity
- Change allocator disconnect timeout
- Migrate ECE on Podman hosts to SELinux in
enforcing
mode
- Securing your installation
- Monitoring your installation
- Administering your installation
- Working with deployments
- Create a deployment
- Access Kibana
- Adding data to Elasticsearch
- Migrating data
- Ingesting data from your application
- Ingest data with Node.js on Elastic Cloud Enterprise
- Ingest data with Python on Elastic Cloud Enterprise
- Ingest data from Beats to Elastic Cloud Enterprise with Logstash as a proxy
- Ingest data from a relational database into Elastic Cloud Enterprise
- Ingest logs from a Python application using Filebeat
- Ingest logs from a Node.js web application using Filebeat
- Manage data from the command line
- Administering deployments
- Change your deployment configuration
- Maintenance mode
- Terminate a deployment
- Restart a deployment
- Restore a deployment
- Delete a deployment
- Migrate to index lifecycle management
- Disable an Elasticsearch data tier
- Access the Elasticsearch API console
- Work with snapshots
- Restore a snapshot across clusters
- Upgrade versions
- Editing your user settings
- Deployment autoscaling
- Configure Beats and Logstash with Cloud ID
- Keep your clusters healthy
- Keep track of deployment activity
- Secure your clusters
- Deployment heap dumps
- Deployment thread dumps
- Traffic Filtering
- Connect to your cluster
- Manage your Kibana instance
- Manage your APM & Fleet Server (7.13+)
- Manage your APM Server (versions before 7.13)
- Manage your Integrations Server
- Switch from APM to Integrations Server payload
- Enable logging and monitoring
- Enable cross-cluster search and cross-cluster replication
- Access other deployments of the same Elastic Cloud Enterprise environment
- Access deployments of another Elastic Cloud Enterprise environment
- Access deployments of an Elasticsearch Service organization
- Access clusters of a self-managed environment
- Enabling CCS/R between Elastic Cloud Enterprise and ECK
- Edit or remove a trusted environment
- Migrate the cross-cluster search deployment template
- Enable App Search
- Enable Enterprise Search
- Enable Graph (versions before 5.0)
- Troubleshooting
- RESTful API
- Authentication
- API calls
- How to access the API
- API examples
- Setting up your environment
- A first API call: What deployments are there?
- Create a first Deployment: Elasticsearch and Kibana
- Applying a new plan: Resize and add high availability
- Updating a deployment: Checking on progress
- Applying a new deployment configuration: Upgrade
- Enable more stack features: Add Enterprise Search to a deployment
- Dipping a toe into platform automation: Generate a roles token
- Customize your deployment
- Remove unwanted deployment templates and instance configurations
- Secure your settings
- API reference
- Changes to index allocation and API
- Script reference
- Release notes
- Elastic Cloud Enterprise 3.7.3
- Elastic Cloud Enterprise 3.7.2
- Elastic Cloud Enterprise 3.7.1
- Elastic Cloud Enterprise 3.7.0
- Elastic Cloud Enterprise 3.6.2
- Elastic Cloud Enterprise 3.6.1
- Elastic Cloud Enterprise 3.6.0
- Elastic Cloud Enterprise 3.5.1
- Elastic Cloud Enterprise 3.5.0
- Elastic Cloud Enterprise 3.4.1
- Elastic Cloud Enterprise 3.4.0
- Elastic Cloud Enterprise 3.3.0
- Elastic Cloud Enterprise 3.2.1
- Elastic Cloud Enterprise 3.2.0
- Elastic Cloud Enterprise 3.1.1
- Elastic Cloud Enterprise 3.1.0
- Elastic Cloud Enterprise 3.0.0
- Elastic Cloud Enterprise 2.13.4
- Elastic Cloud Enterprise 2.13.3
- Elastic Cloud Enterprise 2.13.2
- Elastic Cloud Enterprise 2.13.1
- Elastic Cloud Enterprise 2.13.0
- Elastic Cloud Enterprise 2.12.4
- Elastic Cloud Enterprise 2.12.3
- Elastic Cloud Enterprise 2.12.2
- Elastic Cloud Enterprise 2.12.1
- Elastic Cloud Enterprise 2.12.0
- Elastic Cloud Enterprise 2.11.2
- Elastic Cloud Enterprise 2.11.1
- Elastic Cloud Enterprise 2.11.0
- Elastic Cloud Enterprise 2.10.1
- Elastic Cloud Enterprise 2.10.0
- Elastic Cloud Enterprise 2.9.2
- Elastic Cloud Enterprise 2.9.1
- Elastic Cloud Enterprise 2.9.0
- Elastic Cloud Enterprise 2.8.1
- Elastic Cloud Enterprise 2.8.0
- Elastic Cloud Enterprise 2.7.2
- Elastic Cloud Enterprise 2.7.1
- Elastic Cloud Enterprise 2.7.0
- Elastic Cloud Enterprise 2.6.2
- Elastic Cloud Enterprise 2.6.1
- Elastic Cloud Enterprise 2.6.0
- Elastic Cloud Enterprise 2.5.1
- Elastic Cloud Enterprise 2.5.0
- Elastic Cloud Enterprise 2.4.3
- Elastic Cloud Enterprise 2.4.2
- Elastic Cloud Enterprise 2.4.1
- Elastic Cloud Enterprise 2.4.0
- Elastic Cloud Enterprise 2.3.2
- Elastic Cloud Enterprise 2.3.1
- Elastic Cloud Enterprise 2.3.0
- Elastic Cloud Enterprise 2.2.3
- Elastic Cloud Enterprise 2.2.2
- Elastic Cloud Enterprise 2.2.1
- Elastic Cloud Enterprise 2.2.0
- Elastic Cloud Enterprise 2.1.1
- Elastic Cloud Enterprise 2.1.0
- Elastic Cloud Enterprise 2.0.1
- Elastic Cloud Enterprise 2.0.0
- Elastic Cloud Enterprise 1.1.5
- Elastic Cloud Enterprise 1.1.4
- Elastic Cloud Enterprise 1.1.3
- Elastic Cloud Enterprise 1.1.2
- Elastic Cloud Enterprise 1.1.1
- Elastic Cloud Enterprise 1.1.0
- Elastic Cloud Enterprise 1.0.2
- Elastic Cloud Enterprise 1.0.1
- Elastic Cloud Enterprise 1.0.0
- What’s new with the Elastic Stack
- About this product
How do I resolve node bootlooping?
editHow do I resolve node bootlooping?
editWhen you attempt to apply a configuration change to a deployment, the attempt may fail with an error indicating that the change could not be applied, and deployment resources may be unable to restart. In some cases, bootlooping may result, where the deployment resources cycle through a continual reboot process.

To confirm if your Elasticsearch cluster is bootlooping, you can check the most recent plan under your Deployment Activity page for the error:
Plan change failed: Some instances were not running
Following are some frequent causes of a failed configuration change:
If you’re unable to remediate the failing plan’s root cause, you can reset the deployment to the latest successful Elasticsearch configuration by performing a no-op plan.
Secure settings
editThe most frequent cause of a failed deployment configuration change is due to invalid or mislocated secure settings.
The keystore allows you to safely store sensitive settings, such as passwords, as a key/value pair. You can then access a secret value from a settings file by referencing its key. Importantly, not all settings can be stored in the keystore, and the keystore does not validate the settings that you add. Adding unsupported settings can cause Elasticsearch or other components to fail to restart. To check whether a setting is supported in the keystore, look for a "Secure" qualifier in the lists of reloadable settings.
The following sections detail some secure settings problems that can result in a configuration change error that can prevent a deployment from restarting. You might diagnose these plan failures via the logs or via their related exit codes 1
, 3
, and 78
.
Invalid or outdated values
editThe keystore does not validate any settings that you add, so invalid or outdated values are a common source of errors when you apply a configuration change to a deployment.
To check the current set of stored settings:
- Open the deployment Security page.
- In the Elasticsearch keystore section, check the Security keys list. The list is shown only if you currently have settings configured in the keystore.
One frequent cause of errors is when settings in the keystore are no longer valid, such as when SAML settings are added for a test environment, but the settings are either not carried over or no longer valid in a production environment.
Snapshot repositories
editSometimes, settings added to the keystore to connect to a snapshot repository may not be valid. When this happens, you may get an error such as SettingsException[Neither a secret key nor a shared access token was set.]
For example, when adding an Azure repository storage setting such as azure.client.default.account
to the keystore, the associated setting azure.client.default.key
must also be added for the configuration to be valid.
Third-party authentication
editWhen you configure third-party authentication, it’s important that all required configuration elements that are stored in the keystore are included in the Elasticsearch user settings file. For example, when you create a SAML realm, omitting a field such as idp.entity_id
when that setting is present in the keystore results in a failed configuration change.
Wrong location
editIn some cases, settings may accidentally be added to the keystore that should have been added to the Elasticsearch user settings file. It’s always a good idea to check the lists of reloadable settings to determine if a setting can be stored in the keystore. Settings that can safely be added to the keystore are flagged as Secure
.
Expired custom plugins or bundles
editDuring the process of applying a configuration change, Elastic Cloud Enterprise checks to determine if any uploaded custom plugins or bundles are expired.
To view any added plugins or bundles:
- Go to the Features page and open the Extensions tab.
- Select any extension and then choose Update extension to renew it. No other changes are needed, and any associated configuration change failures should now be able to succeed.
OOM errors
editConfiguration change errors can occur when there is insufficient RAM configured for a data tier. In this case, the cluster typically also shows OOM (out of memory) errors. To resolve these, you need to increase the amount of heap memory, which is half of the amount of memory allocated to a cluster. You might also detect OOM in plan changes via its related exit codes 127
, 137
, and 158
.
As well, you can read our detailed blog Managing and troubleshooting Elasticsearch memory.
Existing index
editIn rare cases, when you attempt to upgrade the version of a deployment and the upgrade fails on the first attempt, subsequent attempts to upgrade may fail due to already existing resources. The problem may be due to the system preventing itself from overwriting existing indices, resulting in an error such as this: Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_2 and restarting Kibana
.
To resolve this:
- Check that you don’t need the content.
-
Run an Elasticsearch Delete index request to remove the existing index.
In this example, the
.kibana_2
index is the rollover of saved objects (such as Kibana visualizations or dashboards) from the original.kibana_1
index. Since.kibana_2
was created as part of the failed upgrade process, this index does not yet contain any pertinent data and it can safely be deleted. - Retry the deployment configuration change.
Insufficient Storage
editConfiguration change errors can occur when there is insufficient disk space for a data tier. To resolve this, you need to increase the size of that tier to ensure it provides enough storage to accommodate the data in your cluster tier considering the high watermark.
On this page