Sending data to Elasticsearch Serverless

IMPORTANT: This documentation is no longer updated. Refer to Elastic's version policy and the latest documentation.

Sending data to Elasticsearch Serverless

When you use Elasticsearch on Elastic Cloud Serverless you don’t need to worry about managing the infrastructure that keeps Elasticsearch distributed and available. These resources are automated on the serverless platform and are designed to scale up and down with your workload.

Logstash to Elastic Cloud Serverless

You’ll use the Logstash Elasticsearch output plugin to send data to Elastic Cloud Serverless. Note these differences between Elasticsearch Serverless and both Elastic Cloud Hosted and self-managed Elasticsearch:

Use API keys to access Elastic Cloud Serverless from Logstash as it does not support native user authentication. Any user-based security settings in your Elasticsearch output plugin configuration are ignored and may cause errors.
Elastic Cloud Serverless uses data streams and data lifecycle management (DLM) instead of index lifecycle management (ILM). Any ILM settings in your Elasticsearch output plugin configuration are ignored and may cause errors.
Logstash monitoring is available through the Logstash Integration in Elastic Observability on Elastic Cloud Serverless.

Known issue for Logstash to Elasticsearch Serverless.
The logstash-output-elasticsearch hosts setting defaults to port :9200.
Set the value to port :443 instead.

Communication between Logstash and Elasticsearch Serverless

edit

Elasticsearch Serverless simplifies safe, secure communication between Logstash and Elasticsearch. When you configure the Elasticsearch output plugin to use cloud_id and an api_key, no additional SSL configuration is needed.

Example:

output {elasticsearch { cloud_id => "<cloud id>" api_key => "<api key>" } }

Note that the value of the api_key option is in the format id:api_key, where id and api_key are the values returned by the Create API key API.

Cloud ID

edit

Logstash uses the Cloud ID, found in the Elastic Cloud web console, to build the Elasticsearch and Kibana hosts settings. It is a base64 encoded text value of about 120 characters made up of upper and lower case letters and numbers. If you have several Cloud IDs, you can add a label, which is ignored internally, to help you tell them apart. To add a label, prefix your Cloud ID with a label and a : separator in this format "<label>:<cloud-id>".

API key

edit

When you create an API key for Logstash, select Logstash from the API key format dropdown. This option formats the API key in the correct id:api_key format required by Logstash.

Using Cloud ID with plugins

edit

The Elasticsearch input, output, and filter plugins, as well as the Elastic_integration filter plugin, support cloud_id in their configurations.

Using Logstash Central Pipeline Management with Elasticsearch Serverless

edit

This setting in the logstash.yml config file can help you get set up to use Central Pipeline management in Elastic Cloud:

xpack.management.elasticsearch.cloud_id

You can use the xpack.management.elasticsearch.cloud_id setting as an alternative to xpack.management.elasticsearch.hosts.

« Sending data to Elastic Cloud Hosted Logstash configuration examples »