Monitor Amazon Kinesis data streams

Amazon Kinesis Data Streams is a real-time data streaming service that you can use to capture, process, and store large amounts of data from a variety of sources, including websites, mobile applications, IoT devices, and more.

Amazon Kinesis Data Streams and Amazon CloudWatch are integrated so that you can collect and monitor CloudWatch metrics for your Kinesis data streams, such as tracking shard usage and recording related operations for each Kinesis data stream. The Elastic Amazon Kinesis Data Stream integration collects metrics from Amazon CloudWatch using Elastic Agent.

By default, Kinesis Data Streams sends stream-level (basic level) metrics to CloudWatch every minute automatically. There is also shard-level data (enhanced level) that is sent to CloudWatch every minute and incurs an additional cost per stream. To get shard-level data, you must specifically enable it for each stream by using the AWS Kinesis enable-enhanced-monitoring API. For example

aws kinesis enable-enhanced-monitoring --stream-name samplestream --shard-level-metrics ALL

For more details, refer to the EnableEnhancedMonitoring documentation.

To collect Kinesis data stream metrics from Amazon CloudWatch, you typically need to install the Elastic Amazon Kinesis Data Stream integration and deploy an Elastic Agent.

Expand the quick guide to learn how, or skip to the next section if your data is already in Elasticsearch.

Elastic Agent is currently the preferred way to add Kinesis data stream metrics. For other ways, refer to Adding data to Elasticsearch.

Kibana provides a full data analytics platform with out-of-the-box dashboards that you can clone and enhance to satisfy your custom visualization use cases. For example, to see an overview of your Kinesis data streams in Kibana, go to the Dashboard app and navigate to the [Metrics AWS] Kinesis Overview dashboard.

This section lists the key metrics that you should watch, organized by category. For a full description of fields exported by the integration, refer to the Amazon Kinesis Data Stream integration docs.