In previous blogs, we explored how Elastic Observability can help you monitor various AWS services and analyze them effectively:
- Managing fundamental AWS services such as Amazon EC2, Amazon RDS, Amazon VPC, and NAT gateway
- Data can be ingested into Elastic observability using a serverless forwarder or Amazon Kinesis Data Firehose
- Ingesting and analyzing AWS VPC Flow logs
One of the more heavily used AWS container services is Amazon ECS (Elastic Container Service). While there is a trend toward using Fargate to simplify the setup and management of ECS clusters, many users still prefer using Amazon ECS with EC2 instances. It may not be as straightforward or efficient as AWS Fargate, but it offers more control over the underlying infrastructure.
In the most recent blog, we explored how Elastic Observability helps manage Amazon ECS with Fargate. However, this blog will review how to manage an Amazon ECS cluster with EC2 instances using Elastic Observability instead.
In general, when setting up Amazon ECS-based clusters with EC2, you may or may not have access to the EC2 instances. This determines what you can use with Elastic Observability in monitoring your EC2-based ECS cluster. Hence, there are two components you can use in monitoring the EC2-based ECS cluster with Elastic Observability:
As you can see in the diagram above, the two components are:
-
_ Baseline setup __ : _ The elastic agent running the AWS integration is configured to obtain ECS metrics and logs from cloud watch. This agent runs on an instance that is not part of the ECS cluster because it allows you to see ALL ECS clusters and other AWS Services, such as EKS, RDS, and EC2.
-
_ Additional setup: _ If you have access to the EC2 instances in the ECS cluster, then you can run Elastic’s docker integration in each EC2 instance. This gives you significantly more details on the containers than AWS container insights. And it does not require AWS Cloudwatch, which can be fairly costly.
Using either just the baseline or the additional setup, you will have to set up AWS CloudWatch Container Insights for the ECS cluster. However, the docker integration with the additional setup can provide additional information to the AWS CloudWatch Container Insights.
Hence, we will review how you can monitor the various components of an EC2-based ECS cluster:
- EC2 instances in the ASG group
- ECS services running in the ECS cluster
- ECS tasks (containers)
Also, we will review how you can obtain metrics and logs from the ECS cluster with and without AWS Cloudwatch. We’ll show you how to use:
- AWS CloudWatch Container Insights (from Cloudwatch)
- Docker metrics (non-Cloudwatch)
- Amazon ECS logs via Cloudwatch
Prerequisites and configuration
If you plan on following this blog, here are some of the components and details we used to set up the configuration:
- An account on Elastic Cloud and a deployed stack (see instructions here) — ensure that you have both.
- A nginx container and a stress container — we will use these two basic containers to help highlight the load on the Elastic ECS Cluster.
- An ECS EC2 Cluster in an Auto Scaling Group — ensure you have access in order to load up the Elastic agent on the EC2 instances, or you can create an AMI and use that as the baseline image for your ECS cluster.
- An EC2 instance anywhere in your account that is not part of the ECS cluster and has public access (to send metrics and logs)
What will you see in Elastic Observability once it's all set up?
If you utilize the baseline configuration with ECS EC2 cluster configured with AWS CloudWatch Container Insights configured, the Elastic Agent configured with the following Elastic agent integrations:
- ECS integration
- EC2 integration
- AWS Cloudwatch Integration with metrics and logging
Then you will be able to get the following information in Elastic dashboards:
- Containers in the cluster (AWS CloudWatch Container Insights via Elastic Agent and AWS Cloudwatch integration)
- Services in the cluster (AWS CloudWatch Container Insights via Elastic Agent and AWS Cloudwatch integration)
- CPU and memory utilization of the ECS Cluster (Elastic Agent with ECS integration)
- EC2 CPU and memory utilization of the instance in the cluster (Elastic Agent with EC2 integration)
- CPU and memory utilization per container (via AWS CloudWatch Container Insights via Elastic Agent and AWS Cloudwatch integration)
If the additional configuration using Elastic agents with docker integration per ECS EC2 instance is used, you will be able to get a direct feed of metrics via docker. The following metrics can be viewed:
Let’s see how to set this all up.
Setting it all up
Over the next few steps, I’ll walk through:
- Getting an account on Elastic Cloud
- Bringing up an ECS EC2 cluster and potentially setting up your own AMI
- Setting up the containers nginx and a stress container
- Setting up the Elastic agent with docker container integration on the ECS EC2 instances
- Setting up the Elastic agent with AWS, Cloudwatch, and ECS integrations on an independent EC2 instance
Step 1: Create an account on Elastic Cloud
Follow the instructions to get started on Elastic Cloud.
Step 2: Set up an ECS Cluster with EC2 instances
When creating a cluster, you have two options when setting it up using the console:
- Create a new ASG group where you will only be allowed to use the preloaded set of Amazon Linux (2 or 2023) based AMIs
- Set up your own ASG Cluster prior to setting up the ECS Cluster and select this from the options. This option will give you more control over what Linux version and the ability to add things like Elastic agents in the AMI used for the instances in the ASG.
Regardless of either option, you will need to turn on Container Insights (see the bottom part of the image below).
Once the cluster is setup, you can go to AWS Cloudwatch where you should see Container Insights for your cluster:
Step 3: Set up Elastic agent with docker integration
Next, you will need to add an Elastic agent to each one of the instances. In the Elastic cloud, set up an Elastic policy with the docker and system integrations as such:
Next, add an agent for the policy, then copy the appropriate install script (in our case it was Linux since we were running Amazon Linux 2), and run it on every EC2 instance in the cluster:
Once this is added you should see agents in the fleet. Each agent will be on each EC2 instance:
If you decide to set up an ECS EC2 cluster with your own ASG and don’t use Amazon Lunix AMIs (2 or 2023 version), you will have to:
- Pick your base image to base an AMI on
- Add an ECS agent and register each instance to the AMI base image manually
- Add the Elastic agent — standalone version — this step will require you to configure your Elastic endpoint and API key (or simply add the script in the “add agent” part of the configuration above when using the UI)
- Create the AMI once all the above components are added
- Use the newly created AMI in creating the ASG for ECS cluster
Step 4: Set up an Elastic agent with the AWS integration
From the integrations tab in Elastic Cloud, select AWS integration and select add agent. You will then have to walk through the configuration of the AWS integration.
At a minimum, ensure that you have the following configuration options turned on:
This will ensure that not only EC2 metrics and logs are ingested but that all CloudWatch metrics and logs are also ingested. ECS metrics and logs are stored in CloudWatch.
If you want to ensure only logs from the specific ECS cluster are ingested, you can also restrict what to ingest by several parameters. In our setup, we are collecting only logs from Log Group with a prefix of /aws/ecs/containerinsights/EC2BasedCluster/.
Once this policy is set up, add an agent like in Step 1.
However, this agent needs to be added to an EC2 instance which is independent of the ECS cluster.
Once installed, this agent will help pull in:
- All EC2 instance metrics across your account (which can be adjusted in the integration policy)
- Ingest AWS CloudWatch Container Insights from ECS
- ECS metrics such as:
- aws.ecs.metrics.CPUReservation.avg
- aws.ecs.metrics.CPUUtilization.avg
- aws.ecs.metrics.GPUReservation.avg
- aws.ecs.metrics.MemoryReservation.avg
- aws.ecs.metrics.MemoryUtilization.avg
- More - see the full list here
Step 5: Setting up services and containers
In running this configuration, we used nginx and a stress container before we go into the task.
In order to initiate service and containers on ECS, you will need to set up a task for each of these containers. But more importantly, you will need to ensure that the roles for both of the following:
"taskRoleArn": "arn:aws:iam::xxxxx:role/ecsTaskExecutionRol"executionRoleArn":,
"arn:aws:iam::xxxxx:role/ecsTaskExecutionRole",
have the following permissions:
Most importantly, you should ensure that this permission is added:
AmazonEC2ContainerServiceforEC2Role
It will ensure containers can be brought up on the EC2 instances in the cluster.
Once you have the right permissions, then set up the following tasks.
Here is the task JSON for NGINX:
{
"family": "NGINX",
"containerDefinitions": [
{
"name": "nginx",
"image": "nginx: latest",
"cpu": 0,
"portMappings": [
{
"name": "nginx-80-tcp",
"containerPort": 80,
"hostPort": 80,
"protocol": "tcp",
"appProtocol": "http"
}
],
"essential": true,
"environment": [],
"environmentFiles": [],
"mountPoints": [],
"volumesFrom": [],
"ulimits": [],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-create-group": "true",
"awslogs-group": "/ecs/",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "ecs"
},
"secretOptions": []
}
}
],
"taskRoleArn": "arn:aws:iam::xxxxxx:role/ecsTaskExecutionRole",
"executionRoleArn": "arn:aws:iam::xxxxx:role/ecsTaskExecutionRole",
"networkMode": "awsvpc",
"requiresCompatibilities": ["EC2"],
"cpu": "256",
"memory": "512",
"runtimePlatform": {
"cpuArchitecture": "X86_64",
"operatingSystemFamily": "LINUX"
}
}
Here is the task JSON for stress container:
{
"family": "stressLoad",
"containerDefinitions": [
{
"name": "stressLoad",
"image": "containerstack/alpine-stress",
"cpu": 0,
"memory": 512,
"memoryReservation": 512,
"portMappings": [],
"essential": true,
"entryPoint": ["sh", "-c"],
"command": [
"/usr/local/bin/stress --cpu 2 --io 2 --vm 1 --vm-bytes 128M --timeout 6000s"
],
"environment": [],
"mountPoints": [],
"volumesFrom": [],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-create-group": "true",
"awslogs-group": "/ecs/",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "ecs"
}
}
}
],
"taskRoleArn": "arn:aws:iam::xxxxx:role/ecsTaskExecutionRole",
"executionRoleArn": "arn:aws:iam::xxxxx:role/ecsTaskExecutionRole",
"networkMode": "awsvpc",
"requiresCompatibilities": ["EC2"],
"cpu": "256",
"memory": "512",
"runtimePlatform": {
"cpuArchitecture": "X86_64",
"operatingSystemFamily": "LINUX"
}
}
Once you have defined the tasks, ensure you bring up each service (one for each task) with the launch type of EC2:
You should have two services running now.
Step 6: Check on metrics and logs in Elastic Cloud
Go to Elastic Cloud and ensure that you are getting metrics and logs from the ECS Cluster. First, check to see if you are receiving metrics by viewing the built-in dashboard called [Metrics Docker] Overview.
_ With some work on this dashboard by adding in container insight metrics and docker metrics, you should be able to see: _
If you only have the ECS integration and the Elastic agent in Step 2, then you will need to create a new dashboard:
This dashboard can be set up with the following metrics:
- Containers in the cluster (containerInsights via Elastic Agent and AWS Cloudwatch integration). Set up a TSVB panel using the following metric: aws.dimensions.ClusterName : "EC2BasedCluster" with aws.containerinsights.metrics.TaskCount.max
- Services in the cluster (containerInsights via Elastic Agent and AWS Cloudwatch integration). Use the following configuration to setup the chart:
- CPU and memory utilization of the ECS Cluster (Elastic Agent with ECS integration). Use the following configuration to set up both CPU and memory utilization charts:
- EC2 CPU and storage utilization of the instance in the cluster (Elastic Agent with EC2 integration). Use the following configuration to set up both CPU and memory utilization charts:
- (Not shown): CPU and memory utilization per container (via containerInsights via Elastic Agent and AWS Cloudwatch integration)
Step 7: Look at logs from your ECS cluster
Since we set up AWS CloudWatch logs collection in Step 2, we can view these logs in Discover by filtering on the logs group arn /aws/ecs/containerinsights/EC2BasedCluster/.
Summary
I hope you’ve gotten an appreciation for how Elastic Observability can help your AWS monitoring ECS service metrics. Here’s a quick recap of lessons and what you learned:
- Elastic Observability supports ingesting and analysis of AWS ECS service metrics and the corresponding EC2 metrics through the AWS integration on the Elastic Agent. It’s easy to set up ingest from AWS Services via the Elastic Agent.
- Elastic Observability can also get container metrics via the Docker integration running on Elastic agents on each of the EC2 instances in the ECS EC2 auto scaling group.
- Elastic has multiple out-of-the-box (OOTB) AWS service dashboards that can be used as baselines to get your own customized view.
Ready to get started? Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.
The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.