aws cloudwatch metricset

edit

The cloudwatch metricset of aws module allows you to monitor various services on AWS. cloudwatch metricset fetches metrics from given namespace periodically by calling GetMetricData api.

AWS Permissions

edit

Some specific AWS permissions are required for IAM user to collect AWS Cloudwatch metrics.

ec2:DescribeRegions
cloudwatch:GetMetricData
cloudwatch:ListMetrics
tag:getResources
sts:GetCallerIdentity
iam:ListAccountAliases

Metricset-specific configuration notes

edit
  • namespace: The namespace used by ListMetrics API to filter against. For example, AWS/EC2, AWS/S3. If wildcard * is given for namespace, metrics from all namespaces will be collected automatically.
  • name: The name of the metric to filter against. For example, CPUUtilization for EC2 instance.
  • dimensions: The dimensions to filter against. For example, InstanceId=i-123.
  • tags.resource_type_filter: The constraints on the resources that you want returned. The format of each resource type is service[:resourceType]. For example, specifying a resource type of ec2 returns all Amazon EC2 resources (which includes EC2 instances). Specifying a resource type of ec2:instance returns only EC2 instances.
  • statistic: Statistics are metric data aggregations over specified periods of time. By default, statistic includes Average, Sum, Count, Maximum and Minimum.

Configuration examples

edit

To be more focused on cloudwatch metricset use cases, the examples below do not include configurations on AWS credentials. Please see AWS credentials options for more details on setting AWS credentials in configurations in order for this metricset to make proper AWS API calls.

Example 1

edit
- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/EBS 
    - namespace: AWS/ELB 
      tags.resource_type_filter: elasticloadbalancing
    - namespace: AWS/EC2 
      name: CPUUtilization
      statistic: ["Average"]
      dimensions:
        - name: InstanceId
          value: i-0686946e22cf9494a

Users can configure the cloudwatch metricset to collect all metrics from one specific namespace, such as AWS/EBS.

cloudwatch metricset also has the ability to collect tags from AWS resources. If user specify tags.resource_type_filter, then tags will be collected and stored as a part of the event. Please see AWS API GetResources for more details about tags.resource_type_filter.

If users knows exactly what are the cloudwatch metrics they want to collect, this configuration format can be used. namespace and metricname need to be specified and dimensions can be used to filter cloudwatch metrics. Please see AWS List Metrics for more details.

Example 2

edit
- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  metrics:
    - namespace: "*"

With this config, metrics from all namespaces will be collected from Cloudwatch. The limitation here is the collection period for all namespaces are all set to be the same, which in this case is 300 second. This will cause extra costs for API calls or data loss. For example, metrics from namespace AWS/Usage are sent to Cloudwatch every 1 minute. With the collection period equals to 300 seconds, data points in between will get lost. Metrics from namespace AWS/Billing are sent to Cloudwatch every several hours. By querying from AWS/Billing namespace every 300 seconds, additional costs will occur.

More examples

edit

With the configuration below, users will be able to collect cloudwatch metrics from EBS, ELB and EC2 without tag information.

- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/EBS
    - namespace: AWS/ELB
    - namespace: AWS/EC2

With the configuration below, users will be able to collect cloudwatch metrics from EBS, ELB and EC2 with tags from these services.

- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/EBS
      tags.resource_type_filter: ebs
    - namespace: AWS/ELB
      tags.resource_type_filter: elasticloadbalancing
    - namespace: AWS/EC2
      tags.resource_type_filter: ec2:instance

With the configuration below, users will be able to collect specific cloudwatch metrics. For example CPUUtilization metric(average) from EC2 instance i-123 and NetworkIn metric(average) from EC2 instance i-456.

- module: aws
  period: 300s
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/EC2
      name: ["CPUUtilization"]
      tags.resource_type_filter: ec2:instance
      dimensions:
        - name: InstanceId
          value: i-123
      statistic: ["Average"]
    - namespace: AWS/EC2
      name: ["NetworkIn"]
      dimensions:
        - name: InstanceId
          value: i-456
      statistic: ["Average"]

This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.

Fields

edit

For a description of each field in the metricset, see the exported fields section.

Here is an example document generated by this metricset:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "aws": {
        "cloudwatch": {
            "namespace": "AWS/RDS"
        },
        "dimensions": {
            "EngineName": "mariadb"
        },
        "rds": {
            "metrics": {
                "BinLogDiskUsage": {
                    "avg": 2803,
                    "count": 5,
                    "max": 3007,
                    "min": 2752,
                    "sum": 14015
                },
                "CPUCreditBalance": {
                    "avg": 144,
                    "count": 1,
                    "max": 144,
                    "min": 144,
                    "sum": 144
                },
                "CPUCreditUsage": {
                    "avg": 0.062006,
                    "count": 1,
                    "max": 0.062006,
                    "min": 0.062006,
                    "sum": 0.062006
                },
                "CPUSurplusCreditBalance": {
                    "avg": 0,
                    "count": 1,
                    "max": 0,
                    "min": 0,
                    "sum": 0
                },
                "CPUSurplusCreditsCharged": {
                    "avg": 0,
                    "count": 1,
                    "max": 0,
                    "min": 0,
                    "sum": 0
                },
                "CPUUtilization": {
                    "avg": 1.2662313605690698,
                    "count": 5,
                    "max": 1.66666666666667,
                    "min": 1.01694915255224,
                    "sum": 6.331156802845349
                },
                "DatabaseConnections": {
                    "avg": 0,
                    "count": 5,
                    "max": 0,
                    "min": 0,
                    "sum": 0
                },
                "DiskQueueDepth": {
                    "avg": 0.00009332666777759262,
                    "count": 5,
                    "max": 0.0002666666666666667,
                    "min": 0,
                    "sum": 0.0004666333388879631
                },
                "FreeStorageSpace": {
                    "avg": 20402470912,
                    "count": 5,
                    "max": 20402470912,
                    "min": 20402470912,
                    "sum": 102012354560
                },
                "FreeableMemory": {
                    "avg": 446488576,
                    "count": 5,
                    "max": 446668800,
                    "min": 446275584,
                    "sum": 2232442880
                },
                "NetworkReceiveThroughput": {
                    "avg": 407.83898884048796,
                    "count": 5,
                    "max": 499.9416676388727,
                    "min": 354.27742870952153,
                    "sum": 2039.1949442024397
                },
                "NetworkTransmitThroughput": {
                    "avg": 2667.0383202542657,
                    "count": 5,
                    "max": 3228.9489909846857,
                    "min": 2274.512091465142,
                    "sum": 13335.191601271328
                },
                "ReadIOPS": {
                    "avg": 0.2333294445092582,
                    "count": 5,
                    "max": 1.166647222546291,
                    "min": 0,
                    "sum": 1.166647222546291
                },
                "ReadLatency": {
                    "avg": 0,
                    "count": 5,
                    "max": 0,
                    "min": 0,
                    "sum": 0
                },
                "ReadThroughput": {
                    "avg": 136.53333333333333,
                    "count": 5,
                    "max": 682.6666666666666,
                    "min": 0,
                    "sum": 682.6666666666666
                },
                "SwapUsage": {
                    "avg": 5287936,
                    "count": 5,
                    "max": 5287936,
                    "min": 5287936,
                    "sum": 26439680
                },
                "WriteIOPS": {
                    "avg": 0.27999083424342597,
                    "count": 5,
                    "max": 0.6999883335277746,
                    "min": 0,
                    "sum": 1.39995417121713
                },
                "WriteLatency": {
                    "avg": 0.00009062937062937063,
                    "count": 5,
                    "max": 0.0003076923076923077,
                    "min": 0,
                    "sum": 0.00045314685314685316
                },
                "WriteThroughput": {
                    "avg": 2621.2741374938682,
                    "count": 5,
                    "max": 7441.066666666667,
                    "min": 0,
                    "sum": 13106.370687469342
                }
            }
        }
    },
    "cloud": {
        "account": {
            "id": "627959692251",
            "name": "elastic-test"
        },
        "provider": "aws",
        "region": "ap-southeast-1"
    },
    "event": {
        "dataset": "aws.cloudwatch",
        "duration": 115000,
        "module": "aws"
    },
    "metricset": {
        "name": "cloudwatch",
        "period": 10000
    },
    "service": {
        "type": "aws"
    }
}