IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Auditd module AWS Fargate module »
Elastic Docs ›Filebeat Reference [7.13] ›Modules

AWS module

edit

AWS module

edit

This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

This is a module for aws logs. It uses filebeat s3 input to get log files from AWS S3 buckets with SQS notification. This module supports reading s3 server access logs with s3access fileset, ELB access logs with elb fileset, VPC flow logs with vpcflow fileset, and CloudTrail logs with cloudtrail fileset.

Access logs contain detailed information about the requests made to these services. VPC flow logs captures information about the IP traffic going to and from network interfaces in AWS VPC. ELB access logs captures detailed information about requests sent to the load balancer. CloudTrail logs contain events that represent actions taken by a user, role or AWS service.

The aws module requires AWS credentials configuration in order to make AWS API calls. Users can either use access_key_id, secret_access_key and/or session_token, or use role_arn AWS IAM role, or use shared AWS credentials file.

Please see AWS credentials options for more details.

Read the quick start to learn how to configure and run modules.

Module configuration

edit

Example config:

- module: aws
  cloudtrail:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.shared_credential_file: /etc/filebeat/aws_credentials
    #var.credential_profile_name: fb-aws
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token
    #var.visibility_timeout: 300s
    #var.api_timeout: 120s
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

  cloudwatch:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.shared_credential_file: /etc/filebeat/aws_credentials
    #var.credential_profile_name: fb-aws
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token
    #var.visibility_timeout: 300s
    #var.api_timeout: 120s
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

  ec2:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.shared_credential_file: /etc/filebeat/aws_credentials
    #var.credential_profile_name: fb-aws
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token
    #var.visibility_timeout: 300s
    #var.api_timeout: 120s
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

  elb:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.shared_credential_file: /etc/filebeat/aws_credentials
    #var.credential_profile_name: fb-aws
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token
    #var.visibility_timeout: 300s
    #var.api_timeout: 120s
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

  s3access:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.shared_credential_file: /etc/filebeat/aws_credentials
    #var.credential_profile_name: fb-aws
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token
    #var.visibility_timeout: 300s
    #var.api_timeout: 120s
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

  vpcflow:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
    #var.shared_credential_file: /etc/filebeat/aws_credentials
    #var.credential_profile_name: fb-aws
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token
    #var.visibility_timeout: 300s
    #var.api_timeout: 120s
    #var.endpoint: amazonaws.com
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb
var.queue_url
(Required) AWS SQS queue url.
var.visibility_timeout
The duration that the received messages are hidden from ReceiveMessage request. Default to be 300 seconds.
var.api_timeout
Maximum duration before AWS API request will be interrupted. Default to be 120 seconds.
var.endpoint
Custom endpoint used to access AWS APIs.
var.shared_credential_file
Filename of AWS credential file.
var.credential_profile_name
AWS credential profile name.
var.access_key_id
First part of access key.
var.secret_access_key
Second part of access key.
var.session_token
Required when using temporary security credentials.
var.role_arn
AWS IAM Role to assume.

cloudtrail fileset

edit

CloudTrail monitors events for the account. If user creates a trail, it delivers those events as log files to a specific Amazon S3 bucket. The cloudtrail fileset does not read the CloudTrail Digest files that are delivered to the S3 bucket when Log File Integrity is turned on, it only reads the CloudTrail logs.

filebeat aws cloudtrail

cloudwatch fileset

edit

Users can use Amazon CloudWatch Logs to monitor, store, and access log files from different sources. Export logs from log groups to an Amazon S3 bucket which has SQS notification setup already. This fileset will parse these logs into timestamp and message field.

ec2 fileset

edit

This fileset is specifically for EC2 logs stored in AWS CloudWatch. Export logs from log groups to Amazon S3 bucket which has SQS notification setup already. With this fileset, EC2 logs will be parsed into fields like ip and program_name. For logs from other services, please use cloudwatch fileset.

elb fileset

edit

Elastic Load Balancing provides access logs that capture detailed information about requests sent to the load balancer. Each log contains information such as the time the request was received, the client’s IP address, latencies, request paths, and server responses. Users can use these access logs to analyze traffic patterns and to troubleshoot issues.

Please follow enable access logs for classic load balancer for sending Classic ELB access logs to S3 bucket. For application load balancer, please follow enable access log for application load balancer. For network load balancer, please follow enable access log for network load balancer.

This fileset comes with a predefined dashboard:

filebeat aws elb overview

s3access fileset

edit

Server access logging provides detailed records for the requests that are made to a bucket. Server access logs are useful for many applications. For example, access log information can be useful in security and access audits. It can also help you learn about customer base and understand Amazon S3 bill.

Please follow how to enable server access logging for sending server access logs to S3 bucket.

This fileset comes with a predefined dashboard:

filebeat aws s3access overview

vpcflow fileset

edit

VPC Flow Logs is a feature in AWS that enables users to capture information about the IP traffic going to and from network interfaces in VPC. Flow log data needs to be published to Amazon S3 in order for vpcflow fileset to retrieve. Flow logs can help users to monitor traffic that is reaching each instance and determine the direction of the traffic to and from the network interfaces.

This fileset comes with a predefined dashboard:

filebeat aws vpcflow overview

AWS Credentials Configuration

edit

To configure AWS credentials, either put the credentials into the Filebeat configuration, or use a shared credentials file, as shown in the following examples.

Configuration parameters

edit
  • access_key_id: first part of access key.
  • secret_access_key: second part of access key.
  • session_token: required when using temporary security credentials.
  • credential_profile_name: profile name in shared credentials file.
  • shared_credential_file: directory of the shared credentials file.
  • role_arn: AWS IAM Role to assume.
  • endpoint: URL of the entry point for an AWS web service. Most AWS services offer a regional endpoint that can be used to make requests. The general syntax of a regional endpoint is protocol://service-code.region-code.endpoint-code. Some services, such as IAM, do not support regions. The endpoints for these services do not include a region. In aws module, endpoint config is to set the endpoint-code part, such as amazonaws.com, amazonaws.com.cn, c2s.ic.gov, sc2s.sgov.gov.

Supported Formats

edit
  • Use access_key_id, secret_access_key and/or session_token

Users can either put the credentials into metricbeat module configuration or use environment variable AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and/or AWS_SESSION_TOKEN instead.

If running on Docker, these environment variables should be added as a part of the docker command. For example, with Metricbeat:

$ docker run -e AWS_ACCESS_KEY_ID=abcd -e AWS_SECRET_ACCESS_KEY=abcd -d --name=metricbeat --user=root --volume="$(pwd)/metricbeat.aws.yml:/usr/share/metricbeat/metricbeat.yml:ro" docker.elastic.co/beats/metricbeat:7.11.1 metricbeat -e -E cloud.auth=elastic:1234 -E cloud.id=test-aws:1234

Sample metricbeat.aws.yml looks like:

metricbeat.modules:
- module: aws
  period: 5m
  access_key_id: ${AWS_ACCESS_KEY_ID}
  secret_access_key: ${AWS_SECRET_ACCESS_KEY}
  session_token: ${AWS_SESSION_TOKEN}
  metricsets:
    - ec2

Environment variables can also be added through a file. For example:

$ cat env.list
AWS_ACCESS_KEY_ID=abcd
AWS_SECRET_ACCESS_KEY=abcd

$ docker run --env-file env.list -d --name=metricbeat --user=root --volume="$(pwd)/metricbeat.aws.yml:/usr/share/metricbeat/metricbeat.yml:ro" docker.elastic.co/beats/metricbeat:7.11.1 metricbeat -e -E cloud.auth=elastic:1234 -E cloud.id=test-aws:1234
  • Use credential_profile_name and/or shared_credential_file

If access_key_id, secret_access_key and role_arn are all not given, then filebeat will check for credential_profile_name. If you use different credentials for different tools or applications, you can use profiles to configure multiple access keys in the same configuration file. If there is no credential_profile_name given, the default profile will be used.

shared_credential_file is optional to specify the directory of your shared credentials file. If it’s empty, the default directory will be used. In Windows, shared credentials file is at C:\Users\<yourUserName>\.aws\credentials. For Linux, macOS or Unix, the file is located at ~/.aws/credentials. When running as a service, the home path depends on the user that manages the service, so the shared_credential_file parameter can be used to avoid ambiguity. Please see Create Shared Credentials File for more details.

  • Use role_arn

role_arn is used to specify which AWS IAM role to assume for generating temporary credentials. If role_arn is given, filebeat will check if access keys are given. If not, filebeat will check for credential profile name. If neither is given, default credential profile will be used. Please make sure credentials are given under either a credential profile or access keys.

If running on Docker, the credential file needs to be provided via a volume mount. For example, with Metricbeat:

docker run -d --name=metricbeat --user=root --volume="$(pwd)/metricbeat.aws.yml:/usr/share/metricbeat/metricbeat.yml:ro" --volume="/Users/foo/.aws/credentials:/usr/share/metricbeat/credentials:ro" docker.elastic.co/beats/metricbeat:7.11.1 metricbeat -e -E cloud.auth=elastic:1234 -E cloud.id=test-aws:1234

Sample metricbeat.aws.yml looks like:

metricbeat.modules:
- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  shared_credential_file: /usr/share/metricbeat/credentials
  metricsets:
    - ec2

  • Use AWS credentials in Filebeat configuration

    filebeat.inputs:
- type: aws-s3
  queue_url: https://sqs.us-east-1.amazonaws.com/123/test-queue
  access_key_id: '<access_key_id>'
  secret_access_key: '<secret_access_key>'
  session_token: '<session_token>'

    or

    filebeat.inputs:
- type: aws-s3
  queue_url: https://sqs.us-east-1.amazonaws.com/123/test-queue
  access_key_id: '${AWS_ACCESS_KEY_ID:""}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
  session_token: '${AWS_SESSION_TOKEN:""}'

  • Use IAM role ARN

    filebeat.inputs:
- type: aws-s3
  queue_url: https://sqs.us-east-1.amazonaws.com/123/test-queue
  role_arn: arn:aws:iam::123456789012:role/test-mb

  • Use shared AWS credentials file

    filebeat.inputs:
- type: aws-s3
  queue_url: https://sqs.us-east-1.amazonaws.com/123/test-queue
  credential_profile_name: test-fb

AWS Credentials Types

edit

There are two different types of AWS credentials can be used: access keys and temporary security credentials.

  • Access keys

AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are the two parts of access keys. They are long-term credentials for an IAM user or the AWS account root user. Please see AWS Access Keys and Secret Access Keys for more details.

  • IAM role ARN

An IAM role is an IAM identity that you can create in your account that has specific permissions that determine what the identity can and cannot do in AWS. A role does not have standard long-term credentials such as a password or access keys associated with it. Instead, when you assume a role, it provides you with temporary security credentials for your role session. IAM role Amazon Resource Name (ARN) can be used to specify which AWS IAM role to assume to generate temporary credentials. Please see AssumeRole API documentation for more details.

Here are the steps to set up IAM role using AWS CLI for Metricbeat. Please replace 123456789012 with your own account ID.

Step 1. Create example-policy.json file to include all permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "sqs:ReceiveMessage"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "sqs:ChangeMessageVisibility",
            "Resource": "arn:aws:sqs:us-east-1:123456789012:test-fb-ks"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "sqs:DeleteMessage",
            "Resource": "arn:aws:sqs:us-east-1:123456789012:test-fb-ks"
        },
        {
            "Sid": "VisualEditor3",
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole",
                "sqs:ListQueues",
                "tag:GetResources",
                "ec2:DescribeInstances",
                "cloudwatch:GetMetricData",
                "ec2:DescribeRegions",
                "iam:ListAccountAliases",
                "sts:GetCallerIdentity",
                "cloudwatch:ListMetrics"
            ],
            "Resource": "*"
        }
    ]
}

Step 2. Create IAM policy using the aws iam create-policy command:

$ aws iam create-policy --policy-name example-policy --policy-document file://example-policy.json

Step 3. Create the JSON file example-role-trust-policy.json that defines the trust relationship of the IAM role

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Principal": { "AWS": "arn:aws:iam::123456789012:root" },
        "Action": "sts:AssumeRole"
    }
}

Step 4. Create the IAM role and attach the policy:

$ aws iam create-role --role-name example-role --assume-role-policy-document file://example-role-trust-policy.json
$ aws iam attach-role-policy --role-name example-role --policy-arn "arn:aws:iam::123456789012:policy/example-policy"

After these steps are done, IAM role ARN can be used for authentication in Metricbeat aws module.

  • Temporary security credentials

Temporary security credentials has a limited lifetime and consists of an access key ID, a secret access key, and a security token which typically returned from GetSessionToken. MFA-enabled IAM users would need to submit an MFA code while calling GetSessionToken. default_region identifies the AWS Region whose servers you want to send your first API request to by default. This is typically the Region closest to you, but it can be any Region. Please see Temporary Security Credentials for more details. sts get-session-token AWS CLI can be used to generate temporary credentials. For example. with MFA-enabled:

aws> sts get-session-token --serial-number arn:aws:iam::1234:mfa/[email protected] --token-code 456789 --duration-seconds 129600

Because temporary security credentials are short term, after they expire, the user needs to generate new ones and modify the aws.yml config file with the new credentials. Unless live reloading feature is enabled for Metricbeat, the user needs to manually restart Metricbeat after updating the config file in order to continue collecting Cloudwatch metrics. This will cause data loss if the config file is not updated with new credentials before the old ones expire. For Metricbeat, we recommend users to use access keys in config file to enable aws module making AWS api calls without have to generate new temporary credentials and update the config frequently.

IAM policy is an entity that defines permissions to an object within your AWS environment. Specific permissions needs to be added into the IAM user’s policy to authorize Metricbeat to collect AWS monitoring metrics. Please see documentation under each metricset for required permissions.

Fields

edit

For a description of each field in the module, see the exported fields section.

« Auditd module AWS Fargate module »