Google Cloud Platform module

This module periodically fetches monitoring metrics from Google Cloud Platform using Stackdriver Monitoring API for Google Cloud Platform services. Note: extra GCP charges on Stackdriver Monitoring API requests will be generated by this module.

This is a list of the possible module parameters you can tune:

Authentication and authorization in Google Cloud Platform can be achieved in many ways. For the current version of the Google Cloud Platform module for Metricbeat, the only supported method is using Service Account JSON files. A typical JSON with a private key looks like this:

{
  "type": "service_account",
  "project_id": "your-project-id",
  "private_key_id": "a_private_key_id",
  "private_key": "-----BEGIN PRIVATE KEY-----your private key\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "123456",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/metricbeat-testing%40your-project-id.iam.gserviceaccount.com"
}

Generally, you have to create a Service Account and assign it the following roles or the permissions described on each role (applies to all metricsets):

You can play in IAM pretty much with your service accounts and Instance level access to your resources (for example, allowing that everything running in an Instance is authorized to use the Compute API). The module uses Google Cloud Platform libraries for authentication so many possibilities are open but the Module is only supported by using the method mentioned above.

Google Cloud Platform offers the Stackdriver Monitoring API to fetch metrics from its services. Those metrics are retrieved one by one.

If you also want to extract service labels (by setting exclude_labels to false, which is the default state). You also make a new API check on the corresponding service. Service labels requires a new API call to extract those metrics. In the worst case the number of API calls will be doubled. In the best case, all metrics come from the same GCP entity and 100% of the required information is included in the first API call (which is cached for subsequent calls).

A recommended period value between fetches is between 5 and 10 minutes, depending on how granular you want your metrics. GCP restricts information for less than 5 minutes.

Google Cloud Platform pricing depends of the number of requests you do to their API’s. Here you have some information that you can use to make an estimation of the pricing you should expect. For example, imagine that you have a Compute Metricset activated and you don’t want to exclude labels. You have a total of 20 instances running in a particular GCP project, region and zone.

For example, if Compute Metricset fetches 14 metrics (which is the number of metrics fetched in the early beta version). Each of those metrics will attempt an API call to Compute API to retrieve also their metadata. Because you have 20 different instances, the total number of API calls that will be done on each refresh period are: 14 metrics + 20 instances = 34 API requests every 5 minutes if that is your current Period. 9792 API requests per day with one zone. If you add 2 zones more with the same amount of instances you’ll have 19584 API requests per day (9792 on each zone) or around 587520 per month for the Compute Metricset. This maths must be done for each different Metricset with slight variations.

Currently, we have compute, loadbalancing, and pubsub metricset in googlecloud module.

Compute Metricset to fetch metrics from Compute Engine Virtual Machines in Google Cloud Platform. The compute metricset contains some of the metrics exported from the Stackdriver API. Extra labels and metadata are also extracted using the Compute API. This is enough to get most of the info associated with a metric like compute labels and metadata and metric specific Labels.

The compute metricset comes with a predefined dashboard:

The Google Cloud Platform module supports the standard configuration options that are described in Modules. Here is an example configuration:

metricbeat.modules:
- module: googlecloud
  metricsets:
    - compute
    - pubsub
    - loadbalancing
  zone: "us-central1-a"
  region: "us-central1"
  project_id: "your project id"
  credentials_file_path: "your JSON credentials file path"
  exclude_labels: false
  period: 300s

- module: googlecloud
  metricsets:
    - storage
  region: "us-central1"
  project_id: "your project id"
  credentials_file_path: "your JSON credentials file path"
  exclude_labels: false
  period: 300s

The following metricsets are available: