Monitor Microsoft Azure

Go to the Azure Management Portal. Search and click on Azure Active Directory.

Click on App registrations in the navigation pane of the selected Active Directory and then click on New registration.

Type the name of your application (this tutorial uses monitor-azure) and click on Register (leave all the other options with the default value).

Copy the Application (client) ID, and save it for future reference. This id is required to configure Metricbeat to connect to your Azure account.

Click on Certificates & secrets. Then, click on New client secret to create a new security key.

Type a key description and select a key duration in the expire list. Click on Add to create a client secret. The next page will display the key value under the Value field. Copy the secret and save it (along with your Client ID) for future reference.

On Azure Portal, search and click on Subscriptions.

In the Select field, type the description name of the configured service principal (monitor-azure).

Use the Metricbeat keystore to store secure settings. Store the Cloud ID in the keystore.

./metricbeat keystore create
echo -n "<Your Deployment Cloud ID>" | ./metricbeat keystore add CLOUD_ID --stdin

To store metrics in Elasticsearch with minimal permissions, create an API key to send data from Metricbeat to Elasticsearch Service. Log into Kibana (you can do so from the Cloud Console without typing in any permissions) and select Management → Dev Tools. Send the following request:

POST /_security/api_key
{
  "name": "metricbeat-monitor",
  "role_descriptors": {
    "metricbeat_writer": {
      "cluster": ["monitor", "read_ilm"],
      "index": [
        {
          "names": ["metricbeat-*"],
          "privileges": ["view_index_metadata", "create_doc"]
        }
      ]
    }
  }
}

The response contains an api_key and an id field, which can be stored in the Metricbeat keystore in the following format: id:api_key.

echo -n "IhrJJHMB4JmIUAPLuM35:1GbfxhkMT8COBB4JWY3pvQ" | ./metricbeat keystore add ES_API_KEY --stdin

To see if both settings have been stored, run the following command:

./metricbeat keystore list

To configure Metricbeat to output to Elasticsearch Service, edit the metricbeat.yml configuration file. Add the following lines to the end of the file.

cloud.id: ${CLOUD_ID}
output.elasticsearch:
  api_key: ${ES_API_KEY}

Finally, test if the configuration is working. If it is not working, verify if you used the right credentials and add them again.

./metricbeat test output

The azure module configuration needs three ids and one secret. Use the commands below to store each one of them in the keystore.

echo -n "<client_id>" | ./metricbeat keystore add AZURE_CLIENT_ID --stdin
echo -n "<client_secret>" | ./metricbeat keystore add AZURE_CLIENT_SECRET --stdin
echo -n "<tenant_id>" | ./metricbeat keystore add AZURE_TENANT_ID --stdin
echo -n "<subscription_id>" | ./metricbeat keystore add AZURE_SUBSCRIPTION_ID --stdin

Enable the Azure module.

./metricbeat modules enable azure

Edit the modules.d/azure.yml file to collect compute_vms metrics.

- module: azure
  metricsets:
  - compute_vm  
  enabled: true
  period: 300s  
  client_id: '${AZURE_CLIENT_ID:""}'  
  client_secret: '${AZURE_CLIENT_SECRET:""}'  
  tenant_id: '${AZURE_TENANT_ID:""}'  
  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'  
  refresh_list_interval: 600s

To check if Metricbeat can collect data, test the input by running the following command:

./metricbeat test modules azure

Metricbeat will print compute_vms metrics to the terminal, if the setup is correct.

Edit the modules.d/azure.yml file to also collect billing metrics.

- module: azure
  metricsets:
  - billing  
  enabled: true
  period: 24h  
  client_id: '${AZURE_CLIENT_ID:""}'
  client_secret: '${AZURE_CLIENT_SECRET:""}'
  tenant_id: '${AZURE_TENANT_ID:""}'
  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
  refresh_list_interval: 600s

When the input and output are ready, start Metricbeat to collect the data.

./metricbeat -e

Finally, log into Kibana and open the [Metricbeat Azure] Compute VMs Overview dashboard.

You can also check the [Metricbeat Azure] Billing overview dashboard, even though it might take longer to collect data.

Use the Filebeat keystore to store secure settings. Store the Cloud ID in the keystore.

./filebeat keystore create
echo -n "<Your Deployment Cloud ID>" | ./filebeat keystore add CLOUD_ID --stdin

To store logs in Elasticsearch with minimal permissions, create an API key to send data from Filebeat to Elasticsearch Service. Log into Kibana (you can do so from the Cloud Console without typing in any permissions) and select Management → Dev Tools. Send the following request:

POST /_security/api_key
{
  "name": "filebeat-monitor-gcp",
  "role_descriptors": {
    "filebeat_writer": {
      "cluster": [
        "monitor",
        "read_ilm",
        "cluster:admin/ingest/pipeline/get", 
        "cluster:admin/ingest/pipeline/put" 
      ],
      "index": [
        {
          "names": ["filebeat-*"],
          "privileges": ["view_index_metadata", "create_doc"]
        }
      ]
    }
  }
}

The response contains an api_key and an id field, which can be stored in the Filebeat keystore in the following format: id:api_key.

echo -n "IhrJJHMB4JmIUAPLuM35:1GbfxhkMT8COBB4JWY3pvQ" | ./filebeat keystore add ES_API_KEY --stdin

To see if both settings have been stored, run the following command:

./filebeat keystore list

To configure Filebeat to output to Elasticsearch Service, edit the filebeat.yml configuration file. Add the following lines to the end of the file.

cloud.id: ${CLOUD_ID}
output.elasticsearch:
  api_key: ${ES_API_KEY}

Finally, test if the configuration is working. If it is not working, verify that you used the right credentials and, if necessary, add them again.

./filebeat test output

Open the Azure PowerShell:

Run the following command to create an Event Hub namespace:

New-AzEventHubNamespace -ResourceGroupName monitor-azure-resource-group `
  -NamespaceName monitor-azure-namespace `
  -Location northeurope

Create an event hub:

New-AzEventHub -ResourceGroupName monitor-azure-resource-group `
  -NamespaceName monitor-azure-namespace `
  -EventHubName monitor-azure-event-hub `
  -MessageRetentionInDays 3 `

Navigate to Activity log.

Click on Diagnostics settings.

Configure it and click on Save.

Navigate to Event Hubs.

Copy the Connection string–primary key value and add it to the keystore.

echo -n "<Your Connection string-primary key>" | ./filebeat keystore add AZURE_CONNECTION_STRING --stdin

Create a storage account. (You can also use an existing one.)

New-AzStorageAccount -ResourceGroupName monitor-azure-resource-group `
  -Name monitorazurestorage `
  -Location northeurope `
  -SkuName Standard_ZRS `
  -Kind StorageV2

Navigate to Storage accounts.

Click on the created storage account and then on Access keys.

Click on Show keys and copy one of the keys.

Add the storage_account_key to the keystore.

echo -n "<Your Storage account key>" | ./filebeat keystore add AZURE_STORAGE_KEY --stdin

Enable the Filebeat Azure module.

./filebeat modules enable azure

Edit the modules.d/azure.yml file with the following configurations.

- module: azure
  # All logs
  activitylogs:
    enabled: true
    var:
      eventhub: "monitor-azure-event-hub"  
      consumer_group: "$Default"  
      connection_string: "${AZURE_CONNECTION_STRING}"  
      storage_account: "monitorazurestorage"  
      storage_account_key: "${AZURE_STORAGE_KEY}"  

Start Filebeat to collect the logs.

./filebeat -e

Finally, log into Kibana and open the [Filebeat Azure] Cloud Overview dashboard.