Metricset Details

This topic provides additional details about creating metricsets.

Each metricset can have its own configuration variables defined. To make use of these variables, you must extend the New method. For example, let’s assume that you want to add a password config option to the metricset. You would extend beat.yml in the following way:

metricbeat.modules:
- module: {module}
  metricsets: ["{metricset}"]
  password: "test1234"

To read in the new password config option, you need to modify the New method. First you define a config struct that contains the value types to be read. You can set default values, as needed. Then you pass the config to the UnpackConfig method for loading the configuration.

Your implementation should look something like this:

type MetricSet struct {
	mb.BaseMetricSet
	password string
}

func New(base mb.BaseMetricSet) (mb.MetricSet, error) {

	// Unpack additional configuration options.
	config := struct {
		Password string `config:"password"`
	}{
		Password: "",
	}
	err := base.Module().UnpackConfig(&config)
	if err != nil {
		return nil, err
	}

	return &MetricSet{
		BaseMetricSet: base,
		password:      config.Password,
	}, nil
}

Each time the Fetch method is called, it makes a request to the service, so it’s important to handle the connections correctly. We recommended that you set up the connections in the New method and persist them in the MetricSet object. This allows connections to be reused.

One very important point is that connections must respect the timeout variable: base.Module().Config().Timeout. If the timeout elapses before the request completes, the request must be ended and an error must be returned to make sure the next request can be started on time. By default the Timeout is set to Period, so one request gets ended before a new request is made.

If a request must be ended or has an error, make sure that you return a useful error message. This error message is also sent to Elasticsearch, making it possible to not only fetch metrics from the service, but also report potential problems or errors with the metricset.

If the data transformation that has to happen in the Fetch method is extensive, we recommend that you create a second file called data.go in the same package as the metricset. The data.go file should contain a function called eventMapping(...). A separate file is not required, but is currently a best practice because it isolates the functionality of the metricset and Fetch method from the data mapping.

The fields.yml file is used for different purposes:

To make sure the Elasticsearch template is correct, it’s important to keep this file up-to-date with all the changes. There is a fields.yml file under module/{module}/_meta/fields.yml that contains the general top level structure for all metricsets. Normally you only need to modify the description in this file.

Here an example for the fields.yml file from the MySQL module.

- key: mysql
  title: "MySQL"
  description: >
    MySQL server status metrics collected from MySQL.
  short_config: false
  release: ga
  fields:
    - name: mysql
      type: group
      description: >
        `mysql` contains the metrics that were obtained from MySQL
        query.
      fields:

There is another fields.yml file under module/{module}/{metricset}/_meta/fields.yml that contains all fields retrieved by the metricset. As field types, each field must have a core data type supported by elasticsearch. Here’s a very basic example that shows one group from the MySQL status metricset:

- name: status
  type: group
  description: >
    `status` contains the metrics that were obtained by the status SQL query.
  fields:
    - name: aborted
      type: group
      description: >
        Aborted status fields.
      fields:
        - name: clients
          type: integer
          description: >
            The number of connections that were aborted because the client died without closing the connection properly.

        - name: connects
          type: integer
          description: >
            The number of failed attempts to connect to the MySQL server.

As you can see, if there are nested fields, you must use the type group.

It’s important to also add tests for your metricset. There are three different types of tests that you need for testing a Beat:

We recommend that you use all three when you create a metricset. Unit tests are written in Go and have no dependencies. Integration tests are also written in Go but require the service from which the module collects metrics to also be running. System tests for Metricbeat also require the service to be running in most cases and are written in Python based on our small Python test framework. We use venv to deal with Python dependencies. You can simply run the command make python-env and then . build/python-env/bin/activate .

You should use a combination of the three test types to test your metricsets because each method has advantages and disadvantages. To get started with your own tests, it’s best to look at the existing tests. You’ll find the unit and integration tests in the _test.go files under existing modules and metricsets. Integration tests usually take the form of TestFetch and TestData. The system tests are under tests/systems.

Integration and system tests need an environment that’s running the service. You can create this environment by using Docker and a docker-compose file. If you add a module that requires a service, you must add the service to the virtual environment. To do this, you:

The docker-compose.yml file is at the root of Metricbeat. Most services have existing Docker modules and can be added as simply as Redis:

redis:
  image: redis:3.2.3

To allow the Beat to access your service, make sure that you define the environment variables in the docker-compose file and add the link to the container:

beat:
  links:
    - redis
  environment:
    - REDIS_HOST=redis
    - REDIS_PORT=6379

To make sure the service is running before the tests are started, modify the docker-entrypoint.sh script to add a check that verifies your service is running. For example, the check for Redis looks like this:

waitFor ${REDIS_HOST} ${REDIS_PORT} Redis

The environment expects your service to be available as soon as it receives a response from the given address and port.

There are normally two integration tests that are part of every metricset: TestFetch and TestData. Both tests will start up a new instance of your metricset and fetch an event. In order to start a metricset, you need to create a configuration object:

func getConfig() map[string]interface{} {
    return map[string]interface{}{
    "module":           "{module}",
    "metricsets":       []string{"{metricset}"},
    "hosts":      []string{GetEnvHost() + ":" + GetEnvPort()}, 
  }
}

func GetEnvHost() string { 
    host := os.Getenv("{module}_HOST")
    if len(host) == 0 {
    host = "127.0.0.1"
  }
  return host
}

func GetEnvPort() string { 
    port := os.Getenv("{module}_PORT")

    if len(port) == 0 {
      port = "1234"
    }
  return port
}

The TestFetch integration test will return a single event from your metricset, which you can use to test the validity of the data. TestData will (re)generate the _meta/data.json file that documents the data reported by the metricset.

import (
	"os"
	"testing"

	"github.com/stretchr/testify/assert"

	"github.com/elastic/beats/libbeat/tests/compose"
	mbtest "github.com/elastic/beats/metricbeat/mb/testing"
)

func TestFetch(t *testing.T) {
	compose.EnsureUp(t, "{module}") 

	f := mbtest.NewReportingMetricSetV2Error(t, getConfig())

	events, errs := mbtest.ReportingFetchV2Error(f)
	if len(errs) > 0 {
		t.Fatalf("Expected 0 errord, had %d. %v\n", len(errs), errs)
	}

	assert.NotEmpty(t, events) 

}

func TestData(t *testing.T) {

	f := mbtest.NewReportingMetricSetV2Error(t, getConfig())

	err := mbtest.WriteEventsReporterV2Error(f, t, "") 
	if !assert.NoError(t, err) {
		t.FailNow()
	}
}

To run all the tests, run make testsuite. To only run unit tests, run make unit-tests or for integration tests make integration-tests-environment. Be aware that a running Docker environment is needed for integration and system tests.

To run TestData and generate the data.json file, run go test -tags=integration -data -run TestData in the directory where your test is located.

Sometimes you may want to run a single integration test, for example, to test a module such as the apache module. To do this, you can:

Each module must be documented. The documentation is based on asciidoc and is in the file module/{module}/_meta/docs.asciidoc for the module and in module/{module}/{metricset}/_meta/docs.asciidoc for the metricset. Basic documentation with the config file and an example output is automatically generated. Use these files to document specific configuration options or usage examples.