HTTP JSON input

The httpjson input keeps a runtime state between requests. This state can be accessed by some configuration options and transforms.

The state has the following elements:

All of the mentioned objects are only stored at runtime during the execution of the periodic request, except cursor, which has values that are persisted between periodic request and restarts.

A transform is an action that lets the user modify the input state. Depending on where the transform is defined, it will have access for reading or writing different elements of the state.

The access limitations are described in the corresponding configuration sections.

Appends a value to an array. If the field does not exist, the first entry will create a new array. If the field exists, the value is appended to the existing field and converted to a list.

- append:
    target: body.foo.bar
    value: '[[.cursor.baz]]'
    default: "a default value"

Deletes the target field.

- delete:
    target: body.foo.bar

Sets a value.

- set:
    target: body.foo.bar
    value: '[[.cursor.baz]]'
    default: "a default value"

Some configuration options and transforms can use value templates. Value templates are Go templates with access to the input state and to some built-in functions. Please note that delimiters are changed from the default {{ }} to [[ ]] to improve interoperability with other templating mechanisms.

To see which state elements and operations are available, see the documentation for the option or transform where you want to use a value template.

A value template looks like:

- set:
    target: body.foo.bar
    value: '[[.cursor.baz]] more data'
    default: "a default value"

The content inside the brackets [[ ]] is evaluated. For more information on Go templates please refer to the Go docs.

Some built-in helper functions are provided to work with the input state inside value templates:

In addition to the provided functions, any of the native functions for time.Time, http.Header, and url.Values types can be used on the corresponding objects. Examples: [[(now).Day]], [[.last_response.header.Get "key"]]

The httpjson input supports the following configuration options plus the Common options described later.

Duration between repeated requests. It may make additional pagination requests in response to the initial request if pagination is enabled. Default: 60s.

When set to false, disables the basic auth configuration. Default: true.

The user to authenticate with.

The password to use.

When set to false, disables the oauth2 configuration. Default: true.

Used to configure supported oauth2 providers. Each supported provider will require specific settings. It is not set by default. Supported providers are: azure, google, okta.

The client ID used as part of the authentication flow. It is always required except if using google as provider. Required for providers: default, azure, okta.

The client secret used as part of the authentication flow. It is always required except if using google or okta as provider. Required for providers: default, azure.

The user used as part of the authentication flow. It is required for authentication - grant type password. It is only available for provider default.

The password used as part of the authentication flow. It is required for authentication - grant type password. It is only available for provider default.

A list of scopes that will be requested during the oauth2 flow. It is optional for all providers.

The endpoint that will be used to generate the tokens during the oauth2 flow. It is required if no provider is specified.

Set of values that will be sent on each request to the token_url. Each param key can have multiple values. Can be set for all providers except google.

- type: httpjson
  auth.oauth2:
    endpoint_params:
      Param1:
        - ValueA
        - ValueB
      Param2:
        - Value

Used for authentication when using azure provider. Since it is used in the process to generate the token_url, it can’t be used in combination with it. It is not required.

For information about where to find it, you can refer to https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal.

The accessed WebAPI resource when using azure provider. It is not required.

The credentials file for Google.

Your credentials information as raw JSON.

The JWT Account Key file for Google.

The JWT Account Key file as raw JSON.

The RSA JWK Private Key file for your Okta Service App which is used for interacting with Okta Org Auth Server to mint tokens with okta.* scopes.

The RSA JWK Private Key JSON for your Okta Service App which is used for interacting with Okta Org Auth Server to mint tokens with okta.* scopes.

The RSA JWK private key PEM block for your Okta Service App which is used for interacting with Okta Org Auth Server to mint tokens with okta.* scopes.

Email of the delegated account used to create the credentials (usually an admin). Used in combination with auth.oauth2.google.jwt_file, auth.oauth2.google.jwt_json, and when defaulting to use ADC.

The URL of the HTTP API. Required.

The API endpoint may be accessed via unix socket and Windows named pipes by adding +unix or +npipe to the URL scheme, for example, http+unix:///var/socket/.

HTTP method to use when making requests. GET or POST are the options. Default: GET.

ContentType used for encoding the request body. If set it will force the encoding in the specified format regardless of the Content-Type header value, otherwise it will honor it if possible or fallback to application/json. By default the requests are sent with Content-Type: application/json. Supported values: application/json and application/x-www-form-urlencoded. application/x-www-form-urlencoded will url encode the url.params and set them as the body. It is not set by default.

An optional HTTP POST body. The configuration value must be an object, and it will be encoded to JSON. This is only valid when request.method is POST. Defaults to null (no HTTP body).

- type: httpjson
  request.method: POST
  request.body:
    query:
      bool:
        filter:
          term:
            type: authentication

Duration before declaring that the HTTP client connection has timed out. Valid time units are ns, us, ms, s, m, h. Default: 30s.

This specifies SSL/TLS configuration. If the ssl section is missing, the host’s CAs are used for HTTPS connections. See SSL for more information.

This specifies proxy configuration in the form of http[s]://<user>:<password>@<server name/ip>:<port>

filebeat.inputs:
# Fetch your public IP every minute.
- type: httpjson
  interval: 1m
  request.url: https://api.ipify.org/?format=json
  request.proxy_url: http://proxy.example:8080

This specifies whether to disable keep-alives for HTTP end-points. Default: true.

The maximum number of idle connections across all hosts. Zero means no limit. Default: 0.

The maximum idle connections to keep per-host. If zero, defaults to two. Default: 0.

The maximum amount of time an idle connection will remain idle before closing itself. Valid time units are ns, us, ms, s, m, h. Zero means no limit. Default: 0s.

The maximum number of retries for the HTTP client. Default: 5.

The minimum time to wait before a retry is attempted. Default: 1s.

The maximum time to wait before a retry is attempted. Default: 60s.

When set to true request headers are forwarded in case of a redirect. Default: false.

When redirect.forward_headers is set to true, all headers except the ones defined in this list will be forwarded. Default: [].

The maximum number of redirects to follow for a request. Default: 10.

The value of the response that specifies the total limit. It is defined with a Go template value. Can read state from: [.last_response.header]

The value of the response that specifies the remaining quota of the rate limit. It is defined with a Go template value. Can read state from: [.last_response.header] If the remaining header is missing from the Response, no rate-limiting will occur.

The value of the response that specifies the epoch time when the rate limit will reset. It is defined with a Go template value. Can read state from: [.last_response.header]

Optionally start rate-limiting prior to the value specified in the Response.

Under the default behavior, Requests will continue while the remaining value is non-zero. Specifying an early_limit will mean that rate-limiting will occur prior to reaching 0.

It is not set by default (by default the rate-limiting as specified in the Response is followed).

List of transforms to apply to the request before each execution.

Available transforms for request: [append, delete, set].

Can read state from: [.first_response.*,.last_response.*, .parent_last_response.* .last_event.*, .cursor.*, .header.*, .url.*, .body.*].

Can write state to: [body.*, header.*, url.*].

filebeat.inputs:
- type: httpjson
  request.url: http://localhost:9200/_search?scroll=5m
  request.method: POST
  request.transforms:
    - set:
        target: body.from
        value: '[[now (parseDuration "-1h")]]'

Example Config:

  filebeat.inputs:
  - type: httpjson
    enabled: true
    id: my-httpjson-id
    request.url: http://xyz.com/services/data/v1.0/export_ids/page
    request.method: POST
    interval: 1h
    request.retry.max_attempts: 2
    request.retry.wait_min: 5s
    request.transforms:
    - set:
        target: body.page
        value: 0
    response.request_body_on_pagination: true
    response.pagination:
      - set:
          target: body.page
          value: '[[ .last_response.body.page ]]'
          fail_on_template_error: true
    chain:
    - step:
          request.url: http://xyz.com/services/data/v1.0/$.exportId/export_ids/$.files[:].id/info
          request.method: POST
          request.transforms:
          - set:
              target: body.exportId
              value: '[[ .parent_last_response.body.exportId ]]'
          replace: $.files[:].id
          replace_with: '$.exportId,.parent_last_response.body.exportId'

Here we can see that the chain step uses .parent_last_response.body.exportId only because response.pagination is present for the parent (root) request. However if response.pagination was not present in the parent (root) request, replace_with clause should have used .first_response.body.exportId. This is because when pagination does not exist at the parent level parent_last_response object is not populated with required values for performance reasons, but the first_response object always stores the very first response in the process chain.

It is possible to log httpjson requests and responses to a local file-system for debugging configurations. This option is enabled by setting the request.tracer.filename value. Additional options are available to tune log rotation behavior.

To differentiate the trace files generated from different input instances, a placeholder * can be added to the filename and will be replaced with the input instance id. For Example, http-request-trace-*.ndjson.

Enabling this option compromises security and should only be used for debugging.

This value sets the maximum size, in megabytes, the log file will reach before it is rotated. By default logs are allowed to reach 1MB before rotation.

This specifies the number days to retain rotated log files. If it is not set, log files are retained indefinitely.

The number of old logs to retain. If it is not set all old logs are retained subject to the request.tracer.maxage setting.

Whether to use the host’s local time rather that UTC for timestamping rotated log file names.

This determines whether rotated logs should be gzip compressed.

ContentType used for decoding the response body. If set it will force the decoding in the specified format regardless of the Content-Type header value, otherwise it will honor it if possible or fallback to application/json. Supported values: application/json, application/x-ndjson, text/csv, application/zip, application/xml and text/xml. It is not set by default.

XML documents may require additional type information to enable correct parsing and ingestion. This information can be provided as an XML Schema Definition (XSD) for the document using the response.xsd option.

List of transforms to apply to the response once it is received.

Available transforms for response: [append, delete, set].

Can read state from: [.last_response.*, .last_event.*, .cursor.*, .header.*, .url.*].

Can write state to: [body.*].

filebeat.inputs:
- type: httpjson
  request.url: http://localhost:9200/_search?scroll=5m
  request.method: POST
  response.transforms:
    - delete:
        target: body.very_confidential
  response.split:
    target: body.hits.hits
  response.pagination:
    - set:
        target: url.value
        value: http://localhost:9200/_search/scroll
    - set:
        target: url.params.scroll_id
        value: '[[.last_response.body._scroll_id]]'
    - set:
        target: body.scroll
        value: 5m

Split operation to apply to the response once it is received. A split can convert a map, array, or string into multiple events. If the split target is empty the parent document will be kept. If documents with empty splits should be dropped, the ignore_empty_value option should be set to true.

Defines the target field upon the split operation will be performed.

Defines the field type of the target. Allowed values: array, map, string. string requires the use of the delimiter options to specify what characters to split the string on. delimiter always behaves as if keep_parent is set to true. Default: array.

A set of transforms can be defined. This list will be applied after response.transforms and after the object has been modified based on response.split[].keep_parent and response.split[].key_field.

Available transforms for response: [append, delete, set].

Can read state from: [.last_response.*, .first_event.*, .last_event.*, .cursor.*, .header.*, .url.*].

Can write state to: [body.*].

If set to true, the fields from the parent document (at the same level as target) will be kept. Otherwise a new document will be created using target as the root. Default: false.

Required if using split type of string. This is the sub string used to split the string. For example if delimiter was "\n" and the string was "line 1\nline 2", then the split would result in "line 1" and "line 2".

Valid when used with type: map. When not empty, defines a new field where the original key value will be stored.

If set to true, empty or missing value will be ignored and processing will pass on to the next nested split operation instead of failing with an error. Default: false.

Nested split operation. Split operations can be nested at will. An event won’t be created until the deepest split operation is applied.

If set to true, the values in request.body are sent for pagination requests. Default: false.

List of transforms that will be applied to the response to every new page request. All the transforms from request.transform will be executed and then response.pagination will be added to modify the next request as needed. For subsequent responses, the usual response.transforms and response.split will be executed normally.

Available transforms for pagination: [append, delete, set].

Can read state from: [.last_response.*, .first_event.*, .last_event.*, .cursor.*, .header.*, .url.*, .body.*].

Can write state to: [body.*, header.*, url.*].

Examples using split:

A chain is a list of requests to be made after the first one.

Contains basic request and response configuration for chained calls.

See request parameters. Required.

Example:

First call: https://example.com/services/data/v1.0/

Second call: https://example.com/services/data/v1.0/1/export_ids

Third call: https://example.com/services/data/v1.0/export_ids/file_1/info

See response split parameter.

+

A JSONPath string to parse values from responses JSON, collected from previous chain steps. Place same replace string in url where collected values from previous call should be placed. Required.

Example:

filebeat.inputs:
- type: httpjson
  enabled: true
  # first call
  request.url: https://example.com/services/data/v1.0/records
  interval: 1h
  chain:
    # second call
    - step:
        request.url: https://example.com/services/data/v1.0/$.records[:].id/export_ids
        request.method: GET
        replace: $.records[:].id
    # third call
    - step:
        request.url: https://example.com/services/data/v1.0/export_ids/$.file_name/info
        request.method: GET
        replace: $.file_name

Example:

filebeat.inputs:
- type: httpjson
  enabled: true
  # first call
  request.url: https://example.com/services/data/v1.0/records
  interval: 1h
  chain:
    # second call
    - step:
        request.url: placeholder:$.records[:]
        request.method: GET
        replace: $.records[:]
    # third call
    - step:
        request.url: placeholder:$.file_name
        request.method: GET
        replace: $.file_name

+

The replace_with: "pattern,value" clause is used to replace a fixed pattern string defined in request.url with the given value. The fixed pattern must have a $. prefix, for example: $.xyz. The value may be hard coded or extracted from context variables like [.last_response.*, .first_response.*, .parent_last_response.*] etc. The replace_with clause can be used in combination with the replace clause thus providing a lot of flexibility in the logic of chain requests.

Example:

filebeat.inputs:
- type: httpjson
  enabled: true
  # first call
  request.url: https://example.com/services/data/v1.0/exports
  interval: 1h
  chain:
    # second call
    - step:
        request.url: https://example.com/services/data/v1.0/$.exportId/files
        request.method: GET
        replace_with: '$.exportId,.first_response.body.exportId'

Example:

Some useful points to remember:-

Contains basic request and response configuration for chained while calls. Chained while calls will keep making the requests for a given number of times until a condition is met or the maximum number of attempts gets exhausted. While chain has an attribute until which holds the expression to be evaluated. Ideally the until field should always be used together with the attributes request.retry.max_attempts and request.retry.wait_min which specifies the maximum number of attempts to evaluate until before giving up and the maximum wait time in between such requests. If request.retry.max_attempts is not specified, it will only try to evaluate the expression once and give up if it fails. If request.retry.wait_min is not specified the default wait time will always be 0 as in successive calls will be made immediately.

See request parameters .

Example:

First call: http://example.com/services/data/v1.0/exports

Second call: http://example.com/services/data/v1.0/9ef0e6a5/export_ids/status

Third call: http://example.com/services/data/v1.0/export_ids/1/info

See response split parameter .

See chain[].step.replace .

Example:

filebeat.inputs:
- type: httpjson
  enabled: true
  # first call
  id: my-httpjson-id
  request.url: http://example.com/services/data/v1.0/exports
  interval: 1h
  chain:
    # second call
    - while:
        request.url: http://example.com/services/data/v1.0/$.exportId/export_ids/status
        request.method: GET
        replace: $.exportId
        until: '[[ eq .last_response.body.status "completed" ]]'
        request.retry.max_attempts: 5
        request.retry.wait_min: 5s
    # third call
    - step:
        request.url: http://example.com/services/data/v1.0/export_ids/$.files[:].id/info
        request.method: GET
        replace: $.files[:].id

Example:

filebeat.inputs:
- type: httpjson
  enabled: true
  # first call
  id: my-httpjson-id
  request.url: http://example.com/services/data/v1.0/exports
  interval: 1h
  chain:
    # second call
    - while:
        request.url: placeholder:$.exportId
        request.method: GET
        replace: $.exportId
        until: '[[ eq .last_response.body.status "completed" ]]'
        request.retry.max_attempts: 5
        request.retry.wait_min: 5s
    # third call
    - step:
        request.url: placeholder:$.files[:]
        request.method: GET
        replace: $.files[:]

See chain[].step.replace_with .

Cursor is a list of key value objects where arbitrary values are defined. The values are interpreted as value templates and a default template can be set. Cursor state is kept between input restarts and updated once all the events for a request are published.

Each cursor entry is formed by:

Can read state from: [.last_response.*, .first_event.*, .last_event.*].

filebeat.inputs:
- type: httpjson
  interval: 1m
  request.url: https://api.ipify.org/?format=json
  response.transforms:
    - set:
        target: body.last_requested_at
        value: '[[.cursor.last_requested_at]]'
        default: "[[now]]"
  cursor:
    last_requested_at:
      value: '[[now]]'
  processors:
    - decode_json_fields:
        fields: ["message"]
        target: "json"

This input exposes metrics under the HTTP monitoring endpoint. These metrics are exposed under the /inputs path. They can be used to observe the activity of the input.

The following configuration options are supported by all inputs.

Use the enabled option to enable and disable inputs. By default, enabled is set to true.

A list of tags that Filebeat includes in the tags field of each published event. Tags make it easy to select specific events in Kibana or apply conditional filtering in Logstash. These tags will be appended to the list of tags specified in the general configuration.

Example:

filebeat.inputs:
- type: httpjson
  . . .
  tags: ["json"]

Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here.

filebeat.inputs:
- type: httpjson
  . . .
  fields:
    app_id: query_engine_12

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields.

A list of processors to apply to the input data.

See Processors for information about specifying processors in your config.

The ingest pipeline ID to set for the events generated by this input.

If this option is set to true, fields with null values will be published in the output document. By default, keep_null is set to false.

If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use output.elasticsearch.index or a processor.

Example value: "%{[agent.name]}-myindex-%{+yyyy.MM.dd}" might expand to "filebeat-myindex-2019.11.01".

By default, all events contain host.name. This option can be set to true to disable the addition of this field to all events. The default value is false.