google_pubsub

For plugins not bundled by default, it is easy to install by running bin/logstash-plugin install logstash-input-google_pubsub. See Working with plugins for more details.

For questions about the plugin, open a topic in the Discuss forums. For bugs or feature requests, open an issue in Github. For the list of Elastic supported plugins, please consult the Elastic Support Matrix.

The main motivation behind the development of this plugin was to ingest Stackdriver Logging messages via the Exported Logs feature of Stackdriver Logging.

You must first create a Google Cloud Platform project and enable the the Google Pub/Sub API. If you intend to use the plugin ingest Stackdriver Logging messages, you must also enable the Stackdriver Logging API and configure log exporting to Pub/Sub. There is plentiful information on https://cloud.google.com/ to get started:

Currently, this module requires you to create a topic manually and specify it in the logstash config file. You must also specify a subscription, but the plugin will attempt to create the pull-based subscription on its own.

All messages received from Pub/Sub will be converted to a logstash event and added to the processing pipeline queue. All Pub/Sub messages will be acknowledged and removed from the Pub/Sub topic (please see more about Pub/Sub concepts.

It is generally assumed that incoming messages will be in JSON and added to the logstash event as-is. However, if a plain text message is received, the plugin will return the raw text in as raw_message in the logstash event.

You have two options for authentication depending on where you run Logstash.

If you intend to use the logstash plugin for Stackdriver Logging message ingestion, you must first manually set up the Export option to Cloud Pub/Sub and the manually create the topic. Please see the more detailed instructions at, https://cloud.google.com/logging/docs/export/using_exported_logs [Exported Logs] and ensure that the necessary permissions have also been manually configured.

Logging messages from Stackdriver Logging exported to Pub/Sub are received as JSON and converted to a logstash event as-is in this format.

Below is a copy of the included example.conf-tmpl file that shows a basic configuration for this plugin.

input {
    google_pubsub {
        # Your GCP project id (name)
        project_id => "my-project-1234"

        # The topic name below is currently hard-coded in the plugin. You
        # must first create this topic by hand and ensure you are exporting
        # logging to this pubsub topic.
        topic => "logstash-input-dev"

        # The subscription name is customizeable. The plugin will attempt to
        # create the subscription (but use the hard-coded topic name above).
        subscription => "logstash-sub"

        # If you are running logstash within GCE, it will use
        # Application Default Credentials and use GCE's metadata
        # service to fetch tokens.  However, if you are running logstash
        # outside of GCE, you will need to specify the service account's
        # JSON key file below.
        #json_key_file => "/home/erjohnso/pkey.json"
    }
}
output { stdout { codec => rubydebug } }

This plugin supports the following configuration options:

Required configuration options:

google_pubsub {
    max_messages => ...
    project_id => ...
    subscription => ...
    topic => ...
}

Available configuration options: