Jms input plugin

This section contains sample configurations for connecting to a JMS provider that is not using JNDI using a combination of the Logstash configuration and the yaml file

This section contains sample configurations for connecting to a JMS provider that is using JNDI using a combination of the Logstash configuration and the yaml file

The most common issue is missing jar files, which may be jar files provided by the JMS vendor, or jar files that the JMS vendor requires to run. This issue can manifest in different ways, depending on where the missing jar file is discovered.

Example log output:

  Failed to load JMS Connection, likely because a JMS Provider is not on the Logstash classpath or correctly
  specified by the plugin's `require_jars` directive
  {:exception=>"cannot load Java class javax.jms.DeliveryMode"

  JMS Consumer Died {:exception=>"Java::JavaxNaming::NoInitialContextException",
                     :exception_message=>"Cannot instantiate class:"

  warning: thread "[main]<jms" terminated with exception (report_on_exception is true):
  java.lang.NoClassDefFoundError: org/apache/commons/lang/exception/NestableException

  JMS Consumer Died {:exception=>"Java::JavaxJms::JMSException", :exception_message=>"io/netty/channel/epoll/Epoll",
  :root_cause=>{:exception=>"Java::JavaLang::NoClassDefFoundError", :exception_message=>"io/netty/channel/epoll/Epoll"}

If any of these issues occur, check the list of require_jars in either the Logstash configuration or yaml configuration files.

Many JMS providers allow or expect System properties to be set to configure certain properties when using JMS, for example, the Apache qpid JMS client allows the connection factory lookup to be stored there, and the Solace JMS client allows many properties, such as number of connection retries to be set as System properties. Any system properties that are set should be set in the Logstash jvm.options file.

The use of multiple JMS consumers and producers in the same Logstash process is unsupported if:

Incorrect message selector syntax can have two effects - either the syntax is incorrect and the selector parser from the JMS provider will throw an exception causing the plugin to fail OR the syntax will be accepted, but the messages will be silently dropped - this can happen with incorrect quoting of string properties in the selector definition. All selector definitions must be double quoted in the Logstash configuration file, and string property values must be single quoted, and numeric property values not quoted at all.

Messages from certain JMS providers may contain headers or properties that Logstash cannot interpret, which can lead to error messages such as:

[2019-11-25T08:04:28,769][ERROR][logstash.inputs.jms      ] Failed to create event {:message=>Java::ComSolacesystemsJmsMessage::SolTextMessage: ...
Attributes: {:jms_correlation_id=>"xxxx", :jms_delivery_mode_sym=>:non_persistent, :jms_destination=>"destination", :jms_expiration=>0, :jms_message_id=>"xxxxxx", :jms_priority=>0, :jms_redelivered=>false, :jms_reply_to=>#<Java::ComSolacesystemsJmsImpl::SolTopicImpl:0xdeadbeef>, :jms_timestamp=>1574669008862, :jms_type=>nil}
Properties: nil, :exception=>org.logstash.MissingConverterException: Missing Converter handling for full class name=com.solacesystems.jms.impl.SolTopicImpl, simple name=SolTopicImpl, :backtrace=>["org.logstash.Valuefier.fallbackConvert(Valuefier.java:98)..."]}
:exception=>org.logstash.MissingConverterException: Missing Converter handling for full class name=com.solacesystems.jms.impl.SolTopicImpl

To get around this, use the skip_headers or skip_properties configuration setting to avoid attempting to process the offending header or property in the message.

In the example shown above, this attribute is causing the MissingConverterException:

jms_reply_to=>#<Java::ComSolacesystemsJmsImpl::SolTopicImpl:0xdeadbeef>

To avoid this error, the configuration should include the following line:

skip_headers => ["jms_reply_to"]

Url to use when connecting to the JMS provider. This is only relevant for non-JNDI configurations.

Name of the destination queue or topic to use.

Setting this value to true will make subscriptions to topics "durable", which allowing messages that arrived on the specified topic while Logstash is not running to still be read. Without setting this value, any messages sent to a topic while Logstash is not actively listening will be lost. A durable subscriber specifies a unique identity consisting of the topic (destination), the client id (durable_subscriber_client_id) and subscriber name (durable_subscriber_subscriber_name). See your JMS Provider documentation for any further requirements/limitations around these settings.

This represents the value of the client ID for a durable subscribtion, and is only used if durable_subscriber is set to true.

This represents the value of the subscriber name for a durable subscribtion, and is only used if durable_subscriber is set to true. Please consult your JMS Provider documentation for constraints and requirements for this setting.

Controls this plugin’s compatibility with the Elastic Common Schema (ECS). The value of this setting affects the default value of headers_target and properties_target.

Full name (including package name) of Java connection factory used to create a connection with your JMS provider.

Hash of implementation specific configuration values to set on the connection factory of the JMS provider. Each JMS Provider will have its own set of parameters that can be used here. These parameters are mapped to set methods on the provided connection factory, and can be supplied in either snake or camel case. For example, a hash including exclusive_consumer => true would call setExclusiveConsumer(true) on the supplied connection factory. See your JMS provider documentation for implementation specific details.

The name of the field under which JMS headers will be added, if include_headers is set.

Include JMS Message Body in the event. Supports TextMessage, MapMessage and BytesMessage.

If the JMS Message is a TextMessage or BytesMessage, then the value will be in the "message" field of the event. If the JMS Message is a MapMessage, then all the key/value pairs will be added at the top-level of the event by default. To avoid pollution of the top-level namespace, when receiving a MapMessage, use the target.

StreamMessage and ObjectMessage are not supported.

Note: This option is deprecated and it will be removed in the next major version of Logstash. Use include_headers instead.

A JMS message has three parts:

You can tell the input plugin which parts should be included in the event produced by Logstash.

Include standard JMS message header field values in the event. Example headers:

    {
        "jms_message_id" => "ID:amqhost-39547-1636977297920-71:1:1:1:1",
        "jms_timestamp" => 1636977329102,
        "jms_expiration" => 0,
        "jms_delivery_mode" => "persistent",
        "jms_redelivered" => false,
        "jms_destination" => "topic://41ad5342149901ad",
        "jms_priority" => 4,
        "jms_type" => "sample",
        "jms_correlation_id" => "28d975cb-14ff-4285-841e-05ef1e0a7ab2"
    }

Include JMS Message Properties Field values in the event.

Polling interval in seconds. This is the time sleeping between asks to a consumed Queue. This parameter has non influence in the case of a subcribed Topic.

Only used if using JNDI lookup. Key value pairs to determine how to connect the JMS message brokers if JNDI is being used. Consult your JMS provider documentation for the correct values to use here.

Only used if using JNDI lookup. Name of JNDI entry at which the Factory can be found.

If you need to use a custom keystore (.jks) specify it here. This does not work with .pem keys

Specify the keystore password here. Note, most .jks files created with keytool require a password

Receive Oracle AQ buffered messages. In this mode persistent Oracle AQ JMS messages will not be received. Only for use with Oracle AQ

Password to use when connecting to the JMS provider.

The name of the field under which JMS properties will be added, if include_properties is set.

If pub-sub (topic) style should be used. Note that if pub_sub is set to true, threads must be set to 1.

If you do not use an yaml configuration use either the factory or jndi_name. An optional array of Jar file names to load for the specified JMS provider. By using this option it is not necessary to put all the JMS Provider specific jar files into the java CLASSPATH prior to starting Logstash.

JMS message selector. Use in conjunctions with message headers or properties to filter messages to be processed. Only messages that match the query specified here will be processed. Check with your JMS provider for the correct JMS message selector syntax.

If include_headers is set, a list of headers to skip processing on can be specified here.

If include_properties is set, a list of properties to skip processing on can be specified here.

Any System properties that the JMS provider requires can be set either in a Hash here, or in jvm.options

The name of the field to assign MapMessage data into. If not specified data will be stored in the root of the event.

Initial connection timeout in seconds.

If you need to use a custom truststore (.jks) specify it here. This does not work with .pem certs.

Specify the truststore password here.

Convert the JMSTimestamp header field to the @timestamp value of the event

Username to use for connecting to JMS provider.

Yaml config file

Yaml config file section name For some known examples, see jms.yml examples.

Add a field to an event

The codec used for input data. Input codecs are a convenient method for decoding your data before it enters the input, without needing a separate filter in your Logstash pipeline.

Disable or enable metric logging for this specific plugin instance by default we record all the metrics we can, but you can disable metrics collection for a specific plugin.

Add a unique ID to the plugin configuration. If no ID is specified, Logstash will generate one. It is strongly recommended to set this ID in your configuration. This is particularly useful when you have two or more plugins of the same type, for example, if you have 2 jms inputs. Adding a named ID in this case will help in monitoring Logstash when using the monitoring APIs.

input {
  jms {
    id => "my_plugin_id"
  }
}

Add any number of arbitrary tags to your event.

This can help with processing later.

Add a type field to all events handled by this input.

Types are used mainly for filter activation.

The type is stored as part of the event itself, so you can also use the type to search for it in Kibana.

If you try to set a type on an event that already has one (for example when you send an event from a shipper to an indexer) then a new input will not override the existing type. A type set at the shipper stays with that event for its life even when sent to another Logstash server.