Networking

Most users will need to configure only the following network settings.

You can configure Elasticsearch to automatically determine its addresses by using the following special values. Use these values when configuring network.host, network.bind_host, network.publish_host, and the corresponding settings for the HTTP and transport interfaces.

Elasticsearch uses network addresses for two distinct purposes known as binding and publishing. Most nodes will use the same address for everything, but more complicated setups may need to configure different addresses for different purposes.

When an application such as Elasticsearch wishes to receive network communications, it must indicate to the operating system the address or addresses whose traffic it should receive. This is known as binding to those addresses. Elasticsearch can bind to more than one address if needed, but most nodes only bind to a single address. Elasticsearch can only bind to an address if it is running on a host that has a network interface with that address. If necessary, you can configure the transport and HTTP interfaces to bind to different addresses.

Each Elasticsearch node has an address at which clients and other nodes can contact it, known as its publish address. Each node has one publish address for its HTTP interface and one for its transport interface. These two addresses can be anything, and don’t need to be addresses of the network interfaces on the host. The only requirements are that each node must be:

If you specify the transport publish address using a hostname then Elasticsearch will resolve this hostname to an IP address once during startup, and other nodes will use the resulting IP address instead of resolving the name again themselves. To avoid confusion, use a hostname which resolves to the node’s address in all network locations.

These advanced settings let you bind to multiple addresses, or to use different addresses for binding and publishing. They are not required in most cases and you should not use them if you can use the commonly used settings instead.

Use the following advanced settings to configure the HTTP interface independently of the transport interface. You can also configure both interfaces together using the network settings.

Use the following advanced settings to configure the transport interface independently of the HTTP interface. Use the network settings to configure both interfaces together.

transport.profiles.default.port: 9300-9400
transport.profiles.default.bind_host: 10.0.0.1
transport.profiles.client.port: 9500-9600
transport.profiles.client.bind_host: 192.168.0.1
transport.profiles.dmz.port: 9700-9800
transport.profiles.dmz.bind_host: 172.16.1.2

Use the following advanced settings to configure the remote cluster interface (API key based model) independently of the transport interface. You can also configure both interfaces together using the network settings.

You can trace individual requests made on the HTTP and transport layers.

response = client.cluster.put_settings(
  body: {
    persistent: {
      "logger.org.elasticsearch.http.HttpTracer": 'TRACE'
    }
  }
)
puts response

PUT _cluster/settings
{
   "persistent" : {
      "logger.org.elasticsearch.http.HttpTracer" : "TRACE"
   }
}

response = client.cluster.put_settings(
  body: {
    persistent: {
      'http.tracer.include' => '*',
      'http.tracer.exclude' => ''
    }
  }
)
puts response

PUT _cluster/settings
{
   "persistent" : {
      "http.tracer.include" : "*",
      "http.tracer.exclude" : ""
   }
}

response = client.cluster.put_settings(
  body: {
    persistent: {
      "logger.org.elasticsearch.http.HttpTracer": 'TRACE',
      "logger.org.elasticsearch.http.HttpBodyTracer": 'TRACE'
    }
  }
)
puts response

PUT _cluster/settings
{
   "persistent" : {
      "logger.org.elasticsearch.http.HttpTracer" : "TRACE",
      "logger.org.elasticsearch.http.HttpBodyTracer" : "TRACE"
   }
}

[TRACE][o.e.h.HttpBodyTracer     ] [master] [276] response body [part 1]: H4sIAAAAAAAA/9...
[TRACE][o.e.h.HttpBodyTracer     ] [master] [276] response body [part 2]: 2oJ93QyYLWWhcD...
[TRACE][o.e.h.HttpBodyTracer     ] [master] [276] response body (gzip compressed, base64-encoded, and split into 2 parts on preceding log lines)

cat httptrace.log | sed -e 's/.*://' | base64 --decode | gzip --decompress

response = client.cluster.put_settings(
  body: {
    persistent: {
      "logger.org.elasticsearch.transport.TransportService.tracer": 'TRACE'
    }
  }
)
puts response

PUT _cluster/settings
{
   "persistent" : {
      "logger.org.elasticsearch.transport.TransportService.tracer" : "TRACE"
   }
}

response = client.cluster.put_settings(
  body: {
    persistent: {
      'transport.tracer.include' => '*',
      'transport.tracer.exclude' => 'internal:coordination/fault_detection/*'
    }
  }
)
puts response

PUT _cluster/settings
{
   "persistent" : {
      "transport.tracer.include" : "*",
      "transport.tracer.exclude" : "internal:coordination/fault_detection/*"
   }
}

This section describes the threading model used by the networking subsystem in Elasticsearch. This information isn’t required to use Elasticsearch, but it may be useful to advanced users who are diagnosing network problems in a cluster.

Elasticsearch nodes communicate over a collection of TCP channels that together form a transport connection. Elasticsearch clients communicate with the cluster over HTTP, which also uses one or more TCP channels. Each of these TCP channels is owned by exactly one of the transport_worker threads in the node. This owning thread is chosen when the channel is opened and remains the same for the lifetime of the channel.

Each transport_worker thread has sole responsibility for sending and receiving data over the channels it owns. Additionally, each http and transport server socket is assigned to one of the transport_worker threads. That worker has the responsibility of accepting new incoming connections to the server socket it owns.

If a thread in Elasticsearch wants to send data over a particular channel, it passes the data to the owning transport_worker thread for the actual transmission.

Normally the transport_worker threads will not completely handle the messages they receive. Instead, they will do a small amount of preliminary processing and then dispatch (hand off) the message to a different threadpool for the rest of their handling. For instance, bulk messages are dispatched to the write threadpool, searches are dispatched to one of the search threadpools, and requests for statistics and other management tasks are mostly dispatched to the management threadpool. However in some cases the processing of a message is expected to be so quick that Elasticsearch will do all of the processing on the transport_worker thread rather than incur the overhead of dispatching it elsewhere.

By default, there is one transport_worker thread per CPU. In contrast, there may sometimes be tens-of-thousands of TCP channels. If data arrives on a TCP channel and its owning transport_worker thread is busy, the data isn’t processed until the thread finishes whatever it is doing. Similarly, outgoing data are not sent over a channel until the owning transport_worker thread is free. This means that we require every transport_worker thread to be idle frequently. An idle transport_worker looks something like this in a stack dump:

"elasticsearch[instance-0000000004][transport_worker][T#1]" #32 daemon prio=5 os_prio=0 cpu=9645.94ms elapsed=501.63s tid=0x00007fb83b6307f0 nid=0x1c4 runnable  [0x00007fb7b8ffe000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait([email protected]/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect([email protected]/EPollSelectorImpl.java:118)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:129)
	- locked <0x00000000c443c518> (a sun.nio.ch.Util$2)
	- locked <0x00000000c38f7700> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:146)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.lang.Thread.run([email protected]/Thread.java:833)

In the Nodes hot threads API an idle transport_worker thread is reported like this:

   0.0% [cpu=0.0%, idle=100.0%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[instance-0000000004][transport_worker][T#1]'
     10/10 snapshots sharing following 9 elements
       [email protected]/sun.nio.ch.EPoll.wait(Native Method)
       [email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:118)
       [email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)
       [email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:146)
       io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:813)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
       io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       [email protected]/java.lang.Thread.run(Thread.java:833)

Note that transport_worker threads should always be in state RUNNABLE, even when waiting for input, because they block in the native EPoll#wait method. The idle= time reports the proportion of time the thread spent waiting for input, whereas the cpu= time reports the proportion of time the thread spent processing input it has received.

If a transport_worker thread is not frequently idle, it may build up a backlog of work. This can cause delays in processing messages on the channels that it owns. It’s hard to predict exactly which work will be delayed:

If the backlog builds up too far, some messages may be delayed by many seconds. The node might even fail its health checks and be removed from the cluster. Sometimes, you can find evidence of busy transport_worker threads using the Nodes hot threads API. However, this API itself sends network messages so may not work correctly if the transport_worker threads are too busy. It is more reliable to use jstack to obtain stack dumps or use Java Flight Recorder to obtain a profiling trace. These tools are independent of any work the JVM is performing.

It may also be possible to identify some reasons for delays from the server logs, particularly looking at warnings from org.elasticsearch.transport.InboundHandler and org.elasticsearch.transport.OutboundHandler. Warnings about long processing times from the InboundHandler are particularly indicative of incorrect threading behaviour, whereas the transmission time reported by the OutboundHandler includes time spent waiting for network congestion and the transport_worker thread is free to do other work during this time.