Hot Threads API

edit

The hot threads API gets the current hot threads for Logstash. A hot thread is a Java thread that has high CPU usage and executes for a longer than normal period of time.

curl -XGET 'localhost:9600/_node/hot_threads?pretty'

The output is a JSON document that contains a breakdown of the top hot threads for Logstash.

Example response:

{
    "time": "2017-01-12T12:09:45-08:00",
    "busiest_threads": 3,
    "threads": [
      {
        "name": "LogStash::Runner",
        "percent_of_cpu_time": 1.07,
        "state": "timed_waiting",
        "traces": [
          "java.lang.Object.wait(Native Method)",
          "java.lang.Thread.join(Thread.java:1253)",
          "org.jruby.internal.runtime.NativeThread.join(NativeThread.java:75)",
          "org.jruby.RubyThread.join(RubyThread.java:697)",
          "org.jruby.RubyThread$INVOKER$i$0$1$join.call(RubyThread$INVOKER$i$0$1$join.gen)",
          "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:663)",
          "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:198)",
          "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)",
          "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)",
          "org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)"
        ]
      },
      {
        "name": "[main]>worker7",
        "percent_of_cpu_time": 0.71,
        "state": "waiting",
        "traces": [
          "sun.misc.Unsafe.park(Native Method)",
          "java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)",
          "java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)",
          "java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:897)",
          "java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1222)",
          "java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)",
          "org.jruby.RubyThread.lockInterruptibly(RubyThread.java:1470)",
          "org.jruby.ext.thread.Mutex.lock(Mutex.java:91)",
          "org.jruby.ext.thread.Mutex.synchronize(Mutex.java:147)",
          "org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)"
        ]
      },
      {
        "name": "[main]>worker3",
        "percent_of_cpu_time": 0.71,
        "state": "waiting",
        "traces": [
          "sun.misc.Unsafe.park(Native Method)",
          "java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)",
          "java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)",
          "java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:897)",
          "java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1222)",
          "java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)",
          "org.jruby.RubyThread.lockInterruptibly(RubyThread.java:1470)",
          "org.jruby.ext.thread.Mutex.lock(Mutex.java:91)",
          "org.jruby.ext.thread.Mutex.synchronize(Mutex.java:147)",
          "org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)"
        ]
      }
    ]
  }
}

The parameters allowed are:

threads

The number of hot threads to return. The default is 3.

human

If true, returns plain text instead of JSON format. The default is false.

ignore_idle_threads

If true, does not return idle threads. The default is true.

See Common Options for a list of options that can be applied to all Logstash monitoring APIs.

You can use the ?human parameter to return the document in a human-readable format.

curl -XGET 'localhost:9600/_node/hot_threads?human=true'

Example of a human-readable response:

 ::: {}
 Hot threads at 2017-01-12T12:10:15-08:00, busiestThreads=3:
 ================================================================================
 1.02 % of cpu usage, state: timed_waiting, thread name: 'LogStash::Runner'
	java.lang.Object.wait(Native Method)
	java.lang.Thread.join(Thread.java:1253)
	org.jruby.internal.runtime.NativeThread.join(NativeThread.java:75)
	org.jruby.RubyThread.join(RubyThread.java:697)
	org.jruby.RubyThread$INVOKER$i$0$1$join.call(RubyThread$INVOKER$i$0$1$join.gen)
	org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:663)
	org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:198)
	org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
	org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
	org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
 --------------------------------------------------------------------------------
 0.71 % of cpu usage, state: waiting, thread name: '[main]>worker7'
	sun.misc.Unsafe.park(Native Method)
	java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:897)
	java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1222)
	java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
	org.jruby.RubyThread.lockInterruptibly(RubyThread.java:1470)
	org.jruby.ext.thread.Mutex.lock(Mutex.java:91)
	org.jruby.ext.thread.Mutex.synchronize(Mutex.java:147)
	org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)
 --------------------------------------------------------------------------------
 0.71 % of cpu usage, state: timed_waiting, thread name: '[main]>worker3'
	sun.misc.Unsafe.park(Native Method)
	java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
	java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
	java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
	sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:497)
	org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:466)
	org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:324)