Tune APM Serveredit
You can fine tune your setup considering following options:
Tune APM Server output parameters for your Elasticsearch clusteredit
If your Elasticsearch cluster is sized properly, but not ingesting the amount of data you expect, you can tweak APM Server options to make better use of the cluster:
-
Adjust
output.elasticsearch.workersto a number that suits your setup. See tune for indexing speed for an overview. -
Ensure
output.elasticsearch.bulk_max_sizeis set to a high value, for example 5120. The default of 50 is very conservative. -
Ensure that
queue.mem.eventsis set to a reasonable value compared to your other settings. A good rule of thumb is thatqueue.mem.eventsshould equaloutput.elasticsearch.workermultiplied byoutput.elasticsearch.bulk_max_size.
Get detailed information on available output configuration options and their default values.
Adjust internal queue sizeedit
APM Server uses an internal queue to allow buffering incoming requests until they can be delievered to Elasticsearch.
A larger internal queue allows Elasticsearch to be unavailable for longer periods,
and it alleviates problems that might result from sudden spikes of data.
You can adjust the queue size by overriding queue.mem.events.
Be aware that increasing queue.mem.events can significantly affect APM Server memory usage.
Adjust concurrent requestsedit
To avoid overflowing the APM Server,
it has an upper limit to how many requests are accepted concurrently.
This limit is determined by the apm-server.concurrent_requests configuration parameter.
As this limit is set to a rather conservative default you might want to set it to some higher value.
Be aware though, that increasing the limit can significantly affect APM Server memory usage.
Add APM Server nodesedit
In case the APM Server cannot process incoming requests quickly enough, you will see a server timeout.
One way to avoid this problem is to add more processing power to your APM Server cluster. This can be easily done by either migrating your APM Server processes to more powerful machines or adding more APM Server nodes. Please refer to the high availability section for general information on how to set up multiple APM Server.
Reduce the payload sizeedit
Large payloads coming from agents may result in a server timeout.
You can reduce the payload size by decreasing the max_queue_size in the agents.
This will result in agents sending smaller payloads to the APM Server,
but the requests will be more frequent.
See the documentation for the Python and Node.js agents for more information.
Optionally you can also reduce the sample rate or reduce the amount of stacktraces collected, both leading to collect less data and decreasing the payload size.
Read more about configuration options directly in the agent documentation.