optimize

Optimize is a bit of a misnomer. It is in actuality a Lucene forceMerge operation. With time-series data in a per-day index, Lucene does a good job of keeping the number of segments low. However, if no new data is being ingested, no further segment merging will happen. There are some minor performance benefits from merging segments down to a smaller count, but a greater benefit when it comes to restarts [e.g. version upgrades, etc.] after a shutdown: with fewer segments to have to validate, the cluster comes back up sooner.

Curator counts the number of segments per shard before performing an optimize call. If the count is at or below the threshold (default is 2 segments per shard) it will skip optimizing that index. In this manner, indices are not optimized repeatedly.

Performing this forceMerge requires a lot of extra disk space: 3x what the original segments consume. Fortunately, this extra space is only needed during the forceMerge operation. Be sure to keep enough disk space available so that this does not become an issue for you.

With some commands (e.g. optimize) the default behavior is to wait until the operation is complete before proceeding with the next step. Since these operations can take quite a long time it is advisable to set --timeout to a high value. If one is not set, a default of 6 hours (21600 seconds) will be applied.

$ curator optimize --help
Usage: curator optimize [OPTIONS] COMMAND [ARGS]...

  Optimize Indices

Options:
  --delay INTEGER             Pause *n* seconds after optimizing an index.
                              [default: 0]
  --max_num_segments INTEGER  Merge to this number of segments per shard.
                              [default: 2]
  --request_timeout INTEGER   Allow this many seconds before the transaction
                              times out.  [default: 21600]
  --help                      Show this message and exit.

Commands:
  indices  Index selection.

Optimize matching indices to 2 segments per shard (the default):

curator optimize indices <<index selection parameters>>

Optimize matching indices older than 2 days to 1 segment per shard, and delay 120 seconds between indices:

curator optimize --max_num_segments 1 --delay 120 indices <<index selection parameters>>