Manage index pattern data fields

edit

Manage index pattern data fields

edit

To customize the data fields in your index pattern, you can add runtime fields to the existing documents, add scrited fields to compute data on the fly, and change how Kibana displays the data fields.

Explore your data with runtime fields

edit

Runtime fields are fields that you add to documents after you’ve ingested your data, and are evaluated at query time. With runtime fields, you allow for a smaller index and faster ingest time so that you can use less resources and reduce your operating costs. You can use runtime fields anywhere index patterns are used, for example, you can explore runtime fields in Discover and create visualizations with runtime fields for your dashboard.

With runtime fields, you can:

  • Define fields for a specific use case without modifying the underlying schema.
  • Override the returned values from index fields.
  • Start working on your data without understanding the structure.
  • Add fields to existing documents without reindexing your data.

Runtime fields can impact Kibana performance. When you run a query, Elasticsearch uses the fields you index first to shorten the response time. Index the fields that you commonly search for and filter on, such as timestamp, then use runtime fields to limit the number of fields Elasticsearch uses to calculate values.

For detailed information on how to use runtime fields with Elasticsearch, refer to Runtime fields.

Add runtime fields

edit

To add runtime fields to your index patterns, open the index pattern you want to change, then define the field values by emitting a single value using the Painless scripting language. You can also add runtime fields in Discover and Lens.

  1. Open the main menu, then click Stack Management > Index Patterns.
  2. Select the index pattern you want to add the runtime field to, then click Add field.
  3. Enter the field Name, then select the Type.
  4. Select Set custom label, then enter the label you want to display where the index pattern is used, such as Discover.
  5. Select Set value, then define the script. The script must match the Type, or the index pattern fails anywhere it is used.
  6. To help you define the script, use the Preview:

    • To view the other available fields, use the Document ID arrows.
    • To filter the fields list, enter the keyword in Filter fields.
    • To pin frequently used fields to the top of the list, hover over the field, then click Icon to pin field to the top of the list.
  7. Click Create field.

Runtime field examples

edit

Try the runtime field examples on your own using the Sample web logs data index pattern.

Return a keyword value

edit

Return Hello World!:

emit("Hello World!");
Perform a calculation on a single field
edit

Calculate kilobytes from bytes:

emit(doc['bytes'].value / 1024)
Return a substring
edit

Return the string that appears after the last slash in the URL:

def path = doc["url.keyword"].value;
if (path != null) {
    int lastSlashIndex = path.lastIndexOf('/');
    if (lastSlashIndex > 0) {
        emit(path.substring(lastSlashIndex+1));
    return;
    }
}
emit("");
Replace nulls with blanks
edit

Replace null values with None:

def source = doc['referer'].value;
if (source != null) {
	emit(source);
	return;
}
else {
	emit("None");
}

Specify the operating system condition:

def source = doc['machine.os.keyword'].value;
if (source != "") {
	emit(source);
}
else {
	emit("None");
}

Manage runtime fields

edit

Edit the settings for runtime fields, or remove runtime fields from index patterns.

  1. Open the main menu, then click Stack Management > Index Patterns.
  2. Select the index pattern that contains the runtime field you want to manage, then open the runtime field edit options or delete the runtime field.

Add scripted fields to index patterns

edit

Deprecated in 7.13.

Use runtime fields instead of scripted fields. Runtime fields support Painless scripts and provide greater flexibility.

Scripted fields compute data on the fly from the data in your Elasticsearch indices. The data is shown on the Discover tab as part of the document data, and you can use scripted fields in your visualizations. You query scripted fields with the Kibana query language, and can filter them using the filter bar. The scripted field values are computed at query time, so they aren’t indexed and cannot be searched using the Kibana default query language.

Computing data on the fly with scripted fields can be very resource intensive and can have a direct impact on Kibana performance. Keep in mind that there’s no built-in validation of a scripted field. If your scripts are buggy, you’ll get exceptions whenever you try to view the dynamically generated data.

When you define a scripted field in Kibana, you have a choice of the Lucene expressions or the Painless scripting language.

You can reference any single value numeric field in your expressions, for example:

doc['field_name'].value

For more information on scripted fields and additional examples, refer to Using Painless in Kibana scripted fields

Create scripted fields

edit

Create and add scripted fields to your index patterns.

  1. Open the main menu, then click Stack Management > Index Patterns.
  2. Select the index pattern you want to add a scripted field to.
  3. Select the Scripted fields tab, then click Add scripted field.
  4. Enter a Name for the scripted field, then enter the Script you want to use to compute a value on the fly from your index data.
  5. Click Create field.

For more information about scripted fields in Elasticsearch, refer to Scripting.

Manage scripted fields

edit
  1. Open the main menu, then click Stack Management > Index Patterns.
  2. Select the index pattern that contains the scripted field you want to manage.
  3. Select the Scripted fields tab, then open the scripted field edit options or delete the scripted field.

Built-in validation is unsupported for scripted fields. When your scripts contain errors, you receive exceptions when you view the dynamically generated data.

Format data fields

edit

Kibana uses the same field types as Elasticsearch, however, some Elasticsearch field types are unsupported in Kibana. To customize how Kibana displays data fields, use the formatting options.

  1. Open the main menu, then click Stack Management > Index Patterns.
  2. Click the index pattern that contains the field you want to change.
  3. Find the field, then open the edit options (Data field edit icon).
  4. Select Set custom label, then enter a Custom label for the field.
  5. Select Set format, then enter the Format for the field.

String field formatters

edit

String fields support String and Url formatters.

The String field formatter enables you to apply transforms to the field.

Supported transformations include:

  • Convert to lowercase
  • Convert to uppercase
  • Convert to title case
  • Apply the short dots transformation, which replaces the content before the . character with the first character of the content. For example:

Original

Becomes

com.organizations.project.ClassName

c.o.p.ClassName

  • Base64 decode
  • URL param decode

You can specify the following types to the Url field formatter:

  • Link — Converts the contents of the field into an URL. You can specify the width and height of the image, while keeping the aspect ratio. When the image is smaller than the specified paramters, the image is unable to upscale.
  • Image — Specifies the image directory.
  • Audio — Specify the audio directory.

To customize URL field formats, use templates. An URL template enables you to add values to a partial URL. To add the contents of the field to a fixed URL, use the {{value}} string.

For example, when:

  • A field contains a user ID
  • A field uses the Url field formatter
  • The URI template is http://company.net/profiles?user_id={­{value}­}

The resulting URL replaces {{value}} with the user ID from the field.

The {{value}} template string URL-encodes the contents of the field. When a field encoded into a URL contains non-ASCII characters, the characters are replaced with a % character and the appropriate hexadecimal code. For example, field contents users/admin result in the URL template adding users%2Fadmin.

When the formatter type is Image, the {{value}} template string specifies the name of an image at the specified URI.

When the formatter type is Audio, the {{value}} template string specifies the name of an audio file at the specified URI.

To pass unescaped values directly to the URL, use the {{rawValue}} string.

A Label template enables you to specify a text string that appears instead of the raw URL. You can use the {{value}} template string normally in label templates. You can also use the {{url}} template string to display the formatted URL.

Date field formatters

edit

Date fields support Date, String, and Url formatters.

The Date formatter enables you to choose the display format of date stamps using the moment.js standard format definitions.

The String field formatter enables you to apply transforms to the field.

Supported transformations include:

  • Convert to lowercase
  • Convert to uppercase
  • Convert to title case
  • Apply the short dots transformation, which replaces the content before the . character with the first character of the content. For example:

Original

Becomes

com.organizations.project.ClassName

c.o.p.ClassName

  • Base64 decode
  • URL param decode

You can specify the following types to the Url field formatter:

  • Link — Converts the contents of the field into an URL. You can specify the width and height of the image, while keeping the aspect ratio. When the image is smaller than the specified paramters, the image is unable to upscale.
  • Image — Specifies the image directory.
  • Audio — Specify the audio directory.

To customize URL field formats, use templates. An URL template enables you to add values to a partial URL. To add the contents of the field to a fixed URL, use the {{value}} string.

For example, when:

  • A field contains a user ID
  • A field uses the Url field formatter
  • The URI template is http://company.net/profiles?user_id={­{value}­}

The resulting URL replaces {{value}} with the user ID from the field.

The {{value}} template string URL-encodes the contents of the field. When a field encoded into a URL contains non-ASCII characters, the characters are replaced with a % character and the appropriate hexadecimal code. For example, field contents users/admin result in the URL template adding users%2Fadmin.

When the formatter type is Image, the {{value}} template string specifies the name of an image at the specified URI.

When the formatter type is Audio, the {{value}} template string specifies the name of an audio file at the specified URI.

To pass unescaped values directly to the URL, use the {{rawValue}} string.

A Label template enables you to specify a text string that appears instead of the raw URL. You can use the {{value}} template string normally in label templates. You can also use the {{url}} template string to display the formatted URL.

Geographic point field formatters

edit

Geographic point fields support the String formatter.

The String field formatter enables you to apply transforms to the field.

Supported transformations include:

  • Convert to lowercase
  • Convert to uppercase
  • Convert to title case
  • Apply the short dots transformation, which replaces the content before the . character with the first character of the content. For example:

Original

Becomes

com.organizations.project.ClassName

c.o.p.ClassName

  • Base64 decode
  • URL param decode

Number field formatters

edit

Numeric fields support Bytes, Color, Duration, Histogram, Number, Percentage, String, and Url formatters.

The Bytes, Number, and Percentage formatters enable you to choose the display formats of numbers in the field using the Elastic numeral pattern syntax that Kibana maintains.

The Histogram formatter is used only for the histogram field type. When you use the Histogram formatter, you can apply the Bytes, Number, or Percentage format to aggregated data.

You can specify the following types to the Url field formatter:

  • Link — Converts the contents of the field into an URL. You can specify the width and height of the image, while keeping the aspect ratio. When the image is smaller than the specified paramters, the image is unable to upscale.
  • Image — Specifies the image directory.
  • Audio — Specify the audio directory.

To customize URL field formats, use templates. An URL template enables you to add values to a partial URL. To add the contents of the field to a fixed URL, use the {{value}} string.

For example, when:

  • A field contains a user ID
  • A field uses the Url field formatter
  • The URI template is http://company.net/profiles?user_id={­{value}­}

The resulting URL replaces {{value}} with the user ID from the field.

The {{value}} template string URL-encodes the contents of the field. When a field encoded into a URL contains non-ASCII characters, the characters are replaced with a % character and the appropriate hexadecimal code. For example, field contents users/admin result in the URL template adding users%2Fadmin.

When the formatter type is Image, the {{value}} template string specifies the name of an image at the specified URI.

When the formatter type is Audio, the {{value}} template string specifies the name of an audio file at the specified URI.

To pass unescaped values directly to the URL, use the {{rawValue}} string.

A Label template enables you to specify a text string that appears instead of the raw URL. You can use the {{value}} template string normally in label templates. You can also use the {{url}} template string to display the formatted URL.

The String field formatter enables you to apply transforms to the field.

Supported transformations include:

  • Convert to lowercase
  • Convert to uppercase
  • Convert to title case
  • Apply the short dots transformation, which replaces the content before the . character with the first character of the content. For example:

Original

Becomes

com.organizations.project.ClassName

c.o.p.ClassName

  • Base64 decode
  • URL param decode

The Duration field formatter displays the numeric value of a field in the following increments:

  • Picoseconds
  • Nanoseconds
  • Microseconds
  • Milliseconds
  • Seconds
  • Minutes
  • Hours
  • Days
  • Weeks
  • Months
  • Years

You can specify these increments with up to 20 decimal places for input and output formats.

The Color field formatter enables you to specify colors with ranges of values for a number field.

When you select the Color formatter, click Add Color, then specify the Range, Text color, and Background color.