« Versioned csv codec plugin docs Csv codec plugin v1.0.0 »

› › ›

Csv codec plugin v1.1.0

edit

Csv codec plugin v1.1.0

edit

Plugin version: v1.1.0
Released on: 2021-07-28
Changelog

For other versions, see the overview list.

To learn more about Logstash, see the Logstash Reference.

Getting help

edit

For questions about the plugin, open a topic in the Discuss forums. For bugs or feature requests, open an issue in Github. For the list of Elastic supported plugins, please consult the Elastic Support Matrix.

Description

edit

The csv codec takes CSV data, parses it and passes it along.

Compatibility with the Elastic Common Schema (ECS)

edit

The plugin behaves the same regardless of ECS compatibility, except giving a warning when ECS is enabled and target isn’t set.

Set the target option to avoid potential schema conflicts.

Csv Codec configuration options

edit

Setting	Input type	Required
`autodetect_column_names`	boolean	No
`autogenerate_column_names`	boolean	No
`charset`	string, one of ["ASCII-8BIT", "UTF-8", "US-ASCII", "Big5", "Big5-HKSCS", "Big5-UAO", "CP949", "Emacs-Mule", "EUC-JP", "EUC-KR", "EUC-TW", "GB2312", "GB18030", "GBK", "ISO-8859-1", "ISO-8859-2", "ISO-8859-3", "ISO-8859-4", "ISO-8859-5", "ISO-8859-6", "ISO-8859-7", "ISO-8859-8", "ISO-8859-9", "ISO-8859-10", "ISO-8859-11", "ISO-8859-13", "ISO-8859-14", "ISO-8859-15", "ISO-8859-16", "KOI8-R", "KOI8-U", "Shift_JIS", "UTF-16BE", "UTF-16LE", "UTF-32BE", "UTF-32LE", "Windows-31J", "Windows-1250", "Windows-1251", "Windows-1252", "IBM437", "IBM737", "IBM775", "CP850", "IBM852", "CP852", "IBM855", "CP855", "IBM857", "IBM860", "IBM861", "IBM862", "IBM863", "IBM864", "IBM865", "IBM866", "IBM869", "Windows-1258", "GB1988", "macCentEuro", "macCroatian", "macCyrillic", "macGreek", "macIceland", "macRoman", "macRomania", "macThai", "macTurkish", "macUkraine", "CP950", "CP951", "IBM037", "stateless-ISO-2022-JP", "eucJP-ms", "CP51932", "EUC-JIS-2004", "GB12345", "ISO-2022-JP", "ISO-2022-JP-2", "CP50220", "CP50221", "Windows-1256", "Windows-1253", "Windows-1255", "Windows-1254", "TIS-620", "Windows-874", "Windows-1257", "MacJapanese", "UTF-7", "UTF8-MAC", "UTF-16", "UTF-32", "UTF8-DoCoMo", "SJIS-DoCoMo", "UTF8-KDDI", "SJIS-KDDI", "ISO-2022-JP-KDDI", "stateless-ISO-2022-JP-KDDI", "UTF8-SoftBank", "SJIS-SoftBank", "BINARY", "CP437", "CP737", "CP775", "IBM850", "CP857", "CP860", "CP861", "CP862", "CP863", "CP864", "CP865", "CP866", "CP869", "CP1258", "Big5-HKSCS:2008", "ebcdic-cp-us", "eucJP", "euc-jp-ms", "EUC-JISX0213", "eucKR", "eucTW", "EUC-CN", "eucCN", "CP936", "ISO2022-JP", "ISO2022-JP2", "ISO8859-1", "ISO8859-2", "ISO8859-3", "ISO8859-4", "ISO8859-5", "ISO8859-6", "CP1256", "ISO8859-7", "CP1253", "ISO8859-8", "CP1255", "ISO8859-9", "CP1254", "ISO8859-10", "ISO8859-11", "CP874", "ISO8859-13", "CP1257", "ISO8859-14", "ISO8859-15", "ISO8859-16", "CP878", "MacJapan", "ASCII", "ANSI_X3.4-1968", "646", "CP65000", "CP65001", "UTF-8-MAC", "UTF-8-HFS", "UCS-2BE", "UCS-4BE", "UCS-4LE", "CP932", "csWindows31J", "SJIS", "PCK", "CP1250", "CP1251", "CP1252", "external", "locale"]	No
`columns`	array	No
`convert`	hash	No
`ecs_compatibility`	string	No
`include_headers`	boolean	No
`quote_char`	string	No
`separator`	string	No
`skip_empty_columns`	boolean	No
`target`	string	No

`autodetect_column_names`

edit

Value type is boolean
Default value is false

Define whether column names should be auto-detected from the header column or not. Defaults to false.

`autogenerate_column_names`

edit

Value type is boolean
Default value is true

Define whether column names should be autogenerated or not. Defaults to true. If set to false, columns not having a header specified will not be parsed.

`charset`

edit

Value can be any of: ASCII-8BIT, UTF-8, US-ASCII, Big5, Big5-HKSCS, Big5-UAO, CP949, Emacs-Mule, EUC-JP, EUC-KR, EUC-TW, GB2312, GB18030, GBK, ISO-8859-1, ISO-8859-2, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-9, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14, ISO-8859-15, ISO-8859-16, KOI8-R, KOI8-U, Shift_JIS, UTF-16BE, UTF-16LE, UTF-32BE, UTF-32LE, Windows-31J, Windows-1250, Windows-1251, Windows-1252, IBM437, IBM737, IBM775, CP850, IBM852, CP852, IBM855, CP855, IBM857, IBM860, IBM861, IBM862, IBM863, IBM864, IBM865, IBM866, IBM869, Windows-1258, GB1988, macCentEuro, macCroatian, macCyrillic, macGreek, macIceland, macRoman, macRomania, macThai, macTurkish, macUkraine, CP950, CP951, IBM037, stateless-ISO-2022-JP, eucJP-ms, CP51932, EUC-JIS-2004, GB12345, ISO-2022-JP, ISO-2022-JP-2, CP50220, CP50221, Windows-1256, Windows-1253, Windows-1255, Windows-1254, TIS-620, Windows-874, Windows-1257, MacJapanese, UTF-7, UTF8-MAC, UTF-16, UTF-32, UTF8-DoCoMo, SJIS-DoCoMo, UTF8-KDDI, SJIS-KDDI, ISO-2022-JP-KDDI, stateless-ISO-2022-JP-KDDI, UTF8-SoftBank, SJIS-SoftBank, BINARY, CP437, CP737, CP775, IBM850, CP857, CP860, CP861, CP862, CP863, CP864, CP865, CP866, CP869, CP1258, Big5-HKSCS:2008, ebcdic-cp-us, eucJP, euc-jp-ms, EUC-JISX0213, eucKR, eucTW, EUC-CN, eucCN, CP936, ISO2022-JP, ISO2022-JP2, ISO8859-1, ISO8859-2, ISO8859-3, ISO8859-4, ISO8859-5, ISO8859-6, CP1256, ISO8859-7, CP1253, ISO8859-8, CP1255, ISO8859-9, CP1254, ISO8859-10, ISO8859-11, CP874, ISO8859-13, CP1257, ISO8859-14, ISO8859-15, ISO8859-16, CP878, MacJapan, ASCII, ANSI_X3.4-1968, 646, CP65000, CP65001, UTF-8-MAC, UTF-8-HFS, UCS-2BE, UCS-4BE, UCS-4LE, CP932, csWindows31J, SJIS, PCK, CP1250, CP1251, CP1252, external, locale
Default value is "UTF-8"

List of valid conversion types used for the convert option The character encoding used in this codec. Examples include "UTF-8" and "CP1252".

`columns`

edit

Value type is array
Default value is []

When decoding: Define a list of column names (in the order they appear in the CSV, as if it were a header line). If columns is not configured, or there are not enough columns specified, the default column names are "column1", "column2", etc.

When encoding: List of fields names to include in the encoded CSV, in the order listed.

`convert`

edit

Value type is hash
Default value is {}

Define a set of datatype conversions to be applied to columns. Possible conversions are: integer, float, date, date_time, boolean

Example

    filter {
      csv {
        convert => { "column1" => "integer", "column2" => "boolean" }
      }
    }

`ecs_compatibility`

edit

Value type is string
Supported values are:
- disabled: CSV data added at root level
- v1,v8: Elastic Common Schema compliant behavior ([event][original] is also added)
Default value depends on which version of Logstash is running:
- When Logstash provides a pipeline.ecs_compatibility setting, its value is used as the default
- Otherwise, the default value is disabled

Controls this plugin’s compatibility with the Elastic Common Schema (ECS).

`include_headers`

edit

Value type is boolean
Default value is false

When encoding in an output plugin, include headers in the encoded CSV once per codec lifecyle (not for every event). Default ⇒ false

`quote_char`

edit

Value type is string
Default value is "\""

Define the character used to quote CSV fields. If this is not specified the default is a double quote ". Optional.

`separator`

edit

Value type is string
Default value is ","

Define the column separator value. If this is not specified, the default is a comma ,. Optional.

`skip_empty_columns`

edit

Value type is boolean
Default value is false

Define whether empty columns should be skipped. Defaults to false. If set to true, columns containing no value will not be included.

`target`

edit

Value type is string
There is no default value for this setting.

Define the target field for placing the row values. If this setting is not set, the CSV data will be stored at the root (top level) of the event.

For example, if you want data to be put under the document field:

    input {
      file {
        codec => csv {
          autodetect_column_names => true
          target => "[document]"
        }
      }
    }

« Versioned csv codec plugin docs Csv codec plugin v1.0.0 »