Breaking changes in 7.0

edit

This section discusses the main changes that you should be aware of if you upgrade the Beats to version 7.0. See the release notes for a complete list of breaking changes, including changes to beta or experimental functionality.

HTML escaping is disabled by default

edit

Starting with verion 7.0, embedded HTML or special symbols like < and > are no longer escaped by default when publishing events. To configure the old behavior of escaping HTML, set escape_html: true in the output configuration.

Filebeat registry

edit

Starting with version 7.0, Filebeat stores the registry in a sub-directory. The directory is configured using the filebeat.registry.path setting. If Filebeat finds an old registry file at the configured location, it will automatically migrate the registry file to the new format.

The settings filebeat.registry_flush and filebeat.registry_file_permission have been renamed to filebeat.registry.flush and filebeat.registry.file_permission.

ILM support

edit

Support for Index Lifecycle Management is GA with Beats version 7.0. This release moved most ILM settings from the output.elasticsearch.ilm namespace to the setup.ilm namespace.

Filebeat apache2 module renamed

edit

The Filebeat apache2 module is renamed to apache in 7.0.

Field name changes

edit

Table 1. Auditbeat renamed fields in 7.0

Old Field New Field

auditd.messages

event.original

auditd.warnings

error.message

beat.hostname

agent.hostname

beat.name

host.name

beat.timezone

event.timezone

beat.version

agent.version

docker.container.id

container.id

docker.container.image

container.image.name

docker.container.labels

container.labels

docker.container.name

container.name

event.type

auditd.message_type

meta.cloud.availability_zone

cloud.availability_zone

meta.cloud.instance_id

cloud.instance.id

meta.cloud.instance_name

cloud.instance.name

meta.cloud.machine_type

cloud.machine.type

meta.cloud.project_id

cloud.project.id

meta.cloud.provider

cloud.provider

meta.cloud.region

cloud.region

process.cwd

process.working_directory

process.exe

process.executable

source.hostname

source.domain

user.auid

user.audit.id

user.egid

user.effective.group.id

user.euid

user.effective.id

user.fsgid

user.filesystem.group.id

user.fsuid

user.filesystem.id

user.gid

user.group.id

user.name_map.auid

user.audit.name

user.name_map.egid

user.effective.group.name

user.name_map.euid

user.effective.name

user.name_map.fsgid

user.filesystem.group.name

user.name_map.fsuid

user.filesystem.name

user.name_map.gid

user.group.name

user.name_map.sgid

user.saved.group.name

user.name_map.suid

user.saved.name

user.name_map.uid

user.name

user.sgid

user.saved.group.id

user.suid

user.saved.id

user.uid

user.id

Table 2. Filebeat renamed fields in 7.0

Old Field New Field

apache2.access.agent

user_agent.original

apache2.access.body_sent.bytes

http.response.body.bytes

apache2.access.geoip.city_name

source.geo.city_name

apache2.access.geoip.continent_name

source.geo.continent_name

apache2.access.geoip.country_iso_code

source.geo.country_iso_code

apache2.access.geoip.location

source.geo.location

apache2.access.geoip.region_iso_code

source.geo.region_iso_code

apache2.access.geoip.region_name

source.geo.region_name

apache2.access.http_version

http.version

apache2.access.method

http.request.method

apache2.access.referrer

http.request.referrer

apache2.access.remote_ip

source.address

apache2.access.response_code

http.response.status_code

apache2.access.url

url.original

apache2.access.user_agent.device

user_agent.device.name

apache2.access.user_agent.major

user_agent.version

apache2.access.user_agent.minor

user_agent.version

apache2.access.user_agent.name

user_agent.name

apache2.access.user_agent.original

user_agent.original

apache2.access.user_agent.os

user_agent.os.full_name

apache2.access.user_agent.os_major

user_agent.os.version

apache2.access.user_agent.os_minor

user_agent.os.version

apache2.access.user_agent.os_name

user_agent.os.name

apache2.access.user_agent.os_patch

user_agent.os.version

apache2.access.user_agent.patch

user_agent.version

apache2.access.user_name

user.name

apache2.error.client

source.address

apache2.error.level

log.level

apache2.error.message

message

apache2.error.pid

process.pid

apache2.error.tid

process.thread.id

auditd.log.acct

user.name

auditd.log.agid

user.audit.group.id

auditd.log.arch

host.architecture

auditd.log.auid

user.audit.id

auditd.log.cmd

process.args

auditd.log.comm

process.name

auditd.log.dst

destination.address

auditd.log.egid

user.effective.group.id

auditd.log.euid

user.effective.id

auditd.log.exe

process.executable

auditd.log.fsgid

user.filesystem.group.id

auditd.log.geoip.city_name

source.geo.city_name

auditd.log.geoip.continent_name

source.geo.continent_name

auditd.log.geoip.country_iso_code

source.geo.country_iso_code

auditd.log.geoip.location

source.geo.location

auditd.log.geoip.region_iso_code

source.geo.region_iso_code

auditd.log.geoip.region_name

source.geo.region_name

auditd.log.gid

user.group.id

auditd.log.msg

message

auditd.log.ogid

user.owner.group.id

auditd.log.ouid

user.owner.id

auditd.log.pid

process.pid

auditd.log.ppid

process.ppid

auditd.log.record_type

event.action

auditd.log.res

event.outcome

auditd.log.sgid

user.saved.group.id

auditd.log.src

source.address

auditd.log.suid

user.saved.id

auditd.log.terminal

user.terminal

auditd.log.uid

user.id

beat.hostname

agent.hostname

beat.name

host.name

beat.timezone

event.timezone

beat.version

agent.version

docker.container.id

container.id

docker.container.image

container.image.name

docker.container.labels

container.labels

docker.container.name

container.name

elasticsearch.audit.origin_address

source.ip

elasticsearch.audit.principal

user.name

elasticsearch.audit.request_body

http.request.body.content

elasticsearch.audit.uri

url.original

elasticsearch.slowlog.took_millis

event.duration

fileset.module

event.module

haproxy.client.ip

source.address

haproxy.client.port

source.port

haproxy.destination.ip

destination.ip

haproxy.destination.port

destination.port

haproxy.geoip.city_name

source.geo.city_name

haproxy.geoip.continent_name

source.geo.continent_name

haproxy.geoip.country_iso_code

source.geo.country_iso_code

haproxy.geoip.location

source.geo.location

haproxy.geoip.region_iso_code

source.geo.region_iso_code

haproxy.geoip.region_name

source.geo.region_name

haproxy.http.request.time_active_ms

event.duration

haproxy.http.response.status_code

http.response.status_code

haproxy.pid

process.pid

haproxy.process_name

process.name

haproxy.total_waiting_time_ms

event.duration

http.response.content_length

http.response.body.bytes

http.response.elapsed_time

event.duration

icinga.debug.message

message

icinga.debug.severity

log.level

icinga.main.message

message

icinga.main.severity

log.level

icinga.startup.message

message

icinga.startup.severity

log.level

iis.access.body_received.bytes

http.request.body.bytes

iis.access.body_sent.bytes

http.response.body.bytes

iis.access.geoip.city_name

source.geo.city_name

iis.access.geoip.continent_name

source.geo.continent_name

iis.access.geoip.country_iso_code

source.geo.country_iso_code

iis.access.geoip.location

source.geo.location

iis.access.geoip.region_iso_code

source.geo.region_iso_code

iis.access.geoip.region_name

source.geo.region_name

iis.access.hostname

destination.domain

iis.access.method

http.request.method

iis.access.port

destination.port

iis.access.query_string

url.query

iis.access.referrer

http.request.referrer

iis.access.remote_ip

source.address

iis.access.request_time_ms

event.duration

iis.access.response_code

http.response.status_code

iis.access.server_ip

destination.address

iis.access.url

url.path

iis.access.user_agent.device

user_agent.device.name

iis.access.user_agent.major

user_agent.version

iis.access.user_agent.minor

user_agent.version

iis.access.user_agent.name

user_agent.name

iis.access.user_agent.original

user_agent.original

iis.access.user_agent.os

user_agent.os.full_name

iis.access.user_agent.os_major

user_agent.os.version

iis.access.user_agent.os_minor

user_agent.os.version

iis.access.user_agent.os_name

user_agent.os.name

iis.access.user_agent.os_patch

user_agent.os.version

iis.access.user_agent.patch

user_agent.version

iis.access.user_name

user.name

iis.error.geoip.city_name

source.geo.city_name

iis.error.geoip.continent_name

source.geo.continent_name

iis.error.geoip.country_iso_code

source.geo.country_iso_code

iis.error.geoip.location

source.geo.location

iis.error.geoip.region_iso_code

source.geo.region_iso_code

iis.error.geoip.region_name

source.geo.region_name

iis.error.http_version

http.version

iis.error.method

http.request.method

iis.error.remote_ip

source.address

iis.error.remote_port

source.port

iis.error.response_code

http.response.status_code

iis.error.server_ip

destination.address

iis.error.server_port

destination.port

iis.error.url

url.original

kafka.log.level

log.level

kafka.log.message

message

kibana.log.meta.meta.statusCode

http.response.status_code

kibana.log.meta.method

http.request.method

kibana.log.meta.req.headers.referer

http.request.referrer

kibana.log.meta.req.headers.user-agent

user_agent.original

kibana.log.meta.req.referer

http.request.referrer

kibana.log.meta.req.remoteAddress

source.address

kibana.log.meta.req.url

url.original

logstash.log.level

log.level

logstash.log.message

message

logstash.slowlog.level

log.level

logstash.slowlog.took_in_nanos

event.duration

meta.cloud.availability_zone

cloud.availability_zone

meta.cloud.instance_id

cloud.instance.id

meta.cloud.instance_name

cloud.instance.name

meta.cloud.machine_type

cloud.machine.type

meta.cloud.project_id

cloud.project.id

meta.cloud.provider

cloud.provider

meta.cloud.region

cloud.region

mongodb.log.message

message

mongodb.log.severity

log.level

mysql.error.level

log.level

mysql.error.message

message

mysql.error.thread_id

mysql.thread_id

mysql.slowlog.host

source.domain

mysql.slowlog.id

mysql.thread_id

mysql.slowlog.ip

source.ip

mysql.slowlog.query_time.sec

event.duration

mysql.slowlog.user

user.name

nginx.access.agent

user_agent.original

nginx.access.body_sent.bytes

http.response.body.bytes

nginx.access.geoip.city_name

source.geo.city_name

nginx.access.geoip.continent_name

source.geo.continent_name

nginx.access.geoip.country_iso_code

source.geo.country_iso_code

nginx.access.geoip.location

source.geo.location

nginx.access.geoip.region_iso_code

source.geo.region_iso_code

nginx.access.geoip.region_name

source.geo.region_name

nginx.access.http_version

http.version

nginx.access.method

http.request.method

nginx.access.referrer

http.request.referrer

nginx.access.remote_ip

source.address

nginx.access.response_code

http.response.status_code

nginx.access.url

url.original

nginx.access.user_agent.device

user_agent.device.name

nginx.access.user_agent.major

user_agent.version

nginx.access.user_agent.minor

user_agent.version

nginx.access.user_agent.name

user_agent.name

nginx.access.user_agent.os

user_agent.os.full_name

nginx.access.user_agent.os_major

user_agent.os.version

nginx.access.user_agent.os_minor

user_agent.os.version

nginx.access.user_agent.os_name

user_agent.os.name

nginx.access.user_agent.os_patch

user_agent.os.version

nginx.access.user_agent.patch

user_agent.version

nginx.access.user_name

user.name

nginx.error.level

log.level

nginx.error.message

message

nginx.error.pid

process.pid

nginx.error.tid

process.thread.id

offset

log.offset

postgresql.log.duration

event.duration

postgresql.log.level

log.level

postgresql.log.message

message

postgresql.log.thread_id

process.pid

postgresql.log.timezone

event.timezone

postgresql.log.user

user.name

process.exe

process.executable

read_timestamp

event.created

redis.log.level

log.level

redis.log.message

message

redis.log.pid

process.pid

source_ecs.geo.city_name

source.geo.city_name

source_ecs.geo.continent_name

source.geo.continent_name

source_ecs.geo.country_iso_code

source.geo.country_iso_code

source_ecs.geo.location

source.geo.location

source_ecs.geo.region_iso_code

source.geo.region_iso_code

source_ecs.geo.region_name

source.geo.region_name

source_ecs.ip

source.ip

source_ecs.port

source.port

suricata.eve.alert.action

event.outcome

suricata.eve.alert.severity

event.severity

suricata.eve.app_proto

network.protocol

suricata.eve.dest_ip

destination.ip

suricata.eve.dest_port

destination.port

suricata.eve.fileinfo.filename

file.path

suricata.eve.fileinfo.size

file.size

suricata.eve.flow.bytes_toclient

destination.bytes

suricata.eve.flow.bytes_toserver

source.bytes

suricata.eve.flow.pkts_toclient

destination.packets

suricata.eve.flow.pkts_toserver

source.packets

suricata.eve.flow.start

event.start

suricata.eve.http.hostname

url.domain

suricata.eve.http.http_method

http.request.method

suricata.eve.http.http_refer

http.request.referrer

suricata.eve.http.http_user_agent

user_agent.original

suricata.eve.http.length

http.response.body.bytes

suricata.eve.http.status

http.response.status_code

suricata.eve.http.url

url.original

suricata.eve.proto

network.transport

suricata.eve.src_ip

source.ip

suricata.eve.src_port

source.port

suricata.eve.timestamp

@timestamp

system.auth.groupadd.gid

group.id

system.auth.groupadd.name

group.name

system.auth.hostname

host.hostname

system.auth.message

message

system.auth.pid

process.pid

system.auth.program

process.name

system.auth.ssh.geoip.city_name

source.geo.city_name

system.auth.ssh.geoip.continent_name

source.geo.continent_name

system.auth.ssh.geoip.country_iso_code

source.geo.country_iso_code

system.auth.ssh.geoip.location

source.geo.location

system.auth.ssh.geoip.region_iso_code

source.geo.region_iso_code

system.auth.ssh.geoip.region_name

source.geo.region_name

system.auth.ssh.ip

source.ip

system.auth.ssh.port

source.port

system.auth.timestamp

@timestamp

system.auth.user

user.name

system.auth.useradd.gid

group.id

system.auth.useradd.name

user.name

system.auth.useradd.uid

user.id

system.syslog.hostname

host.hostname

system.syslog.message

message

system.syslog.pid

process.pid

system.syslog.program

process.name

traefik.access.agent

user_agent.original

traefik.access.body_sent.bytes

http.response.body.bytes

traefik.access.duration

event.duration

traefik.access.geoip.city_name

source.geo.city_name

traefik.access.geoip.continent_name

source.geo.continent_name

traefik.access.geoip.country_iso_code

source.geo.country_iso_code

traefik.access.geoip.location

source.geo.location

traefik.access.geoip.region_iso_code

source.geo.region_iso_code

traefik.access.geoip.region_name

source.geo.region_name

traefik.access.http_version

http.version

traefik.access.method

http.request.method

traefik.access.referrer

http.request.referrer

traefik.access.remote_ip

source.address

traefik.access.response_code

http.response.status_code

traefik.access.url

url.original

traefik.access.user_agent.device

user_agent.device.name

traefik.access.user_agent.major

user_agent.version

traefik.access.user_agent.minor

user_agent.version

traefik.access.user_agent.name

user_agent.name

traefik.access.user_agent.original

user_agent.original

traefik.access.user_agent.os

user_agent.os.full_name

traefik.access.user_agent.os_major

user_agent.os.version

traefik.access.user_agent.os_minor

user_agent.os.version

traefik.access.user_agent.os_name

user_agent.os.name

traefik.access.user_agent.os_patch

user_agent.os.version

traefik.access.user_agent.patch

user_agent.version

traefik.access.user_name

user.name

Table 3. Heartbeat renamed fields in 7.0

Old Field New Field

beat.hostname

agent.hostname

beat.name

host.name

beat.timezone

event.timezone

beat.version

agent.version

docker.container.id

container.id

docker.container.image

container.image.name

docker.container.labels

container.labels

docker.container.name

container.name

http.url

url.full

meta.cloud.availability_zone

cloud.availability_zone

meta.cloud.instance_id

cloud.instance.id

meta.cloud.instance_name

cloud.instance.name

meta.cloud.machine_type

cloud.machine.type

meta.cloud.project_id

cloud.project.id

meta.cloud.provider

cloud.provider

meta.cloud.region

cloud.region

monitor.host

url.domain

monitor.scheme

url.scheme

process.exe

process.executable

resolve.host

url.domain

tcp.port

url.port

Table 4. Journalbeat renamed fields in 7.0

Old Field New Field

beat.hostname

agent.hostname

beat.name

host.name

beat.timezone

event.timezone

beat.version

agent.version

docker.container.id

container.id

docker.container.image

container.image.name

docker.container.labels

container.labels

docker.container.name

container.name

host.name

host.hostname

meta.cloud.availability_zone

cloud.availability_zone

meta.cloud.instance_id

cloud.instance.id

meta.cloud.instance_name

cloud.instance.name

meta.cloud.machine_type

cloud.machine.type

meta.cloud.project_id

cloud.project.id

meta.cloud.provider

cloud.provider

meta.cloud.region

cloud.region

process.exe

process.executable

read_timestamp

event.created

Table 5. Metricbeat renamed fields in 7.0

Old Field New Field

beat.hostname

agent.hostname

beat.name

host.name

beat.timezone

event.timezone

beat.version

agent.version

docker.container.id

container.id

docker.container.image

container.image.name

docker.container.labels

container.labels

docker.container.name

container.name

haproxy.info.pid

process.pid

haproxy.stat.process_id

process.pid

http.request.body

http.request.body.content

kibana.stats.transport_address

service.address

kibana.stats.uuid

service.id

kibana.stats.version

service.version

kibana.status.uuid

service.id

kibana.status.version.number

service.version

logstash.node.host

service.hostname

logstash.node.jvm.pid

process.pid

logstash.node.version

service.version

meta.cloud.availability_zone

cloud.availability_zone

meta.cloud.instance_id

cloud.instance.id

meta.cloud.instance_name

cloud.instance.name

meta.cloud.machine_type

cloud.machine.type

meta.cloud.project_id

cloud.project.id

meta.cloud.provider

cloud.provider

meta.cloud.region

cloud.region

metricset.host

service.address

metricset.module

event.module

metricset.namespace

event.dataset

metricset.rrt

event.duration

mongodb.status.process

process.name

mongodb.status.version

service.version

php_fpm.status.content_length

http.response.body.bytes

php_fpm.status.pid

process.pid

php_fpm.status.request_method

http.request.method

php_fpm.status.request_uri

url.original

php_fpm.status.user

http.response.user.name

process.exe

process.executable

rabbitmq.connection.node

rabbitmq.node.name

rabbitmq.connection.user

user.name

rabbitmq.connection.vhost

rabbitmq.vhost

rabbitmq.exchange.user

user.name

rabbitmq.exchange.vhost

rabbitmq.vhost

rabbitmq.queue.node

rabbitmq.node.name

rabbitmq.queue.vhost

rabbitmq.vhost

redis.info.server.os

os.full

redis.info.server.process_id

process.pid

redis.info.server.version

service.version

system.process.cwd

process.working_directory

system.process.name

process.name

system.process.pgid

process.pgid

system.process.pid

process.pid

system.process.ppid

process.ppid

system.process.username

user.name

system.socket.direction

network.direction

system.socket.family

network.type

system.socket.process.command

process.name

system.socket.process.exe

process.executable

system.socket.process.pid

process.pid

system.socket.user.id

user.id

system.socket.user.name

user.full_name

zookeeper.mntr.version

service.version

Table 6. Packetbeat renamed fields in 7.0

Old Field New Field

beat.hostname

agent.hostname

beat.name

host.name

beat.timezone

event.timezone

beat.version

agent.version

bytes_in

source.bytes

bytes_out

destination.bytes

dest.stats.net_bytes_total

destination.bytes

dest.stats.net_packets_total

destination.packets

docker.container.id

container.id

docker.container.image

container.image.name

docker.container.labels

container.labels

docker.container.name

container.name

final

flow.final

flow_id

flow.id

http.request.body

http.request.body.content

http.request.params

url.query

http.response.body

http.response.body.content

http.response.code

http.response.status_code

http.response.phrase

http.response.status_phrase

last_time

event.end

meta.cloud.availability_zone

cloud.availability_zone

meta.cloud.instance_id

cloud.instance.id

meta.cloud.instance_name

cloud.instance.name

meta.cloud.machine_type

cloud.machine.type

meta.cloud.project_id

cloud.project.id

meta.cloud.provider

cloud.provider

meta.cloud.region

cloud.region

method

http.request.method

mysql.iserror

status

no_request

cassandra.no_request

notes

error.message

path

url.path

process.exe

process.executable

real_ip

network.forwarded_ip

responsetime

event.duration

rpc.call_size

source.bytes

rpc.reply_size

destination.bytes

rpc.time

event.duration

source.stats.net_bytes_total

source.bytes

source.stats.net_packets_total

source.packets

start_time

event.start

transport

network.transport

Table 7. Winlogbeat renamed fields in 7.0

Old Field New Field

activity_id

winlog.activity_id

beat.hostname

agent.hostname

beat.name

host.name

beat.timezone

event.timezone

beat.version

agent.version

computer_name

winlog.computer_name

docker.container.id

container.id

docker.container.image

container.image.name

docker.container.labels

container.labels

docker.container.name

container.name

event_id

winlog.event_id

keywords

winlog.keywords

level

log.level

log_name

winlog.channel

message_error

error.message

meta.cloud.availability_zone

cloud.availability_zone

meta.cloud.instance_id

cloud.instance.id

meta.cloud.instance_name

cloud.instance.name

meta.cloud.machine_type

cloud.machine.type

meta.cloud.project_id

cloud.project.id

meta.cloud.provider

cloud.provider

meta.cloud.region

cloud.region

opcode

winlog.opcode

process.exe

process.executable

process_id

winlog.process.pid

provider_guid

winlog.provider_guid

record_number

winlog.record_id

related_activity_id

winlog.related_activity_id

source_name

winlog.provider_name

task

winlog.task

thread_id

winlog.process.thread.id

type

winlog.api

user.domain

winlog.user.domain

user.identifier

winlog.user.identifier

user.type

winlog.user.type

version

winlog.version

xml

event.original

Auditbeat type changes

edit

The Auditbeat JSON data types produced by the output have been changed to align with the data types used in the Elasticsearch index template.

Table 8. Auditbeat Type Changes in 7.0

Field Old Type New Type

file.gid

number

string

file.uid

number

string

process.pid

string

number

process.ppid

string

number