Skip to content

Latest commit

 

History

History
311 lines (238 loc) · 19.8 KB

CHANGELOG.next.asciidoc

File metadata and controls

311 lines (238 loc) · 19.8 KB

Beats version HEAD

Breaking changes

Affecting all Beats

  • Make error message about locked data path actionable. 18667

  • Fix panic with inline SSL when the certificate or key were small than 256 bytes. 23820

  • Remove the deprecated xpack.monitoring. settings. Going forward only monitoring. settings may be used. 9424 18608

  • Skip add_kubernetes_metadata processor when kubernetes metadata are already present 27689

  • Remove deprecated/undocumented IncludeCreatorMetadata setting from kubernetes metadata config options 28006

  • Remove deprecated fields from kubernetes module 28046

  • Remove deprecated config option aws_partition. 28120

  • Improve stats API 27963

  • Enable IMDSv2 support for add_cloud_metadata processor on AWS. 22101 28285

  • Libbeat: logp package forces ECS compliant logs. Logs are JSON formatted. Options to enable ECS/JSON have been removed. 15544 28573

  • Previously, RE2 and thus Golang had a bug where (|a)* matched more characters than (|a)+. To stay consistent with PCRE, the bug was fixed. Configurations that rely on the old, buggy behaviour has to be adjusted. See more about Golang bug: golang/go#46123 27543

  • Update docker client. 28716

  • Remove auto from the available options of setup.ilm.enabled and set the default value to true. 28671

  • add_process_metadata processor: Replace usage of deprecated process.ppid field with process.parent.pid. 28620

  • add_docker_metadata processor: Replace usage of deprecated process.ppid field with process.parent.pid. 28620

  • Use data streams instead of indices for storing events from Beats. 28450

  • Remove option setup.template.type and always load composable template with data streams. 28450

  • Remove several ILM options (rollover_alias and pattern) as data streams does not require index aliases. 28450

  • Index template’s default_fields setting is only populated with ECS fields. 28596 28215

  • Remove deprecated --template and --ilm-policy flags. Use --index-management instead. 28870

  • Remove options logging.files.suffix and default to datetime endings. 28927

  • Remove Journalbeat. Use journald input of Filebeat instead. 29131

Auditbeat

  • File integrity dataset (macOS): Replace unnecessary file.origin.raw (type keyword) with file.origin.text (type text). 12423 15630

  • Change event.kind=error to event.kind=event to comply with ECS. 18870 20685

Filebeat

  • Preserve case of http.request.method. ECS prior to 1.6 specified normalizing to lowercase, which lost information. Affects filesets: apache/access, elasticsearch/audit, iis/access, iis/error, nginx/access, nginx/ingress_controller, aws/elb, suricata/eve, zeek/http. 18154 18359

  • Disable the option of running --machine-learning on its own. 20241

  • Add support for GMT timezone offsets in decode_cef. 20993

Heartbeat

Metricbeat

  • kubernetes.container.cpu.limit.cores and kubernetes.container.cpu.requests.cores are now floats. 11975

Packetbeat

Winlogbeat

  • Add support to Sysmon file delete events (event ID 23). 18094

  • Improve ECS field mappings in Sysmon module. related.hash, related.ip, and related.user are now populated. 18364

  • Improve ECS field mappings in Sysmon module. Hashes are now also populated to the corresponding process.hash, process.pe.imphash, file.hash, or file.pe.imphash. 18364

  • Improve ECS field mappings in Sysmon module. file.name, file.directory, and file.extension are now populated. 18364

  • Improve ECS field mappings in Sysmon module. rule.name is populated for all events when present. 18364

  • Fix unprefixed fields in fields.yml for Powershell module 18984

Functionbeat

Bugfixes

Affecting all Beats

  • Fix a race condition with the Kafka pipeline client, it is possible that Close() get called before Connect() . 11945

  • Allow users to configure only cluster_uuid setting under monitoring namespace. 14338

  • Update replicaset group to apps/v1 15802

  • Fix missing output in dockerlogbeat 15719

  • Add ssl.ca_sha256 option to the supported TLS option, this allow to check that a specific certificate is used as part of the verified chain. 15717

  • Improve some logging messages for add_kubernetes_metadata processor https://github.com/elastic/beats/pull/16866{16866}

  • Do not rotate log files on startup when interval is configured and rotateonstartup is disabled. 17613

  • Fix setup.dashboards.index setting not working. 17749

  • Fix Elasticsearch license endpoint URL referenced in error message. 17880 18030

  • Change decode_json_fields processor, to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

  • Gives monitoring reporter hosts, if configured, total precedence over corresponding output hosts. 17937 17991

  • [Autodiscover] Check if runner is already running before starting again. 18564

  • Fix regression in add_kubernetes_metadata, so configured indexers and matchers are used if defaults are not disabled. 18481 18818

  • Fix the translate_sid processor’s handling of unconfigured target fields. 18990 18991

  • Fixed a service restart failure under Windows. 18914 18916

  • Fix terminating pod autodiscover issue. 20084

  • Fix seccomp policy for calls to chmod and chown. 20054

  • Output errors when Kibana index pattern setup fails. 20121

  • Fix issue in autodiscover that kept inputs stopped after config updates. 20305

  • Add service resource in k8s cluster role. 20546

  • Periodic metrics in logs will now report libbeat.output.events.active and beat.memstats.rss as gauges (rather than counters). 22877

  • Fix discovery of Nomad allocations with multiple events during startup. 28700

  • Allows disable pod events enrichment with deployment name 28521

  • Fix fingerprint processor to give it access to the @timestamp field. 28683

  • Fix the wrong beat name on monitoring and state endpoint 27755

  • Skip configuration checks in autodiscover for configurations that are already running 29048

  • Fix decode_json_processor to always respect add_error_key 29107

  • Fix add_labels flattening of array values. 29211

  • Overwrite index name in index template correctly. 28571 29299

Auditbeat

  • system/package: Fix parsing of Installed-Size field of DEB packages. 16661 17188

  • system module: Fix panic during initialisation when /proc/stat can’t be read. 17569

  • system/package: Fix an error that can occur while trying to persist package metadata. 18536 18887

  • Fix handling of root and relative paths 24430 28354

  • system/socket: Fix bugs leading to wrong process being attributed to flows. 29166 17165

Filebeat

  • cisco/asa fileset: Fix parsing of 302021 message code. 14519

  • Fix filebeat azure dashboards, event category should be Alert. 14668

  • Fix s3 input with cloudtrail fileset reading json file. 16374 16441

  • Add queue_url definition in manifest file for aws module. https://github.com/elastic/beats/pull/16640{16640}

  • Fix elasticsearch.gc fileset to not collect all logs when Elasticsearch is running in Docker. 13164 16583 17164

  • Fixed a mapping exception when ingesting CEF logs that used the spriv or dpriv extensions. 17216 17220

  • Fix issue 17734 to retry on rate-limit error in the Filebeat httpjson input. 17734 17735

  • Remove migrationVersion map 7.7.0 reference from Kibana dashboard file to fix backward compatibility issues. 17425

  • Fixed cloudfoundry.access to have the correct cloudfoundry.app.id contents. 17847

  • Fixing ingress_controller. fields to be of type keyword instead of text. 17834

  • Fixed typo in log message. 17897

  • Fix o365 module ignoring var.api settings. 18948

  • Fix S3 input to trim delimiter /n from each log line. 19972

  • Fix s3 input parsing json file without expand_event_list_from_field. 19902 19962

  • Fix s3 input parsing json file without expand_event_list_from_field. 19902 19962 20370

  • Fix millisecond timestamp normalization issues in CrowdStrike module 20035, 20138

  • Fix support for message code 106100 in Cisco ASA and FTD. 19350 20245

  • Fix fortinet setting event.timezone to the system one when no tz field present 20273

  • Fix okta geoip lookup in pipeline for destination.ip 20454

  • Fix mapping exception in the googlecloud/audit dataset pipeline. 18465 20465

  • Fix cisco asa and ftd parsing of messages 106102 and 106103. 20469

  • Update indentation for azure filebeat configuration. 26604

  • Add support for passing a prefix on S3 bucket list mode for AWS-S3 input 28252 27965

  • Tolerate faults when Windows Event Log session is interrupted 27947 28191

  • Add support for username in cisco asa security negotiation logs 26975

  • Relax time parsing and capture group and session type in Cisco ASA module 24710 28325

  • Correctly track bytes read when max_bytes is exceeded. 28317 28352

  • Fix parsing of apache log levels including numbers. 28717

  • Upgrade azure-eventhub sdk reference, contains potential checkpoint fixes. 28919

  • Revert usageDetails api version to 2019-01-01. 28995

  • Fix in aws-s3 input regarding provider discovery through endpoint 28963

  • Fix threatintel.misp filters configuration. 27970

  • Fix opening files on Windows in filestream so open files can be deleted. 29113 29180

  • Fix handling of escaped newlines in the decode_cef processor. 16995 29268

  • Fix panw module ingest errors for GLOBALPROTECT logs 29154

Heartbeat

Metricbeat

  • Fix checking tagsFilter using length in cloudwatch metricset. 14525

  • Log bulk failures from bulk API requests to monitoring cluster. 14303 14356

  • Fix skipping protocol scheme by light modules. pull

  • Revert changes in docker module: add size flag to docker.container. 16600

  • Fix detection and logging of some error cases with light modules. 14706

  • Reduce memory usage in elasticsearch/index metricset. 16503 16538

  • Fix issue in Jolokia module when mbean contains multiple quoted properties. 17375 17374

  • Fix azure storage dashboards. 17590

  • Metricbeat no longer needs to be started strictly after Logstash for logstash-xpack module to report correct data. 17261 17497

  • Fix pubsub metricset to collect all GA stage metrics from gcp stackdriver. 17154 17600

  • Add privileged option so as mb to access data dir in Openshift. 17606

  • Fix "ID" event generator of Google Cloud module 17160 17608

  • Add privileged option for Auditbeat in Openshift 17637

  • Fix storage metricset to allow config without region/zone. 17623 17624

  • Fix overflow on Prometheus rates when new buckets are added on the go. 17753

  • Remove specific win32 api errors from events in perfmon. 18292 18361

  • Remove required for region/zone and make stackdriver a metricset in googlecloud. 16785 18398

  • Fix application_pool metricset after pdh changes. 18477

  • Fix panic on metricbeat test modules when modules are configured in metricbeat.modules. 18789 18797

  • Fix getting gcp compute instance metadata with partial zone/region in config. 18757

  • Add missing network.sent_packets_count metric into compute metricset in googlecloud module. 18802

  • Fix compute and pubsub dashboard for googlecloud module. 18962 18980

  • Fix crash on vsphere module when Host information is not available. 18996 19078

  • Modify doc for app_insights metricset to contain example of config. 20185

  • Add required option for metrics in app_insights. 20406

  • Groups same timestamp metric values to one event in the app_insights metricset. 20403

Packetbeat

Winlogbeat

Functionbeat

Elastic Logging Plugin

Added

Affecting all Beats

  • Decouple Debug logging from fail_on_error logic for rename, copy, truncate processors 12451

  • Fingerprint processor adds a new xxhash hashing algorithm 15418

  • Update documentation for system.process.memory fields to include clarification on Windows os’s. 17268

  • Add keystore support for autodiscover static configurations. {pull]16306[16306]

  • When using the decode_json_fields processor, decoded fields are now deep-merged into existing event. 17958

  • Add keystore support for autodiscover static configurations. {pull]16306[16306]

  • Add TLS support to Kerberos authentication in Elasticsearch. 18607

  • Set index.max_docvalue_fields_search in index template to increase value to 200 fields. 20215

Auditbeat

  • Reference kubernetes manifests include configuration for auditd and enrichment with kubernetes metadata. 17431

Filebeat

  • container and docker inputs now support reading of labels and env vars written by docker JSON file logging driver. 8358

  • Add index option to all inputs to directly set a per-input index value. 14010

  • move create-[module,fileset,fields] to mage and enable in x-pack/filebeat 15836

  • Work on e2e ACK’s for the azure-eventhub input 15671 16215

  • Add a TLS test and more debug output to httpjson input 16315

  • Add an SSL config example in config.yml for filebeat MISP module. 16320

  • Added documentation for running Filebeat in Cloud Foundry. 17275

  • Release Google Cloud module as GA. 17511

  • Update filebeat httpjson input to support pagination via Header and Okta module. 16354

  • Change the json.* input settings implementation to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

  • Add support for array parsing in azure-eventhub input. 18585

  • Improved performance of PANW sample dashboards. 19031 19032

  • Add event.ingested for CrowdStrike module 20138

  • Add support for additional fields and FirewallMatchEvent type events in CrowdStrike module 20138

  • Added feature to modules to adapt Ingest Node pipelines for compatibility with older Elasticsearch versions by

  • Add timezone config option to the decode_cef processor. 27232 27727

  • Add timezone config option to the syslog input. 27727

  • Added support for parsing syslog dates containing a leading 0 (e.g. Sep 01) rather than a space. 27775

  • Add base64 Encode functionality to httpjson input. 27681

  • Add join and sprintf functions to httpjson input. 27735

  • Improve memory usage of line reader of log and filestream input. 27782

  • Add ignore_empty_value flag to httpjson split processor. 27880

  • Update Cisco ASA/FTD ingest pipeline grok/dissect patterns for multiple message IDs. 26869 26879

  • Add write access to url.value from request.transforms in httpjson input. 27937

  • Add Base64 encoded HMAC and UUID template functions to httpjson input 27873

  • Release checkpoint module as GA. 27814

  • Make aws-cloudwatch input GA. 28161

  • Move processing to ingest node for AWS vpcflow fileset. 28168

  • Release zoom module as GA. 28106

  • Add support for secondary object attribute handling in ThreatIntel MISP module 28124

  • Azure signinlogs - Add support for ManagedIdentitySignInLogs, NonInteractiveUserSignInLogs, and ServicePrincipalSignInLogs. 23653

  • Add base64Decode and base64DecodeNoPad functions to httpsjon templates. 28385

  • Add 'early_limit' config option for Rate-Limiting httpjson. Default rate-limiting for Okta will start when remaining is 1. 28513

  • Add latency config option for aws-cloudwatch input. 28509

  • Added proxy support to threatintel/malwarebazaar. 28533

  • Add text/csv decoder to httpjson input 28564

  • Update aws-s3 input to connect to non AWS S3 buckets 28222 28234

  • Sophos UTM: Support logs containing hostname in syslog header. 28638

  • Moving Oracle Filebeat module to GA. 28754

  • Add support for '/var/log/pods/' path for add_kubernetes_metadata processor with resource_type: pod. 28868

  • Add documentation for add_kubernetes_metadata processors log_path matcher. 28868

  • Add support in aws-s3 input for s3 notification from SNS to SQS. 28800

  • Add support in aws-s3 input for custom script parsing of s3 notifications. 28946

  • Improve error handling in aws-s3 input for malformed s3 notifications. 28828 28946

  • Add support for parsers on journald input 29070

  • Add elapsed time information to aws-s3 input errors and log messages. 29328

  • Add support in httpjson input for oAuth2ProviderDefault of password grant_type. 29087

Heartbeat

Metricbeat

  • Move the windows pdh implementation from perfmon to a shared location in order for future modules/metricsets to make use of. 15503

  • Add key/value mode for SQL module. 15770 {pull]15845[15845]

  • Add database_account azure metricset. 15758

  • Release Zookeeper/connection module as GA. 14281 17043

  • Add dashboard for pubsub metricset in googlecloud module. 17161

  • Added documentation for running Metricbeat in Cloud Foundry. 17275

  • Add memory metrics into compute googlecloud. 18802

Packetbeat

Functionbeat

Winlogbeat

  • Add configuration option for registry file flush timeout 29001 29053

Elastic Log Driver

Deprecated

Affecting all Beats

Filebeat

Heartbeat

Metricbeat

Packetbeat

Winlogbeat

Functionbeat

Known Issue

Journalbeat