Merge master into 7.x 2019-10-29 #14298
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…0476) (elastic#10641) Kubernetes autodiscover only emits events for containers with an ID in pods with an IP, but when a pod is being terminated, their containers can lack of ID and the pod itself can lack of IP. This leads to modules that are never stopped because the delete event that should stop them lacks of the needed information. This change makes two things to avoid this problem: * Don't require the pod to have an IP on stop events. * Use IDs for containers that don't depend on its state. (cherry picked from commit 15f2f26)
Two renamings has happened which lead to the problem: * the Filebeat module apache2 has been renamed to apache * the ID of ML modules has been suffixed with _ecs As it is still a fragile solution, I have opened an issue for the ML team to add the possibility of listing all available modules with additional meta data: elastic/kibana#30934 (cherry picked from commit 7542db1)
… signal handlers (elastic#10797) Cherry-pick of PR elastic#10694 to 7.x branch. Original message: Disable librpm signal handlers. Resolves elastic#10633.
…ics, safeguard array access (elastic#10881) Cherry-pick of PR elastic#10872 to 7.x branch. Original message: Fixes two concerns in the login dataset: 1. Since it's using an internal goroutine it should recover from any panics. 2. Puts in a safeguard for any byte arrays coming from C.
…lastic#10889) This changes the mongoDB decoder reporting unknown opcodes to report each unknown opcode only once, to avoid flooding the log file with errors. (cherry picked from commit 1e76915)
…gin types (elastic#10910) Cherry-pick of PR elastic#10865 to 7.x branch. Original message: Depending on the distro and the type of login attempt (e.g. ssh, local login) the `ut_type` value in `/var/log/btmp` is different. So far, the login dataset only responded to the rarer login type `7` (`USER_PROCESS`). The more common one (seems to be exclusively used on Fedora 29, but also used on Ubuntu 18.04 for failed SSH login attempts) is `6` (`LOGIN_PROCESS`) that we are currently ignoring. This changes the code to have a separate function to process UTMP records from btmp files that treats both `USER_PROCESS` and `LOGIN_PROCESS` the same. It also adds a unit test for failed logins including a btmp test file from Ubuntu 18.04 with three bad login attempts.
…ibrpm code compatible across CentOS 6.x, 7.x, and Fedora 29 (elastic#10842) Cherry-pick of PR elastic#10796 to 7.x branch. Original message: Librpm version 4.14.2.1 on Fedora 29 no longer contains the `headerGetEntry` method we are currently using. It was deprecated and then removed in version 4.14 (rpm-software-management/rpm@c68fa9a). Also, the much older version 4.8.0 of librpm on CentOS 6.10 (Final) does not yet contain newer data structures for tags like `rpm_tag_t/rpmTag/rpmTagVal`. This PR makes two changes that should allow this code to work on all three distros (CentOS 6.x, 7.x, Fedora 29 - and hopefully anything in between): 1. Use `headerGetString/headerGetNumber` instead of `headerGetEntry`. 2. Use `int32_t` instead of `rpm_tag_t/rpmTag/rpmTagVal`. Luckily, this seems to work on all three distros. I'd prefer something like a typedef, but unfortunately, C99 does not allow repeating a typedef (C11 does) and so backporting them is not easily possible. It also makes the code more lenient with errors during data collection: Only when no package name can be found do we return an error. Together with elastic#10694 this will hopefully allow RPM package collection to work well.
…eta (elastic#10902) Cherry-pick of PR elastic#10800 to 7.x branch. Original message: The System module has been marked as `experimental` so far. With 7.0 we're moving it to `beta`.
* Fix errors in filebeat Zeek dashboard and README files. Update field descriptions. Add notice.log support. (elastic#10916)
…#11000) Sometimes the httptest package when using fancy TLS options doesn't put the server up as fast as it should (at least that's the theory), and we hit before it's ready, causing a false test failure. This patch makes those tests more resilient. It's possible there's something else at work here, but this bug is only seen on CI, and impossible to repro on my laptop. Fixes elastic#10722 (cherry picked from commit eaf0889)
…elastic#11006) (elastic#11041) Golang's xml parser is pretty strict about the presence of control characters in the XML it is fed. This patch replaces those characters with an unicode escape sequence: "\uNNNN". (cherry picked from commit a6102a8)
…#10950) (elastic#11033) Original field name conversion was buggy. (cherry picked from commit 85e470e)
`type` is moving to `winlog.api`, but the new field was incorrectly named `winlog.type` instead.
….yml (elastic#11061) Cherry-pick of PR elastic#11016 to 7.x branch. Original message: The `add_host_metadata` processor (default since 6.5) fills the fields `host.os.build` and `host.containerized`, but there is no entry for them in `fields.yml` so they are missing from the generated index patterns. This adds them to `add_host_metadata/_meta/fields.yml`. elastic#10992 already addressed this for 6.7.
…unify bucket closing logic (elastic#11026) Cherry-pick of PR elastic#10897 to 7.x branch. Original message: The `host` dataset is erroneously trying to save state in its `Close()` method. It should have saved the state earlier - usually at the end of `Fetch()` - and then should only close the bucket (something it is not doing at all). At the same time, it is not saving state in its `reportState()` method. Combined, this can lead to an error when the dataset is terminated before the first regular `reportChanges()` is run. This fixes both issues and furthermore unifies the bucket closing logic across all six datasets of the System module.
Bug fix was missing an entry in CHANGELOG.next (cherry picked from commit b6c05a2)
…nerating index pattern (elastic#11089) Cherry-pick of PR elastic#10995 to 7.x branch. Original message: A number of fields in our `fields.yml` files now are `type: ip` (e.g. `source.ip`) and `type: boolean`, but the code generating index patterns does not know about these types yet and so does not add a `type` to the index pattern for those fields at all. This leads to errors in Kibana when looking at dashboards that contain references to those fields.
There were no significant conflicts.
…-04-03 Merge master into 7.x 2019-04-03
…11772) Cherry-pick of PR elastic#11768 to 7.x branch. Original message:
No conflicts. PR MUST BE MERGED, DO USE "Create a merge commit" OPTION!
) This reverts commit 8b5012e.
…-04-08 Merge master into 7.x 2019 04 08
Merge master into 7.x 2019-08-14
…-08-29 Merge master into 7.x 2019-08-29
…tic#13405) This reverts commit f28055f.
Merge master into 7.x 2019-09-12
Merge master into 7.x 2019-10-03
This reverts commit 58bd06d.
Merge master into 7.x 2019-10-16
…s Guide (elastic#14158) (elastic#14267)" This reverts commit d2d87b0.
|
@dedemorton please check libbeat/docs/version.asciidoc. |
dedemorton
requested changes
Oct 29, 2019
libbeat/docs/version.asciidoc
Outdated
| :go-version: 1.12.12 | ||
| :release-state: unreleased | ||
| :python: 2.7.9 | ||
| :docker: 1.12 | ||
| :docker-compose: 1.11 | ||
| :branch: 7.x |
There was a problem hiding this comment.
Delete this line - we're getting the setting for branch from the shared file in the docs repo now.
libbeat/docs/version.asciidoc
Outdated
| :go-version: 1.12.12 | ||
| :release-state: unreleased | ||
| :python: 2.7.9 | ||
| :docker: 1.12 | ||
| :docker-compose: 1.11 | ||
| :branch: 7.x | ||
| :ecs_version: 1.1 |
There was a problem hiding this comment.
Delete this line - we're getting the setting for ecs_version from the shared file in the docs repo now.
|
@urso There seems to have quite a few modules failing in this backport for filebeat. Early check this might be related to a change in the format or a difference between master ES and 7.x ES. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.