[DOCS] Add output.console to Functionbeat doc and Functionbeat reference file #18965

EamonnTP · 2020-06-04T09:28:25Z

What does this PR do?

This PR adds the output.console content to the Functionbeat doc and to the functionbeat.reference.yml file.

Why is it important?

output.console was added in 7.6.0 to help users debug, but it was not documented. While this new output mode is intended for troubleshooting/debugging only, it can still benefit developers/admins. The functionbeat.reference.yml file is also updated so developers know that the console output configuration exists.

Related issues

Closes #16855

HTML output

Coming soon. Doc build is currently failing and requires further investigation.

elasticmachine · 2020-06-04T09:49:55Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #18965 updated]
Start Time: 2020-06-11T20:24:43.126+0000
Duration: 78 min 44 sec

Test stats 🧪

Test	Results
Failed	0
Passed	9437
Skipped	1574
Total	11011

Steps errors

Expand to view the steps failures

Name: Report to Codecov
- Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done
- Duration: 1 min 17 sec
- Start Time: 2020-06-11T21:18:31.477+0000
- log
Name: Report to Codecov
- Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done
- Duration: 2 min 22 sec
- Start Time: 2020-06-11T21:05:25.433+0000
- log

elasticmachine · 2020-06-09T13:32:45Z

Pinging @elastic/integrations-services (Team:Services)

bmorelli25 · 2020-06-09T16:22:44Z

@EamonnTP Merge this branch with master and push to retry the unsuccessful checks

bmorelli25

Other than the failing tests, this LGTM. I see both the console output and codec documentation in the HTML preview.

https://beats_18965.docs-preview.app.elstc.co/guide/en/beats/functionbeat/master/configuration-output-codec.html

Retry unsuccessful checks

bmorelli25 · 2020-06-09T20:37:37Z

Here's the error from beats-ci/pr-merge

[2020-06-09T20:25:50.292Z] diff --git a/x-pack/functionbeat/functionbeat.reference.yml b/x-pack/functionbeat/functionbeat.reference.yml
[2020-06-09T20:25:50.292Z] index 0a3d6c5..61cf8a2 100644
[2020-06-09T20:25:50.292Z] --- a/x-pack/functionbeat/functionbeat.reference.yml
[2020-06-09T20:25:50.292Z] +++ b/x-pack/functionbeat/functionbeat.reference.yml
[2020-06-09T20:25:50.292Z] @@ -927,19 +927,9 @@ output.elasticsearch:
[2020-06-09T20:25:50.292Z]    # timing out. The default is 30s.
[2020-06-09T20:25:50.292Z]    #timeout: 30s
[2020-06-09T20:25:50.292Z]  
[2020-06-09T20:25:50.292Z] -# =============================== Console Output ===============================
[2020-06-09T20:25:50.292Z]  
[2020-06-09T20:25:50.292Z] -#output.console:
[2020-06-09T20:25:50.292Z] -  # Boolean flag to enable or disable the output module.
[2020-06-09T20:25:50.292Z] -  #enabled: true
[2020-06-09T20:25:50.292Z]  
[2020-06-09T20:25:50.292Z] -  # Configure JSON encoding
[2020-06-09T20:25:50.292Z] -  #codec.json:
[2020-06-09T20:25:50.292Z] -    # Pretty-print JSON event
[2020-06-09T20:25:50.292Z] -    #pretty: false
[2020-06-09T20:25:50.292Z]  
[2020-06-09T20:25:50.292Z] -    # Configure escaping HTML symbols in strings.
[2020-06-09T20:25:50.292Z] -    #escape_html: false
[2020-06-09T20:25:50.292Z]  
[2020-06-09T20:25:50.292Z]  # =================================== Paths ====================================
[2020-06-09T20:25:50.292Z]  
[2020-06-09T20:25:50.292Z] x-pack/functionbeat/functionbeat.reference.yml: needs update

@dedemorton do you know what's wrong?

dedemorton · 2020-06-09T22:14:33Z

Aha. So I should have caught this during the review, but it's been awhile since I've updated the config reference, and I'd forgotten that they're now using templating to create the reference files.

So instead of manually updating the functionbeat.reference.yml, you need to make a code change. Remove this line:

https://github.com/elastic/beats/blob/master/x-pack/functionbeat/scripts/mage/config.go#L16

Then run make update from the beats directory. Note that you need to have your dev
environment set up correctly to run this command.

dedemorton · 2020-06-09T22:19:20Z

(or I guess you could set it to false but I don't see it set anywhere else, so removing probably makes more sense).

ppf2 · 2020-06-10T00:18:15Z

One thing we might want to add (open for discussion) is a warning about output.console to be used for debugging only for it can produce a large amounts of logging (this can translate to $$ if the output of the logging ends up in Cloudwatch logs). Not sure if we need to explicitly call this out though or if it's implied :)

urso · 2020-06-10T12:41:13Z

One thing we might want to add (open for discussion) is a warning about output.console to be used for debugging only for it can produce a large amounts of logging (this can translate to $$ if the output of the logging ends up in Cloudwatch logs). Not sure if we need to explicitly call this out though or if it's implied :)

I'd be in favor of adding a warning.

urso · 2020-06-11T10:37:27Z

libbeat/outputs/console/docs/console.asciidoc

@@ -7,6 +7,8 @@

 The Console output writes events in JSON format to stdout.


One can configure a custom formatter to print messages with a custom layout. JSON is just the default formatter (see codec setting).

Kick off tests

…nce file (elastic#18965) * Add outputconsole to Functionbeat doc * Remove no_codec attirbute * Updates following review # Conflicts: # libbeat/outputs/console/docs/console.asciidoc

…nce file (elastic#18965) * Add outputconsole to Functionbeat doc * Remove no_codec attirbute * Updates following review # Conflicts: # libbeat/outputs/console/docs/console.asciidoc # x-pack/functionbeat/functionbeat.reference.yml # x-pack/functionbeat/scripts/mage/config.go

…nce file (elastic#18965) * Add outputconsole to Functionbeat doc * Remove no_codec attirbute * Updates following review # Conflicts: # libbeat/outputs/console/docs/console.asciidoc

…nce file (elastic#18965) * Add outputconsole to Functionbeat doc * Remove no_codec attirbute * Updates following review # Conflicts: # libbeat/outputs/console/docs/console.asciidoc # x-pack/functionbeat/functionbeat.reference.yml # x-pack/functionbeat/scripts/mage/config.go

…ngs-archive * upstream/master: (119 commits) Update filebeat input docs (elastic#19110) Add ECS fields from log pipeline of PostgreSQL (elastic#19127) Init package libbeat/statestore (elastic#19117) [Ingest Manager] Retryable downloads of beats (elastic#19102) [DOCS] Add output.console to Functionbeat doc and Functionbeat reference file (elastic#18965) Add compatibility info (elastic#18929) Set ecszap version to v0.2.0 (elastic#19106) [filebeat][httpjson] Fix unit test function call (elastic#19124) [Filebeat][httpjson] Adds oauth2 support for httpjson input (elastic#18892) Allow host.* fields to be disabled in Suricata module (elastic#19107) Make selector string casing configurable (elastic#18854) Switch the GRPC communication where Agent is running the server and the beats are connecting back to Agent (elastic#18973) Disable host.* fields by default for netflow module (elastic#19087) Automatically fill zube teams on backports if available (elastic#18924) Fix crash on vsphere module (elastic#19078) [Ingest Manager] Download snapshot artifacts from snapshots repo (elastic#18685) [Ingest Manager] Basic Elastic Agent documentation (elastic#19030) Make user.id a string in system/users, in line with ECS (elastic#19019) [docs] Add 7.8 release highlights placeholder file (elastic#18493) Fix translate_sid's empty target field handling (elastic#18991) ...

…nce file (#18965) (#19145) * Add outputconsole to Functionbeat doc * Remove no_codec attirbute * Updates following review # Conflicts: # libbeat/outputs/console/docs/console.asciidoc

…nce file (#18965) (#19143) * Add outputconsole to Functionbeat doc * Remove no_codec attirbute * Updates following review # Conflicts: # libbeat/outputs/console/docs/console.asciidoc

…reference file (#18965) (#19147) * [DOCS] Add output.console to Functionbeat doc and Functionbeat reference file (#18965) * Add outputconsole to Functionbeat doc * Remove no_codec attirbute * Updates following review # Conflicts: # libbeat/outputs/console/docs/console.asciidoc # x-pack/functionbeat/functionbeat.reference.yml # x-pack/functionbeat/scripts/mage/config.go * Update x-pack/functionbeat/scripts/mage/config.go Co-authored-by: Brandon Morelli <bmorelli25@gmail.com> * Add output console to ref file Co-authored-by: Brandon Morelli <bmorelli25@gmail.com>

…nce file (elastic#18965) * Add outputconsole to Functionbeat doc * Remove no_codec attirbute * Updates following review

…reference file (elastic#18965) (elastic#19147) * [DOCS] Add output.console to Functionbeat doc and Functionbeat reference file (elastic#18965) * Add outputconsole to Functionbeat doc * Remove no_codec attirbute * Updates following review # Conflicts: # libbeat/outputs/console/docs/console.asciidoc # x-pack/functionbeat/functionbeat.reference.yml # x-pack/functionbeat/scripts/mage/config.go * Update x-pack/functionbeat/scripts/mage/config.go Co-authored-by: Brandon Morelli <bmorelli25@gmail.com> * Add output console to ref file Co-authored-by: Brandon Morelli <bmorelli25@gmail.com>

Add outputconsole to Functionbeat doc

d95ec1c

EamonnTP added docs v7.7.0 labels Jun 4, 2020

EamonnTP requested review from bmorelli25 and ppf2 June 4, 2020 09:28

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 4, 2020

EamonnTP self-assigned this Jun 4, 2020

andresrc added the Team:Services (Deprecated) Label for the former Integrations-Services team label Jun 5, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 5, 2020

Remove no_codec attirbute

f570c72

EamonnTP marked this pull request as ready for review June 9, 2020 13:32

bmorelli25 approved these changes Jun 9, 2020

View reviewed changes

Merge branch 'master' into add-output-console-functionbeat

e4da4ed

Retry unsuccessful checks

Updates following review

a29a791

urso reviewed Jun 11, 2020

View reviewed changes

urso approved these changes Jun 11, 2020

View reviewed changes

Merge branch 'master' into add-output-console-functionbeat

8bb1112

Kick off tests

EamonnTP merged commit 5cd55f9 into elastic:master Jun 11, 2020

EamonnTP deleted the add-output-console-functionbeat branch June 11, 2020 21:43

EamonnTP mentioned this pull request Jun 11, 2020

[7.8] [DOCS] Add output.console to Functionbeat doc and Functionbeat reference file (#18965) #19143

Merged

This was referenced Jun 11, 2020

[7.7] [DOCS] Add output.console to Functionbeat doc and Functionbeat reference file (#18965) #19144

Closed

[7.x] [DOCS] Add output.console to Functionbeat doc and Functionbeat reference file (#18965) #19145

Merged

EamonnTP mentioned this pull request Jun 11, 2020

[7.7] [DOCS] Add output.console to Functionbeat doc and Functionbeat reference file (#18965) #19147

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DOCS] Add output.console to Functionbeat doc and Functionbeat reference file #18965

[DOCS] Add output.console to Functionbeat doc and Functionbeat reference file #18965

EamonnTP commented Jun 4, 2020 •

edited

Loading

elasticmachine commented Jun 4, 2020 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jun 9, 2020

bmorelli25 commented Jun 9, 2020

bmorelli25 left a comment

bmorelli25 commented Jun 9, 2020 •

edited

Loading

dedemorton commented Jun 9, 2020 •

edited

Loading

dedemorton commented Jun 9, 2020

ppf2 commented Jun 10, 2020

urso commented Jun 10, 2020

urso Jun 11, 2020

		@@ -7,6 +7,8 @@

		The Console output writes events in JSON format to stdout.

[DOCS] Add output.console to Functionbeat doc and Functionbeat reference file #18965

[DOCS] Add output.console to Functionbeat doc and Functionbeat reference file #18965

Conversation

EamonnTP commented Jun 4, 2020 • edited Loading

What does this PR do?

Why is it important?

Related issues

HTML output

elasticmachine commented Jun 4, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

elasticmachine commented Jun 9, 2020

bmorelli25 commented Jun 9, 2020

bmorelli25 left a comment

Choose a reason for hiding this comment

bmorelli25 commented Jun 9, 2020 • edited Loading

dedemorton commented Jun 9, 2020 • edited Loading

dedemorton commented Jun 9, 2020

ppf2 commented Jun 10, 2020

urso commented Jun 10, 2020

urso Jun 11, 2020

Choose a reason for hiding this comment

EamonnTP commented Jun 4, 2020 •

edited

Loading

elasticmachine commented Jun 4, 2020 •

edited

Loading

bmorelli25 commented Jun 9, 2020 •

edited

Loading

dedemorton commented Jun 9, 2020 •

edited

Loading