Add documentation for deploying filebeat and metricbeat in Cloud Foun…

…dry. (#17275)

* Add running-on-cloudfoundry for filebeat.

* Fix typo.

* Add running on cloudfoundry for metricbeat.

* Add changelog.

* Add missing newline at end of metricbeat manifest.

* Fix filebeat being referenced in metricbeat docs.

CHANGELOG.next.asciidoc

@@ -210,6 +210,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add pattern for Cisco ASA / FTD Message 734001 {issue}16212[16212] {pull}16612[16612]
 - Added new module `o365` for ingesting Office 365 management activity API events. {issue}16196[16196] {pull}16386[16386]
 - Improve AWS cloudtrail field mappings {issue}16086[16086] {issue}16110[16110] {pull}17155[17155]
+- Added documentation for running Filebeat in Cloud Foundry. {pull}17275[17275]
 
 *Heartbeat*
 
@@ -275,6 +276,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Release Oracle module as GA. {issue}14279[14279] {pull}16833[16833]
 - Release vsphere module as GA. {issue}15798[15798] {pull}17119[17119]
 - Add Storage metricsets to GCP module {pull}15598[15598]
+- Added documentation for running Metricbeat in Cloud Foundry. {pull}17275[17275]
 
 *Packetbeat*
 

deploy/cloudfoundry/README.md

@@ -0,0 +1,11 @@
+# Beats Cloud Foundry manifests examples
+
+## Getting started
+
+This is the list of officially supported Beats, with example manifests to run
+them in Cloud Foundry:
+
+Beat | Description
+---- | ----
+[filebeat](filebeat) | Ships logs from loggregator
+[metricbeat](metricbeat) | Ships metrics from loggregator

deploy/cloudfoundry/filebeat/filebeat.yml

@@ -0,0 +1,54 @@
+#=========================== Filebeat inputs =============================
+
+# Configure the input to access loggregator to forward the log events.
+
+filebeat.inputs:
+  - &cloudfoundry
+    type: cloudfoundry
+    client_id: ${CLIENT_ID:filebeat}
+    client_secret: ${CLIENT_SECRET:changeme}
+    api_address: ${API_ADDRESS}
+    #doppler_address: ${DOPPLER_ADDRESS}
+    #uaa_address: ${UAA_ADDRESS}
+    #rlp_address: ${RLP_ADDRESS}
+    #shard_id: ${SHARD_ID}
+
+
+#================================ Outputs =====================================
+
+# Configure the Elasticsearch output either to a specific host or using
+# Elastic Cloud.
+
+#-------------------------- Elasticsearch output ------------------------------
+output.elasticsearch:
+  # Array of hosts to connect to.
+  hosts: ["${ELASTICSEARCH_OUTPUT}:9200"]
+
+  # Protocol - either `http` (default) or `https`.
+  #protocol: "https"
+
+  # Authentication credentials - either API key or username/password.
+  #api_key: "id:api_key"
+  #username: "elastic"
+  #password: "changeme"
+
+#----------------------------- Elastic Cloud ----------------------------------
+
+# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).
+
+# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
+# `setup.kibana.host` options.
+# You can find the `cloud.id` in the Elastic Cloud web UI.
+#cloud.id:
+
+# The cloud.auth setting overwrites the `output.elasticsearch.username` and
+# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
+#cloud.auth:
+
+#================================ Processors =====================================
+
+# Configure processors to enhance or manipulate events generated by the beat.
+
+processors:
+  - add_cloudfoundry_metadata:
+    <<: *cloudfoundry

deploy/cloudfoundry/filebeat/manifest.yml

@@ -0,0 +1,10 @@
+applications:
+- name: filebeat
+  memory: 256M
+  instances: 1
+  buildpacks:
+  - binary_buildpack
+  command: ./filebeat -e -c ~/filebeat.yml
+  stack: cflinuxfs3
+  health-check-type: process
+  no-route: true

deploy/cloudfoundry/metricbeat/manifest.yml

@@ -0,0 +1,10 @@
+applications:
+  - name: metricbeat
+    memory: 256M
+    instances: 1
+    buildpacks:
+    - binary_buildpack
+    command: ./metricbeat -e -c ~/metricbeat.yml
+    stack: cflinuxfs3
+    health-check-type: process
+    no-route: true

deploy/cloudfoundry/metricbeat/metricbeat.yml

@@ -0,0 +1,54 @@
+#=========================== Metricbeat modules =============================
+
+# Configure the module to forward metrics from loggregator.
+
+metricbeat.modules:
+  - &cloudfoundry
+    module: cloudfoundry
+    client_id: ${CLIENT_ID}
+    client_secret: ${CLIENT_SECRET}
+    api_address: ${API_ADDRESS}
+    #doppler_address: ${DOPPLER_ADDRESS}
+    #uaa_address: ${UAA_ADDRESS}
+    #rlp_address: ${RLP_ADDRESS}
+    #shard_id: ${SHARD_ID}
+
+
+#================================ Outputs =====================================
+
+# Configure the Elasticsearch output either to a specific host or using
+# Elastic Cloud.
+
+#-------------------------- Elasticsearch output ------------------------------
+output.elasticsearch:
+  # Array of hosts to connect to.
+  hosts: ["${ELASTICSEARCH_OUTPUT}:9200"]
+
+  # Protocol - either `http` (default) or `https`.
+  #protocol: "https"
+
+  # Authentication credentials - either API key or username/password.
+  #api_key: "id:api_key"
+  #username: "elastic"
+  #password: "changeme"
+
+#----------------------------- Elastic Cloud ----------------------------------
+
+# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).
+
+# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
+# `setup.kibana.host` options.
+# You can find the `cloud.id` in the Elastic Cloud web UI.
+#cloud.id:
+
+# The cloud.auth setting overwrites the `output.elasticsearch.username` and
+# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
+#cloud.auth:
+
+#================================ Processors =====================================
+
+# Configure processors to enhance or manipulate events generated by the beat.
+
+processors:
+  - add_cloudfoundry_metadata:
+    <<: *cloudfoundry

filebeat/docs/getting-started.asciidoc

@@ -109,6 +109,11 @@ See <<running-on-docker, Running on Docker>> for deploying Docker containers.
 
 See <<running-on-kubernetes, Running on Kubernetes>> for deploying with Kubernetes.
 
+[[cloudfoundry]]
+*cloudfoundry:*
+
+See <<running-on-cloudfoundry, Running on Cloud Foundry>> for deploying with Cloud Foundry.
+
 [[win]]
 *win:*
 

filebeat/docs/running-on-cloudfoundry.asciidoc

@@ -0,0 +1,80 @@
+[[running-on-cloudfoundry]]
+=== Running {beatname_uc} on Cloud Foundry
+
+You can use {beatname_uc} on Cloud Foundry to retrieve and ship logs.
+
+ifeval::["{release-state}"=="unreleased"]
+
+However, version {version} of {beatname_uc} has not yet been
+released, no build is currently available for this version.
+
+endif::[]
+
+
+[float]
+==== Cloud Foundry credentials
+
+{beatname_uc} needs credentials created with UAA so it can connect to loggregator to receive the logs. The uaac
+command will create the required credentials for connecting to loggregator.
+
+["source", "sh"]
+------------------------------------------------
+uaac client add {beatname_lc} --name {beatname_lc} --secret changeme --authorized_grant_types client_credentials,refresh_token --authorities doppler.firehose,cloud_controller.admin_read_only
+------------------------------------------------
+
+[WARNING]
+=======================================
+*Use a unique secret:* The uaac command above is just an example and the secret should be changed and the
+`{beatname_lc}.yml` should be updated with your choosen secret.
+=======================================
+
+
+[float]
+==== Cloud Foundry deploy manifests
+
+You deploy {beatname_uc} as an application with no route.
+
+Cloud Foundry requires that 3 files exist inside of a directory to allow {beatname_uc} to be pushed. The commands
+below provide the basic steps for getting it up and running.
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+curl -L -O https://artifacts.elastic.co/downloads/beats/{beatname_lc}/{beatname_lc}-{version}-linux-x86_64.tar.gz
+tar xzvf {beatname_lc}-{version}-linux-x86_64.tar.gz
+cd {beatname_lc}-{version}-linux-x86_64
+curl -L -O https://raw.githubusercontent.com/elastic/beats/{branch}/deploy/cloudfoundry/{beatname_lc}/{beatname_lc}.yml
+# Update api_address, client_id, client_secret in downloaded {beatname_lc}.
+curl -L -O https://raw.githubusercontent.com/elastic/beats/{branch}/deploy/cloudfoundry/{beatname_lc}/manifest.yml
+------------------------------------------------
+
+
+[float]
+==== Deploy
+
+To deploy {beatname_uc} to Cloud Foundry, run:
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+cf push
+------------------------------------------------
+
+To check the status, run:
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+$ cf apps
+
+name       requested state   instances   memory   disk   urls
+filebeat   started           1/1         256M     1G
+------------------------------------------------
+
+Log events should start flowing to Elasticsearch. The events are annotated with
+metadata added by the <<add-cloudfoundry-metadata>> processor.
+
+
+[WARNING]
+=======================================
+*Set shard_id to scale:* By default {beatname_uc} will generate a random `shard_id` when it starts. In the case that
+{beatname_uc} needs to be scaled passed 1 instance, be sure to set a static `shard_id`. Not setting a static `shard_id`
+will result in duplicate events being pushed to Elasticsearch.
+=======================================

filebeat/docs/setting-up-running.asciidoc

@@ -26,6 +26,8 @@ This section includes additional information on how to set up and run
 
 * <<running-on-kubernetes>>
 
+* <<running-on-cloudfoundry>>
+
 * <<running-with-systemd>>
 
 
@@ -41,6 +43,8 @@ include::./running-on-docker.asciidoc[]
 
 include::./running-on-kubernetes.asciidoc[]
 
+include::./running-on-cloudfoundry.asciidoc[]
+
 include::{libbeat-dir}/shared-systemd.asciidoc[]
 
 include::{libbeat-dir}/shared-shutdown.asciidoc[]

metricbeat/docs/gettingstarted.asciidoc

@@ -119,6 +119,11 @@ See <<running-on-docker, Running on Docker>> for deploying Docker containers.
 
 See <<running-on-kubernetes, Running on Kubernetes>> for deploying with Kubernetes.
 
+[[cloudfoundry]]
+*cloudfoundry:*
+
+See <<running-on-cloudfoundry, Running on Cloud Foundry>> for deploying with Cloud Foundry.
+
 [[win]]
 *win:*
 

metricbeat/docs/running-on-cloudfoundry.asciidoc

@@ -0,0 +1,80 @@
+[[running-on-cloudfoundry]]
+=== Running {beatname_uc} on Cloud Foundry
+
+You can use {beatname_uc} on Cloud Foundry to retrieve and ship metrics.
+
+ifeval::["{release-state}"=="unreleased"]
+
+However, version {version} of {beatname_uc} has not yet been
+released, no build is currently available for this version.
+
+endif::[]
+
+
+[float]
+==== Cloud Foundry credentials
+
+{beatname_uc} needs credentials created with UAA so it can connect to loggregator to receive the logs. The uaac
+command will create the required credentials for connecting to loggregator.
+
+["source", "sh"]
+------------------------------------------------
+uaac client add {beatname_lc} --name {beatname_lc} --secret changeme --authorized_grant_types client_credentials,refresh_token --authorities doppler.firehose,cloud_controller.admin_read_only
+------------------------------------------------
+
+[WARNING]
+=======================================
+*Use a unique secret:* The uaac command above is just an example and the secret should be changed and the
+`{beatname_lc}.yml` should be updated with your choosen secret.
+=======================================
+
+
+[float]
+==== Cloud Foundry deploy manifests
+
+You deploy {beatname_uc} as an application with no route.
+
+Cloud Foundry requires that 3 files exist inside of a directory to allow {beatname_uc} to be pushed. The commands
+below provide the basic steps for getting it up and running.
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+curl -L -O https://artifacts.elastic.co/downloads/beats/{beatname_lc}/{beatname_lc}-{version}-linux-x86_64.tar.gz
+tar xzvf {beatname_lc}-{version}-linux-x86_64.tar.gz
+cd {beatname_lc}-{version}-linux-x86_64
+curl -L -O https://raw.githubusercontent.com/elastic/beats/{branch}/deploy/cloudfoundry/{beatname_lc}/{beatname_lc}.yml
+# Update api_address, client_id, client_secret in downloaded {beatname_lc}.
+curl -L -O https://raw.githubusercontent.com/elastic/beats/{branch}/deploy/cloudfoundry/{beatname_lc}/manifest.yml
+------------------------------------------------
+
+
+[float]
+==== Deploy
+
+To deploy {beatname_uc} to Cloud Foundry, run:
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+cf push
+------------------------------------------------
+
+To check the status, run:
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+$ cf apps
+
+name         requested state   instances   memory   disk   urls
+metricbeat   started           1/1         256M     1G
+------------------------------------------------
+
+Metrics should start flowing to Elasticsearch. The events are annotated with
+metadata added by the <<add-cloudfoundry-metadata>> processor.
+
+
+[WARNING]
+=======================================
+*Set shard_id to scale:* By default {beatname_uc} will generate a random `shard_id` when it starts. In the case that
+{beatname_uc} needs to be scaled passed 1 instance, be sure to set a static `shard_id`. Not setting a static `shard_id`
+will result in duplicate events being pushed to Elasticsearch.
+=======================================

metricbeat/docs/setting-up-running.asciidoc

@@ -26,6 +26,8 @@ This section includes additional information on how to set up and run
 
 * <<running-on-kubernetes>>
 
+* <<running-on-cloudfoundry>>
+
 * <<running-with-systemd>>
 
 //MAINTAINERS: If you add a new file to this section, make sure you update the bulleted list ^^ too.
@@ -40,6 +42,8 @@ include::./running-on-docker.asciidoc[]
 
 include::./running-on-kubernetes.asciidoc[]
 
+include::./running-on-cloudfoundry.asciidoc[]
+
 include::{libbeat-dir}/shared-systemd.asciidoc[]
 
 include::{libbeat-dir}/shared-shutdown.asciidoc[]