Merge branch 'master' into systemtest-migrate-profiling

Author: axw

.go-version

@@ -1 +1 @@
-1.14.7
+1.14.12

Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.14.7
+FROM golang:1.14.12
 MAINTAINER Nicolas Ruflin <ruflin@elastic.co>
 
 RUN set -x && \

README.md

@@ -17,7 +17,7 @@ To get started with APM please see our [Getting Started Guide](https://www.elast
 
 ### Requirements
 
-* [Golang](https://golang.org/dl/) 1.14.7
+* [Golang](https://golang.org/dl/) 1.14.12
 
 ### Install
 

_meta/beat.yml

@@ -27,6 +27,10 @@ apm-server:
   # Maximum number of new connections to accept simultaneously (0 means unlimited).
   #max_connections: 0
 
+  # Custom HTTP headers to add to all HTTP responses, e.g. for security policy compliance.
+  #response_headers:
+  #  X-My-Header: Contents of the header
+
   # If true (default), APM Server captures the IP of the instrumented service
   # or the IP and User Agent of the real user (RUM requests).
   #capture_personal_data: true

apm-server.docker.yml

@@ -27,6 +27,10 @@ apm-server:
   # Maximum number of new connections to accept simultaneously (0 means unlimited).
   #max_connections: 0
 
+  # Custom HTTP headers to add to all HTTP responses, e.g. for security policy compliance.
+  #response_headers:
+  #  X-My-Header: Contents of the header
+
   # If true (default), APM Server captures the IP of the instrumented service
   # or the IP and User Agent of the real user (RUM requests).
   #capture_personal_data: true

apm-server.yml

@@ -27,6 +27,10 @@ apm-server:
   # Maximum number of new connections to accept simultaneously (0 means unlimited).
   #max_connections: 0
 
+  # Custom HTTP headers to add to all HTTP responses, e.g. for security policy compliance.
+  #response_headers:
+  #  X-My-Header: Contents of the header
+
   # If true (default), APM Server captures the IP of the instrumented service
   # or the IP and User Agent of the real user (RUM requests).
   #capture_personal_data: true

apmpackage/README.md

@@ -0,0 +1,125 @@
+## Developer documentation
+
+### ~Requirements
+
+- Checkout `elastic/package-registry`, `elastic/package-storage` and `elastic/beats`
+- Have `elastic/package-spec` at hand
+
+### Guide
+
+#### Update / fix a package
+
+1. Actual work
+    - Make changes in `apmpackage/apm/<version>`, `apmpackage/docs/README.template.md` and/or code as needed
+    - Run `make update`. That will update fields, pipeline and doc files based on apm-server fields and pipeline defs.
+
+2. Run the registry
+    - Checkout a fresh master from the registry and run `mage build`
+    - Copy `apmpackage/apm` in apm-server to `build/package-storage/packages` in the registry
+    - `go run .`
+
+3. Run the stack
+    - Update Kibana settings with `xpack.fleet.registryUrl: http://localhost:8080`
+    - Start Kibana and Elasticsearch with X-Pack enabled. One way to do it is with a local Kibana:
+        - `yarn es snapshot --license trial --ssl -E xpack.security.authc.api_key.enabled=true`
+        - `yarn start --ssl`
+
+4. Test
+    - Go to the Fleet UI, install the integration and test what you need. You generally will want to have a look at the
+   installed assets (ie. templates and pipelines), and the generated `apm` input in the policy.
+    - If you need to change the package, you *must* remove the installed integration first. You can use the UI
+    or the API, eg: `curl -X DELETE -k -u elastic:changeme https://localhost:5601/abc/api/fleet/epm/packages/apm-0.1.0 -H 'kbn-xsrf: xyz'`
+    See [API docs](https://github.com/elastic/kibana/tree/master/x-pack/plugins/fleet/dev_docs/api) for details.
+    You normally don't need to restart the registry (an exception to this is eg. if you change a `hbs` template file).
+
+5. Upload to the snapshot registry
+    - When everything works and `apmpackage/apm/` changes have been merged to `master`, copy the new package to
+    `package-storage/packages` in the `package-storage` repo, `snapshot` branch.
+    Do *NOT* override any existing packages. Instead, bump the qualifier version. Eg: rename `apm/0.1.0-dev.1` to `apm/0.1.0-dev.2`
+    - Bump the qualifier in the contents too: `find . -type f  -print0 | xargs -0 sed -i ""  "s/0.1.0-dev.1/0.1.0-dev.2/g"`
+    This step can be done in a separate commit to facilitate review.
+
+
+#### Create a new package version
+
+Follow steps described above, except:
+
+1. New local version
+    - Copy `apmpackage/apm/<latest-version>` to `apmpackage/apm/<next-version>`.
+    - Add a new version mapping in the `versionMapping` variable at the top of `apmpackage/cmd/gen-package/main.go`.
+      Most likely this won't be needed in the future.
+    - Then do any changes in the new folder. The rest of the steps are the same.
+
+2. First dev version
+    - When copying to the `package-storage`, create the first version qualifier instead of bumping the last one.
+      Eg: `apm/0.2.0` -> `apm/0.2.0-dev.1`
+
+
+#### Run the Elastic Agent
+
+If you do code changes or a whole new version, you need to run the Elastic Agent locally.
+Most of the work here is done in `beats/x-pack/elastic-agent`
+
+0. Optional: Update the spec
+
+   The spec informs whether the Elastic Agent should or should not start apm-server based on the policy file,
+   and what settings to pass via GRPC call.
+    - Edit `spec/apm-server.yml`
+    - `mage update`
+
+1. Build / Package
+
+    *First time*
+    - `DEV=true PLATFORMS=darwin mage package` (replace platform as needed)
+    - Untar `build/distributions` contents
+
+    *Every time after*
+    - `DEV=true mage build`
+    - Copy `build/elastic-agent` to `build/distributions/elastic-agent-<version>-<platform>/data/elastic-agent-<hash>/`
+
+2. Override policy / apm-server
+    - Use the right `elastic-agent.yml` policy
+
+      It might be one you just generated with the UI, or one you have at hand with an apm input.
+      Copy to `build/distributions/elastic-agent-<version>-<platform>/elastic-agent.yml`
+
+    - Override apm-server in `install` and `downloads` folders. Approximately:
+      ```
+      # compile apm-server
+      cd ~/<path>/apm-server
+      make && make update
+
+      # tar and compress
+      cp build/fields/fields.yml .
+      tar cvf apm-server-<stack-version>-<platform>.tar apm-server LICENSE.txt NOTICE.txt README.md apm-server.yml ingest fields.yml
+      gzip apm-server-<stack-version>-<platform>.tar
+      sha512sum apm-server-<stack-version>-<platform>.tar.gz | tee apm-server-<stack-version>-<platform>.tar.gz.sha512
+
+      # delete old stuff
+      cd ~/<path>/beats/x-pack/elastic-agent/build/distributions/elastic-agent-<version>-<platform>/data/elastic-agent-<hash>/downloads
+      rm apm*
+      rm -rf ../install/apm*
+
+      # copy new files
+      mv <path>/apm-server-<stack-version>-<platform>.tar* .
+      mkdir -p ../install/apm-server-<stack-version>-<platform>
+      tar zxvf apm-server-<stack-version>-<platform> -C ../install/apm-server-<stack-version>-<platform>
+      ```
+3. Run the Elastic Agent
+    - `./build/distributions/<blablabla>/elastic-agent -e`
+    - Check apm-server logs at `build/distributions/<blablabla>/data/<blablabla>/logs/default`
+
+      (The last default in the path comes from the namespace in the policy)
+
+#### Promote a package
+
+Generally it should be done between FF and release.
+1. Remove the qualifier version from the package
+2. Push to the corresponding production branch(es)
+
+
+### Caveats
+
+Fleet is under active development and this guide might become obsolete quickly.
+
+Take everything with a grain of salt.

apmpackage/apm/0.1.0/data_stream/app_metrics/elasticsearch/ingest_pipeline/apm_ingest_timestamp.json

@@ -0,0 +1,12 @@
+{
+  "description": "Add an ingest timestamp for APM events",
+  "processors": [
+    {
+      "set": {
+        "field": "event.ingested",
+        "if": "ctx.processor?.event != 'span'",
+        "value": "{{_ingest.timestamp}}"
+      }
+    }
+  ]
+}

apmpackage/apm/0.1.0/data_stream/app_metrics/elasticsearch/ingest_pipeline/apm_remove_span_metadata.json

@@ -0,0 +1,25 @@
+{
+  "description": "Removes metadata fields available already on the parent transaction, to save storage",
+  "processors": [
+    {
+      "remove": {
+        "field": [
+          "host",
+          "process",
+          "user",
+          "user_agent",
+          "container",
+          "kubernetes",
+          "service.node",
+          "service.version",
+          "service.language",
+          "service.runtime",
+          "service.framework"
+        ],
+        "if": "ctx.processor?.event == 'span'",
+        "ignore_failure": true,
+        "ignore_missing": true
+      }
+    }
+  ]
+}

apmpackage/apm/0.1.0/data_stream/app_metrics/elasticsearch/ingest_pipeline/apm_user_agent.json

@@ -0,0 +1,13 @@
+{
+  "description": "Add user agent information for APM events",
+  "processors": [
+    {
+      "user_agent": {
+        "field": "user_agent.original",
+        "ignore_failure": true,
+        "ignore_missing": true,
+        "target_field": "user_agent"
+      }
+    }
+  ]
+}