added section on adding matrix tests to CONTRIBUTING.md (#967)

Co-authored-by: Colton Myers <colton.myers@gmail.com>

Author: beniwohli

CONTRIBUTING.md

@@ -108,6 +108,65 @@ be used. For example, whenever a test has `elasticapm_client` as an argument,
 that is a fixture which is defined
 [here](https://github.com/elastic/apm-agent-python/blob/ed4ce5fd5db3cc091a54d3328384fbce62635bbb/tests/fixtures.py#L150).
 
+#### Adding new instrumentations to the matrix build
+
+For tests that require external dependencies like databases, or for testing different versions of the same library,
+we use a matrix build that leverages Docker and docker-compose.
+
+The setup requires a little bit of boilerplate to get started.
+In this example, we will create an instrumentation for the "foo" database, by instrumenting its Python driver, `foodriver`.
+
+1. mark your tests with a pytest marker that describes the new instrumentation at the top of your tests file:
+
+       pytestmark = pytest.mark.foo
+
+   `pytestmark` can also be a list if you need to define more than one mark (e.g. to mark tests as integration tests):
+
+       pytestmark = [pytest.mark.foo, pytest.mark.integrationtest]
+
+1. make sure to use `pytest.importorskip` to import any dependencies that are only required by your tests:
+
+       foodriver = pytest.importorskip("foodriver")
+
+1. Create one or more requirements files in `tests/requirements` that list the dependencies that are to be installed specifically for your tests:
+   To only test the newest version of the library, create a file `tests/requirements/reqs-foo-newest.txt` and add something like this to it:
+
+       foodriver
+       -r reqs-base.txt
+
+   This tells the matrix runner to install the newest version of `foodriver`, as well as the base requirements needed to run the test suite.
+   To test more than one version of the library, create additional `reqs-foo-X.Y.txt` files with specific versions of your instrumented package.
+
+1. Create a file called `foo.sh` in `tests/scripts/envs/foo.sh`.
+   Here you can define environment variables that are required to run your tests.
+   As a minimum, you'll have to set the `PYTEST_MARKER` variable to the same value you used above for the pytest marker, e.g.
+
+       export PYTEST_MARKER="-m foo"
+
+1. Add entries in `.ci/jenkins_framework.yml` (for pull requests) and `.ci/jenkins_framework_full.yml` (for nightly builds).
+   Generally, we only test the newest version of an instrumentation with every pull request:
+
+       - foo-newest
+
+   To test other versions in the nightly build, add them to `.ci/jenkins_framework_full.yml`.
+
+1. OPTIONAL: If you need a real service to test against (e.g. an actual foo database), add an entry in `tests/docker-compose.yml` under `services`:
+
+         foo:
+            image: foobase:latest
+
+   You'll also have to add a `DOCKER_DEPS` environment variable to `tests/scripts/envs/foo.sh` which tells the matrix
+   to spin up the given docker-compose service before running your tests.
+   You may also need to add things like hostname configuration here.
+
+       DOCKER_DEPS="foo"
+       FOO_CONNECTION_URL="http://foo:4711"
+
+1. OPTIONAL: If your instrumented package does not support all Python versions we test with, you can exclude certain combinations by adding them to `.ci/jenkins_exclude.yml`:
+
+       - PYTHON_VERSION: python-3.5  # foo doesn't support Python 3.5
+         FRAMEWORK: foo-newest
+
 ### Workflow
 
 All feature development and most bug fixes hit the master branch first.