Add support for tornado.web

.ci/.jenkins_exclude.yml

@@ -207,6 +207,19 @@ exclude:
     FRAMEWORK: aiohttp-newest
   - PYTHON_VERSION: python-3.6
     FRAMEWORK: aiohttp-newest
+  # tornado, only supported in Python 3.7+
+  - PYTHON_VERSION: python-2.7
+    FRAMEWORK: tornado-newest
+  - PYTHON_VERSION: pypy-2
+    FRAMEWORK: tornado-newest
+  - PYTHON_VERSION: pypy-3
+    FRAMEWORK: tornado-newest
+  - PYTHON_VERSION: python-3.4
+    FRAMEWORK: tornado-newest
+  - PYTHON_VERSION: python-3.5
+    FRAMEWORK: tornado-newest
+  - PYTHON_VERSION: python-3.6
+    FRAMEWORK: tornado-newest
   # aiopg
   - PYTHON_VERSION: python-2.7
     FRAMEWORK: aiopg-newest

.ci/.jenkins_framework.yml

@@ -35,3 +35,4 @@ FRAMEWORK:
   - pymysql-newest
   - aiohttp-newest
   - aiopg-newest
+  - tornado-newest

.ci/.jenkins_framework_full.yml

@@ -65,3 +65,4 @@ FRAMEWORK:
   - aiohttp-4.0
   - aiohttp-newest
   - aiopg-newest
+  - tornado-newest

.flake8

@@ -1,4 +1,11 @@
 [flake8]
-exclude=elasticapm/utils/wrapt,build,src,tests,dist,conftest.py,setup.py
+exclude=
+    elasticapm/utils/wrapt/**,
+    build/**,
+    src/**,
+    tests/**,
+    dist/**,
+    conftest.py,
+    setup.py
 max-line-length=120
 ignore=E731,W503,E203,BLK100,B301

.gitignore

@@ -28,3 +28,5 @@ venv
 benchmarks/result*
 coverage.xml
 tests/python-agent-junit.xml
+*.code-workspace
+.pytest_cache/

CHANGELOG.asciidoc

@@ -43,6 +43,7 @@ This will be the last minor release to support the following versions:
 ===== New Features
 
  * Added support for aiohttp client and server {pull}659[#659]
+ * Added support for tornado web framework {pull}661[#661]
  * Added support for W3C `traceparent` and `tracestate` headers {pull}660[#660]
  * Added Django 3.0 and Flask 1.1 to the support matrix {pull}667[#667]
  * Added support for aiopg {pull}668[#668]

docs/set-up.asciidoc

@@ -6,6 +6,7 @@ To get you off the ground, we’ve prepared guides for setting up the Agent with
  * <<django-support,Django>>
  * <<flask-support,Flask>>
  * <<aiohttp-server-support,aiohttp>>
+ * <<tornado-support,tornado>>
 
 For custom instrumentation, see the <<api>>.
 
@@ -14,3 +15,5 @@ include::./django.asciidoc[]
 include::./flask.asciidoc[]
 
 include::./aiohttp-server.asciidoc[]
+
+include::./tornado.asciidoc[]

docs/supported-technologies.asciidoc

@@ -6,11 +6,11 @@ The Elastic APM Python Agent comes with support for the following frameworks:
 
  * <<django-support,Django>>
  * <<flask-support,Flask>>
+ * <<supported-aiohttp,Aiohttp Server>>
+ * <<supported-tornado,Tornado>>
 
 For other frameworks and custom Python code, the agent exposes a set of <<api,APIs>> for integration.
 
-NOTE: The Elastic APM Python agent does currently not support asynchronous frameworks like Twisted or Tornado.
-
 [float]
 [[supported-python]]
 === Python
@@ -62,6 +62,14 @@ We support these aiohttp versions:
 
  * 3.0+
 
+[float]
+[[supported-tornado]]
+=== Tornado
+
+We support these tornado versions:
+
+ * 6.0+
+
 [float]
 [[automatic-instrumentation]]
 == Automatic Instrumentation

docs/tornado.asciidoc

@@ -0,0 +1,125 @@
+[[tornado-support]]
+=== Tornado Support
+
+Incorporating Elastic APM into your Tornado project only requires a few easy
+steps.
+
+[float]
+[[tornado-installation]]
+==== Installation
+
+Install the Elastic APM agent using pip:
+
+[source,bash]
+----
+$ pip install elastic-apm
+----
+
+or add `elastic-apm` to your project's `requirements.txt` file.
+
+
+[float]
+[[tornado-setup]]
+==== Setup
+
+To set up the agent, you need to initialize it with appropriate settings.
+
+The settings are configured either via environment variables,
+the application's settings, or as initialization arguments.
+
+You can find a list of all available settings in the
+<<configuration, Configuration>> page.
+
+To initialize the agent for your application using environment variables:
+
+[source,python]
+----
+import tornado.web
+from elasticapm.contrib.tornado import ElasticAPM
+
+app = tornado.web.Application()
+apm = ElasticAPM(app)
+----
+
+To configure the agent using `ELASTIC_APM` in your application's settings:
+
+[source,python]
+----
+import tornado.web
+from elasticapm.contrib.tornado import ElasticAPM
+
+app = tornado.web.Application()
+app.settings['ELASTIC_APM'] = {
+    'SERVICE_NAME': '<SERVICE-NAME>',
+    'SECRET_TOKEN': '<SECRET-TOKEN>',
+}
+apm = ElasticAPM(app)
+----
+
+[float]
+[[tornado-usage]]
+==== Usage
+
+Once you have configured the agent, it will automatically track transactions
+and capture uncaught exceptions within tornado.
+
+Capture an arbitrary exception by calling
+<<client-api-capture-exception,`capture_exception`>>:
+
+[source,python]
+----
+try:
+    1 / 0
+except ZeroDivisionError:
+    apm.client.capture_exception()
+----
+
+Log a generic message with <<client-api-capture-message,`capture_message`>>:
+
+[source,python]
+----
+apm.client.capture_message('hello, world!')
+----
+
+[float]
+[[tornado-performance-metrics]]
+==== Performance metrics
+
+If you've followed the instructions above, the agent has installed our
+instrumentation within the base RequestHandler class in tornado.web. This will
+measure response times, as well as detailed performance data for all supported
+technologies.
+
+NOTE: Due to the fact that `asyncio` drivers are usually separate from their
+synchronous counterparts, specific instrumentation is needed for all drivers.
+The support for asynchronous drivers is currently quite limited.
+
+[float]
+[[tornado-ignoring-specific-views]]
+===== Ignoring specific routes
+
+You can use the
+<<config-transactions-ignore-patterns,`TRANSACTIONS_IGNORE_PATTERNS`>>
+configuration option to ignore specific routes. The list given should be a
+list of regular expressions which are matched against the transaction name:
+
+[source,python]
+----
+app.settings['ELASTIC_APM'] = {
+    # ...
+    'TRANSACTIONS_IGNORE_PATTERNS': ['^GET SecretHandler', 'MainHandler']
+    # ...
+}
+----
+
+This would ignore any requests using the `GET SecretHandler` route
+and any requests containing `MainHandler`.
+
+
+[float]
+[[supported-tornado-and-python-versions]]
+==== Supported tornado and Python versions
+
+A list of supported <<supported-tornado,tornado>> and <<supported-python,Python>> versions can be found on our <<supported-technologies,Supported Technologies>> page.
+
+NOTE: Elastic APM only supports `asyncio` when using Python 3.7+

elasticapm/contrib/tornado/__init__.py

@@ -0,0 +1,66 @@
+#  BSD 3-Clause License
+#
+#  Copyright (c) 2019, Elasticsearch BV
+#  All rights reserved.
+#
+#  Redistribution and use in source and binary forms, with or without
+#  modification, are permitted provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this
+#    list of conditions and the following disclaimer.
+#
+#  * Redistributions in binary form must reproduce the above copyright notice,
+#    this list of conditions and the following disclaimer in the documentation
+#    and/or other materials provided with the distribution.
+#
+#  * Neither the name of the copyright holder nor the names of its
+#    contributors may be used to endorse or promote products derived from
+#    this software without specific prior written permission.
+#
+#  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+#  AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+#  IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+#  DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+#  FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+#  DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+#  SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+#  CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+#  OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+"""
+Framework integration for Tornado
+
+Note that transaction creation is actually done in the tornado
+instrumentation. This module only creates the client for later use by the
+that instrumentation, and triggers the global instrumentation itself.
+"""
+import elasticapm
+import tornado
+from elasticapm import Client
+
+
+class ElasticAPM:
+    def __init__(self, app, client=None, **config):
+        """
+        Create the elasticapm Client object and store in the app for later
+        use.
+
+        ElasticAPM configuration is sent in via the **config kwargs, or
+        optionally can be added to the application via the Application object
+        (as a dictionary under the "ELASTIC_APM" key in the settings).
+        """
+        if "ELASTIC_APM" in app.settings and isinstance(app.settings["ELASTIC_APM"], dict):
+            settings = app.settings["ELASTIC_APM"]
+            settings.update(config)
+            config = settings
+        if not client:
+            config.setdefault("framework_name", "tornado")
+            config.setdefault("framework_version", tornado.version)
+            client = Client(config)
+        self.app = app
+        self.client = client
+        app.elasticapm_client = client
+
+        # Don't instrument if debug=True in tornado, unless client.config.debug is True
+        if (not self.app.settings.get("debug") or client.config.debug) and client.config.instrument:
+            elasticapm.instrument()

elasticapm/contrib/tornado/utils.py

@@ -0,0 +1,81 @@
+#  BSD 3-Clause License
+#
+#  Copyright (c) 2019, Elasticsearch BV
+#  All rights reserved.
+#
+#  Redistribution and use in source and binary forms, with or without
+#  modification, are permitted provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this
+#    list of conditions and the following disclaimer.
+#
+#  * Redistributions in binary form must reproduce the above copyright notice,
+#    this list of conditions and the following disclaimer in the documentation
+#    and/or other materials provided with the distribution.
+#
+#  * Neither the name of the copyright holder nor the names of its
+#    contributors may be used to endorse or promote products derived from
+#    this software without specific prior written permission.
+#
+#  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+#  AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+#  IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+#  DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+#  FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+#  DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+#  SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+#  CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+#  OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+import elasticapm
+from elasticapm.conf import constants
+from elasticapm.utils import compat
+
+try:
+    import tornado
+except ImportError:
+    pass
+
+
+def get_data_from_request(request_handler, request, capture_body=False, capture_headers=True):
+    """
+    Capture relevant data from a tornado.httputil.HTTPServerRequest
+    """
+    result = {
+        "method": request.method,
+        "socket": {"remote_address": request.remote_ip, "encrypted": request.protocol == "https"},
+        "cookies": request.cookies,
+        "http_version": request.version,
+    }
+    if capture_headers:
+        result["headers"] = dict(request.headers)
+    if request.method in constants.HTTP_WITH_BODY:
+        if tornado.web._has_stream_request_body(request_handler.__class__):
+            # Body is a future and streaming is expected to be handled by
+            # the user in the RequestHandler.data_received function.
+            # Currently not sure of a safe way to get the body in this case.
+            result["body"] = "[STREAMING]" if capture_body else "[REDACTED]"
+        else:
+            body = None
+            try:
+                body = tornado.escape.json_decode(request.body)
+            except Exception:
+                body = request.body
+
+            if body is not None:
+                result["body"] = body if capture_body else "[REDACTED]"
+
+    result["url"] = elasticapm.utils.get_url_dict(request.full_url())
+    return result
+
+
+def get_data_from_response(request_handler, capture_headers=True):
+    result = {}
+
+    result["status_code"] = request_handler.get_status()
+
+    if capture_headers and request_handler._headers:
+        headers = request_handler._headers
+        result["headers"] = {key: ";".join(headers.get_list(key)) for key in compat.iterkeys(headers)}
+        pass
+    return result