Appinsights telemetry for aiohttp (#634)

* Initial version of aiohttp telemetry middleware

* black: Initial version of aiohttp telemetry middleware

* created separate package

* pylint: created separate package

* black: created separate package

* namespace renamed and tests added

* testing yaml pipeline

* removing pipeline

* Update ci-pr-pipeline.yml for Azure Pipelines

Updating pipeline for new integration package (appinsights-aiohttp)

Author: axelsrz

ci-pr-pipeline.yml

@@ -41,17 +41,14 @@ jobs:
       pip install -e ./libraries/botbuilder-dialogs
       pip install -e ./libraries/botbuilder-azure
       pip install -e ./libraries/botbuilder-testing
+      pip install -e ./libraries/botbuilder-integration-applicationinsights-aiohttp
       pip install -r ./libraries/botframework-connector/tests/requirements.txt
       pip install -r ./libraries/botbuilder-core/tests/requirements.txt
       pip install coveralls
       pip install pylint
       pip install black
     displayName: 'Install dependencies'
 
-  - script: 'pip install requests_mock'
-    displayName: 'Install requests mock (REMOVE AFTER MERGING INSPECTION)'
-    enabled: false
-
   - script: |
       pip install pytest
       pip install pytest-cov

libraries/botbuilder-integration-applicationinsights-aiohttp/README.rst

@@ -0,0 +1,87 @@
+
+========================================================
+BotBuilder-ApplicationInsights SDK extension for aiohttp
+========================================================
+
+.. image:: https://fuselabs.visualstudio.com/SDK_v4/_apis/build/status/Python/SDK_v4-Python-CI?branchName=master
+   :target:  https://fuselabs.visualstudio.com/SDK_v4/_apis/build/status/Python/SDK_v4-Python-CI
+   :align: right
+   :alt: Azure DevOps status for master branch
+.. image:: https://badge.fury.io/py/botbuilder-applicationinsights.svg
+   :target: https://badge.fury.io/py/botbuilder-applicationinsights
+   :alt: Latest PyPI package version
+
+Within the Bot Framework, BotBuilder-ApplicationInsights enables the Azure Application Insights service.
+
+Application Insights is an extensible Application Performance Management (APM) service for developers on multiple platforms. 
+Use it to monitor your live bot application. It includes powerful analytics tools to help you diagnose issues and to understand 
+what users actually do with your bot.
+
+How to Install
+==============
+
+.. code-block:: python
+  
+  pip install botbuilder-applicationinsights-aiohttp
+
+
+Documentation/Wiki
+==================
+
+You can find more information on the botbuilder-python project by visiting our `Wiki`_.
+
+Requirements
+============
+
+* `Python >= 3.7.0`_
+
+
+Source Code
+===========
+The latest developer version is available in a github repository:
+https://github.com/Microsoft/botbuilder-python/
+
+
+Contributing
+============
+
+This project welcomes contributions and suggestions.  Most contributions require you to agree to a
+Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
+the rights to use your contribution. For details, visit https://cla.microsoft.com.
+
+When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
+a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
+provided by the bot. You will only need to do this once across all repos using our CLA.
+
+This project has adopted the `Microsoft Open Source Code of Conduct`_.
+For more information see the `Code of Conduct FAQ`_ or
+contact `opencode@microsoft.com`_ with any additional questions or comments.
+
+Reporting Security Issues
+=========================
+
+Security issues and bugs should be reported privately, via email, to the Microsoft Security
+Response Center (MSRC) at `secure@microsoft.com`_. You should
+receive a response within 24 hours. If for some reason you do not, please follow up via
+email to ensure we received your original message. Further information, including the
+`MSRC PGP`_ key, can be found in
+the `Security TechCenter`_.
+
+License
+=======
+
+Copyright (c) Microsoft Corporation. All rights reserved.
+
+Licensed under the MIT_ License.
+
+.. _Wiki: https://github.com/Microsoft/botbuilder-python/wiki
+.. _Python >= 3.7.0: https://www.python.org/downloads/
+.. _MIT: https://github.com/Microsoft/vscode/blob/master/LICENSE.txt
+.. _Microsoft Open Source Code of Conduct: https://opensource.microsoft.com/codeofconduct/
+.. _Code of Conduct FAQ: https://opensource.microsoft.com/codeofconduct/faq/
+.. _opencode@microsoft.com: mailto:opencode@microsoft.com
+.. _secure@microsoft.com: mailto:secure@microsoft.com
+.. _MSRC PGP: https://technet.microsoft.com/en-us/security/dn606155
+.. _Security TechCenter: https://github.com/Microsoft/vscode/blob/master/LICENSE.txt
+
+.. <https://technet.microsoft.com/en-us/security/default>`_

libraries/botbuilder-integration-applicationinsights-aiohttp/botbuilder/integration/applicationinsights/aiohttp/__init__.py

@@ -0,0 +1,7 @@
+from .aiohttp_telemetry_middleware import bot_telemetry_middleware
+from .aiohttp_telemetry_processor import AiohttpTelemetryProcessor
+
+__all__ = [
+    "bot_telemetry_middleware",
+    "AiohttpTelemetryProcessor",
+]

libraries/botbuilder-integration-applicationinsights-aiohttp/botbuilder/integration/applicationinsights/aiohttp/about.py

@@ -0,0 +1,15 @@
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License.
+"""Bot Framework Application Insights integration package for aiohttp library."""
+
+import os
+
+__title__ = "botbuilder-integration-applicationinsights-aiohttp"
+__version__ = (
+    os.environ["packageVersion"] if "packageVersion" in os.environ else "4.4.0b1"
+)
+__uri__ = "https://www.github.com/Microsoft/botbuilder-python"
+__author__ = "Microsoft"
+__description__ = "Microsoft Bot Framework Bot Builder"
+__summary__ = "Microsoft Bot Framework Bot Builder SDK for Python."
+__license__ = "MIT"

libraries/botbuilder-integration-applicationinsights-aiohttp/botbuilder/integration/applicationinsights/aiohttp/aiohttp_telemetry_middleware.py

@@ -0,0 +1,26 @@
+from threading import current_thread
+from aiohttp.web import middleware
+
+# Map of thread id => POST body text
+_REQUEST_BODIES = {}
+
+
+def retrieve_aiohttp_body():
+    """ retrieve_flask_body
+    Retrieve the POST body text from temporary cache.
+    The POST body corresponds with the thread id and should resides in
+    cache just for lifetime of request.
+    """
+    result = _REQUEST_BODIES.pop(current_thread().ident, None)
+    return result
+
+
+@middleware
+async def bot_telemetry_middleware(request, handler):
+    """Process the incoming Flask request."""
+    if "application/json" in request.headers["Content-Type"]:
+        body = await request.json()
+        _REQUEST_BODIES[current_thread().ident] = body
+
+    response = await handler(request)
+    return response

libraries/botbuilder-integration-applicationinsights-aiohttp/botbuilder/integration/applicationinsights/aiohttp/aiohttp_telemetry_processor.py

@@ -0,0 +1,24 @@
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License.
+"""Telemetry processor for aiohttp."""
+import sys
+
+from botbuilder.applicationinsights.processor.telemetry_processor import (
+    TelemetryProcessor,
+)
+from .aiohttp_telemetry_middleware import retrieve_aiohttp_body
+
+
+class AiohttpTelemetryProcessor(TelemetryProcessor):
+    def can_process(self) -> bool:
+        return self.detect_aiohttp()
+
+    def get_request_body(self) -> str:
+        if self.detect_aiohttp():
+            return retrieve_aiohttp_body()
+        return None
+
+    @staticmethod
+    def detect_aiohttp() -> bool:
+        """Detects if running in aiohttp."""
+        return "aiohttp" in sys.modules

libraries/botbuilder-integration-applicationinsights-aiohttp/setup.py

@@ -0,0 +1,62 @@
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License.
+
+import os
+from setuptools import setup
+
+REQUIRES = [
+    "applicationinsights>=0.11.9",
+    "botbuilder-schema>=4.4.0b1",
+    "botframework-connector>=4.4.0b1",
+    "botbuilder-core>=4.4.0b1",
+    "botbuilder-applicationinsights>=4.4.0b1",
+]
+TESTS_REQUIRES = [
+    "aiounittest==1.3.0",
+    "aiohttp==3.5.4",
+]
+
+root = os.path.abspath(os.path.dirname(__file__))
+
+with open(
+    os.path.join(
+        root, "botbuilder", "integration", "applicationinsights", "aiohttp", "about.py"
+    )
+) as f:
+    package_info = {}
+    info = f.read()
+    exec(info, package_info)
+
+with open(os.path.join(root, "README.rst"), encoding="utf-8") as f:
+    long_description = f.read()
+
+setup(
+    name=package_info["__title__"],
+    version=package_info["__version__"],
+    url=package_info["__uri__"],
+    author=package_info["__author__"],
+    description=package_info["__description__"],
+    keywords=[
+        "BotBuilderApplicationInsights",
+        "bots",
+        "ai",
+        "botframework",
+        "botbuilder",
+        "aiohttp",
+    ],
+    long_description=long_description,
+    long_description_content_type="text/x-rst",
+    license=package_info["__license__"],
+    packages=["botbuilder.integration.applicationinsights.aiohttp"],
+    install_requires=REQUIRES + TESTS_REQUIRES,
+    tests_require=TESTS_REQUIRES,
+    include_package_data=True,
+    classifiers=[
+        "Programming Language :: Python :: 3.7",
+        "Intended Audience :: Developers",
+        "License :: OSI Approved :: MIT License",
+        "Operating System :: OS Independent",
+        "Development Status :: 5 - Production/Stable",
+        "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    ],
+)

libraries/botbuilder-integration-applicationinsights-aiohttp/tests/test_aiohttp_processor.py

@@ -0,0 +1,26 @@
+from unittest.mock import Mock
+from aiounittest import AsyncTestCase
+
+import aiohttp  # pylint: disable=unused-import
+
+from botbuilder.integration.applicationinsights.aiohttp import (
+    aiohttp_telemetry_middleware,
+    AiohttpTelemetryProcessor,
+)
+
+
+class TestAiohttpTelemetryProcessor(AsyncTestCase):
+    # pylint: disable=protected-access
+    def test_can_process(self):
+        assert AiohttpTelemetryProcessor.detect_aiohttp()
+        assert AiohttpTelemetryProcessor().can_process()
+
+    def test_retrieve_aiohttp_body(self):
+        aiohttp_telemetry_middleware._REQUEST_BODIES = Mock()
+        aiohttp_telemetry_middleware._REQUEST_BODIES.pop = Mock(
+            return_value="test body"
+        )
+        assert aiohttp_telemetry_middleware.retrieve_aiohttp_body() == "test body"
+
+        assert AiohttpTelemetryProcessor().get_request_body() == "test body"
+        aiohttp_telemetry_middleware._REQUEST_BODIES = {}

libraries/botbuilder-integration-applicationinsights-aiohttp/tests/test_aiohttp_telemetry_middleware.py

@@ -0,0 +1,35 @@
+from asyncio import Future
+from unittest.mock import Mock, MagicMock
+from aiounittest import AsyncTestCase
+
+from botbuilder.integration.applicationinsights.aiohttp import (
+    bot_telemetry_middleware,
+    aiohttp_telemetry_middleware,
+)
+
+
+class TestAiohttpTelemetryMiddleware(AsyncTestCase):
+    # pylint: disable=protected-access
+    async def test_bot_telemetry_middleware(self):
+        req = Mock()
+        req.headers = {"Content-Type": "application/json"}
+        req.json = MagicMock(return_value=Future())
+        req.json.return_value.set_result("mock body")
+
+        async def handler(value):
+            return value
+
+        sut = await bot_telemetry_middleware(req, handler)
+
+        assert "mock body" in aiohttp_telemetry_middleware._REQUEST_BODIES.values()
+        aiohttp_telemetry_middleware._REQUEST_BODIES.clear()
+        assert req == sut
+
+    def test_retrieve_aiohttp_body(self):
+        aiohttp_telemetry_middleware._REQUEST_BODIES = Mock()
+        aiohttp_telemetry_middleware._REQUEST_BODIES.pop = Mock(
+            return_value="test body"
+        )
+        assert aiohttp_telemetry_middleware.retrieve_aiohttp_body() == "test body"
+
+        aiohttp_telemetry_middleware._REQUEST_BODIES = {}