Async Handlers/Jobs Table (#1082)

* Adds async support for bokeh handlers and for the jobs table endpoints
Fixes a security issue with the jobs table and enhances the formatting of logs

* fix typos and test coverage for map view

* docs

* fix checkboxes on PR template

* async safe db queries for jobs table views

* add permission to access jobs that user does not own

* OAuth2 Provider docs update
Support OAUTH2_PROVIDER_URL_NAMESPACE setting

.github/pull_request_template.md

@@ -11,7 +11,7 @@ This merge request addresses...
 -  
 
 ### Quality Checks
- - [] New code is 100% tested
- - [] Code has been formated
- - [] Code has been linted
- - [] Docstrings for new methods have been added
+ - [ ] New code is 100% tested
+ - [ ] Code has been formated
+ - [ ] Code has been linted
+ - [ ] Docstrings for new methods have been added

docs/installation/resources/example-portal-config.yml

@@ -122,6 +122,12 @@ settings:
   #    SOCIAL_AUTH_ARCGIS_PORTAL_SECRET: ''
   #    SOCIAL_AUTH_ARCGIS_PORTAL_URL: ''
 
+  #    OAUTH2_PROVIDER:
+  #        URL_NAMESPACE: o
+  #        SCOPES:
+  #          read: Read scope
+
+
   #  ANALYTICS_CONFIG:
   #    CLICKMAP_TRACKER_ID: False
   #    CLICKY_SITE_ID: False

docs/supplementary/optional_features.rst

@@ -104,6 +104,8 @@ Enables a Tethys Portal to be a provider of OAuth2 authentication.
 **dependencies**
     - ``django-oauth-toolkit``
 
+See :ref:`oauth2_provider_settings`
+
 Portal Enhancements
 ===================
 

docs/tethys_portal/configuration.rst

@@ -179,7 +179,7 @@ CAPTCHA_CONFIG
 
 .. important::
 
-    These Captcha feature requires either the ``django-simple-captcha`` library or the ``django-recaptcha2`` library to be installed. Starting with Tethys 5.0 or if you are using ``micro-tethys-platform``, you will need to install one of these libraries using conda or pip as follows:
+    The Captcha feature requires either the ``django-simple-captcha`` library or the ``django-recaptcha2`` library to be installed. Starting with Tethys 5.0 or if you are using ``micro-tethys-platform``, you will need to install one of these libraries using conda or pip as follows:
 
     .. code-block:: bash
 
@@ -285,6 +285,36 @@ SOCIAL_AUTH_ONELOGIN_OIDC_SUBDOMAIN                    Your OneLogin Subdomain.
 SOCIAL_AUTH_ONELOGIN_OIDC_TOKEN_ENDPOINT_AUTH_METHOD   The authentication method to use when requesting tokens from the token endpoint. See :ref:`social_auth_onelogin` SSO Setup.
 ====================================================== ================================================================================
 
+.. _oauth2_provider_settings:
+
+OAUTH2_PROVIDER
++++++++++++++++
+
+.. important::
+
+    The OAuth2 Provider feature requires the ``django-oauth-toolkit`` library to be installed. Starting with Tethys 5.0 or if you are using ``micro-tethys-platform``, you will need to install one of these libraries using conda or pip as follows:
+
+    .. code-block:: bash
+
+        # conda: conda-forge channel strongly recommended
+        conda install -c conda-forge django-oauth-toolkit
+
+        # pip
+        pip install django-oauth-toolkit
+
+.. note::
+
+    The ``OAUTH2_PROVIDER`` heading can be listed under the ``OAUTH_CONFIG`` heading or it can be it's own heading.
+
+====================================================== ================================================================================
+Setting                                                Description
+====================================================== ================================================================================
+URL_NAMESPACE                                          The URL prefix to use to register the ``oauth2_provider`` urls. Default is ``o`` which produces URL endpoints like `<http://127.0.0.1:8000/o/applications/register/>`_.
+====================================================== ================================================================================
+
+For additional ``OAUTH2_PROVIDER`` refer to the `Django OAuth Toolkit documentation <https://django-oauth-toolkit.readthedocs.io/en/stable/settings.html>`_.
+
+
 MFA_CONFIG
 ++++++++++
 

docs/tethys_sdk/jobs.rst

@@ -71,7 +71,6 @@ You can now use the job manager to create a new job, or retrieve an existing job
 Creating and Executing a Job
 ----------------------------
 To create a new job call the ``create_job`` method on the job manager. The required arguments are:
-
     * ``name``: A unique string identifying the job
     * ``user``: A user object, usually from the request argument: `request.user`
     * ``job_type``: A string specifying on of the supported job types (see `Job Types`_)
@@ -119,13 +118,11 @@ Common Attributes
 Job attributes can be passed into the `create_job` method of the job manager or they can be specified after the job is instantiated. All jobs have a common set of attributes. Each job type may have additional attributes specific that are to that job type.
 
 The following attributes can be defined for *all* job types:
-
     * ``name`` (string, required): a unique identifier for the job. This should not be confused with the job template name. The template name identifies a template from which jobs can be created and is set when the template is created. The job ``name`` attribute is defined when the job is created (see `Creating and Executing a Job`_).
     * ``description`` (string): a short description of the job.
     * ``workspace`` (string): a path to a directory that will act as the workspace for the job. Each job type may interact with the workspace differently. By default the workspace is set to the user's workspace in the app that is creating the job.
     * ``extended_properties`` (dict): a dictionary of additional properties that can be used to create custom job attributes.
-    * ``status`` (string): a string representing the state of the job. Possible statuses are:
-
+    * ``status`` (string): a string representing the state of the job. When accessed the status will be updated if necessary. Possible statuses are:
         - 'Pending'
         - 'Submitted'
         - 'Running'
@@ -138,10 +135,12 @@ The following attributes can be defined for *all* job types:
         - 'Other'\**
 
         \*used for job types with multiple sub-jobs (e.g. CondorWorkflow).
+
         \**When  a custom job status is set the official status is 'Other', but the custom status is stored as an extended property of the job.
 
-All job types also have the following **read-only** attributes:
+    * ``cached_status`` (string): Same as the ``status`` attribute, except that the status is not actively updated. Rather the last known status is returned.
 
+All job types also have the following **read-only** attributes:
     * ``user`` (User): the user who created the job.
     * ``label`` (string): the package name of the Tethys App used to created the job.
     * ``creation_time`` (datetime): the time the job was created.
@@ -152,8 +151,7 @@ All job types also have the following **read-only** attributes:
 Job Types
 ---------
 
-The Jobs API is designed to support multiple job types. Each job type provides a different framework and environment for executing jobs. When creating a new job you must specify its type by passing in the `job_type` argument. Supported values for `job_type` are:
-
+The Jobs API is designed to support multiple job types. Each job type provides a different framework and environment for executing jobs. When creating a new job you must specify its type by passing in the ``job_type`` argument. Supported values for ``job_type`` are:
     * 'BASIC'
     * 'CONDOR' or 'CONDORJOB'
     * 'CONDORWORKFLOW'
@@ -193,7 +191,7 @@ Two methods are provided to retrieve jobs: ``list_jobs`` and ``get_job``. Jobs a
 
 Jobs Table Gizmo
 ----------------
-The Jobs Table Gizmo facilitates job management through the web interface and is designed to be used in conjunction with the Job Manager. It can be configured to list any of the properties of the jobs, and will automatically update the job status. It also can provide a list of actions that can be done on the a job. In addition to several build-in actions (including run, delete, viewing job results, etc.), developers can also create custom actions to include in the actions dropdown list. The following code sample shows how to use the job manager to populate the jobs table:
+The Jobs Table Gizmo facilitates job management through the web interface and is designed to be used in conjunction with the Job Manager. It can be configured to list any of the properties of the jobs, and will automatically update the job status. It also can provide a list of actions that can be done on the a job. In addition to several build-in actions (including run, delete, viewing job results, etc.), developers can also create custom actions to include in the actions dropdown list. Note that while none of the built-in actions are asynchronous on any of the built-in `Job Types`_, the Jobs Table supports both synchronous and asynchronous actions. Custom actions or the built-in actions of custom job types may be asynchronous. The following code sample shows how to use the job manager to populate the jobs table:
 
 ::
 

docs/tethys_sdk/routing.rst

@@ -118,7 +118,7 @@ Bokeh Integration in Tethys takes advantage of :ref:`websockets` and ``Django Ch
 
     Interactive ``Bokeh`` visualization tools can be entirely created using only Python with the help of ``Bokeh Server``. However, this usually requires the use of an additional server (``Tornado``). One of the alternatives to ``Tornado`` is using ``Django Channels``, which is already supported with Tethys. Therefore, interactive ``Bokeh`` models along with the all the advantages of using ``Bokeh Server`` can be leveraged in Tethys without the need of an additional server.
 
-Even though Bokeh uses :ref:`websockets`, routing with Bokeh endpoints is handled differently from other :ref:`websockets` that would normally be handled by a ``Consumer`` class and use the :ref:`consumer-decorator`. In contrast, Bokeh endpoints use a ``handler`` function that contains the main logic needed for a Bokeh model to be displayed. It contains the model or group of models as well as the callback functions that will help link them to the client. A ``handler`` function should be registered with the :ref:`handler-decorator`.
+Even though Bokeh uses :ref:`websockets`, routing with Bokeh endpoints is handled differently from other :ref:`websockets` that would normally be handled by a ``Consumer`` class and use the :ref:`consumer-decorator`. In contrast, Bokeh endpoints use a ``handler`` function that contains the main logic needed for a Bokeh model to be displayed. It contains the model or group of models as well as the callback functions that will help link them to the client. A ``handler`` function should be registered with the :ref:`handler-decorator`. Note that the :ref:`handler-decorator` supports both synchronous and asynchronous functions.
 
 ``Handlers`` are added to the ``Bokeh Document``, the smallest serialization unit in ``Bokeh Server``. This same ``Document`` is retrieved and added to the template variables in a ``controller`` function that is linked to the ``Handler function`` using Bokeh's ``server_document`` function. The ``controller`` function is created and registered automatically with the :ref:`handler-decorator`. However, you can manually create a ``controller`` function if custom logic is needed. In this case the ``controller`` function should not be decorated, but rather passed in as an argument to the :ref:`handler-decorator`.
 

tests/unit_tests/test_tethys_apps/test_base/test_bokeh_handler.py

@@ -17,7 +17,7 @@
 from tethys_apps.base.paths import TethysPath
 
 
-class TestBokehHandler(unittest.TestCase):
+class TestBokehHandler(unittest.IsolatedAsyncioTestCase):
     def setUp(self) -> None:
         self.app = baa.Application()
         self.doc = self.app.create_document()
@@ -45,6 +45,17 @@ def with_request_decorated(doc: Document):
         self.assertIsNotNone(getattr(ret_doc.request, "user", None))
         self.assertIsInstance(ret_doc.request, HttpRequest)
 
+    async def test_with_request_decorator_async(self):
+        @with_request
+        async def with_request_decorated(doc: Document):
+            return doc
+
+        ret_doc = await with_request_decorated(self.doc)
+
+        self.assertIsNotNone(getattr(ret_doc, "request", None))
+        self.assertIsNotNone(getattr(ret_doc.request, "user", None))
+        self.assertIsInstance(ret_doc.request, HttpRequest)
+
     @mock.patch("tethys_apps.base.bokeh_handler.deprecation_warning")
     @mock.patch("tethys_quotas.utilities.log")
     @mock.patch("tethys_apps.base.workspace.log")
@@ -117,6 +128,58 @@ def with_paths_decorated(doc: Document):
         self.assertEqual(TethysPath("resources").path, ret_doc.app_resources_path.path)
         self.assertEqual(TethysPath("public").path, ret_doc.app_public_path.path)
 
+    @mock.patch("tethys_apps.base.paths.Path.mkdir")
+    @mock.patch("tethys_quotas.utilities.log")
+    @mock.patch("tethys_apps.base.workspace.log")
+    @mock.patch("tethys_apps.utilities.get_active_app")
+    @mock.patch("tethys_apps.base.paths._get_app_workspace_root")
+    @mock.patch("tethys_apps.base.paths._get_app_media_root")
+    @mock.patch("tethys_apps.base.paths._resolve_app_class")
+    @mock.patch("tethys_apps.base.paths._resolve_username")
+    @override_settings(USE_OLD_WORKSPACES_API=False)
+    async def test_with_paths_decorator_async(
+        self, username, rac, mock_gamr, mock_gaw, _, __, ___, ____
+    ):
+        mock_gaw.return_value = Path("workspaces")
+        mock_gamr.return_value = Path("app-media-root/media")
+
+        mock_app = mock.MagicMock()
+        mock_app.package = "mock-app-package"
+        mock_app.resources_path = TethysPath("resources")
+        mock_app.public_path = TethysPath("public")
+        rac.return_value = mock_app
+
+        username.return_value = "mock-username"
+
+        @with_paths
+        async def with_paths_decorated(doc: Document):
+            return doc
+
+        ret_doc = await with_paths_decorated(self.doc)
+        self.assertIsNotNone(getattr(ret_doc, "user_workspace", None))
+        self.assertIsNotNone(getattr(ret_doc, "app_workspace", None))
+        self.assertIsNotNone(getattr(ret_doc, "app_media_path", None))
+        self.assertIsNotNone(getattr(ret_doc, "user_media_path", None))
+        self.assertIsNotNone(getattr(ret_doc, "app_resources_path", None))
+
+        self.assertEqual(
+            TethysPath("workspaces/user_workspaces/mock-username").path,
+            ret_doc.user_workspace.path,
+        )
+        self.assertEqual(
+            TethysPath("workspaces/app_workspace").path, ret_doc.app_workspace.path
+        )
+        self.assertEqual(
+            TethysPath("app-media-root/media/app").path,
+            ret_doc.app_media_path.path,
+        )
+        self.assertEqual(
+            TethysPath("app-media-root/media/user/mock-username").path,
+            ret_doc.user_media_path.path,
+        )
+        self.assertEqual(TethysPath("resources").path, ret_doc.app_resources_path.path)
+        self.assertEqual(TethysPath("public").path, ret_doc.app_public_path.path)
+
     @mock.patch("tethys_apps.base.bokeh_handler.render")
     @mock.patch("tethys_apps.base.bokeh_handler.server_document")
     def test_get_bokeh_controller(self, mock_server_doc, mock_render):

tests/unit_tests/test_tethys_compute/test_models/test_TethysJob.py

@@ -6,7 +6,7 @@
 from datetime import datetime, timedelta
 from pytz import timezone as pytz_timezone
 from django.utils import timezone as django_timezone
-from unittest import mock
+from unittest import mock, IsolatedAsyncioTestCase
 
 
 def test_function():
@@ -409,3 +409,10 @@ def test_lt(self):
         ret = sorted((self.tethysjob, self.tethysjob_execute_time))
         expected_value = [self.tethysjob, self.tethysjob_execute_time]
         self.assertListEqual(ret, expected_value)
+
+
+class AsyncTethysJobTest(IsolatedAsyncioTestCase):
+    async def test_safe_close(self):
+        job = TethysJob()
+        ret = await job.safe_close()
+        self.assertIsNone(ret)

tests/unit_tests/test_tethys_gizmos/test_gizmo_options/test_map_view.py

@@ -10,10 +10,33 @@ def setUp(self):
     def tearDown(self):
         pass
 
-    def test_MapView(self):
+    @mock.patch("tethys_gizmos.gizmo_options.map_view.deprecation_warning")
+    def test_MapView(self, mock_deprecation):
         height = "500px"
         width = "90%"
-        basemap = "Aerial"
+        basemap = [
+            "Aerial",
+            "OpenStreetMap",
+            "CartoDB",
+            [
+                {"CartoDB": {"style": "dark"}},
+                {
+                    "CartoDB": {
+                        "style": "light",
+                        "labels": False,
+                        "control_label": "CartoDB-light-no-labels",
+                    }
+                },
+            ],
+            {
+                "XYZ": {
+                    "url": "https://maps.wikimedia.org/osm-intl/{z}/{x}/{y}.png",
+                    "control_label": "Wikimedia",
+                }
+            },
+            "ESRI",
+            "BING",
+        ]
         controls = ["ZoomSlider", "Rotate", "FullScreen", "ScaleLine"]
 
         result = gizmo_map_view.MapView(
@@ -23,13 +46,14 @@ def test_MapView(self):
         # Check Result
         self.assertIn(height, result["height"])
         self.assertIn(width, result["width"])
-        self.assertIn(basemap, result["basemap"])
+        self.assertListEqual(basemap, result["basemap"])
         self.assertEqual(controls, result["controls"])
 
         self.assertIn(".js", gizmo_map_view.MapView.get_vendor_js()[0])
         self.assertIn(".js", gizmo_map_view.MapView.get_gizmo_js()[0])
         self.assertIn(".css", gizmo_map_view.MapView.get_vendor_css()[0])
         self.assertIn(".css", gizmo_map_view.MapView.get_gizmo_css()[0])
+        mock_deprecation.assert_called_once()
 
     def test_MVView(self):
         projection = "EPSG:4326"