Allow aborting of on-going harvesting tasks #8176

ricardogsilva · 2021-09-27T17:41:40Z

This PR implements aborting of on-going harvesting-related asynchronous tasks.

It improves the existing concept of a harvesting session by implementing the AsynchronousHarvestingSession model. This model is now used for the two types of async operations supported by a harvester:

Refreshing the list of harvestable resources that are available for being harvested on the remote;
Performing the actual harvesting of the selected remote resources.

Each session also stores some useful properties and can be used to get a sense of progress. For now, some of these properties are also being reflected on the Harvester model. In the future we may remove them from the harvester and use the session exclusively, as it seems like a more natural place to store them.

Despite introducing better support for harvesting-related sessions, the HarvesterAdmin continues to be the main place to kickoff/abort new sessions. As such, the admin section for harvesting.harvesters gains new actions for aborting on-going sessions. An aborted session is signaled as aborted by means of its status field. On the API side this means that the harvesting sessions continue to be read-only and that the harvesters/{harvester-id} continues to be the endpoint to reach for whenever starting/aborting a new session - this is done via sending HTTP PATCH requests that modify the harvester's status field and was not changed.

Another notable change introduced by this PR is that when a harvester is created via the django admin it no longer automatically updates its underlying harvestable resources list. The user must now explicitly ask for a refresh of the harvestable resources. This change was introduced to both a) make the code cleaner and b) give the user one more chance to review the harvester parameters after creation, without forcing him to wait for a potentially long time before the harvester becomes ready again.

fixes #7955

Merge from upsteram

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.2 to 1.26.3. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/1.26.3/CHANGES.rst) - [Commits](urllib3/urllib3@1.26.2...1.26.3) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni <toni.schoenbuchner@csgis.de>

…6881) * [Fixes GeoNode#6880] Circle CI upload tests fail irregulary * CircleCI test fix: sometimes expires due to upload timeout in the test environment * - Avoid infinite loop on upload testing * Revert "CircleCI test fix: sometimes expires due to upload timeout in the test environment" This reverts commit 66139fd. Co-authored-by: Alessio Fabiani <alessio.fabiani@geo-solutions.it> Co-authored-by: afabiani <alessio.fabiani@gmail.com>

…ps (GeoNode#6915)

…de#6911) * get meaningful document filenames on download * - Strip extension from document title before slugify it (e.g.: image.jpg instead of imagejpg.jpg) Co-authored-by: afabiani <alessio.fabiani@gmail.com> Co-authored-by: Alessio Fabiani <alessio.fabiani@geo-solutions.it>

…loop on "wait_for_progress"

…ng slash at the end of GEOSERVER_LOCATION (GeoNode#6913) * [Fixes GeoNode#6916] gsimporter.api.NotFound caused by missing trailing slash at the end of GEOSERVER_LOCATION * [Fixes GeoNode#6916] unit test for GEOSERVER_LOCATION

Bumps [django-cors-headers](https://github.com/adamchainz/django-cors-headers) from 3.6.0 to 3.7.0. - [Release notes](https://github.com/adamchainz/django-cors-headers/releases) - [Changelog](https://github.com/adamchainz/django-cors-headers/blob/master/HISTORY.rst) - [Commits](adamchainz/django-cors-headers@3.6.0...3.7.0) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [amqp](https://github.com/celery/py-amqp) from 5.0.3 to 5.0.5. - [Release notes](https://github.com/celery/py-amqp/releases) - [Changelog](https://github.com/celery/py-amqp/blob/master/Changelog) - [Commits](celery/py-amqp@v5.0.3...v5.0.5) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [pip](https://github.com/pypa/pip) from 21.0 to 21.0.1. - [Release notes](https://github.com/pypa/pip/releases) - [Changelog](https://github.com/pypa/pip/blob/master/NEWS.rst) - [Commits](pypa/pip@21.0...21.0.1) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [coverage](https://github.com/nedbat/coveragepy) from 5.3.1 to 5.4. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](coveragepy/coveragepy@coverage-5.3.1...coverage-5.4) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.1 to 6.2.2. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst) - [Commits](pytest-dev/pytest@6.2.1...6.2.2) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [djangorestframework-gis](https://github.com/openwisp/django-rest-framework-gis) from 0.16 to 0.17. - [Release notes](https://github.com/openwisp/django-rest-framework-gis/releases) - [Changelog](https://github.com/openwisp/django-rest-framework-gis/blob/master/CHANGES.rst) - [Commits](openwisp/django-rest-framework-gis@v0.16.0...v0.17.0) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

… it has… (GeoNode#6923) * [Fixes GeoNode#6922][REST API v2] Expose the curated thumbnail URL if it has been uploaded * - Add REST APIs test suite to CircleCI

* [Cleanup and Refactor] Remove QGIS server backend dependencies * [Cleanup and Refactor] Remove QGIS server backend dependencies * - Fix LGTM issues

…iddleware Feature#650 basic auth middleware

Upstream master

…olutions_master

lgtm-com · 2021-09-30T09:35:42Z

This pull request introduces 1 alert when merging d9f3a3d into 2164b02 - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

lgtm-com · 2021-09-30T10:06:13Z

This pull request introduces 1 alert when merging da9a933 into 2164b02 - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

ricardogsilva · 2021-09-30T11:25:25Z

@afabiani the drop in coverage seems to be unrelated to this PR. Perhaps due to other parts of the codebase changing? Or maybe I am reading things wrong?

lgtm-com · 2021-09-30T11:34:08Z

This pull request introduces 1 alert when merging 795c90c into 2164b02 - view on LGTM.com

new alerts:

1 for Signature mismatch in overriding method

afabiani · 2021-09-30T12:44:42Z

@ricardogsilva no worries, I really don't think it is depending by this PR

afabiani · 2021-09-30T12:47:54Z

@ricardogsilva should we worried about that
https://lgtm.com/projects/g/GeoNode/geonode/rev/pr-188ed5bc3da3aca1edeeb3a2355e430771ca961b

ricardogsilva · 2021-09-30T13:53:27Z

@afabiani yeah let me fix that before you move on

Add the `AsynchronousHarvestingSession` model, which is used to implement sessions for both refreshing of a harvester's harvestable resources and for the harvesting of remote resources. Refactor the `admin`, `api` and `tasks` to use this new model. Moved some functions out of `harvesting.utils` module in order to avoid circular imports

…esolution

ricardogsilva · 2021-09-30T14:06:48Z

@afabiani should be fixed now

ricardogsilva · 2021-09-30T14:30:14Z

@afabiani I accidentally left a commented line in settings.py (I'm using the default test runner) - let me quickly fix that and then I think it would be good to merge.

…as the one to be used

ricardogsilva · 2021-09-30T14:32:23Z

done!

giohappy and others added 30 commits January 20, 2021 09:37

Merge pull request #626 from geosolutions-it/merge_from_upsteram

f654111

Merge from upsteram

Merge branch 'master' of https://github.com/GeoNode/geonode

0df9357

[Fixes GeoNode#6914] Remove "add to basket" tool for documents and ma…

65c6267

…ps (GeoNode#6915)

Added malnajdi as contributor

158511a

- CircleCI Upload Tests: trying to reduce more the risk of infinite …

91f1a4e

…loop on "wait_for_progress"

- Algin setup.cfg to requirements.txt

ae6f6c1

[Fixes GeoNode#6922][REST API v2] Expose the curated thumbnail URL if…

0dad216

… it has… (GeoNode#6923) * [Fixes GeoNode#6922][REST API v2] Expose the curated thumbnail URL if it has been uploaded * - Add REST APIs test suite to CircleCI

[Fixes GeoNode#6918] Removal of QGIS support (GeoNode#6919)

30910b2

* [Cleanup and Refactor] Remove QGIS server backend dependencies * [Cleanup and Refactor] Remove QGIS server backend dependencies * - Fix LGTM issues

Merge branch 'master' of https://github.com/GeoNode/geonode

dbb4ac0

allow Basic authenticated requests in LOCKDOWN mode

0bcf2fb

fix to avoid circular import

af22a3c

flake8 check fix

c7ede4f

added tests

b557f6d

Merge pull request #652 from geosolutions-it/feature#650_basic_auth_m…

0a82a14

…iddleware Feature#650 basic auth middleware

Merge pull request #678 from geosolutions-it/upstream_master

11f40ba

Upstream master

Merge remote-tracking branch 'upstream/master'

482080a

Merge remote-tracking branch 'upstream/master'

2f70532

Merge branch 'master' of https://github.com/geosolutions-it/geonode

de43e90

Merge remote-tracking branch 'origin/master' into geosolutions-master

40408a3

Merge branch 'master' of https://github.com/GeoNode/geonode into geos…

4ce20ab

…olutions_master

ricardogsilva force-pushed the 7955-allow-stopping-tasks branch from 8f6b489 to d9f3a3d Compare September 30, 2021 09:26

ricardogsilva marked this pull request as ready for review September 30, 2021 11:24

ricardogsilva requested a review from afabiani September 30, 2021 11:24

Ricardo Garcia Silva and others added 11 commits September 30, 2021 15:04

Merge branch 'master' of github.com:GeoNode/geonode

bd8de19

[Fixes GeoNode#7945] Ingest harvested layer data to geonode

4fa8283

Improve harvesting session and the admin

dd43d6c

fix migration files conflict

4edd83d

Initial work for implementing stoppable harvesting sessions

51eb810

fix tests

4f3c228

fix conflicts

28e25e9

Remove accidental duplication of code that crept in during conflict r…

aa39ffd

…esolution

Add a couple more tests

9e1f59e

Update signature of get_resource method in WMS harvester worker

e091f05

ricardogsilva force-pushed the 7955-allow-stopping-tasks branch from c007191 to e091f05 Compare September 30, 2021 14:06

Uncomment settings.py line that designates the GeoNode test runner …

8359065

…as the one to be used

afabiani added this to the 4.0.0 milestone Oct 1, 2021

afabiani assigned ricardogsilva Oct 1, 2021

afabiani approved these changes Oct 1, 2021

View reviewed changes

afabiani merged commit 0867642 into GeoNode:master Oct 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Allow aborting of on-going harvesting tasks #8176

Allow aborting of on-going harvesting tasks #8176

Uh oh!

ricardogsilva commented Sep 27, 2021 •

edited

Loading

Uh oh!

lgtm-com bot commented Sep 30, 2021

Uh oh!

lgtm-com bot commented Sep 30, 2021

Uh oh!

ricardogsilva commented Sep 30, 2021

Uh oh!

lgtm-com bot commented Sep 30, 2021

Uh oh!

afabiani commented Sep 30, 2021

Uh oh!

afabiani commented Sep 30, 2021

Uh oh!

ricardogsilva commented Sep 30, 2021

Uh oh!

ricardogsilva commented Sep 30, 2021

Uh oh!

ricardogsilva commented Sep 30, 2021

Uh oh!

ricardogsilva commented Sep 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Allow aborting of on-going harvesting tasks #8176

Allow aborting of on-going harvesting tasks #8176

Uh oh!

Conversation

ricardogsilva commented Sep 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lgtm-com bot commented Sep 30, 2021

Uh oh!

lgtm-com bot commented Sep 30, 2021

Uh oh!

ricardogsilva commented Sep 30, 2021

Uh oh!

lgtm-com bot commented Sep 30, 2021

Uh oh!

afabiani commented Sep 30, 2021

Uh oh!

afabiani commented Sep 30, 2021

Uh oh!

ricardogsilva commented Sep 30, 2021

Uh oh!

ricardogsilva commented Sep 30, 2021

Uh oh!

ricardogsilva commented Sep 30, 2021

Uh oh!

ricardogsilva commented Sep 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

ricardogsilva commented Sep 27, 2021 •

edited

Loading