[Fixes #7945] Ingest harvested layer data to geonode

[meomancer]

References: #7945

This PR implements copying of remote layers and documents in the GeonodeLegacyHarvester. It also includes a number of bug fixes.

Copying remote data is configured by means of editing a Harvester's harvester type specific configuration parameter. The following additional configuration parameters are now available:

• copy_documents
• copy_datasets

They both default to False, which means that by default a harvester will not copy remote data. Setting them to True will cause GeoNode to copy the remote data when performing harvesting.

The video below shows the proposed implementation in action:

2021-09-21.17-35-34_18.6.12.mp4

In the video I am demonstrating harvesting with two harvesters simultaneously. One of the harvesters (copys pcrafi) performs a copy of the remote data onto the local GeoNode, while the other (links pcrafi) merely copies the metadata. The video also demonstrates an error with the copying of a remote raster (the error is intended here) and how this is recorded in the corresponding harvesting session. The harvested resources appear on the GeoNode frontend.

Improvements

• The HarvestingSession model is now more useful, as it records the progress of harvesting operations as they are on-going
• The harvesting section of the django admin now includes a number of additional fields in order to ease navigation between harvesters, harvesting sessions and harvestable resources.
• It is now possible to harvest individual harvestable resources directly from the admin section without needing to go to the harvester page. This is mainly useful for testing.

Fixes

Metadata fields for resource's thumbnail, URL extension (for documents) and subtype (for datasets) are now correctly provided.

When harvesting is merely referencing the remote resources (i.e. we are not copying the actual data, just the metadata):

• Only metadata is copied over to the local GeoNode
• Remote thumbnail is used (fixes geosolutions/nexus-geonode#450)
• Remote resource URL is used

Notes

In the above video we can see that for remote documents, the GeoNode frontend offers a download of the file in the main page. This seems like a bug of the frontend, so I'm leaving it out of the current PR's scope (which is already quite big).

fixes #7945

[afabiani]

So far it almost works fine but...

ISSUE 1

I noticed that in the case I leave the option

{
   "harvest_datasets": true,
   "copy_datasets": false
}

The layer won't be populated not accessible.

E.g.: by using the demo instance https://development.demo.geonode.org/ as a source I get the following

The dataset is empty.

ISSUE 2

Trying to harvest a Map fails with an exception, the session and the harvester are then blocker forever.

Internal Server Error: /en/admin/harvesting/harvestableresource/
Traceback (most recent call last):
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/django/contrib/admin/options.py", line 616, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/django/utils/decorators.py", line 130, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/django/contrib/admin/sites.py", line 232, in inner
    return view(request, *args, **kwargs)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/django/utils/decorators.py", line 43, in _wrapper
    return bound_method(*args, **kwargs)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/django/utils/decorators.py", line 130, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/django/contrib/admin/options.py", line 1723, in changelist_view
    response = self.response_action(request, queryset=cl.get_queryset(request))
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/django/contrib/admin/options.py", line 1408, in response_action
    response = func(self, request, queryset)
  File "/mnt/c/Work/geonode/geonode/harvesting/admin.py", line 365, in harvest_selected_resources
    tasks.harvest_resources.apply_async(args=(harvestable_resource_ids, harvesting_session.pk))
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/task.py", line 569, in apply_async
    return self.apply(args, kwargs, task_id=task_id or uuid(),
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/task.py", line 790, in apply
    ret = tracer(task_id, args, kwargs, request)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/trace.py", line 467, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/trace.py", line 450, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/mnt/c/Work/geonode/geonode/harvesting/tasks.py", line 147, in harvest_resources
    harvesting_workflow.apply_async()
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/canvas.py", line 1408, in apply_async
    return self.apply(args, kwargs,
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/canvas.py", line 1427, in apply
    args=(tasks.apply(args, kwargs).get(propagate=propagate),),
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/canvas.py", line 1117, in apply
    return app.GroupResult(group_id, [
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/canvas.py", line 1118, in <listcomp>
    sig.apply(args=args, kwargs=kwargs, **options) for sig, _, _ in tasks
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/canvas.py", line 186, in apply
    return self.type.apply(args, kwargs, **options)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/task.py", line 790, in apply
    ret = tracer(task_id, args, kwargs, request)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/trace.py", line 467, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/home/afabiani/.virtualenvs/geonode/lib/python3.8/site-packages/celery/app/trace.py", line 450, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/mnt/c/Work/geonode/geonode/harvesting/tasks.py", line 174, in _harvest_resource
    harvested_resource_info = worker.get_resource(harvestable_resource, harvesting_session_id)
  File "/mnt/c/Work/geonode/geonode/harvesting/harvesters/geonodeharvester.py", line 299, in get_resource
    resource_descriptor = self._get_resource_details(
  File "/mnt/c/Work/geonode/geonode/harvesting/harvesters/geonodeharvester.py", line 444, in _get_resource_details
    result = self._get_resource_descriptor(
  File "/mnt/c/Work/geonode/geonode/harvesting/harvesters/geonodeharvester.py", line 547, in _get_resource_descriptor
    additional_params_handler = {
KeyError: 'maps'

ISSUE 3

Whenever an harvest session fails, there's no way to recover the harvester, which remains busy forever.

[giohappy]

Ragarding ISSUE 3 this is a long standing request that was already raised in the past @ricardogsilva. If a job breaks there's no way to recover it but going into the DB and manually changing the status back to ready. If we need a dedicated issue for this please open it @ricardogsilva.

[ricardogsilva]

@afabiani I will make the necessary adjustments. Some clarification wrt the issues you raise:

for Issue 1, the current GeonodeLegacyHarvester has been designed for harvesting older GeoNode (read pre-v2 API) instances. The fact that it does not work on the development demo of GeoNode, while unfortunate, is not too surprising. The plan is to create another worker that is able to serve modern GeoNode instances - the code will be a lot cleaner, since this legacy harvester has some additional complexity to compensate for the fact that the GeoNode API used to be a bit rough. I think the ideal solution will be to have a single harvester that is able to distinguish between older and newer GeoNode deployments. We ar e not there yet though.

For issue 2, that is a clear bug, which I will fix ASAP. The current harvester worker does not claim to support maps, so it should just fail in an orderly fashion.

For issue 3, that is a by-product of the bug described above probably. Generally speaking, the code is trying to catch known exceptions in the code so that it avoids entering that invalid state. The fact that it is still happening is a sign of some error in the code which is not being properly caught.

[giohappy]

@ricardogsilva regarding ISSUE 1, I expect the legacy harvester to work even on 3.3.x (the development demo) and master, since nothing from the old API and CSW has been dropped so far.

[afabiani]

@ricardogsilva just to be more clear a bit, notice that about ISSUE #1, if I enable copy_datasets=True it works fine.
The issue seems to be happening in the case I wanted to leave the data on the remote service.

[afabiani]

btw, any chance to fix also this one while looking for the other issues?

======================================================================
FAIL: test_perform_harvesting (geonode.harvesting.tests.test_admin.HarvesterAdminTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/unittest/mock.py", line 1325, in patched
    return func(*newargs, **newkeywargs)
  File "/usr/src/geonode/geonode/harvesting/tests/test_admin.py", line 73, in test_perform_harvesting
    self.assertEqual(self.harvester.status, models.Harvester.STATUS_PERFORMING_HARVESTING)
AssertionError: 'ready' != 'harvesting-resources'
- ready
+ harvesting-resources

[afabiani]

ISSUE 4

Raster datasets seem to be not harvested correctly even when the copy_dataeset option has been set to True

[lgtm-com]

This pull request introduces 2 alerts when merging 45fc7c2 into bf23880 - view on LGTM.com

new alerts:

• 2 for Clear-text logging of sensitive information

[ricardogsilva]

@afabiani

My latest commit should fix the issues you reported.

I fixed some missing metadata fields not being correctly passed to the resource manager.

I also disabled harvesting of maps for the GeoNode harvester. The reason for this is that API does not really provide enough information for us to be able to replicate a map locally - there is no info about the map's layers, etc.

To the best of my knowledge documents and datasets are now correctly harvested, regardless of being of type vector or raster and regardless of them being copied or just referenced from the remote.

I cannot guarantee 100% that this works for all GeoNode remote deployments, as I'm not sure how stable the old API had been between releases. Seems to work OK with those deployments that I am using as a test bed.

As for the failing test, I don't seem to get it locally. I'm waiting to let the CI build be done to check it output.

[lgtm-com]

This pull request introduces 2 alerts when merging a9ec56a into bf23880 - view on LGTM.com

new alerts:

• 2 for Clear-text logging of sensitive information

[afabiani]

@ricardogsilva there still something not fully working with the remote resources and in particular with the WMS harvester, but I'll postpone those fixes to #8149 which will include this PR too.