Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate create_plugin_registries #781

Open
wants to merge 14 commits into
base: main
Choose a base branch
from

Conversation

evrose54
Copy link
Contributor

@evrose54 evrose54 commented Sep 16, 2024

Reviewer Checklist

  • Required existing tests pass (ie full_test.sh, others as appropriate)
  • Required unit tests added and pass for new/modified functionality
  • NO REQUIRED integration tests (explain why not required)
  • Required documentation added for new/modified functionality
  • Required release notes added for new/modified functionality
  • Required updates to other repos added for new/modified functionality

Related Issues

fixes #647
Correlated PR in data_fusion is required to fix a slowdown to data_fusion procflows which was introduced by this PR.

We can close the correlated PR if we decide to keep this PR as-is. A discussion needs to occur about the downstream impacts of defaulting, or not defaulting, to rebuilding the plugin registries on a plugin retrieval failure.

Testing Instructions

Run ./tests/integration_tests/full_test.sh

Summary

The command 'create_plugin_registries' has greatly improved the time it takes to
operate GeoIPS under the hood. This is because all plugins are no longer loaded into
memory, as we can now traverse a plugin registry to load the exact plugins we need
to, when we need them. Unfortuneately, this requires the user to run
'create_plugin_registries' anytime a new plugin has been added to one or more plugin
packages. We've been thinking for a long time of how to automate this process, and
this update does just that.

'create_plugin_registries' can now either be ran manually, or will attempt to be
done automatically by GeoIPS when a plugin is requested. If a user has no registries
in any of their plugin packages, then this is ran the first time get_plugin is
called when certain attributes of the associated interface are being initialized.

If the registries already exist, but a plugin is missing or seemingly out of sync,
then 'create_plugin_registries' will be ran, and get_plugin will be called again. If
the failure persists, then it is likely a problem on the user's end that they need
to fix before running GeoIPS. Now, users no longer have to run
'create_plugin_registries', unless they choose to do so manually.

File Updates:

added:
  - tests/unit_tests/plugins/test_get_plugin.py
modified:
  - geoips/commandline/geoips_test.py
  - geoips/interfaces/base.py
  - geoips/interfaces/module_based/output_checkers.py
  - geoips/interfaces/yaml_based/products.py

Output

from geoips.interfaces import readers

test_plg = readers.get_plugin("fake_plugin")

# OUTPUT

Running create_plugin_registries due to a missing plugin.
16_203926 create_plugin_registries.py:318  INTERACTIVE: Writing /local/home/evan/geoips/geoips_packages/geoips_clavrx/geoips_clavrx/registered_plugins.json
16_203927 create_plugin_registries.py:318  INTERACTIVE: Writing /local/home/evan/geoips/geoips_packages/data_fusion/data_fusion/registered_plugins.json
16_203927 create_plugin_registries.py:318  INTERACTIVE: Writing /local/home/evan/geoips/geoips_packages/recenter_tc/recenter_tc/registered_plugins.json
16_203927 create_plugin_registries.py:318  INTERACTIVE: Writing /local/home/evan/geoips/geoips_packages/geoips_plugin_example/geoips_plugin_example/registered_plugins.json
16_203928 create_plugin_registries.py:820  INTERACTIVE: Skipping module 'interp_scipy' from 'geoips', interface_name is 'None'
16_203928 create_plugin_registries.py:820  INTERACTIVE: Skipping module 'boxdefinitions' from 'geoips', interface_name is 'None'
16_203928 create_plugin_registries.py:820  INTERACTIVE: Skipping module 'interp_pyresample' from 'geoips', interface_name is 'None'
16_203928 create_plugin_registries.py:820  INTERACTIVE: Skipping module 'geostationary_geolocation' from 'geoips', interface_name is 'None'
16_203928 create_plugin_registries.py:820  INTERACTIVE: Skipping module 'hrit_reader' from 'geoips', interface_name is 'None'
16_203928 create_plugin_registries.py:820  INTERACTIVE: Skipping module 'remss_reader' from 'geoips', interface_name is 'None'
16_203928 create_plugin_registries.py:820  INTERACTIVE: Skipping module 'tc_file_naming' from 'geoips', interface_name is 'None'
16_203928 create_plugin_registries.py:318  INTERACTIVE: Writing /local/home/evan/geoips/geoips_packages/geoips/geoips/registered_plugins.json

PluginError: Plugin 'fake_plugin', from interface 'readers' appears to not exist.
Create plugin, then call create_plugin_registries?

@evrose54 evrose54 marked this pull request as draft September 16, 2024 20:41
@evrose54 evrose54 added Dev Update Minor developmental releases (Ie, X.Y.X.devN) refactor Code refactoring updates labels Sep 18, 2024
@evrose54 evrose54 marked this pull request as ready for review September 18, 2024 20:48
Copy link
Contributor

@jsolbrig jsolbrig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should have a CLI flag added to allow turning on automatic registry creation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Dev Update Minor developmental releases (Ie, X.Y.X.devN) refactor Code refactoring updates
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Automating create_plugin_registries
2 participants