DM-52399: Remove daf_relation from registry.certify() and decertify() #1260
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Move RemoteButler's registry query implementations to a new base class. This prepares for using this implementation in DirectButler as well.
Previously, most registry tests were running against `Butler._registry`. For DirectButler `_registry` is the internal `SqlRegistry` implementation, and not the user-facing `RegistryShim` implementation. We will soon be gutting large portions of `SqlRegistry`, so the tests need to be running against the public `RegistryShim` object rather than the internals.
Add registry query shims for findDatasets and findRelatedDatasets, which are not yet deprecated but which were not yet implemented in RemoteButler. Also added a stub for materialize() as a no-op. The one place we are aware of where this was being used is now gone, but this is not yet deprecated.
Implement the 'check' parameter for skipping governor constraint validation in the shims from the old registry query system to the new query system.
Switch the implementation of `Registry.queryDatasets`, `queryDataIds`, `queryDimensionRecords`, and `queryDatasetAssociations` to use shims to the new query system, shared with the `RemoteButler` implementation.
Remove unit test code paths that only applied to the old query system.
Ahead of reworking its implementation, add a few tests to nail down the behavior of edge cases for findDataset.
It turns out that some downstream code (in pipe_tasks and possibly others) relies on undocumented behavior where dimension records will be preserved from the input data ID to the output ref.
Fix an issue where attempting to query datasets with a timespan overlap constraint but no calibration collections in the search would cause the query builder to raise "sqlalchemy.exc.ArgumentError: Only '=', '!=', 'is_()', 'is_not()', 'is_distinct_from()', 'is_not_distinct_from()' operators can be used with None/True/False"
Replace the implementation of Registry.findDataset with the new query system, to allow us to retire daf_relation.
dhirving
force-pushed
the
tickets/DM-52399
branch
from
debc402 to
a78f5cf
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## tickets/DM-52398 #1260 +/- ##
====================================================
- Coverage 88.33% 88.32% -0.02%
====================================================
Files 379 379
Lines 51174 51182 +8
Branches 6116 6117 +1
====================================================
- Hits 45205 45204 -1
- Misses 4522 4534 +12
+ Partials 1447 1444 -3 ☔ View full report in Codecov by Sentry. |
Modify registry.certify() to use the new query system instead of daf_relation, because we plan to retire daf_relation.
dhirving
force-pushed
the
tickets/DM-52399
branch
from
a78f5cf to
c50b8d3
Compare
dhirving
force-pushed
the
tickets/DM-52398
branch
3 times, most recently
from
c8830ea to
79c994b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist
doc/changesconfigs/old_dimensions