TIMX 491 - remove parquet feature flags #363
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Why these changes are being introduced: As of pipenv 2025.0.1 the use of `pipenv check` would throw an error, indicating that the library `safety` was not installed. It worked to run `pipenv check --auto-install` which would temporarily install `safety`, but this was not ideal for multiple reasons. First, we anticipate potentially moving away from `pipenv`. Second, it appears that `safety` is moving to a pay / subscription model. Third, it remains a little obfuscated what `pipenv check` is actually doing. As this new situation affects all builds in Github Actions CI, we need a way to scan for vulnerabilities that ideally is not a massive overhaul of our vulnerability scanning approach. How this addresses that need: `pip-audit` is a nice standalone, open-source library that performs very similar work to `safety`. This commit replaces `pipenv check` (which was `safety` under the hood) with `pip-audit`. Side effects of this change: * Builds will be successful in Github Actions Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/IN-1240
Why these changes are being introduced: During development work of establishing the parquet dataset architecture for TIMDEX ETL, feature flags were used in various applications to remain backwards compatible. Now that the parquet dataset architecture is established, the feature flags are no longer needed. How this addresses that need: * Removes 'bulk-index' and 'bulk-delete' CLI commands, preferring only 'bulk-update' which expects a subset of records from the parquet dataset to index and delete (per the 'action' column in the dataset). Side effects of this change: * Indexing from a JSON array of TIMDEX Record objects is no longer supported, neither is a deleting from a JSON array of objects with a single 'timdex_record_id' field. All indexing and deleting is expected to originate from parquet dataset records. Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/TIMX-491
Why these changes are being introduced: With the removal of CLI commands bulk-index and bulk-delete, the test suite needed some updates. How this addresses that need: * Recreate VCR cassettes for bulk indexing and deleting by using local parquet dataset fixtures * Remove helpers no longer needed; parsing previous JSON and TXT files for input Side effects of this change: * None Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/TIMX-491
Pull Request Test Coverage Report for Build 14909955215Details
💛 - Coveralls |
ehanson8
approved these changes
May 9, 2025
Contributor
ehanson8
left a comment
ehanson8
left a comment
There was a problem hiding this comment.
Looks good and works as expected!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose and background context
This PR removes feature flags from TIM.
Before the parquet dataset, indexing and deleting were performed separately via CLI commands:
bulk-indexbulk-deletetimdex_record_idto remove from the indexNow that we are using the parquet dataset, which includes an
actioncolumn indicating if that record should be indexed or deleted, the need for distinct CLI commands is not needed. Now a singlebulk-updatecommand exists, expecting arun_idwhich provides a subset of records from the parquet dataset to index and delete from Opensearch.Note the commit 280262f which updates the VCR cassettes for indexing and deleting records. This commit has lots of file churn, but is really just re-recording the VCR cassettes to use the parquet dataset as input vs JSON + TXT files.
How can a reviewer manually see the effects of these changes?
1- Set AWS Dev credentials for
TimdexManagers2- Set env vars:
3- Run
bulk-updatecommand:Successful StepFunction run with feature flag removed TIM code in Dev: https://222053980223-lritm6hn.us-east-1.console.aws.amazon.com/states/home?region=us-east-1#/v2/executions/details/arn:aws:states:us-east-1:222053980223:execution:timdex-ingest-dev:85cfe316-089c-4639-a5af-c861a7321493.
Includes new or updated dependencies?
YES | NO
Changes expectations for external applications?
YES | NO
What are the relevant tickets?
Developer
Code Reviewer(s)