Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug 1889156: script to load processed crashes into Elasticsearch #6609

Merged
merged 3 commits into from
May 22, 2024

Conversation

biancadanforth
Copy link
Contributor

@biancadanforth biancadanforth commented May 8, 2024

Because:

This PR:

  • Adds a script that loads processed crashes from either S3 or GCS (depending on settings.CRASH_SOURCE) into Elasticsearch by either date (YYYY-MM-DD) or crash ID.
How to test this locally

I have painstakingly added test cases for all the major scenarios, so running the tests in socorro/tests/test_load_processed_crashes_into_es.py (or replicating the setup before running the script with similar args) should be sufficient.

The test cases conditionally skip based on settings.CLOUD_PROVIDER, so to test the script with GCS as the crash storage source locally, you can edit your my.env so that CLOUD_PROVIDER=GCP and rebuild the test container (which in my case is the devcontainer). Just don't forget to revert that change when you're done. See #6609 (comment) -- this PR now runs both test cases against both cloud providers thanks to some clever pytest config.

I can add some manual testing instructions if things aren't clear from the test. I might just add those here tomorrow.

@biancadanforth biancadanforth force-pushed the bug-1889156 branch 3 times, most recently from ecddb5a to ed713a7 Compare May 8, 2024 16:12
@biancadanforth biancadanforth force-pushed the bug-1889156 branch 12 times, most recently from 7720a12 to cbe3687 Compare May 20, 2024 21:25
@biancadanforth biancadanforth marked this pull request as ready for review May 20, 2024 21:37
@biancadanforth biancadanforth requested a review from a team as a code owner May 20, 2024 21:37
@biancadanforth biancadanforth requested a review from relud May 20, 2024 21:37
socorro/tests/test_load_processed_crashes_into_es.py Outdated Show resolved Hide resolved
Comment on lines 195 to 199
except Exception as exc:
click.echo(
f"Unable to load crash with ID {crash_id!r} from {type(crash_source).__name__!r}; error: {exc}."
)
continue
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't think of any special types of exceptions that we'd want to handle differently -- are there any?

Also, please let me know if we'd like to handle exceptions in the general case differently -- right now we just log to stdout that we couldn't load it and move on.

@biancadanforth
Copy link
Contributor Author

Okay I've updated the PR for all your feedback except for #6609 (comment) . I'll submit a separate commit for that tomorrow, so it's easier to revert if willkg weighs in and says it's a bad idea.

Copy link
Member

@relud relud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@biancadanforth biancadanforth requested a review from relud May 22, 2024 15:52
@biancadanforth biancadanforth merged commit 4c6a914 into main May 22, 2024
2 checks passed
@biancadanforth biancadanforth deleted the bug-1889156 branch May 22, 2024 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants