feat: Introduce new create_data_warehouse_table_from_csv helper
#30900
Conversation
Testing queries against data warehouse tables is extremely hard right now. We're doing it in different ways in different places. Let's standardize it by providing a common interface we can use everywhere: CSV files + single function to handle creating tables, sources, credentials, etc. Extracted from #30895
We're storing our data warehouse data inside `objectstorage` locally but that's not properly resolved by default because we didn't add it to `/etc/hosts`. There are 2 solutions to it: - run tests inside docker - expose that outside docker via `/etc/hosts` We're doing the latter :) I've also fixed the script slightly. We were previously only appending that to the hosts file if running on an interactive shell, but that's not right, we should always run it
rafaeelaudibert
changed the title
create_data_warehouse_table_from_csv helper
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR Summary
This PR standardizes data warehouse table creation in tests by introducing a new CSV-based helper and refactoring existing test setups for consistency and easier cleanup.
- In
/posthog/hogql_queries/experiments/test/experiment_query_runner/base.py, replaced duplicated S3 logic with the new helper, dynamically assigning cleanup functions. - In
/posthog/hogql_queries/experiments/test/experiment_query_runner/utils.py, removed DW table creation code and external dependencies. - In
/posthog/hogql_queries/experiments/test/test_experiment_trends_query_runner.py, integrated the helper for payments, usage, subscriptions, and customer tables. - Added the helper in
/posthog/warehouse/test/utils.pyto centralize CSV ingestion and cleanup logic.
6 file(s) reviewed, 1 comment(s)
Edit PR Review Bot Settings | Greptile
rafaeelaudibert
force-pushed
the
improve-data-warehouse-test-setup
branch
from
9554dfc to
d73285b
Compare
…0900) Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
|
Hey, was this line supposed to be removed from I was setting up things locally and this is where I was blocked. Should I add it back? |
|
Hey @Radu-Raicea, welcome! 🫡 It's unusual you're the first person hit by this in 4 months, and we've had way over 20 people onboarding that time, so... sounds like we dont really need it? If it blocked you tho, yeah, let's just add it back. PRs > issues :) |
|
And no, I dont remember why I removed it. So... happy to just have it back |
1 similar comment
|
And no, I dont remember why I removed it. So... happy to just have it back |
|
There you go! #35324 Maybe we all hit it but none decided to put a sign! |
Testing queries against data warehouse tables is extremely hard right now. We're doing it in different ways in different places. Let's standardize it by providing a common interface we can use everywhere: CSV files + a single function to create tables, sources, credentials, etc.
Extracted from #30895