Skip to content

Ele 5122 cli stop using deprecated tests#2022

Merged
haritamar merged 11 commits intomasterfrom
ele-5122-cli-stop-using-deprecated-tests
Oct 9, 2025
Merged

Ele 5122 cli stop using deprecated tests#2022
haritamar merged 11 commits intomasterfrom
ele-5122-cli-stop-using-deprecated-tests

Conversation

@haritamar
Copy link
Collaborator

@haritamar haritamar commented Oct 8, 2025

null

Summary by CodeRabbit

  • Tests

    • Added a full end-to-end dbt test project (models, macros, snapshots, packages, seeds, singular tests), a synthetic data generator, debug helper, and a docker-compose setup.
    • Updated CI to a multi-step E2E flow (deps, seed, run, validate allowing one designated model failure, test) and removed manual gating flags; added new E2E environment variables.
  • Documentation

    • Added README notes describing the E2E project and how monitoring/reporting consume its data.
  • Chores

    • Broadened ignore rules to exclude dbt_packages repository-wide.

@linear
Copy link

linear bot commented Oct 8, 2025

@github-actions
Copy link
Contributor

github-actions bot commented Oct 8, 2025

👋 @haritamar
Thank you for raising your pull request.
Please make sure to add tests and document all user-facing changes.
You can do this by editing the docs files in this pull request.

@coderabbitai
Copy link

coderabbitai bot commented Oct 8, 2025

Walkthrough

Replaces a single package integration step with an explicit multi-step E2E pipeline, removes legacy workflow inputs/env vars, adds a full E2E dbt project (configs, packages, models, macros, snapshots, tests, data generator, docker-compose), broadens .gitignore for dbt_packages, and updates workflow references to the new E2E project path.

Changes

Cohort / File(s) Summary
CI Workflow
​.github/workflows/test-warehouse.yml
Removed should-run-tests/clear-tests inputs; replaced legacy env vars with ELEMENTARY_DBT_PACKAGE_PATH, CLI_INTERNAL_DBT_PKG_DIR, E2E_DBT_PROJECT_DIR; added explicit E2E steps: dbt deps, dbt seed, dbt run, validation of run_results.json (allow single designated failure), dbt test, monitor/report/artifact steps; updated --project-dir / working-directory references to E2E_DBT_PROJECT_DIR.
Git Ignore
.gitignore
Replaced a specific ignore path with recursive rule **/dbt_packages/ to ignore any dbt_packages directory.
E2E Project: Root & Docs
tests/e2e_dbt_project/README.md, tests/e2e_dbt_project/dbt_project.yml, tests/e2e_dbt_project/packages.yml
Added E2E dbt_project.yml (paths, vars, clean-targets, seed config), packages.yml (local package + templated dbt_utils version selection), and README describing the E2E project and workflow usage.
E2E Project: Data & Runtime
tests/e2e_dbt_project/generate_data.py, tests/e2e_dbt_project/debug.sh, tests/e2e_dbt_project/docker-compose.yml
Added Python data generator producing many training/validation CSVs, debug.sh exporting DBT debug envs, and docker-compose.yml defining Postgres and ClickHouse services.
Macros: Generic Tests
tests/e2e_dbt_project/macros/generic_tests/*
Added generic_test_on_model.sql and generic_test_on_column.sql macros that render a constant-row test query.
Macros: System
tests/e2e_dbt_project/macros/system/*
Added dbg() macro, generate_schema_name(custom_schema_name, node) for schema resolution, and two test materializations (default and Snowflake-specific).
Models: Test Models & Utilities
tests/e2e_dbt_project/models/*.sql, tests/e2e_dbt_project/models/nested/.../nested.sql
Added many models: unions of training/validation for anomaly types, pass-through/copy models, ephemeral model, non_dbt materialized model, intentional syntax-error error_model.sql, simple constant models like one.sql, and nested example.
Model Schema & Tests Config
tests/e2e_dbt_project/models/schema.yml
Added comprehensive schema.yml with models, sources, exposures, and extensive tests covering anomalies, freshness, schema changes, seasonality, config-level validations, and expectations.
Snapshots
tests/e2e_dbt_project/snapshots/failed_snapshot.sql
Added failed_snapshot snapshot (timestamp strategy, unique_key, updated_at, target_schema='snapshots').
Singular Tests
tests/e2e_dbt_project/tests/*.sql
Added singular SQL tests covering model refs, source refs, API.Relation-based relation creation, multi-ref unions/filters, and simple assertions.
Packages & Dependencies
tests/e2e_dbt_project/packages.yml
Local package reference to ../../ and templated remote dbt_utils version selection via Jinja.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor Trigger as Workflow Trigger
  participant Runner as GitHub Runner
  participant DBT as dbt CLI
  participant E2E as tests/e2e_dbt_project
  participant Validator as run_results.json Validator
  participant Monitor as EDR Monitor/Report

  Trigger->>Runner: start test-warehouse
  Runner->>DBT: (wd: $E2E_DBT_PROJECT_DIR) dbt deps
  Runner->>DBT: (wd: $E2E_DBT_PROJECT_DIR) dbt seed
  DBT-->>Runner: seeds loaded
  Runner->>DBT: (wd: $E2E_DBT_PROJECT_DIR) dbt run
  DBT-->>Runner: writes run_results.json
  Runner->>Validator: validate run_results.json (only designated model may fail)
  alt validation OK
    Runner->>DBT: (wd: $E2E_DBT_PROJECT_DIR) dbt test
    DBT-->>Runner: test results
    Runner->>Monitor: run edr monitor/report and collect artifacts
  else validation FAIL
    Runner-->>Trigger: fail workflow (unexpected failures)
  end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Suggested reviewers

  • elazarlachkar

Poem

I hopped through seeds and models new,
I stitched up macros, tests, and view.
One model fails by careful art,
The runner checks each tiny part.
Carrots for CI — a rabbit's heart. 🥕

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check ✅ Passed The title refers to ceasing use of deprecated tests in the CLI, which aligns with the removal of gating inputs around Python e2e tests, but it does not capture the broader migration to a full end-to-end DBT project and workflow updates introduced in this changeset. While it reflects a genuine aspect of the PR, it is overly narrow and includes a ticket ID that could be omitted for clarity.
Docstring Coverage ✅ Passed No functions found in the changes. Docstring coverage check skipped.
✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch ele-5122-cli-stop-using-deprecated-tests

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

Disabled knowledge base sources:

  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 2baebad and 8ce421e.

📒 Files selected for processing (1)
  • .github/workflows/test-warehouse.yml (6 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
  • GitHub Check: test / test
  • GitHub Check: code-quality

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🧹 Nitpick comments (6)
tests/e2e_dbt_project/models/test_alerts_union.sql (1)

12-18: Avoid SELECT * across UNIONs; enumerate columns to prevent drift.

Explicitly list columns for dbt, anomalies, and schema_changes to avoid brittle breakages if upstream models change column order.

tests/e2e_dbt_project/models/backfill_days_column_anomalies.sql (1)

15-23: Consistency: select explicit columns in the final projection.

Keep the outer query selecting the explicit column list instead of select * to guard against upstream changes.

tests/e2e_dbt_project/models/no_timestamp_anomalies.sql (1)

15-30: Nit: remove leading space before final CTE and avoid select * in outer query.

Minor readability/style; and explicitly list columns in the outer select to prevent schema drift issues.

tests/e2e_dbt_project/models/any_type_column_anomalies.sql (1)

15-30: Mirror the projection in the outer query; avoid select *.

Explicit outer select prevents brittle coupling to CTE internals.

tests/e2e_dbt_project/generate_data.py (2)

38-42: CSV writer: add newline='' and encoding to avoid extra blank lines and ensure UTF-8.

On some platforms CSVs will contain blank lines without newline=''. Also set encoding="utf-8" explicitly.

-    with open(csv_path, "w") as csv_file:
+    with open(csv_path, "w", newline="", encoding="utf-8") as csv_file:
         writer = csv.DictWriter(csv_file, fieldnames=header)
         writer.writeheader()
         writer.writerows(rows)

404-421: Silence lints: unused parameters in nested row builders.

row_index and rows_count aren’t used in these closures; rename to _/__ to satisfy linters.

-def get_training_row(date, row_index, rows_count):
+def get_training_row(date, _row_index, _rows_count):
@@
-def get_validation_row(date, row_index, rows_count):
+def get_validation_row(date, _row_index, _rows_count):
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

Disabled knowledge base sources:

  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 440fd9d and b4f6ea2.

⛔ Files ignored due to path filters (16)
  • tests/e2e_dbt_project/data/training/any_type_column_anomalies_training.csv is excluded by !**/*.csv
  • tests/e2e_dbt_project/data/training/backfill_days_column_anomalies_training.csv is excluded by !**/*.csv
  • tests/e2e_dbt_project/data/training/dimension_anomalies_training.csv is excluded by !**/*.csv
  • tests/e2e_dbt_project/data/training/groups_training.csv is excluded by !**/*.csv
  • tests/e2e_dbt_project/data/training/numeric_column_anomalies_training.csv is excluded by !**/*.csv
  • tests/e2e_dbt_project/data/training/stats_players_training.csv is excluded by !**/*.csv
  • tests/e2e_dbt_project/data/training/stats_team_training.csv is excluded by !**/*.csv
  • tests/e2e_dbt_project/data/training/string_column_anomalies_training.csv is excluded by !**/*.csv
  • tests/e2e_dbt_project/data/validation/any_type_column_anomalies_validation.csv is excluded by !**/*.csv
  • tests/e2e_dbt_project/data/validation/backfill_days_column_anomalies_validation.csv is excluded by !**/*.csv
  • tests/e2e_dbt_project/data/validation/dimension_anomalies_validation.csv is excluded by !**/*.csv
  • tests/e2e_dbt_project/data/validation/groups_validation.csv is excluded by !**/*.csv
  • tests/e2e_dbt_project/data/validation/numeric_column_anomalies_validation.csv is excluded by !**/*.csv
  • tests/e2e_dbt_project/data/validation/stats_players_validation.csv is excluded by !**/*.csv
  • tests/e2e_dbt_project/data/validation/stats_team_validation.csv is excluded by !**/*.csv
  • tests/e2e_dbt_project/data/validation/string_column_anomalies_validation.csv is excluded by !**/*.csv
📒 Files selected for processing (38)
  • .github/workflows/test-warehouse.yml (1 hunks)
  • .gitignore (1 hunks)
  • tests/e2e_dbt_project/README.md (1 hunks)
  • tests/e2e_dbt_project/dbt_project.yml (1 hunks)
  • tests/e2e_dbt_project/debug.sh (1 hunks)
  • tests/e2e_dbt_project/generate_data.py (1 hunks)
  • tests/e2e_dbt_project/macros/generic_tests/generic_test_on_column.sql (1 hunks)
  • tests/e2e_dbt_project/macros/generic_tests/generic_test_on_model.sql (1 hunks)
  • tests/e2e_dbt_project/macros/system/dbg.sql (1 hunks)
  • tests/e2e_dbt_project/macros/system/generate_schema_name.sql (1 hunks)
  • tests/e2e_dbt_project/macros/system/materializations.sql (1 hunks)
  • tests/e2e_dbt_project/models/any_type_column_anomalies.sql (1 hunks)
  • tests/e2e_dbt_project/models/backfill_days_column_anomalies.sql (1 hunks)
  • tests/e2e_dbt_project/models/config_levels_project.sql (1 hunks)
  • tests/e2e_dbt_project/models/config_levels_test_and_model.sql (1 hunks)
  • tests/e2e_dbt_project/models/copy_numeric_column_anomalies.sql (1 hunks)
  • tests/e2e_dbt_project/models/dimension_anomalies.sql (1 hunks)
  • tests/e2e_dbt_project/models/ephemeral_model.sql (1 hunks)
  • tests/e2e_dbt_project/models/error_model.sql (1 hunks)
  • tests/e2e_dbt_project/models/groups.sql (1 hunks)
  • tests/e2e_dbt_project/models/nested/models/tree/nested.sql (1 hunks)
  • tests/e2e_dbt_project/models/no_timestamp_anomalies.sql (1 hunks)
  • tests/e2e_dbt_project/models/non_dbt_model.sql (1 hunks)
  • tests/e2e_dbt_project/models/numeric_column_anomalies.sql (1 hunks)
  • tests/e2e_dbt_project/models/one.sql (1 hunks)
  • tests/e2e_dbt_project/models/schema.yml (1 hunks)
  • tests/e2e_dbt_project/models/stats_players.sql (1 hunks)
  • tests/e2e_dbt_project/models/stats_team.sql (1 hunks)
  • tests/e2e_dbt_project/models/string_column_anomalies.sql (1 hunks)
  • tests/e2e_dbt_project/models/test_alerts_union.sql (1 hunks)
  • tests/e2e_dbt_project/models/users_per_day_weekly_seasonal.sql (1 hunks)
  • tests/e2e_dbt_project/models/users_per_hour_daily_seasonal.sql (1 hunks)
  • tests/e2e_dbt_project/packages.yml (1 hunks)
  • tests/e2e_dbt_project/snapshots/failed_snapshot.sql (1 hunks)
  • tests/e2e_dbt_project/tests/singular_test_with_no_ref.sql (1 hunks)
  • tests/e2e_dbt_project/tests/singular_test_with_one_ref.sql (1 hunks)
  • tests/e2e_dbt_project/tests/singular_test_with_source_ref.sql (1 hunks)
  • tests/e2e_dbt_project/tests/singular_test_with_two_refs.sql (1 hunks)
🧰 Additional context used
🪛 Ruff (0.13.3)
tests/e2e_dbt_project/generate_data.py

59-59: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


59-59: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


62-62: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


62-62: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


64-64: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


67-67: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


69-69: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


70-70: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


78-78: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


78-78: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


81-81: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


81-81: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


84-84: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


84-84: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


88-88: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


90-90: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


91-91: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


131-131: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


132-132: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


135-135: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


137-137: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


138-138: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


139-139: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


140-140: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


141-141: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


142-142: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


150-150: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


151-151: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


154-154: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


156-156: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


157-157: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


158-158: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


159-159: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


160-160: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


161-161: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


206-206: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


208-208: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


209-209: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


212-212: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


214-214: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


215-215: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


218-218: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


220-220: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


221-221: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


226-226: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


236-236: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


238-238: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


239-239: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


242-242: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


244-244: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


245-245: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


248-248: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


250-250: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


251-251: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


256-256: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


303-303: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


311-311: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


344-344: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


363-363: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


404-404: Unused function argument: row_index

(ARG001)


404-404: Unused function argument: rows_count

(ARG001)


409-409: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


409-409: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


413-413: Unused function argument: row_index

(ARG001)


413-413: Unused function argument: rows_count

(ARG001)


418-418: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)


418-418: Standard pseudo-random generators are not suitable for cryptographic purposes

(S311)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
  • GitHub Check: test / test
  • GitHub Check: code-quality
🔇 Additional comments (5)
tests/e2e_dbt_project/models/stats_team.sql (1)

1-1: LGTM for passthrough model dependency.
This passthrough keeps stats_team aligned with stats_team_validation so downstream tests see identical columns—looks good.

tests/e2e_dbt_project/models/stats_players.sql (1)

1-1: LGTM for passthrough model dependency.
Mirrors the validation model cleanly; no issues spotted.

tests/e2e_dbt_project/models/groups.sql (1)

1-1: Pass-through model looks good.

Matches the existing pattern of proxy models and correctly references groups_validation.

tests/e2e_dbt_project/macros/system/materializations.sql (1)

1-7: Confirm overriding dbt’s built-in test materialization is intentional and compatible.

dbt defines materialization test by default. Overriding it can affect all tests. Ensure elementary.materialization_test_default/snowflake return a valid materialization result and align with the dbt-core version under test.

tests/e2e_dbt_project/generate_data.py (1)

53-93: Ruff S311 warnings are expected here.

Use of random is fine for non-crypto test data generation. No action needed.

If you want to suppress S311 in CI for this file, we can add # noqa: S311 selectively or configure Ruff to ignore S311 under tests/.

Also applies to: 126-163, 199-259, 297-331, 334-377, 380-396, 398-401, 403-447

@haritamar haritamar force-pushed the ele-5122-cli-stop-using-deprecated-tests branch from 2baebad to 8ce421e Compare October 8, 2025 14:40
@elazarlachkar
Copy link
Contributor

@haritamar I see you didn't copy all the macros from the dbt pacakge (the unit_tests dir, for example). Is this by design?

@haritamar haritamar merged commit 382b865 into master Oct 9, 2025
58 of 61 checks passed
@haritamar haritamar deleted the ele-5122-cli-stop-using-deprecated-tests branch October 9, 2025 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants