Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: update fixtures scope that are loading data into the analytical db #17780

Merged
merged 11 commits into from
Dec 17, 2021

Conversation

bkyryliuk
Copy link
Member

@bkyryliuk bkyryliuk commented Dec 16, 2021

SUMMARY

Play with fixture scopes to see if tests will be faster
Hive and presto DBs are slow, we should load test data only once rather than do it for every module / test function

Speed improvements:
Python Presto/Hive / test-postgres-presto (3.8) 38m vs 92 m
Python Presto/Hive / test-postgres-hive 39m vs 53 m

There are probably other tests as well that can benefit from a similar improvement

@bkyryliuk bkyryliuk force-pushed the bogdan/change_fixture_scope branch 5 times, most recently from f341163 to 1fde0ce Compare December 16, 2021 20:20
@codecov
Copy link

codecov bot commented Dec 16, 2021

Codecov Report

Merging #17780 (c69af00) into master (bcd698d) will increase coverage by 0.07%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #17780      +/-   ##
==========================================
+ Coverage   67.72%   67.79%   +0.07%     
==========================================
  Files        1605     1605              
  Lines       64205    64207       +2     
  Branches     6790     6790              
==========================================
+ Hits        43481    43529      +48     
+ Misses      18868    18822      -46     
  Partials     1856     1856              
Flag Coverage Δ
hive 81.80% <ø> (+<0.01%) ⬆️
mysql 82.18% <ø> (+<0.01%) ⬆️
postgres 82.23% <ø> (+<0.01%) ⬆️
presto 82.10% <ø> (?)
python 82.72% <ø> (+0.15%) ⬆️
sqlite 81.91% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/dashboards/dao.py 95.55% <0.00%> (-0.75%) ⬇️
superset/dashboards/commands/update.py 84.28% <0.00%> (+0.46%) ⬆️
superset/models/core.py 90.00% <0.00%> (+0.73%) ⬆️
superset/sql_lab.py 81.64% <0.00%> (+0.78%) ⬆️
superset/extensions.py 93.90% <0.00%> (+1.21%) ⬆️
superset/connectors/sqla/models.py 88.42% <0.00%> (+1.34%) ⬆️
superset/db_engine_specs/presto.py 90.39% <0.00%> (+6.05%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bcd698d...c69af00. Read the comment docs.

@bkyryliuk bkyryliuk changed the title chore: update db setup fixture scope [WIP] chore: update fixtures scope that are loading data into the analytical db Dec 16, 2021
@bkyryliuk bkyryliuk requested review from betodealmeida, villebro, amitmiran137 and suddjian and removed request for villebro December 16, 2021 22:58
Copy link
Member

@betodealmeida betodealmeida left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great!

I've been working on converting integration tests to unit tests, hopefully we can keep the number of integration tests down to a bare minimum in the near future.

@bkyryliuk bkyryliuk merged commit afd2e9f into apache:master Dec 17, 2021
@bkyryliuk bkyryliuk deleted the bogdan/change_fixture_scope branch December 17, 2021 00:11
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.5.0 labels Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/L 🚢 1.5.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants