Polars remove notebook #173

armgilles · 2024-11-07T12:56:37Z

Closed #172

On enlève toute référence à LFS (pour les gros notebook il y longtemps...) et reprise d'historique git (très chiant de gérer ces veilles erreurs....)

Signed-off-by: Gillesa <arm.gilles@gmail.com>

Test api bdx

Signed-off-by: Gillesa <arm.gilles@gmail.com>

) Signed-off-by: Gillesa <arm.gilles@gmail.com>

Test transformation json to DataFrame

Signed-off-by: Armand <arm.gilles@gmail.com>

…ect #85 Signed-off-by: Armand <arm.gilles@gmail.com>

Signed-off-by: Armand <arm.gilles@gmail.com>

* Bench pipeline transf (#155) * Update for front (#111) * Fix install in front and got to be in PROD Signed-off-by: Armand <arm.gilles@gmail.com> * Just to check in site-package on dir below ROOT_DIR Signed-off-by: Armand <arm.gilles@gmail.com> * Update Ruff version in pre commit Signed-off-by: Armand <arm.gilles@gmail.com> * Change check for prod with new structure and Front Signed-off-by: Armand <arm.gilles@gmail.com> * Remove non usefull reset_index in oslandia API data crunsh Signed-off-by: Armand <arm.gilles@gmail.com> * Format Signed-off-by: Armand <arm.gilles@gmail.com> * Add matplotlib as dep (visualisation.py) Signed-off-by: Armand <arm.gilles@gmail.com> * Add seaborn as dep for deploy Signed-off-by: Armand <arm.gilles@gmail.com> * Try to fix error install ci watcher 'error: Your local changes to the following files would be overwritten by checkout' Signed-off-by: Armand <arm.gilles@gmail.com> * minor update on notebook Signed-off-by: Armand <arm.gilles@gmail.com> --------- Signed-off-by: Armand <arm.gilles@gmail.com> * Bump to version 1.2.2a (#112) Signed-off-by: Armand <arm.gilles@gmail.com> * Add benchmark for pipeline transf from json API data #154 Signed-off-by: Armand <arm.gilles@gmail.com> * Improve speed of creation of simulated data & fix big data creation setting #154 Signed-off-by: Armand <arm.gilles@gmail.com> * Optimise creation of simulated data and reduce volume test #154 Signed-off-by: Armand <arm.gilles@gmail.com> * Codspeed run forever, only on small test #154 Signed-off-by: Armand <arm.gilles@gmail.com> * Codspeed run forever, only on small test #154 Signed-off-by: Armand <arm.gilles@gmail.com> * Codspeed previous commit ok with no new test, check with one Signed-off-by: Armand <arm.gilles@gmail.com> * Reduce volume for #154 to run on codspeed Signed-off-by: Armand <arm.gilles@gmail.com> * Reduce volume for #154 to run on codspeed Signed-off-by: Armand <arm.gilles@gmail.com> * Reduce again volume for codspeed, reduce CI time Signed-off-by: Armand <arm.gilles@gmail.com> --------- Signed-off-by: Armand <arm.gilles@gmail.com> * specify timezone to be as real data #154 Signed-off-by: Armand <arm.gilles@gmail.com> --------- Signed-off-by: Armand <arm.gilles@gmail.com>

Signed-off-by: Armand <arm.gilles@gmail.com>

* Try lazy + expression fonction to check perf bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * Update code with get_transactions_out expr #146 Signed-off-by: Armand <arm.gilles@gmail.com> * have to collect with lazy #146 Signed-off-by: Armand <arm.gilles@gmail.com> * Lazy read_activity_vcub #148 and update notebook transactions_out Signed-off-by: Armand <arm.gilles@gmail.com> * Update bench test with lazy vcub_keeper_py312 #148 Signed-off-by: Armand <arm.gilles@gmail.com> * update docstring #148 Signed-off-by: Armand <arm.gilles@gmail.com> * collect lazy df to be a fair bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * collect lazy df to be a fair bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * collect lazy df to be a fair bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * Update transactions_in to be lazy and expr fonction #149 Signed-off-by: Armand <arm.gilles@gmail.com> * Try to improve bench test on big result Signed-off-by: Armand <arm.gilles@gmail.com> * lazy and Expr function for transactions_all function #150 Signed-off-by: Armand <arm.gilles@gmail.com> * try to fix bad perf on big dataset lazy Signed-off-by: Armand <arm.gilles@gmail.com> * try to fix bad perf on big dataset lazy Signed-off-by: Armand <arm.gilles@gmail.com> * try to fix bad perf on big dataset lazy Signed-off-by: Armand <arm.gilles@gmail.com> * Lazy expr for get_consecutive_no_transactions_out #151 Signed-off-by: Armand <arm.gilles@gmail.com> * Lazy transform_json_api_bdx_station_data_to_df function #152 Signed-off-by: Armand <arm.gilles@gmail.com> * Encoding time in Expr function and process_data_cluster in lazy mode #153 Signed-off-by: Armand <arm.gilles@gmail.com> * add todo for ML with pandas Signed-off-by: Armand <arm.gilles@gmail.com> * Adapt code for pipeline bench lazy Signed-off-by: Armand <arm.gilles@gmail.com> * Try new lazy for pipeline bench Signed-off-by: Armand <arm.gilles@gmail.com> * Try new lazy for pipeline bench Signed-off-by: Armand <arm.gilles@gmail.com> * process data with with_columns style & lazy #161 Signed-off-by: Armand <arm.gilles@gmail.com> * forget previous commit Signed-off-by: Armand <arm.gilles@gmail.com> * have to collect this tests Signed-off-by: Armand <arm.gilles@gmail.com> * Small test bench are in eager mode, big in lazy mode to faire comparaison Signed-off-by: Armand <arm.gilles@gmail.com> * Small test bench are in eager mode, big in lazy mode to faire comparaison Signed-off-by: Armand <arm.gilles@gmail.com> * Update notebook with with_colums style for feature creation Signed-off-by: Armand <arm.gilles@gmail.com> * Using pipe style with lazy Signed-off-by: Armand <arm.gilles@gmail.com> * cleaning Signed-off-by: Armand <arm.gilles@gmail.com> --------- Signed-off-by: Armand <arm.gilles@gmail.com>

Signed-off-by: Armand <arm.gilles@gmail.com>

…#166) Signed-off-by: Armand <arm.gilles@gmail.com>

* Update important viz fucntion to polars #164 Signed-off-by: Armand <arm.gilles@gmail.com> * Upate notebook with viz, some pandas are still here but it's ok #164 Signed-off-by: Armand <arm.gilles@gmail.com> --------- Signed-off-by: Armand <arm.gilles@gmail.com>

Signed-off-by: Armand <arm.gilles@gmail.com>

* Fix bad condition for consecutive_no_transactions_out and available bike in station #170 Signed-off-by: Armand <arm.gilles@gmail.com> * Update test to test available_bike less or equal 2 is consecutive_no_transactions_out = 0 #170 Signed-off-by: Armand <arm.gilles@gmail.com> * Small update to give same type in test #170 Signed-off-by: Armand <arm.gilles@gmail.com> --------- Signed-off-by: Armand <arm.gilles@gmail.com>

Signed-off-by: Armand <arm.gilles@gmail.com>

codecov · 2024-11-07T12:59:20Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.70%. Comparing base (ab1fa8c) to head (a42f2f0).

Additional details and impacted files

@@             Coverage Diff             @@
##           dev_polars     #173   +/-   ##
===========================================
  Coverage       68.70%   68.70%           
===========================================
  Files              10       10           
  Lines             310      310           
===========================================
  Hits              213      213           
  Misses             97       97

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

codspeed-hq · 2024-11-07T13:03:19Z

CodSpeed Performance Report

Merging #173 will degrade performances by 10.11%

_{Comparing polars_remove_notebook (bf50e6a) with polars_remove_notebook (4717996)}

Summary

⚡ 1 improvements
❌ 1 regressions
✅ 11 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	`polars_remove_notebook`	`polars_remove_notebook`	Change
❌	`test_benchmark_get_transaction_all`	1.5 ms	1.7 ms	-10.11%
⚡	`test_benchmark_get_transaction_in`	1.9 ms	1.8 ms	+7.17%

…/vcub_keeper into polars_remove_notebook

Signed-off-by: Armand <arm.gilles@gmail.com>

armgilles added 30 commits March 24, 2023 20:51

update again CI. Create branch & PR you FOOL (#72)

99daea3

Signed-off-by: Gillesa <arm.gilles@gmail.com>

update again CI. Create branch & PR you FOOL (#72)

2f22651

Signed-off-by: Gillesa <arm.gilles@gmail.com>

Merge pull request #73 from armgilles/test_api_bdx

63f69f4

Test api bdx

Merge pull request #73 from armgilles/test_api_bdx

4a10f52

Test api bdx

To add .csv file for tests purpose (#74)

808eb50

Signed-off-by: Gillesa <arm.gilles@gmail.com>

To add .csv file for tests purpose (#74)

5d201e8

Signed-off-by: Gillesa <arm.gilles@gmail.com>

Script / data & notebooks to test transf fonction json to DataFrame (#74

2ff337b

) Signed-off-by: Gillesa <arm.gilles@gmail.com>

Script / data & notebooks to test transf fonction json to DataFrame (#74

7720bbc

) Signed-off-by: Gillesa <arm.gilles@gmail.com>

Merge pull request #75 from armgilles/test_transf_json_df

42ef496

Test transformation json to DataFrame

Merge pull request #75 from armgilles/test_transf_json_df

c767519

Test transformation json to DataFrame

Pyproject.toml file for package

1027c1c

Signed-off-by: Armand <arm.gilles@gmail.com>

Pyproject.toml file for package

a45b461

Signed-off-by: Armand <arm.gilles@gmail.com>

Aff pre-commit config #85

4136f89

Signed-off-by: Armand <arm.gilles@gmail.com>

Aff pre-commit config #85

3650831

Signed-off-by: Armand <arm.gilles@gmail.com>

Update project #85

6058ef2

Signed-off-by: Armand <arm.gilles@gmail.com>

Update project #85

218f1ef

Signed-off-by: Armand <arm.gilles@gmail.com>

Change project withi pyproject & pre-commit, change structure fo proj…

1f9d551

…ect #85 Signed-off-by: Armand <arm.gilles@gmail.com>

Change project withi pyproject & pre-commit, change structure fo proj…

c7c8bc0

…ect #85 Signed-off-by: Armand <arm.gilles@gmail.com>

Format & Ruff check pass #85

2de3b79

Signed-off-by: Armand <arm.gilles@gmail.com>

Format & Ruff check pass #85

e6e4d1d

Signed-off-by: Armand <arm.gilles@gmail.com>

remove print

de8e6cf

Signed-off-by: Armand <arm.gilles@gmail.com>

remove print

bef9d5a

Signed-off-by: Armand <arm.gilles@gmail.com>

Update test with new package version #85

b51e48d

Signed-off-by: Armand <arm.gilles@gmail.com>

Update test with new package version #85

b3f2d80

Signed-off-by: Armand <arm.gilles@gmail.com>

No more warning with resample and using min VS T #85

1ddad6c

Signed-off-by: Armand <arm.gilles@gmail.com>

No more warning with resample and using min VS T #85

eb9b67c

Signed-off-by: Armand <arm.gilles@gmail.com>

Update git action #85

1b972cf

Signed-off-by: Armand <arm.gilles@gmail.com>

Update git action #85

855c132

Signed-off-by: Armand <arm.gilles@gmail.com>

Update git action #85

ea114a5

Signed-off-by: Armand <arm.gilles@gmail.com>

Update git action #85

d80fd13

Signed-off-by: Armand <arm.gilles@gmail.com>

armgilles added 21 commits October 10, 2024 16:06

Merge branch 'dev_pandas' into dev_polars

3664110

merging from dev_polars to have bench pipeline tests

99ddd1a

Signed-off-by: Armand <arm.gilles@gmail.com>

merging from dev_polars to have bench pipeline tests

f903829

Signed-off-by: Armand <arm.gilles@gmail.com>

Update ML pipeple to Polars #162 (#163)

e4406a8

Signed-off-by: Armand <arm.gilles@gmail.com>

Update ML pipeple to Polars #162 (#163)

fe645a5

Signed-off-by: Armand <arm.gilles@gmail.com>

Fix new value status from API Bordeaux, improve mapping and test #165 (…

ac800a7

…#166) Signed-off-by: Armand <arm.gilles@gmail.com>

Fix new value status from API Bordeaux, improve mapping and test #165 (…

59156d6

…#166) Signed-off-by: Armand <arm.gilles@gmail.com>

Polars viz (#167)

58c6ab2

* Update important viz fucntion to polars #164 Signed-off-by: Armand <arm.gilles@gmail.com> * Upate notebook with viz, some pandas are still here but it's ok #164 Signed-off-by: Armand <arm.gilles@gmail.com> --------- Signed-off-by: Armand <arm.gilles@gmail.com>

Polars viz (#167)

7426cf6

* Update important viz fucntion to polars #164 Signed-off-by: Armand <arm.gilles@gmail.com> * Upate notebook with viz, some pandas are still here but it's ok #164 Signed-off-by: Armand <arm.gilles@gmail.com> --------- Signed-off-by: Armand <arm.gilles@gmail.com>

Change date type into Bdx pipeline (unset time zone) #168) (#169)

71dc189

Signed-off-by: Armand <arm.gilles@gmail.com>

Change date type into Bdx pipeline (unset time zone) #168) (#169)

0fe074a

Signed-off-by: Armand <arm.gilles@gmail.com>

Remove notebook from package #172

32ae201

Signed-off-by: Armand <arm.gilles@gmail.com>

Remove notebook from package #172

fda6fbf

Signed-off-by: Armand <arm.gilles@gmail.com>

Remove .gitattributes & LFS stuff #172

3509004

Signed-off-by: Armand <arm.gilles@gmail.com>

Remove .gitattributes & LFS stuff #172

a954774

Signed-off-by: Armand <arm.gilles@gmail.com>

armgilles added the bug Something isn't working label Nov 7, 2024

armgilles added this to the V1.3 milestone Nov 7, 2024

armgilles self-assigned this Nov 7, 2024

armgilles added 2 commits November 7, 2024 13:36

Merge branch 'polars_remove_notebook' of https://github.com/armgilles…

68f8346

…/vcub_keeper into polars_remove_notebook

Reduction taille notebook

4717996

Signed-off-by: Armand <arm.gilles@gmail.com>

armgilles closed this Nov 7, 2024

armgilles force-pushed the polars_remove_notebook branch from a42f2f0 to 4717996 Compare November 7, 2024 14:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Polars remove notebook #173

Polars remove notebook #173

armgilles commented Nov 7, 2024 •

edited

Loading

codecov bot commented Nov 7, 2024 •

edited

Loading

codspeed-hq bot commented Nov 7, 2024 •

edited

Loading

Polars remove notebook #173

Polars remove notebook #173

Conversation

armgilles commented Nov 7, 2024 • edited Loading

codecov bot commented Nov 7, 2024 • edited Loading

Codecov Report

codspeed-hq bot commented Nov 7, 2024 • edited Loading

CodSpeed Performance Report

Merging #173 will degrade performances by 10.11%

Summary

Benchmarks breakdown

armgilles commented Nov 7, 2024 •

edited

Loading

codecov bot commented Nov 7, 2024 •

edited

Loading

codspeed-hq bot commented Nov 7, 2024 •

edited

Loading