Lazy expression get_transactions_all() #150
Labels
Milestone
Comments
armgilles
added a commit
that referenced
this issue
Oct 8, 2024
Signed-off-by: Armand <arm.gilles@gmail.com>
armgilles
added a commit
that referenced
this issue
Oct 16, 2024
* Try lazy + expression fonction to check perf bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * Update code with get_transactions_out expr #146 Signed-off-by: Armand <arm.gilles@gmail.com> * have to collect with lazy #146 Signed-off-by: Armand <arm.gilles@gmail.com> * Lazy read_activity_vcub #148 and update notebook transactions_out Signed-off-by: Armand <arm.gilles@gmail.com> * Update bench test with lazy vcub_keeper_py312 #148 Signed-off-by: Armand <arm.gilles@gmail.com> * update docstring #148 Signed-off-by: Armand <arm.gilles@gmail.com> * collect lazy df to be a fair bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * collect lazy df to be a fair bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * collect lazy df to be a fair bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * Update transactions_in to be lazy and expr fonction #149 Signed-off-by: Armand <arm.gilles@gmail.com> * Try to improve bench test on big result Signed-off-by: Armand <arm.gilles@gmail.com> * lazy and Expr function for transactions_all function #150 Signed-off-by: Armand <arm.gilles@gmail.com> * try to fix bad perf on big dataset lazy Signed-off-by: Armand <arm.gilles@gmail.com> * try to fix bad perf on big dataset lazy Signed-off-by: Armand <arm.gilles@gmail.com> * try to fix bad perf on big dataset lazy Signed-off-by: Armand <arm.gilles@gmail.com> * Lazy expr for get_consecutive_no_transactions_out #151 Signed-off-by: Armand <arm.gilles@gmail.com> * Lazy transform_json_api_bdx_station_data_to_df function #152 Signed-off-by: Armand <arm.gilles@gmail.com> * Encoding time in Expr function and process_data_cluster in lazy mode #153 Signed-off-by: Armand <arm.gilles@gmail.com> * add todo for ML with pandas Signed-off-by: Armand <arm.gilles@gmail.com> * Adapt code for pipeline bench lazy Signed-off-by: Armand <arm.gilles@gmail.com> * Try new lazy for pipeline bench Signed-off-by: Armand <arm.gilles@gmail.com> * Try new lazy for pipeline bench Signed-off-by: Armand <arm.gilles@gmail.com> * process data with with_columns style & lazy #161 Signed-off-by: Armand <arm.gilles@gmail.com> * forget previous commit Signed-off-by: Armand <arm.gilles@gmail.com> * have to collect this tests Signed-off-by: Armand <arm.gilles@gmail.com> * Small test bench are in eager mode, big in lazy mode to faire comparaison Signed-off-by: Armand <arm.gilles@gmail.com> * Small test bench are in eager mode, big in lazy mode to faire comparaison Signed-off-by: Armand <arm.gilles@gmail.com> * Update notebook with with_colums style for feature creation Signed-off-by: Armand <arm.gilles@gmail.com> * Using pipe style with lazy Signed-off-by: Armand <arm.gilles@gmail.com> * cleaning Signed-off-by: Armand <arm.gilles@gmail.com> --------- Signed-off-by: Armand <arm.gilles@gmail.com>
|
On n'utilise pas les |
armgilles
added a commit
that referenced
this issue
Nov 7, 2024
* Try lazy + expression fonction to check perf bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * Update code with get_transactions_out expr #146 Signed-off-by: Armand <arm.gilles@gmail.com> * have to collect with lazy #146 Signed-off-by: Armand <arm.gilles@gmail.com> * Lazy read_activity_vcub #148 and update notebook transactions_out Signed-off-by: Armand <arm.gilles@gmail.com> * Update bench test with lazy vcub_keeper_py312 #148 Signed-off-by: Armand <arm.gilles@gmail.com> * update docstring #148 Signed-off-by: Armand <arm.gilles@gmail.com> * collect lazy df to be a fair bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * collect lazy df to be a fair bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * collect lazy df to be a fair bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * Update transactions_in to be lazy and expr fonction #149 Signed-off-by: Armand <arm.gilles@gmail.com> * Try to improve bench test on big result Signed-off-by: Armand <arm.gilles@gmail.com> * lazy and Expr function for transactions_all function #150 Signed-off-by: Armand <arm.gilles@gmail.com> * try to fix bad perf on big dataset lazy Signed-off-by: Armand <arm.gilles@gmail.com> * try to fix bad perf on big dataset lazy Signed-off-by: Armand <arm.gilles@gmail.com> * try to fix bad perf on big dataset lazy Signed-off-by: Armand <arm.gilles@gmail.com> * Lazy expr for get_consecutive_no_transactions_out #151 Signed-off-by: Armand <arm.gilles@gmail.com> * Lazy transform_json_api_bdx_station_data_to_df function #152 Signed-off-by: Armand <arm.gilles@gmail.com> * Encoding time in Expr function and process_data_cluster in lazy mode #153 Signed-off-by: Armand <arm.gilles@gmail.com> * add todo for ML with pandas Signed-off-by: Armand <arm.gilles@gmail.com> * Adapt code for pipeline bench lazy Signed-off-by: Armand <arm.gilles@gmail.com> * Try new lazy for pipeline bench Signed-off-by: Armand <arm.gilles@gmail.com> * Try new lazy for pipeline bench Signed-off-by: Armand <arm.gilles@gmail.com> * process data with with_columns style & lazy #161 Signed-off-by: Armand <arm.gilles@gmail.com> * forget previous commit Signed-off-by: Armand <arm.gilles@gmail.com> * have to collect this tests Signed-off-by: Armand <arm.gilles@gmail.com> * Small test bench are in eager mode, big in lazy mode to faire comparaison Signed-off-by: Armand <arm.gilles@gmail.com> * Small test bench are in eager mode, big in lazy mode to faire comparaison Signed-off-by: Armand <arm.gilles@gmail.com> * Update notebook with with_colums style for feature creation Signed-off-by: Armand <arm.gilles@gmail.com> * Using pipe style with lazy Signed-off-by: Armand <arm.gilles@gmail.com> * cleaning Signed-off-by: Armand <arm.gilles@gmail.com> --------- Signed-off-by: Armand <arm.gilles@gmail.com>
armgilles
added a commit
that referenced
this issue
Nov 7, 2024
* Try lazy + expression fonction to check perf bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * Update code with get_transactions_out expr #146 Signed-off-by: Armand <arm.gilles@gmail.com> * have to collect with lazy #146 Signed-off-by: Armand <arm.gilles@gmail.com> * Lazy read_activity_vcub #148 and update notebook transactions_out Signed-off-by: Armand <arm.gilles@gmail.com> * Update bench test with lazy vcub_keeper_py312 #148 Signed-off-by: Armand <arm.gilles@gmail.com> * update docstring #148 Signed-off-by: Armand <arm.gilles@gmail.com> * collect lazy df to be a fair bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * collect lazy df to be a fair bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * collect lazy df to be a fair bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * Update transactions_in to be lazy and expr fonction #149 Signed-off-by: Armand <arm.gilles@gmail.com> * Try to improve bench test on big result Signed-off-by: Armand <arm.gilles@gmail.com> * lazy and Expr function for transactions_all function #150 Signed-off-by: Armand <arm.gilles@gmail.com> * try to fix bad perf on big dataset lazy Signed-off-by: Armand <arm.gilles@gmail.com> * try to fix bad perf on big dataset lazy Signed-off-by: Armand <arm.gilles@gmail.com> * try to fix bad perf on big dataset lazy Signed-off-by: Armand <arm.gilles@gmail.com> * Lazy expr for get_consecutive_no_transactions_out #151 Signed-off-by: Armand <arm.gilles@gmail.com> * Lazy transform_json_api_bdx_station_data_to_df function #152 Signed-off-by: Armand <arm.gilles@gmail.com> * Encoding time in Expr function and process_data_cluster in lazy mode #153 Signed-off-by: Armand <arm.gilles@gmail.com> * add todo for ML with pandas Signed-off-by: Armand <arm.gilles@gmail.com> * Adapt code for pipeline bench lazy Signed-off-by: Armand <arm.gilles@gmail.com> * Try new lazy for pipeline bench Signed-off-by: Armand <arm.gilles@gmail.com> * Try new lazy for pipeline bench Signed-off-by: Armand <arm.gilles@gmail.com> * process data with with_columns style & lazy #161 Signed-off-by: Armand <arm.gilles@gmail.com> * forget previous commit Signed-off-by: Armand <arm.gilles@gmail.com> * have to collect this tests Signed-off-by: Armand <arm.gilles@gmail.com> * Small test bench are in eager mode, big in lazy mode to faire comparaison Signed-off-by: Armand <arm.gilles@gmail.com> * Small test bench are in eager mode, big in lazy mode to faire comparaison Signed-off-by: Armand <arm.gilles@gmail.com> * Update notebook with with_colums style for feature creation Signed-off-by: Armand <arm.gilles@gmail.com> * Using pipe style with lazy Signed-off-by: Armand <arm.gilles@gmail.com> * cleaning Signed-off-by: Armand <arm.gilles@gmail.com> --------- Signed-off-by: Armand <arm.gilles@gmail.com>
armgilles
added a commit
that referenced
this issue
Nov 8, 2024
* Try lazy + expression fonction to check perf bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * Update code with get_transactions_out expr #146 Signed-off-by: Armand <arm.gilles@gmail.com> * have to collect with lazy #146 Signed-off-by: Armand <arm.gilles@gmail.com> * Lazy read_activity_vcub #148 and update notebook transactions_out Signed-off-by: Armand <arm.gilles@gmail.com> * Update bench test with lazy vcub_keeper_py312 #148 Signed-off-by: Armand <arm.gilles@gmail.com> * update docstring #148 Signed-off-by: Armand <arm.gilles@gmail.com> * collect lazy df to be a fair bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * collect lazy df to be a fair bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * collect lazy df to be a fair bench #146 Signed-off-by: Armand <arm.gilles@gmail.com> * Update transactions_in to be lazy and expr fonction #149 Signed-off-by: Armand <arm.gilles@gmail.com> * Try to improve bench test on big result Signed-off-by: Armand <arm.gilles@gmail.com> * lazy and Expr function for transactions_all function #150 Signed-off-by: Armand <arm.gilles@gmail.com> * try to fix bad perf on big dataset lazy Signed-off-by: Armand <arm.gilles@gmail.com> * try to fix bad perf on big dataset lazy Signed-off-by: Armand <arm.gilles@gmail.com> * try to fix bad perf on big dataset lazy Signed-off-by: Armand <arm.gilles@gmail.com> * Lazy expr for get_consecutive_no_transactions_out #151 Signed-off-by: Armand <arm.gilles@gmail.com> * Lazy transform_json_api_bdx_station_data_to_df function #152 Signed-off-by: Armand <arm.gilles@gmail.com> * Encoding time in Expr function and process_data_cluster in lazy mode #153 Signed-off-by: Armand <arm.gilles@gmail.com> * add todo for ML with pandas Signed-off-by: Armand <arm.gilles@gmail.com> * Adapt code for pipeline bench lazy Signed-off-by: Armand <arm.gilles@gmail.com> * Try new lazy for pipeline bench Signed-off-by: Armand <arm.gilles@gmail.com> * Try new lazy for pipeline bench Signed-off-by: Armand <arm.gilles@gmail.com> * process data with with_columns style & lazy #161 Signed-off-by: Armand <arm.gilles@gmail.com> * forget previous commit Signed-off-by: Armand <arm.gilles@gmail.com> * have to collect this tests Signed-off-by: Armand <arm.gilles@gmail.com> * Small test bench are in eager mode, big in lazy mode to faire comparaison Signed-off-by: Armand <arm.gilles@gmail.com> * Small test bench are in eager mode, big in lazy mode to faire comparaison Signed-off-by: Armand <arm.gilles@gmail.com> * Update notebook with with_colums style for feature creation Signed-off-by: Armand <arm.gilles@gmail.com> * Using pipe style with lazy Signed-off-by: Armand <arm.gilles@gmail.com> * cleaning Signed-off-by: Armand <arm.gilles@gmail.com> --------- Signed-off-by: Armand <arm.gilles@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Utilisation de LazyFrame afin d'optimiser encore les performances / mémoire des process data pour la fonction get_transactions_all()
The text was updated successfully, but these errors were encountered: