Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Always use the executemany method when inserting rows in DbApiHook as it's way much faster #38715

Merged
merged 69 commits into from
Apr 12, 2024
Merged

Always use the executemany method when inserting rows in DbApiHook as it's way much faster #38715

merged 69 commits into from
Apr 12, 2024

Commits on Apr 3, 2024

  1. refactor: Always use the fast_executemany method when inserting rows … 

    …as it's much faster than inserting each row separately and committing every once in a while in between
    @davidblain-infrabel
    davidblain-infrabel committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    0312a8f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 3, 2024
    Configuration menu
    Copy the full SHA
    219bd43 View commit details
    Browse the repository at this point in the history

  3. refactor: Only set fast_executemany option if explicitly defined in c… 

    …onstructor of Hook as this is not a standard supported option by all ODBC drivers
    @davidblain-infrabel
    davidblain-infrabel committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    64b7a85 View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2024

  1. refactor: Fixed tests related to insert_rows for Postgres

    @davidblain-infrabel
    davidblain-infrabel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    c183404 View commit details
    Browse the repository at this point in the history

  2. refactor: Fixed assertions on executemany for insert rows in Postgres… 

    … test
    @davidblain-infrabel
    davidblain-infrabel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    bc3895e View commit details
    Browse the repository at this point in the history

  3. refactor: Deprecated bulk_insert_rows in Teradata as the insert_rows … 

    …does the same by default, no need for a specialized method and thus delegate to insert_rows method
    @davidblain-infrabel
    davidblain-infrabel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    3af4c7c View commit details
    Browse the repository at this point in the history

  4. refactor: Removed duplicate calls object definition in test_bulk_inse… 

    …rt_rows_with_commit_every
    @davidblain-infrabel
    davidblain-infrabel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    ef36ed0 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 4, 2024
    Configuration menu
    Copy the full SHA
    e222af5 View commit details
    Browse the repository at this point in the history

  6. refactor: Use DeprecationWarning instead

    @davidblain-infrabel
    davidblain-infrabel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    d0fd763 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 4, 2024
    Configuration menu
    Copy the full SHA
    4dbdff5 View commit details
    Browse the repository at this point in the history

  8. refactor: Use AirflowProviderDeprecationWarning instead

    @davidblain-infrabel
    davidblain-infrabel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    42b7ab7 View commit details
    Browse the repository at this point in the history

  9. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 4, 2024
    Configuration menu
    Copy the full SHA
    58d3cfa View commit details
    Browse the repository at this point in the history

  10. refactor: Ignore deprecation warnings for TestTeradataHook

    @davidblain-infrabel
    davidblain-infrabel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    04d0a47 View commit details
    Browse the repository at this point in the history

  11. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 4, 2024
    Configuration menu
    Copy the full SHA
    6b0ce23 View commit details
    Browse the repository at this point in the history

  12. refactor: Re-added check on rows parameter for bulk_insert_rows metho… 

    …d and changed some rows values to string for the TestTeradataHook
    @davidblain-infrabel
    davidblain-infrabel committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    1cb5e5e View commit details
    Browse the repository at this point in the history

  13. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 4, 2024
    Configuration menu
    Copy the full SHA
    57318a9 View commit details
    Browse the repository at this point in the history

  14. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 4, 2024
    Configuration menu
    Copy the full SHA
    4831868 View commit details
    Browse the repository at this point in the history

  15. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 4, 2024
    Configuration menu
    Copy the full SHA
    6bb724d View commit details
    Browse the repository at this point in the history

  16. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 4, 2024
    Configuration menu
    Copy the full SHA
    35580fe View commit details
    Browse the repository at this point in the history

  17. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 4, 2024
    Configuration menu
    Copy the full SHA
    3a9fcbf View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2024

  1. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 6, 2024
    Configuration menu
    Copy the full SHA
    1e577c7 View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2024

  1. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 8, 2024
    Configuration menu
    Copy the full SHA
    96f5019 View commit details
    Browse the repository at this point in the history

  2. refactor: Fixed logging statement regarding total rows in insert_rows… 

    … method
    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    b64351f View commit details
    Browse the repository at this point in the history

  3. refactor: Changed string values back to int's but adapted expected du… 

    …e to _serialize_cell
    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    92775af View commit details
    Browse the repository at this point in the history

  4. refactor: Added deprecation warning for executemany parameter in inse… 

    …rt_rows method of DbApiHook
    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    1369944 View commit details
    Browse the repository at this point in the history

  5. refactor: Reformatted mocked call for test_bulk_insert_rows_with_comm… 

    …it_every
    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    ac185d2 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 8, 2024
    Configuration menu
    Copy the full SHA
    7216c3e View commit details
    Browse the repository at this point in the history

  7. refactor: Added explicit fast_executemany parameter to constructor of… 

    … DbApiHook
    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    f8d8d92 View commit details
    Browse the repository at this point in the history

  8. refactor: Reformatted second mocked call in test_bulk_insert_rows_wit… 

    …h_commit_every
    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    fee8e01 View commit details
    Browse the repository at this point in the history

  9. refactor: Re-added executemany param to insert_rows method

    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    39fe36b View commit details
    Browse the repository at this point in the history

  10. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 8, 2024
    Configuration menu
    Copy the full SHA
    7b65015 View commit details
    Browse the repository at this point in the history

  11. refactor: Put executemany between single quote to avoid spelling chec… 

    …k error
    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    5634360 View commit details
    Browse the repository at this point in the history

  12. Revert "refactor: Added deprecation warning for executemany parameter… 

    … in insert_rows method of DbApiHook"

This reverts commit 1369944
    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    cdfeaf2 View commit details
    Browse the repository at this point in the history

  13. refactor: Still support original implementation without executemany i… 

    …n insert_rows method, override insert_rows method for TeraDataHook and OdbcHook and set executemany parameter to True by default
    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    934239a View commit details
    Browse the repository at this point in the history

  14. refactor: Reformatted insert_rows method in DbApiHook

    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    b3ad6a6 View commit details
    Browse the repository at this point in the history

  15. fix: Fixed insert_rows method in DbApiHook and extracted common _seri… 

    …alize_cells helper method
    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    49e923c View commit details
    Browse the repository at this point in the history

  16. fix: Re-added check on fast_executemany in insert_rows method

    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    4e6df4e View commit details
    Browse the repository at this point in the history

  17. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 8, 2024
    Configuration menu
    Copy the full SHA
    a216fe9 View commit details
    Browse the repository at this point in the history

  18. refactor: Put docstring of _closing_supporting_autocommit method on o… 

    …ne line
    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    82ca138 View commit details
    Browse the repository at this point in the history

  19. refactor: Added autocommit parameter which is False by default to _cl… 

    …osing_supporting_autocommit so we can pass the autocommit parameter when used in run method
    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    be17da1 View commit details
    Browse the repository at this point in the history

  20. refactor: Added supports_executemany class variable which allows to s… 

    …pecify the used strategy with insert_rows
    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    8e8b692 View commit details
    Browse the repository at this point in the history

  21. refactor: test_insert_rows should use executemany for Postgres

    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    4d6d240 View commit details
    Browse the repository at this point in the history

  22. refactor: Deprecation warning for test_insert_rows_replace_executeman… 

    …y_hana_dialect should also be ignored
    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    564b5ef View commit details
    Browse the repository at this point in the history

  23. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 8, 2024
    Configuration menu
    Copy the full SHA
    e6dd1a5 View commit details
    Browse the repository at this point in the history

  24. refactor: Deprecation warning for test_insert_rows_executemany should… 

    … also be ignored
    @davidblain-infrabel
    davidblain-infrabel committed Apr 8, 2024
    Configuration menu
    Copy the full SHA
    215d719 View commit details
    Browse the repository at this point in the history

  25. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 8, 2024
    Configuration menu
    Copy the full SHA
    45b3a94 View commit details
    Browse the repository at this point in the history

Commits on Apr 9, 2024

  1. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 9, 2024
    Configuration menu
    Copy the full SHA
    161f3d2 View commit details
    Browse the repository at this point in the history

  2. refactor: Removed fast_executemany

    @davidblain-infrabel
    davidblain-infrabel committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    8566316 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 9, 2024
    Configuration menu
    Copy the full SHA
    74e0659 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 9, 2024
    Configuration menu
    Copy the full SHA
    ec16892 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 9, 2024
    Configuration menu
    Copy the full SHA
    65d2ae7 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 9, 2024
    Configuration menu
    Copy the full SHA
    4590409 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 9, 2024
    Configuration menu
    Copy the full SHA
    f6fc1be View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2024

  1. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 10, 2024
    Configuration menu
    Copy the full SHA
    4b9e47b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 10, 2024
    Configuration menu
    Copy the full SHA
    d2b0629 View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. refactor: Removed unused fast_executemany param from DbApiHook constr… 

    …uctor
    @davidblain-infrabel
    davidblain-infrabel committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    5580451 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    b242539 View commit details
    Browse the repository at this point in the history

  3. refactor: Removed placeholder attribute from DbApiHook interface

    @davidblain-infrabel
    davidblain-infrabel committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    0250d49 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    e17aca5 View commit details
    Browse the repository at this point in the history

  5. Revert unnecessary format change

    @uranusjr
    uranusjr authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    15f76dc View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    7814ee3 View commit details
    Browse the repository at this point in the history

  7. docs: Updated docstring for excutemany parameter in insert_rows method 

    Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
    @dabla @uranusjr
    dabla and uranusjr authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    65f0518 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    8850f82 View commit details
    Browse the repository at this point in the history

  9. refactor: Renamed _closing_supporting_autocommit method to _create_au… 

    …tocommit_connection in DbApiHook
    @davidblain-infrabel
    davidblain-infrabel committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    3b029e1 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    85536d9 View commit details
    Browse the repository at this point in the history

  11. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    b9ccfc1 View commit details
    Browse the repository at this point in the history

  12. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    1d295c0 View commit details
    Browse the repository at this point in the history

  13. Use comprehension instead of map

    @uranusjr
    uranusjr authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    ad65b9b View commit details
    Browse the repository at this point in the history

  14. Merge branch 'main' into feature/sql-performance-enhancement-insertmany

    @dabla
    dabla authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    1c58f99 View commit details
    Browse the repository at this point in the history