Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

27787 fine tune sync parameters for ctid with large tables #27792

Draft
wants to merge 74 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
194a62c
initial ctid for testing
rodireich Jun 10, 2023
ec7198c
initial ctid for testing
rodireich Jun 11, 2023
1459acb
initial ctid for testing
rodireich Jun 13, 2023
8681c8c
Automated Commit - Format and Process Resources Changes
rodireich Jun 13, 2023
14371ea
add version and state type to xmin status
subodh1810 Jun 13, 2023
a4ecc7e
Merge remote-tracking branch 'origin/26486-initial-sync-using-ctid' i…
subodh1810 Jun 13, 2023
eb7255c
add logic to swtich between xmin and ctid sync
subodh1810 Jun 13, 2023
69e6989
Merge branch 'master' into 26486-initial-sync-using-ctid
subodh1810 Jun 13, 2023
a796a81
Merge branch '26486-initial-sync-using-ctid' into state-structure-cha…
subodh1810 Jun 13, 2023
2215f64
npe fixes
subodh1810 Jun 13, 2023
52d09a0
use enum
subodh1810 Jun 13, 2023
c6bebb5
refactor
subodh1810 Jun 15, 2023
286c5cd
add relation node logic + validation for vacuuming + more refactor
subodh1810 Jun 16, 2023
7950be8
Merge branch 'master' into state-structure-change-xmin-ctid
subodh1810 Jun 18, 2023
7ef08e3
refine test + make PR ready for review
subodh1810 Jun 18, 2023
cba94db
remove un-wanted changes
subodh1810 Jun 18, 2023
378357a
missed this one
subodh1810 Jun 18, 2023
cb5bd39
remove irrelevant comments
subodh1810 Jun 18, 2023
0689f30
add more assertions
subodh1810 Jun 19, 2023
7ea3fdf
remove jdbc log
subodh1810 Jun 19, 2023
3ab062c
Merge branch 'master' into state-structure-change-xmin-ctid
subodh1810 Jun 20, 2023
f7e9939
address review comments
subodh1810 Jun 20, 2023
06f3ea7
🤖 Auto format source-postgres code [skip ci]
octavia-squidington-iii Jun 20, 2023
643aff7
skip streams under vacuum
subodh1810 Jun 20, 2023
0756579
🤖 Auto format source-postgres code [skip ci]
octavia-squidington-iii Jun 20, 2023
134637c
update log message
subodh1810 Jun 20, 2023
1f67fac
Merge branch 'state-structure-change-xmin-ctid' of https://github.com…
subodh1810 Jun 20, 2023
1e57a69
🤖 Auto format source-postgres code [skip ci]
octavia-squidington-iii Jun 20, 2023
f7dddd8
comment
rodireich Jun 20, 2023
06dcae4
Merge branch 'master' into state-structure-change-xmin-ctid
subodh1810 Jun 21, 2023
854b414
latest round of review comments
subodh1810 Jun 21, 2023
65f4711
missed this file
subodh1810 Jun 21, 2023
63a14f7
Merge branch 'master' into state-structure-change-xmin-ctid
subodh1810 Jun 21, 2023
52eb61d
🤖 Auto format source-postgres code [skip ci]
octavia-squidington-iii Jun 21, 2023
27fb1a2
Merge branch 'master' into state-structure-change-xmin-ctid
subodh1810 Jun 21, 2023
e7e89c0
initial drop for testing
rodireich Jun 24, 2023
0653433
Merge branch 'master' into 27558-break-a-long-running-ctid-query-into…
rodireich Jun 24, 2023
c5caac6
test
rodireich Jun 26, 2023
fa7db79
refactor query plan
rodireich Jun 27, 2023
6c18678
Merge branch 'master' into 27558-break-a-long-running-ctid-query-into…
rodireich Jun 27, 2023
3571e98
Add testing for query plan
rodireich Jun 27, 2023
0fa0da7
sanity
rodireich Jun 27, 2023
f203c63
sanity
rodireich Jun 27, 2023
43039fc
sanity
rodireich Jun 27, 2023
5d5052a
remove change in spec json
rodireich Jun 27, 2023
89c4ed0
sanity
rodireich Jun 27, 2023
d22e395
test
rodireich Jun 27, 2023
ac78f12
comment
rodireich Jun 27, 2023
2ffef6f
remove hardcoded testing value
rodireich Jun 27, 2023
3b6c7c8
add test case
rodireich Jun 27, 2023
a6e1ea8
revert json spec
rodireich Jun 27, 2023
53aff2f
test
rodireich Jun 28, 2023
2d4bca0
test
rodireich Jun 28, 2023
09b7a06
Merge branch 'master' into 27558-break-a-long-running-ctid-query-into…
alafanechere Jun 28, 2023
4cc5bea
Merge branch '27558-break-a-long-running-ctid-query-into-smaller-size…
alafanechere Jun 28, 2023
de85f62
test
rodireich Jun 28, 2023
a7939fa
test
rodireich Jun 28, 2023
689ced7
test
rodireich Jun 28, 2023
1b78b0f
test
rodireich Jun 28, 2023
bb37555
test
rodireich Jun 28, 2023
2c416a1
Automated Commit - Format and Process Resources Changes
rodireich Jun 28, 2023
3fa733e
test
rodireich Jun 28, 2023
d13ecff
Merge remote-tracking branch 'origin/27787-fine-tune-sync-parameters-…
rodireich Jun 28, 2023
f249110
test
rodireich Jun 29, 2023
403c865
small imrovements
rodireich Jun 30, 2023
622f4a1
Merge branch 'master' into 27558-break-a-long-running-ctid-query-into…
rodireich Jun 30, 2023
a4a6d00
typo
rodireich Jun 30, 2023
8d660f8
test
rodireich Jun 30, 2023
353192d
Merge branch '27558-break-a-long-running-ctid-query-into-smaller-size…
rodireich Jun 30, 2023
c96424f
fix test
rodireich Jun 30, 2023
75c933b
Merge branch '27558-break-a-long-running-ctid-query-into-smaller-size…
rodireich Jun 30, 2023
fe79713
Merge branch 'master' into 27787-fine-tune-sync-parameters-for-ctid-w…
rodireich Jun 30, 2023
0523484
test
rodireich Jul 2, 2023
78dd4e7
test
rodireich Jul 3, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
initial ctid for testing
  • Loading branch information
rodireich committed Jun 11, 2023
commit ec7198ce24e0e92eed05afaf7c2643223e37c8af
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,7 @@ public DataSource build() {
config.setPassword(password);
config.setUsername(username);

config.setIdleTimeout(1_000); // TODO: check here
/*
* Disable to prevent failing on startup. Applications may start prior to the database container
* being available. To avoid failing to create the connection pool, disable the fail check. This
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ protected AirbyteMessage computeNext() {
}

if (messageIterator.hasNext()) {
if (count % 10_000 == 0 && StringUtils.isNotBlank(lastCtid)) {
if (count % 1_000_000 == 0 && StringUtils.isNotBlank(lastCtid)) {
LOGGER.info("saving ctid state with {}", this.lastCtid);
return CtidStateManager.createStateMessage(pair, new CtidStatus().withCtid(lastCtid));
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ private PreparedStatement createCtidQueryStatement(
preparedStatement.setObject(1, "(0,0)");
}
LOGGER.info("Executing query for table {}: {}", tableName, preparedStatement);
preparedStatement.setQueryTimeout(120);
return preparedStatement;
} catch (final SQLException e) {
throw new RuntimeException(e);
Expand Down Expand Up @@ -161,7 +162,7 @@ private AutoCloseableIterator<AirbyteMessage> augmentWithLogs(final AutoCloseabl
AirbyteStreamUtils.convertFromNameAndNamespace(pair.getName(), pair.getNamespace()),
r -> {
final long count = recordCount.incrementAndGet();
if (count % 10000 == 0) {
if (count % 1_000_000 == 0) {
LOGGER.info("Reading stream {}. Records read: {}", streamName, count);
}
return r;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -418,7 +418,7 @@ private AutoCloseableIterator<AirbyteMessage> createReadIterator(final Database
AirbyteStreamUtils.convertFromNameAndNamespace(pair.getName(), pair.getNamespace()),
r -> {
final long count = recordCount.incrementAndGet();
if (count % 10000 == 0) {
if (count % 1_000_000 == 0) {
LOGGER.info("Reading stream {}. Records read: {}", streamName, count);
}
return r;
Expand Down