Skip to content

Conversation

gongxun0928
Copy link
Contributor

  1. Explicit inline functions.
  2. Instead of checking the physical size of the file every time a tuple is inserted, it is checked every 16 tuples.

performance result

create table t1(a int, b int, c int, d int, e text, f text,g text, h text) using pax with(compresstype
=zstd,compresslevel=5);

gpadmin=# insert into t1 select i, i+1,i+2,i+3, i::text, i::text, i::text, i::text from generate_series(1,5000000) i;
INSERT 0 5000000
Time: 6124.535 ms (00:06.125)
gpadmin=# insert into t1 select i, i+1,i+2,i+3, i::text, i::text, i::text, i::text from generate_series(1,5000000) i;
INSERT 0 5000000
Time: 5993.682 ms (00:05.994)

-- optimized with this commit
create table t1(a int, b int, c int, d int, e text, f text,g text, h text) using pax with(compresstype
=zstd,compresslevel=5);
gpadmin=# insert into t1 select i, i+1,i+2,i+3, i::text, i::text, i::text, i::text from generate_series(1,5000000) i;
INSERT 0 5000000
Time: 5713.184 ms (00:05.713)
gpadmin=# insert into t1 select i, i+1,i+2,i+3, i::text, i::text, i::text, i::text from generate_series(1,5000000) i;
INSERT 0 5000000
Time: 5430.221 ms (00:05.430)

Fixes #ISSUE_Number

What does this PR do?

Type of Change

  • Bug fix (non-breaking change)
  • New feature (non-breaking change)
  • Breaking change (fix or feature with breaking changes)
  • Documentation update

Breaking Changes

Test Plan

  • Unit tests added/updated
  • Integration tests added/updated
  • Passed make installcheck
  • Passed make -C src/test installcheck-cbdb-parallel

Impact

Performance:

User-facing changes:

Dependencies:

Checklist

Additional Context

CI Skip Instructions


@gongxun0928 gongxun0928 force-pushed the performance/import-tuple-insert-performance-and-eliminate-some-call-overhead branch from 7f0085c to a342620 Compare September 21, 2025 13:54
@gongxun0928 gongxun0928 force-pushed the performance/import-tuple-insert-performance-and-eliminate-some-call-overhead branch from a342620 to 72d88a1 Compare September 22, 2025 15:58
@gongxun0928 gongxun0928 force-pushed the performance/import-tuple-insert-performance-and-eliminate-some-call-overhead branch 2 times, most recently from 688b7cf to 50ef57c Compare September 23, 2025 07:25
@gongxun0928 gongxun0928 force-pushed the performance/import-tuple-insert-performance-and-eliminate-some-call-overhead branch from 50ef57c to 254eb7b Compare September 26, 2025 03:39
@gongxun0928 gongxun0928 force-pushed the performance/import-tuple-insert-performance-and-eliminate-some-call-overhead branch 2 times, most recently from bc3ca05 to d479902 Compare October 9, 2025 07:07
Copy link
Contributor

@jiaqizho jiaqizho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with single comment

if (strategy_->ShouldSplit(writer_->PhysicalSize(), num_tuples_)) {
// Sampled split check to reduce PhysicalSize() overhead
// We first perform a sampled pre-write check to avoid empty files.
if ((num_tuples_ % PAX_SPLIT_CHECK_INTERVAL) == 0) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be?

  if ((num_tuples_ % PAX_SPLIT_CHECK_INTERVAL) == 0) {
    cur_physical_size_ = writer_->PhysicalSize();
    if (strategy_->ShouldSplit(cur_physical_size_, num_tuples_)) {
      ... 
    }
  }

Copy link
Contributor Author

@gongxun0928 gongxun0928 Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because of the CTID constraint, we have to strictly enforce the accuracy of the tuple count and make sure it doesn't exceed PAX_MAX_NUM_TUPLES_PER_FILE. That's why we kept this precise check here.

On the other hand, the biggest performance hit here is the PhysicalSize() function. So to reduce the overhead of calling it so often, we only check the file size every PAX_SPLIT_CHECK_INTERVAL tuples.

@gongxun0928 gongxun0928 force-pushed the performance/import-tuple-insert-performance-and-eliminate-some-call-overhead branch from d479902 to eee39f5 Compare October 10, 2025 12:33
1. Explicit inline functions.
2. Instead of checking the physical size of the file every time a tuple
is inserted, it is checked every 16 tuples.

performance result
```
create table t1(a int, b int, c int, d int, e text, f text,g text, h text) using pax with(compresstype
=zstd,compresslevel=5);

gpadmin=# insert into t1 select i, i+1,i+2,i+3, i::text, i::text, i::text, i::text from generate_series(1,5000000) i;
INSERT 0 5000000
Time: 6124.535 ms (00:06.125)
gpadmin=# insert into t1 select i, i+1,i+2,i+3, i::text, i::text, i::text, i::text from generate_series(1,5000000) i;
INSERT 0 5000000
Time: 5993.682 ms (00:05.994)

-- optimized with this commit
create table t1(a int, b int, c int, d int, e text, f text,g text, h text) using pax with(compresstype
=zstd,compresslevel=5);
gpadmin=# insert into t1 select i, i+1,i+2,i+3, i::text, i::text, i::text, i::text from generate_series(1,5000000) i;
INSERT 0 5000000
Time: 5713.184 ms (00:05.713)
gpadmin=# insert into t1 select i, i+1,i+2,i+3, i::text, i::text, i::text, i::text from generate_series(1,5000000) i;
INSERT 0 5000000
Time: 5430.221 ms (00:05.430)

```
@gongxun0928 gongxun0928 force-pushed the performance/import-tuple-insert-performance-and-eliminate-some-call-overhead branch from eee39f5 to 4cc222b Compare October 10, 2025 12:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants