Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(pageserver): split delta writer automatically determines key range #8850

Merged
merged 2 commits into from
Sep 9, 2024

Conversation

skyzh
Copy link
Member

@skyzh skyzh commented Aug 27, 2024

Problem

close #8838

Summary of changes

This patch modifies the split delta layer writer to avoid taking start_key and end_key when creating/finishing the layer writer. The start_key for the delta layers will be the first key provided to the layer writer, and the end_key would be the last_key.next(). This simplifies the delta layer writer API.

On that, the layer key hack is removed. Image layers now use the full key range, and delta layers use the first/last key provided by the user.

Checklist before requesting a review

  • I have performed a self-review of my code.
  • If it is a core feature, I have added thorough tests.
  • Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
  • If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

  • Do not forget to reformat commit message to not include the above checklist

@skyzh skyzh requested a review from a team as a code owner August 27, 2024 17:15
@skyzh skyzh requested a review from problame August 27, 2024 17:15
Copy link

github-actions bot commented Aug 27, 2024

3883 tests run: 3768 passed, 1 failed, 114 skipped (full report)


Failures on Postgres 14

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_storage_controller_heartbeats[release-pg14-failure4]"
Flaky tests (2)

Postgres 15

Postgres 14

Code coverage* (full report)

  • functions: 31.7% (7343 of 23156 functions)
  • lines: 49.9% (59528 of 119196 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
a2138c2 at 2024-09-09T20:44:28.340Z :recycle:

Copy link
Contributor

@problame problame left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The NON_L0_MAX can be removed in this PR

pub const NON_L0_MAX: Key = Key {

pageserver/src/tenant/storage_layer/split_writer.rs Outdated Show resolved Hide resolved
pageserver/src/tenant/storage_layer/split_writer.rs Outdated Show resolved Hide resolved
Signed-off-by: Alex Chi Z <chi@neon.tech>
@skyzh skyzh force-pushed the skyzh/remove-key-range-hack branch from 4a4fcd9 to a2138c2 Compare September 9, 2024 13:56
Copy link

@orca-security-us orca-security-us bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

@skyzh skyzh enabled auto-merge (squash) September 9, 2024 14:09
@skyzh skyzh merged commit e158df4 into main Sep 9, 2024
76 of 77 checks passed
@skyzh skyzh deleted the skyzh/remove-key-range-hack branch September 9, 2024 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pageserver: remove image/delta layer range hack in gc-compaction and
2 participants