Skip to content

Parquet design proposal #703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open

Parquet design proposal #703

wants to merge 17 commits into from

Conversation

yeya24
Copy link
Owner

@yeya24 yeya24 commented Apr 21, 2025

What this PR does:

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

dependabot bot and others added 13 commits April 9, 2025 13:34
Bumps the actions-dependencies group with 1 update: [github/codeql-action](https://github.com/github/codeql-action).


Updates `github/codeql-action` from 3.28.13 to 3.28.14
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@1b549b9...fc7e4a0)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 3.28.14
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
…ithub_actions/actions-dependencies-7e8f67935b

Bump github/codeql-action from 3.28.13 to 3.28.14 in the actions-dependencies group
…ls (cortexproject#6698)

* fix query frontend per tenant metrics leak when cleaning up user labels

Signed-off-by: Ben Ye <benye@amazon.com>

* changelog

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
…6661)

Bumps [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt) from 5.2.1 to 5.2.2.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](golang-jwt/jwt@v5.2.1...v5.2.2)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Add support for percentage based shard size

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

* Refactor implementation and add unit tests

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

* Update CHANGELOG

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

* Add support for percentage based shard size

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

* Refactor implementation and add unit tests

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

* Update configuration doc

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

* Remove wrong rebase change

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

* Refactor shard size logic and add more unit tests

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

* Minor refactoring

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

* Update CHANGELOG to trigger test re-run

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

* Remove default shardsize value

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

* Addressing PR comments

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

* Minor refactor

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

* Minor comment update

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

* Update documentation

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

* Update docs

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>

---------

Signed-off-by: Wilbert Guo <wilbeguo@amazon.com>
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
* Include KubeCon London Talks 2025
…tive # of NH series (cortexproject#6695)

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
…` in Ingesters and StoreGateways (cortexproject#6674)

* Add resource based throttling to ingesters and store gateways

Signed-off-by: Justin Jung <jungjust@amazon.com>

* doc

Signed-off-by: Justin Jung <jungjust@amazon.com>

* Add automaxprocs

Signed-off-by: Justin Jung <jungjust@amazon.com>

* nit

Signed-off-by: Justin Jung <jungjust@amazon.com>

* Add test for monitor

Signed-off-by: Justin Jung <jungjust@amazon.com>

* fix tests

Signed-off-by: Justin Jung <jungjust@amazon.com>

* changelog

Signed-off-by: Justin Jung <jungjust@amazon.com>

* fix test

Signed-off-by: Justin Jung <jungjust@amazon.com>

* remove interface

Signed-off-by: Justin Jung <jungjust@amazon.com>

* address comments

Signed-off-by: Justin Jung <jungjust@amazon.com>

* rename doc

Signed-off-by: Justin Jung <jungjust@amazon.com>

* Make monitor more generic + separate scanners

Signed-off-by: Justin Jung <jungjust@amazon.com>

* fix tests

Signed-off-by: Justin Jung <jungjust@amazon.com>

* fix more tests

Signed-off-by: Justin Jung <jungjust@amazon.com>

* remove monitor_test.go

Signed-off-by: Justin Jung <jungjust@amazon.com>

* move noop scanner to darwin scanner

Signed-off-by: Justin Jung <jungjust@amazon.com>

* doc update

Signed-off-by: Justin Jung <jungjust@amazon.com>

* doc

Signed-off-by: Justin Jung <jungjust@amazon.com>

* lint

Signed-off-by: Justin Jung <jungjust@amazon.com>

* add debugging log on unsupported resource type

Signed-off-by: Justin Jung <jungjust@amazon.com>

* test

Signed-off-by: Justin Jung <jungjust@amazon.com>

* add more error handling + resource_based_limiter_limit metric

Signed-off-by: Justin Jung <jungjust@amazon.com>

* fix test

Signed-off-by: Justin Jung <jungjust@amazon.com>

* fix test

Signed-off-by: Justin Jung <jungjust@amazon.com>

* update changelog

Signed-off-by: Justin Jung <jungjust@amazon.com>

* Move noopScanner to scanner.go and fix RegisterFlagsWithPrefix

Signed-off-by: Justin Jung <jungjust@amazon.com>

* Add limit breached metric + wrap error with 429

Signed-off-by: Justin Jung <jungjust@amazon.com>

* Add more validation and test on instance_limits

Signed-off-by: Justin Jung <jungjust@amazon.com>

* Added _total to counter metric

Signed-off-by: Justin Jung <jungjust@amazon.com>

---------

Signed-off-by: Justin Jung <jungjust@amazon.com>
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.37.0 to 0.38.0.
- [Commits](golang/net@v0.37.0...v0.38.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.38.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Add support for rule group labels

Signed-off-by: Mustafain Ali Khan <mustalik@amazon.com>

* fix test

Signed-off-by: Mustafain Ali Khan <mustalik@amazon.com>

* Include group labels in backup and update integration test

Signed-off-by: Mustafain Ali Khan <mustalik@amazon.com>

---------

Signed-off-by: Mustafain Ali Khan <mustalik@amazon.com>
@yeya24 yeya24 force-pushed the parquet-design-proposal branch from df5227f to 792964b Compare April 21, 2025 17:24
Signed-off-by: Ben Ye <benye@amazon.com>
@yeya24 yeya24 force-pushed the parquet-design-proposal branch from 792964b to ef43b28 Compare April 21, 2025 18:18
yeya24 added 3 commits April 21, 2025 20:37
Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants