Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Logs new schema support #5528

Closed
wants to merge 40 commits into from
Closed

Conversation

nityanandagohain
Copy link
Member

@nityanandagohain nityanandagohain commented Jul 19, 2024

PR for new logs schema

Fixes #5555

current limitations

  • Attribute search is case sensitive, while resources and body search is case insensitive.
  • Minmax Index on numbers don't work (clickhouse limitation)

How to go about this PR

Section 1:- Logs v3 and v4 builder

  • V3 is the existing builder and has no changes
  • V4 is the new builder that supports the new schema, the test cases are similar to v3 with the changes related to the schema
  • Resource filters are only added to the resource table and not to the logs_v2 table

Section 2:- The materialize api's and attribute api's in reader.go

  • These APIs just houses changes to support new schema through the feature flags

Section 3:- runLogsListQuery in querier.go
We have updated the logs on how list data is fetched for logs, first we check the last 30 mins data to find the 100 logs and then increase the window size until 100 logs are found.

Section 4: - Use of the useLogsNewSchema flag.
Changes to different files are added just so that useLogsNewSchema can be passed around.

Copy link

Build Error! No Linked Issue found. Please link an issue or mention it in the body using #<issue_id>

@github-actions github-actions bot added the enhancement New feature or request label Jul 19, 2024
@nityanandagohain nityanandagohain mentioned this pull request Jul 25, 2024
21 tasks
@nityanandagohain nityanandagohain marked this pull request as ready for review August 22, 2024 16:21
@nityanandagohain nityanandagohain marked this pull request as draft August 26, 2024 05:02
@nityanandagohain nityanandagohain marked this pull request as ready for review August 27, 2024 12:50
@nityanandagohain
Copy link
Member Author

I have added a description of how to go about the PR, let me know if a walkthrough is required.

@srikanthccv
Copy link
Member

Even with a walkthrough, I won't be able to review this (I made a couple of attempts to skim through but it's too big to comprehend). It is 4k+ additions. Please send them in multiple stacked PRs.

@nityanandagohain
Copy link
Member Author

Sure, I will try to break it down.

@nityanandagohain nityanandagohain deleted the fix/new_schema branch September 16, 2024 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[EPIC]: Logs new schema
3 participants