Compact: Display Planned and Running Compactions #7590

amandaguan-ag · 2024-08-02T17:30:02Z

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

planned blocks

Verification

Signed-off-by: amandaguan-ag <amandaguan1314@gmail.com>

…tching planned blocks Signed-off-by: amandaguan-ag <amandaguan1314@gmail.com>

…ndicator, and Blocks component with API integration Signed-off-by: amandaguan-ag <amandaguan1314@gmail.com>

Signed-off-by: amandaguan-ag <amandaguan1314@gmail.com>

…hod planner)

saswatamcode

Thanks! Some feedback

saswatamcode · 2024-08-08T15:17:03Z

pkg/api/blocks/v1.go

+	}
+
+	metasByMinTime := []*metadata.Meta{&mockBlocks[0], &mockBlocks[1]}
+	plannedMetas, err := bapi.planner.Plan(context.TODO(), metasByMinTime, nil, nil)


Do we define bapi.planner anywhere?

Just defined bapi.planner in the BlocksAPI struct

saswatamcode · 2024-08-08T15:20:31Z

.bin/gopls

Probably shouldn't be committing these? :)

should we add these files in .gitignore, i got these files everytime i run make react-app

What is your GOBIN set to? I think this causes issues

my says /Users/amandaguan/Desktop/Epicodus/thanos/.bin, as instructed by CONTRIBUTING.md

I see, those are older instructions, usually don't need to.

pkg/api/blocks/v1.go

saswatamcode · 2024-08-08T15:22:06Z

pkg/ui/react-app/src/thanos/pages/blocks/Blocks.tsx

+  if (globalBlocksLoading || planLoading) return <div>Loading...</div>;
+  if (globalBlocksError) return <UncontrolledAlert color="danger">{globalBlocksError.toString()}</UncontrolledAlert>;
+  if (planError) return <UncontrolledAlert color="danger">{planError.toString()}</UncontrolledAlert>;
+


Would we need this, I think the WithStatusIndicator handles this for us?

I see. withStatusIndicator HOC will be handling the loading and error states, i will remove the redundant checks

saswatamcode · 2024-08-08T15:22:52Z

pkg/ui/react-app/src/thanos/pages/blocks/Blocks.tsx

+
+  if (err) {
+    return (
+      <UncontrolledAlert color="danger">{err.toString()}</UncontrolledAlert>


Probably done with status indicator already?

I will look in withStatusIndicator for this one. I see BlocksContent also has this direct handling

I see withStatusIndicator is handling error in general and Displaying an error message (. I will take off from PlanBlocksContent and maybe from BlocksContent

saswatamcode · 2024-08-08T15:23:18Z

pkg/ui/react-app/src/thanos/pages/blocks/Blocks.tsx

+            key={key}
+            data={blocksPool[key]}
+            title={`Plan View - ${key}`}
+            selectBlock={() => { }}  // No-op function if selection is not applicable


Why no-op here?

saswatamcode · 2024-08-08T15:24:15Z

pkg/ui/react-app/src/thanos/pages/blocks/Blocks.tsx

-      isLoading={isLoading}
-    />
+    <div>
+      <h2>Planned Blocks</h2>


I think...this would make sense as a separate page rather than a component above the Global Blocks screen?

…n assigned to bapi.planner

…ithStatusIndicator HOC should be handling the loading and error states

The selectBlock function is passed down to SourceView where it can be invoked, typically through some user interaction like clicking on a block.

- Add new routes for Global Blocks and Planned Blocks in App.tsx - Update navigation configuration to include links to new pages - Create GlobalBlocks and PlannedBlocks components - Implement separate views for global and planned blocks - Ensure proper routing and prop passing for new components This commit introduces dedicated pages for viewing Global Blocks and Planned Blocks, improving the organization and accessibility of block information in the Thanos UI. Signed-off-by: amandaguan-ag <amandaguan1314@gmail.com>

Signed-off-by: amandaguan-ag <amandaguan1314@gmail.com>

This commit introduces the `UpdateOnPlanned` method in the `tsdbBasedPlanner` struct, allowing for dynamic updates of planned block metadata. The callback mechanism is integrated into the compaction process to handle planned blocks efficiently. The main changes include: - Definition and implementation of `UpdateOnPlanned` in `tsdbBasedPlanner`. - Adjustment in `runCompact` to instantiate `tsdbPlanner` at the beginning of the function, ensuring accessibility throughout the compaction process and avoiding repeated declaration. - Use of `UpdateOnPlanned` to set planned blocks via the API, enhancing the flexibility and responsiveness of block management. By moving the `tsdbPlanner` instantiation earlier in `runCompact`, we avoid duplication and enhance code clarity, ensuring the planner is accessible where needed without multiple initializations. Signed-off-by: amandaguan-ag<amandaguan1314@gmail.com>

…an the tsdbBasedPlanner struct

Signed-off-by: amandaguan-ag <amandaguan1314@gmail.com>

amandaguan-ag · 2024-08-21T16:13:40Z

pkg/compact/compact.go

@@ -1135,6 +1140,9 @@ func (cg *Group) compact(ctx context.Context, dir string, planner Planner, comp
 	var toCompact []*metadata.Meta
 	if err := tracing.DoInSpanWithErr(ctx, "compaction_planning", func(ctx context.Context) (e error) {
 		toCompact, e = planner.Plan(ctx, cg.metasByMinTime, errChan, cg.extensions)
+		planner.UpdateOnPlanned(func(metas []metadata.Meta, err error) {
+			level.Info(cg.logger).Log("msg", "planner updated", "metas", len(metas), "err", err)


planner.UpdateOnPlanned(func(blocks []metadata.Meta, err error) {
api.SetPlanned(blocks, err)
})

amandaguan-ag · 2024-08-21T16:15:03Z

pkg/api/blocks/v1.go

+// SetPlanned updates the plan blocks' metadata in the API.
+func (bapi *BlocksAPI) SetPlanned(blocks []metadata.Meta, err error) {
+
+	bapi.plannedBlocksInfo.set(blocks, err)


should be a append operation

saswatamcode · 2024-08-15T16:34:57Z

pkg/compact/planner.go

@@ -53,6 +54,10 @@ func (p *tsdbBasedPlanner) Plan(_ context.Context, metasByMinTime []*metadata.Me
 	return p.plan(p.noCompBlocksFunc(), metasByMinTime)
 }

+func (p *tsdbBasedPlanner) UpdateOnPlanned(f func([]metadata.Meta, error)) {
+	p.updateOnPlanned = f


updateOnPlanned needs to be called from somewhere..to figure out.

Also, let's do this on the planner interface instead?

saswatamcode · 2024-08-15T16:35:30Z

pkg/compact/planner.go

@@ -25,6 +25,7 @@ type tsdbBasedPlanner struct {
 	ranges []int64

 	noCompBlocksFunc func() map[ulid.ULID]*metadata.NoCompactMark
+	updateOnPlanned  func([]metadata.Meta, error)


Maybe listener instead of updateOnPlanned?

saswatamcode · 2024-08-21T16:12:08Z

cmd/thanos/compact.go

@@ -288,6 +295,7 @@ func runCompact(
 		cf.UpdateOnChange(func(blocks []metadata.Meta, err error) {
 			api.SetLoaded(blocks, err)
 		})
+


We should pass in blocksAPI.SetPlanned into UpdateOnPlanned callback?

saswatamcode · 2024-08-21T16:13:18Z

pkg/compact/compact.go

@@ -1135,6 +1140,9 @@ func (cg *Group) compact(ctx context.Context, dir string, planner Planner, comp
 	var toCompact []*metadata.Meta
 	if err := tracing.DoInSpanWithErr(ctx, "compaction_planning", func(ctx context.Context) (e error) {
 		toCompact, e = planner.Plan(ctx, cg.metasByMinTime, errChan, cg.extensions)
+		planner.UpdateOnPlanned(func(metas []metadata.Meta, err error) {
+			level.Info(cg.logger).Log("msg", "planner updated", "metas", len(metas), "err", err)


SetPlanned over here?

saswatamcode · 2024-08-21T16:20:29Z

pkg/compact/compact.go

@@ -1135,6 +1140,9 @@ func (cg *Group) compact(ctx context.Context, dir string, planner Planner, comp
 	var toCompact []*metadata.Meta
 	if err := tracing.DoInSpanWithErr(ctx, "compaction_planning", func(ctx context.Context) (e error) {
 		toCompact, e = planner.Plan(ctx, cg.metasByMinTime, errChan, cg.extensions)
+		planner.UpdateOnPlanned(func(metas []metadata.Meta, err error) {


Let's call this on group level, and get plans of every group, and set it once

1. Add BlocksAPI to Compact method 2. Update BlocksAPI initialization - Return plannedBlocksInfo in BlocksAPI constructor 3. Enhance compact package - Add blocksAPI parameter to Group.compact method This change improves the integration of BlocksAPI with the compaction process. Signed-off-by: [amandaguan-ag] <amandaguan1314@gmail.com>

1. Introduce `blocksAPI` fields to the `BucketCompactor` struct. 2. Implement logic to retrieve metas for the current group and convert them into `[]metadata.Meta`. 3. Invoke `Syncer.SetPlanned` prior to compaction to update the syncer's state. 4. Invoke `BlocksAPI.SetPlanned` to synchronize the API with the planned metas. These updates enhance the tracking and management of planned compactions, streamlining the overall process and ensuring that external systems are informed of upcoming compactions. Signed-off-by: [amandaguan-ag] <amandaguan1314@gmail.com>

saswatamcode · 2024-08-27T16:23:26Z

pkg/compact/compact.go

+					groupMetas := g.Metas()
+
+					// Delete before merge: Convert []*metadata.Meta to []metadata.Meta
+					plannedMetas := make([]metadata.Meta, len(groupMetas))


Ok, I think we are just setting blocks of the group, instead of the planned blocks? We need planner.Plan output

saswatamcode · 2024-08-27T16:38:19Z

pkg/compact/compact.go

+					}
+
+					// Delete before merge: Call Syncer.SetPlanned before compaction
+					if err := c.sy.SetPlanned(plannedMetas); err != nil {


I think this will be overriden on each group

amandaguan-ag · 2024-08-27T16:47:26Z

pkg/compact/compact.go

@@ -1475,7 +1490,33 @@ func (c *BucketCompactor) Compact(ctx context.Context, v1 *v1.BlocksAPI) (rerr e
 			go func() {
 				defer wg.Done()
 				for g := range groupChan {
-					shouldRerunGroup, _, err := g.Compact(workCtx, c.compactDir, c.planner, c.comp, c.blockDeletableChecker, c.compactionLifecycleCallback, v1)
+					// Delete before merge: Get the metas for the current group
+					groupMetas := g.Metas()


do i need change the source from g to planner.plan, move out not for group

read go routine, and channels ->concurrency

…anned

pull-request-size bot added the size/M label Aug 2, 2024

amandaguan-ag added 4 commits August 2, 2024 10:33

feat(test): Add sleep delay and browser opening for e2einteractive data

6ceb3b7

Signed-off-by: amandaguan-ag <amandaguan1314@gmail.com>

feat(blocksAPI): Add plannedBlocksInfo, plannedLock, and route for fe…

112bc0d

…tching planned blocks Signed-off-by: amandaguan-ag <amandaguan1314@gmail.com>

feat(ui): Add placeholders for PlanBlocksContent, PlanViewWithStatusI…

1dca384

…ndicator, and Blocks component with API integration Signed-off-by: amandaguan-ag <amandaguan1314@gmail.com>

chore(build): Update auto-generated React build files

e7c15bf

Signed-off-by: amandaguan-ag <amandaguan1314@gmail.com>

amandaguan-ag force-pushed the ag-compact-plan-ui branch from 8cb3769 to e7c15bf Compare August 2, 2024 17:33

Add mock data and simply list out planned blocks

dbe5d56

pull-request-size bot added size/L and removed size/M labels Aug 8, 2024

amandaguan-ag added 2 commits August 8, 2024 08:14

TODO: fix bapi.planner undefined (type *BlocksAPI has no field or met…

d43a456

…hod planner)

TODO: fix Type 'Block[][]' is not assignable to type 'BlocksPool'.

c6cf932

amandaguan-ag force-pushed the ag-compact-plan-ui branch from af0c7b3 to c6cf932 Compare August 8, 2024 15:17

saswatamcode reviewed Aug 8, 2024

View reviewed changes

amandaguan-ag added 12 commits August 8, 2024 08:52

planner is passed as an argument to the NewBlocksAPI function and the…

71b94c6

…n assigned to bapi.planner

simplify the Blocks component by removing the redundant checks, The w…

72942d0

…ithStatusIndicator HOC should be handling the loading and error states

add useState to manage the selected block.

2a486c9

The selectBlock function is passed down to SourceView where it can be invoked, typically through some user interaction like clicking on a block.

remove SetPlanned placeholder

f9777dc

feat(api): Add mock data for planned blocks

3c7b555

remove planner

f343a47

Update auto-generated React build files

08592ae

Signed-off-by: amandaguan-ag <amandaguan1314@gmail.com>

fix prettier error

ea98f7c

Move GlobalBlocks and PlannedBlocks components to the blocks directory

394d56e

Signed-off-by: amandaguan-ag <amandaguan1314@gmail.com>

Update auto-generated React build files

7c46fc2

Signed-off-by: amandaguan-ag <amandaguan1314@gmail.com>

fix comment indent

28ec713

amandaguan-ag force-pushed the ag-compact-plan-ui branch 2 times, most recently from c461c53 to 28ec713 Compare August 15, 2024 15:26

amandaguan-ag added 3 commits August 15, 2024 08:26

define the UpdateOnPlanned method on the Planner interface, rather th…

06cfa3a

…an the tsdbBasedPlanner struct

call UpdateOnPlanned after each Plan call.

87134d3

Replace mock data with plannedBlocksInfo

ee15442

Signed-off-by: amandaguan-ag <amandaguan1314@gmail.com>

amandaguan-ag commented Aug 21, 2024

View reviewed changes

saswatamcode reviewed Aug 21, 2024

View reviewed changes

amandaguan-ag force-pushed the ag-compact-plan-ui branch from 61711a7 to ac2253a Compare August 23, 2024 15:59

amandaguan-ag added 2 commits August 26, 2024 11:02

update compact_e2e_test with api field in NewBucketCompactor

2c7d008

saswatamcode reviewed Aug 27, 2024

View reviewed changes

amandaguan-ag commented Aug 27, 2024

View reviewed changes

amandaguan-ag added 3 commits August 28, 2024 23:19

replace g.Metas() with planner.Plan

82a07e7

accumulate all the planned metas from all groups before calling SetPl…

b4ffc90

…anned

add mergeFunc to NewBucketCompactor

1046dfa

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compact: Display Planned and Running Compactions #7590

Compact: Display Planned and Running Compactions #7590

amandaguan-ag commented Aug 2, 2024 •

edited

Loading

saswatamcode left a comment

saswatamcode Aug 8, 2024

amandaguan-ag Aug 8, 2024

saswatamcode Aug 8, 2024

amandaguan-ag Aug 8, 2024

saswatamcode Aug 9, 2024

amandaguan-ag Aug 9, 2024

saswatamcode Aug 9, 2024

saswatamcode Aug 8, 2024

amandaguan-ag Aug 8, 2024

saswatamcode Aug 8, 2024

amandaguan-ag Aug 9, 2024

amandaguan-ag Aug 9, 2024

saswatamcode Aug 8, 2024

saswatamcode Aug 8, 2024

amandaguan-ag Aug 21, 2024

amandaguan-ag Aug 21, 2024

saswatamcode Aug 15, 2024

saswatamcode Aug 15, 2024

saswatamcode Aug 21, 2024

saswatamcode Aug 21, 2024

saswatamcode Aug 21, 2024

saswatamcode Aug 27, 2024

saswatamcode Aug 27, 2024

amandaguan-ag Aug 27, 2024

amandaguan-ag Aug 27, 2024

Compact: Display Planned and Running Compactions #7590

Are you sure you want to change the base?

Compact: Display Planned and Running Compactions #7590

Conversation

amandaguan-ag commented Aug 2, 2024 • edited Loading

Changes

Verification

saswatamcode left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amandaguan-ag commented Aug 2, 2024 •

edited

Loading