GC+/ Milestone 2 execution plan #4801

eden-ohana · 2022-12-12T22:09:19Z

nopcoder · 2022-12-13T06:34:57Z

design/accepted/gc_plus/gc-plus-milestone2-execution-plan.md

+## Milestone 1
+The first beta version that was released included:


Suggested change

## Milestone 1

The first beta version that was released included:

## Milestone 1

The first beta version that was released included:

nopcoder · 2022-12-13T06:36:36Z

design/accepted/gc_plus/gc-plus-milestone2-execution-plan.md

+1. Implementation of the clean run flow for old and new repository structures (without optimizations)
+2. Mark & Sweep
+3. Integration tests
+4. Backup & Restore


minimal support using rclone

nopcoder · 2022-12-13T06:43:04Z

design/accepted/gc_plus/gc-plus-milestone2-execution-plan.md

+## Milestone 2
+
+### Goals
+1. Removing the limitation of a read-only lakeFS during the job run


job run == gc?

nopcoder · 2022-12-13T06:44:03Z

design/accepted/gc_plus/gc-plus-milestone2-execution-plan.md

+2. Performance improvements - better parallelization of the storage namespace traversal
+3. Implementing Optimized Run


2 and 3 are performance and optimizations - the first item is self explained, I didn't understand the second one.
Do you mean incremental uncommitted gc?

@nopcoder Optimized run means incremental
@eden-ohana I think we should focus this milestone on optimization of the full run and adding more testing (unit / integration) - and create a separate milestone for the incremental/optimized run

nopcoder · 2022-12-13T06:48:32Z

design/accepted/gc_plus/gc-plus-milestone2-execution-plan.md

+        * [lakeFSFS renameObject method](https://github.com/treeverse/lakeFS/issues/4479)
+    * [Track copied objects in ref-store](https://github.com/treeverse/lakeFS/issues/4562)
+
+3. Performance improvements:


These are all uncommitted GC? maybe we can specify it as there are lakefs, common code and etc.

Suggested change

3. Performance improvements:

2. Performance improvements:

nopcoder · 2022-12-13T06:51:31Z

design/accepted/gc_plus/gc-plus-milestone2-execution-plan.md

+3. Performance improvements:
+    * [Optimaized listing on old repository structure](https://github.com/treeverse/lakeFS/issues/4620)
+    * [Efficient listing on committed entries](https://github.com/treeverse/lakeFS/issues/4600)
+    * [Optimized listing on slices](https://github.com/treeverse/lakeFS/issues/4614)


Thought we already doing this one

@nopcoder Yes you did :) I think we can close this issue

N-o-Z · 2022-12-13T08:51:51Z

design/accepted/gc_plus/gc-plus-milestone2-execution-plan.md

+3. Implementing Optimized Run
+
+### Non-Goals
+1. Support for non-S3 repositories


This is a big bullet - I think it should be itemized by storage providers. Also I really think we should do it incremental.
For example - for this milestone, starting with Azure - and afterwards working on GCS

N-o-Z

@eden-ohana Thanks for creating this plan.
It looks well put through - I think we learned a lot from the first milestone, especially about our expectations and we should put this into this plan.
There is a lot of work hidden inside the milestone goals lines in my opinion.
I believe we should focus on improving our current solution for the 2nd milestone - structuring it, creating more tests and add optimizations.
We should also start to create the ground for incremental runs but I think the client implementation itself should be planned for milestone 3

eden-ohana · 2022-12-13T12:23:32Z

@ozkatz can you give context from a product perspective? Is there any requirement to prioritize the GC+ for Azure?

ozkatz · 2022-12-13T12:51:11Z

@eden-ohana I was under the impression that since the "Sweep" part of GC+ is re-used (i.e. it's the same "sweep" that is already implemented for Azure) we should support this ootb?

There are active, production installations that we know of that are running on Azure, with compliance requirements that are very hard to meet without GC+..

eden-ohana · 2022-12-14T09:34:40Z

@eden-ohana I was under the impression that since the "Sweep" part of GC+ is re-used (i.e. it's the same "sweep" that is already implemented for Azure) we should support this ootb?

There are active, production installations that we know of that are running on Azure, with compliance requirements that are very hard to meet without GC+..

The sweep part is reused but we added more functionalities that weren't tesed on Azure

itaiad200

Thanks! It feels like a very-stressed milestone. I think we should settle for the "safe-write" lakeFS goal and leave the rest for the next milestones. The timeline itself is too short for all three, I think...

itaiad200 · 2022-12-15T11:46:43Z

design/accepted/gc_plus/gc-plus-milestone2-execution-plan.md

+
+* marks dependency
+
+1. Required changes by lakeFS:


I think we're missing a non-trivial test issue here. I'm not sure how to test that lakeFS is safe while GC is running. Feels like we should plan for that here too.

design/accepted/gc_plus/gc-plus-milestone2-execution-plan.md

N-o-Z · 2022-12-18T12:00:42Z

design/accepted/gc_plus/gc-plus-milestone2-execution-plan.md

+   * [Optimaized listing on old repository structure](https://github.com/treeverse/lakeFS/issues/4620)
+   * [Efficient listing on committed entries](https://github.com/treeverse/lakeFS/issues/4600)
+   * Benchmarks - verify uncommitted GC [performance requirements](https://github.com/treeverse/lakeFS/blob/e316cafe7717bb3203e4018837a41415aa61f74b/design/accepted/gc_plus/uncommitted-gc.md?plain=1#L185) are kept
+2. [Implement optimized run flow](https://github.com/treeverse/lakeFS/issues/4489)


Are we implementing the incremental run? I thought we decided to do it in the next milestone

N-o-Z · 2022-12-18T12:01:35Z

design/accepted/gc_plus/gc-plus-milestone2-execution-plan.md

+3. Support for non-S3 repositories
+   * Azure
+   * GCP
+4. Incorporation of committed & uncommitted GC into a single job


Same here - I thought we were going to push it towards the end of development

N-o-Z · 2022-12-18T12:02:45Z

design/accepted/gc_plus/gc-plus-milestone2-execution-plan.md

+
+By the end of this milestone, we will release a new beta version that includes all the additions.
+
+**Due date: 15/01/2023**


IMO, this due date as a bit ambitious. We still left a lot of work inside this milestone

It isn't anymore :)

itaiad200

LGTM

N-o-Z

👍🙏

add execution plan

996f554

eden-ohana added exclude-changelog PR description should not be included in next release changelog GC+ labels Dec 12, 2022

eden-ohana requested a review from a team December 12, 2022 22:09

nopcoder reviewed Dec 13, 2022

View reviewed changes

N-o-Z reviewed Dec 13, 2022

View reviewed changes

N-o-Z requested changes Dec 13, 2022

View reviewed changes

eden-ohana added 2 commits December 13, 2022 12:59

add execution plan

346bbad

pr comments

1a6bcf9

itaiad200 requested changes Dec 15, 2022

View reviewed changes

pr comments

214b060

eden-ohana requested review from nopcoder, N-o-Z and itaiad200 December 18, 2022 11:51

N-o-Z reviewed Dec 18, 2022

View reviewed changes

itaiad200 approved these changes Dec 18, 2022

View reviewed changes

pr comments

ba888e9

N-o-Z approved these changes Dec 18, 2022

View reviewed changes

eden-ohana merged commit fa39e88 into master Dec 18, 2022

eden-ohana deleted the gc+/mileston2-execution-plan branch December 18, 2022 14:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GC+/ Milestone 2 execution plan #4801

GC+/ Milestone 2 execution plan #4801

eden-ohana commented Dec 12, 2022 •

edited

Loading

nopcoder Dec 13, 2022

eden-ohana Dec 13, 2022

nopcoder Dec 13, 2022

eden-ohana Dec 13, 2022

nopcoder Dec 13, 2022

eden-ohana Dec 13, 2022

nopcoder Dec 13, 2022

N-o-Z Dec 13, 2022

nopcoder Dec 13, 2022

eden-ohana Dec 13, 2022

nopcoder Dec 13, 2022

N-o-Z Dec 13, 2022

eden-ohana Dec 13, 2022

N-o-Z Dec 13, 2022

N-o-Z left a comment

eden-ohana commented Dec 13, 2022

ozkatz commented Dec 13, 2022

eden-ohana commented Dec 14, 2022

itaiad200 left a comment

itaiad200 Dec 15, 2022

N-o-Z Dec 18, 2022

N-o-Z Dec 18, 2022

N-o-Z Dec 18, 2022

itaiad200 Dec 18, 2022

itaiad200 left a comment

N-o-Z left a comment

		## Milestone 1
		The first beta version that was released included:

		2. Performance improvements - better parallelization of the storage namespace traversal
		3. Implementing Optimized Run


		By the end of this milestone, we will release a new beta version that includes all the additions.

		Due date: 15/01/2023

GC+/ Milestone 2 execution plan #4801

GC+/ Milestone 2 execution plan #4801

Conversation

eden-ohana commented Dec 12, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

N-o-Z left a comment

Choose a reason for hiding this comment

eden-ohana commented Dec 13, 2022

ozkatz commented Dec 13, 2022

eden-ohana commented Dec 14, 2022

itaiad200 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itaiad200 left a comment

Choose a reason for hiding this comment

N-o-Z left a comment

Choose a reason for hiding this comment

eden-ohana commented Dec 12, 2022 •

edited

Loading