Feature/deployment concurrency

[erikkrieg]

What issue type does this pull request address?
/kind enhancement
/kind feature

What does this pull request do? Which issues does it resolve?
Enables deployments to opt into deploying concurrently in order to improve performance for configs with many deployments.

resolves #1877

We did some tests with a modified version of examples/quickstart and found a roughly 35% - 50% reduction in deployment time. The more deployments, the greater the improvement.

• 3 concurrent vs 3 sequential deployments saw ~35% reduction in time.
• 6 concurrent vs 6 sequential deployments saw ~50% reduction in time.

Please provide a short message that should be published in the DevSpace release notes
Allow select deployments to deploy concurrently by setting concurrent: true.

What else do we need to know?
The logging was changed to resemble how concurrent image build logs are handled. We attempted to capture the effect the sequential deploys get from using StartWait but this is something we were less certain about.

Example snippet of logging output:

[redis] Done: Successfully deployed redis with helm
[pgsql] Done: Deployed helm chart (Release revision: 1)
[pgsql] Done: Successfully deployed pgsql with helm
[wait] ⠋ Deploying 2 deployments concurrently (13s)

Something we didn't love but are not sure how to address is that as the number of remaining deployments decrements a new StartWait call is made and therefore the count of seconds starts back at 0.

We also wanted to wait for feedback before updating any documentation and will certainly do this if/when things are looking good :D

[pratikjagrut]

@erikkrieg Thanks a lot for the PR! :)

[pratikjagrut]

It'll also help if you could add e2e tests for this feature.
Maybe two scenarios:

1. All deployments are concurrent.
2. Some deployments are concurrent and some are not.

You can refer to some existing e2e tests under ./e2e/tests/deploy and add a new one to the same dir.

@FabianKramm WDYT?

[kuuji]

Thanks for the suggestion @pratikjagrut
I just added some e2e tests for both all concurrent and some concurrent some sequential.

I also included some vscode config to run e2e, similar to what vcluster has. Let me know if you prefer that specific part to be in a different PR since it's not really related to this PR.

[pratikjagrut]

Thank you @kuuji :) E2E tests LGTM just some minor changes.

[pratikjagrut]

I also included some vscode config to run e2e, similar to what vcluster has. Let me know if you prefer that specific part to be in a different PR since it's not really related to this PR.

Personally, I would have prefered separate PR for this change as it is not an intrinsic part of this feature.

[FabianKramm]

@erikkrieg @kuuji thanks so much for this PR! Thats really an improvement for DevSpace! Besides some small things in the e2e test the PR looks great and is ready to get merged!

[kuuji]

I also included some vscode config to run e2e, similar to what vcluster has. Let me know if you prefer that specific part to be in a different PR since it's not really related to this PR.

Personally, I would have prefered separate PR for this change as it is not an intrinsic part of this feature.

That's fair, I'll put a separate PR 👍

[erikkrieg]

Included documentation. Feeling pretty good about most of it but not sure whether the warning about an edge case with Helm hooks and concurrency is too specific or not.

[FabianKramm]

@erikkrieg thanks a lot, LGTM now!

[erikkrieg]

Thanks, @FabianKramm and @pratikjagrut!