Skip to content

Assembler: clone-all from link-index registry #1311

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 23, 2025

Conversation

reakaleek
Copy link
Member

@reakaleek reakaleek commented May 22, 2025

Context

With these changes the assembler build is more resilient, as it only fetches successfully built commits.

Changes

  • Change the RepositorySourcesFetcher to be able to fetch a specific commit.
  • clone-all command not fetches the commits from the link-index registry.

How to test

Try the following commands

dotnet run --project src/tooling/docs-assembler -- repo clone-all --environment prod --fetch-latest
dotnet run --project src/tooling/docs-assembler -- repo clone-all --environment prod
dotnet run --project src/tooling/docs-assembler -- repo clone-all --environment staging --fetch-latest
dotnet run --project src/tooling/docs-assembler -- repo clone-all --environment staging

@reakaleek reakaleek requested a review from Copilot May 22, 2025 22:52
@reakaleek reakaleek self-assigned this May 22, 2025
@reakaleek reakaleek marked this pull request as ready for review May 22, 2025 22:53
@reakaleek reakaleek requested a review from a team as a code owner May 22, 2025 22:53
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR reworks the repository cloning logic to fetch specific commits from the link-index registry and updates the CLI command accordingly.

  • Renamed and refactored the repository fetching method to support commit-specific cloning.
  • Updated the CLI command to use the new cloning logic and added configuration documentation for fetchLatest behavior.
  • Changed the repository configuration by updating the CheckoutStrategy property to an enum.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
src/tooling/docs-assembler/Sourcing/RepositorySourcesFetcher.cs Refactored method renamed to CloneAll with commit-specific fetching via the link-index registry.
src/tooling/docs-assembler/Cli/RepositoryCommands.cs Updated clone-all CLI command to handle the new fetchLatest parameter and call the updated cloning method.
src/Elastic.Documentation.Configuration/Assembler/Repository.cs Converted CheckoutStrategy from string to enum for improved type safety.
Comments suppressed due to low confidence (1)

src/tooling/docs-assembler/Sourcing/RepositorySourcesFetcher.cs:75

  • The variable 'context' used later (e.g., context.Collector.EmitError(...)) is not defined. Replace it with the appropriate diagnostics collector instance or variable that is in scope.
if (!linkRegistry.Repositories.TryGetValue(repo.Key, out var entry))

Copy link
Member

@Mpdreamz Mpdreamz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! just a few q?'s.

@@ -39,11 +41,13 @@ private void AssignOutputLogger()
/// <summary> Clones all repositories </summary>
/// <param name="strict"> Treat warnings as errors and fail the build on warnings</param>
/// <param name="environment"> The environment to build</param>
/// <param name="fetchLatest"> If true fetch the latest commit of the branch instead of the link registry entry ref</param>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whats our use-case for this? I like the flag but just wondering :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For local QAing, if you specify a different branch for a repo e.g. feature/new-content, you want to be able to fetch the latest.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general I wanted to keep the old behaviour. Just in case.

"--branch", branch
);
case CheckoutStrategy.Full:
ExecIn(checkoutFolder, "git", "sparse-checkout", "disable");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this necessary? (not questioning it just looking to learn).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's actually unlikely.. But I was just thinking, if you ever change the checkout strategy and if you run the command, while the repo is still checked out (cached) you want to also disabled it.

Comment on lines -158 to -161
"git", "clone", "--filter=blob:none", "--no-checkout", repository.Origin, checkoutFolder.FullName
);

ExecIn(checkoutFolder, "git", "sparse-checkout", "set", "--cone");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suprised how little effect --cone mode actually has on the size on the disk. Runs contrary to: https://github.blog/open-source/git/bring-your-monorepo-down-to-size-with-sparse-checkout/#sparse-checkout-and-partial-clones

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think because cone mode is the default according to https://git-scm.com/docs/git-sparse-checkout#_internalscone_mode_handling

@reakaleek reakaleek merged commit c73a00a into main May 23, 2025
15 checks passed
@reakaleek reakaleek deleted the feature/clone-all-from-link-index branch May 23, 2025 10:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants