Add support for grouping dependency updates

[alejandrohdezma]

💻 How to review this PR?

This PR was created with the idea of being reviewed commit by commit.

Each commit contains an incremental change (and its tests, if applicable) that makes it easier to review. Also some of the commits contain additional information in their description to help understand why the change was made.

I also recommend checking "Hide whitespace" when reviewing this PR!

🎯 What's the goal of this PR?

This PR adds support for a long-requested feature of Scala Steward, the ability to group pull-request updates.

❓ How this will be done?

A new configuration value (pullRequests.grouping) can be added to the .scala-steward.conf file. This config will represent the desired PR groups.

A group will be an object with:

• name (mandatory): the name of the group, can be used for things like naming the branch.
• title (optional): if provided it will be used as the title for the PR.
• filter (mandatory): a non-empty list containing the filters to use to know if an update falls into this group. Filters would have this format:
  • {version = "major" | "minor" | "patch" | "pre-release" | "build-metadata"}, group = "{group}", artifact = "{artifact}"}.

For filters every component is optional but at least one must be provided. For grouping every update a filter like {group = "*"} can be provided.

Examples

pullRequests.grouping = [
  { name = "patches", title = "Patch updates", filter = [{version = "patch"}] },
  { name = "minor_major", title = "Minor/major updates", filter = [{version = "minor"}, {version = "major"}] }
]

☝🏼 This configuration will group updates in two PRs: one titled "Patch updates" that will contain patch version updates and another one titled "Minor/major updates" with the rest of them.

pullRequests.grouping = [
  { name = "typelevel", title = "Typelevel updates", filter = [{group = "org.typelevel"}, {group = "org.http4s"}] }
]

☝🏼 This configuration will group Typelevel and Http4s updates into one PR. Since there is no other group, any update not falling into this group will follow normal behavior.

pullRequests.grouping = [
  { name = "my_libraries", filter = [{"artifact" = "my-library-*"}, {"artifact" = "my-other-library", "group" = "my-org.*"}] },
  { name = "all", title = "Not my libraries, filter = [{group = "*"] }
]

☝🏼 This configuration will group updates whose artifact is named my-library or my-org:my-other-library in one PR titled my_libraries (no title provided, so it will use name) and any other update under one PR titled "Not my libraries"

🔖 Summary of changes made

• Created class for representing configuration for grouping updates pull-requests
• Add method for grouping Update objects based on configuration.
• Fork the current method to process updates:
  • Updates not falling into any group will follow normal update mechanism.
  • A new mechanism will be created for creating pull-requests with grouped updates. This "grouped" pull requests should contain most features from the current mechanism (list of changes, scalafix migrations, hooks...) but there are some (closing a PR ignores the version) that won't be supported.
• Adding tests for new paths in the code.
• Update documentation for new configuration.

[alejandrohdezma]

@fthomas @exoego WDYT about this approach? Should I take something into account before I proceed with the next steps?

[codecov]

Codecov Report

Base: 80.62% // Head: 80.28% // Decreases project coverage by -0.33% ⚠️

Coverage data is based on head (095af21) compared to base (a80b77c).
Patch coverage: 71.25% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2714      +/-   ##
==========================================
- Coverage   80.62%   80.28%   -0.34%     
==========================================
  Files         147      149       +2     
  Lines        2740     2831      +91     
  Branches      175      191      +16     
==========================================
+ Hits         2209     2273      +64     
- Misses        531      558      +27     

Impacted Files	Coverage Δ
.../scala/org/scalasteward/core/data/UpdateData.scala	50.00% <ø> (ø)
...ala/org/scalasteward/core/nurture/NurtureAlg.scala	0.00% <0.00%> (ø)
...lasteward/core/repoconfig/PullRequestsConfig.scala	91.66% <ø> (ø)
.../scala/org/scalasteward/core/vcs/VCSExtraAlg.scala	100.00% <ø> (ø)
...main/scala/org/scalasteward/core/data/SemVer.scala	88.00% <62.50%> (-12.00%)	⬇️
...scalasteward/core/repoconfig/GroupRepoConfig.scala	83.33% <75.00%> (+8.33%)	⬆️
...main/scala/org/scalasteward/core/util/logger.scala	88.23% <75.00%> (-4.63%)	⬇️
...main/scala/org/scalasteward/core/data/Update.scala	96.29% <90.90%> (-1.38%)	⬇️
...ward/core/repoconfig/PullRequestUpdateFilter.scala	94.44% <94.44%> (ø)
...in/scala/org/scalasteward/core/git/CommitMsg.scala	100.00% <100.00%> (ø)
... and 7 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[exoego]

Great. I like the overall idea since it can cover multiple scenario (1. Grouping to reduce number of PR/CI, 2. Grouping relevant dependencies like Typelevel OSSs).

[alejandrohdezma]

@exoego @fthomas this is ready to review now. Even though there are a ton of changes, they should be easy to follow if following the How to review this PR? section. The names for GroupedUpdate and AnUpdate are not fixed at all, we've only used those to make them easily distinguishable from the previous classes. We can update those in a future PR and hopefully find some common stuff to pull out of individual classes (GroupedUpdate, Update.Single and Update.Group) into the super-type AnUpdate. LMK what you think.

[exoego]

@alejandrohdezma I think it is better to explain the differences of these version enums.
Specially, I do not get build-metadata 🙇

[alejandrohdezma]

Maybe it can be a good idea to refer to the semver.org docs on these docs. WDYT?

On build-metadata from point 10:

Build metadata MAY be denoted by appending a plus sign and a series of dot separated identifiers immediately following the patch or pre-release version. Identifiers MUST comprise only ASCII alphanumerics and hyphens [0-9A-Za-z-]. Identifiers MUST NOT be empty. Build metadata MUST be ignored when determining version precedence. Thus two versions that differ only in the build metadata, have the same precedence. Examples: 1.0.0-alpha+001, 1.0.0+20130313144700, 1.0.0-beta+exp.sha.5114f85, 1.0.0+21AF26D3—-117B344092BD.

[exoego]

Ah it is sem-ver terms.
Yeah, a link to that sounds enough

[alejandrohdezma]

Done! See f0158c6

[exoego]

Though I think AnUpdate and GroupedUpdate is a confusing name since there are already Update.Single and Update.Group, the feature itself looks great to me.
We may consider refactoring later.

I left a comment to improve document.

[exoego]

LGTM

[alejandrohdezma]

@exoego Do you think we can merge this so we can test it thoroughly with a snapshot version?

[exoego]

Yes. Let's do

[rtyley]

• A new mechanism will be created for creating pull-requests with grouped updates. This "grouped" pull requests should contain most features from the current mechanism (list of changes, scalafix migrations, hooks...) but there are some (closing a PR ignores the version) that won't be supported.

Hi, love this feature! Does it only perform creation of grouped pull requests, or is it capable of dealing with updating existing pull requests when additional updates for the group come along? For instance, given this config, grouping com.amazonaws artifacts:

pullRequests.grouping = [
  { name = "aws", "title" = "AWS Scala dependency updates", "filter" = [{"group" = "software.amazon"}, {"group" = "com.amazonaws"}] },
  { name = "non_aws", "title" = "Non AWS Scala dependency updates", "filter" = [{"group" = "*"}] }
]

...should we expect the first PR raised in our repo (PR 4629, for aws-java-sdk-cloudwatch) to get updated with the subsequent 3 new artifacts from the the com.amazonaws group? Ideally, that's what we'd like, but it doesn't seem to be happening - we get ungrouped PRs instead:

If it's of interest, you can see the GitHub Action runs of Scala Steward that are happening here: https://github.com/guardian/scala-steward-public-repos/actions/workflows/public-repos-scala-steward.yml

[alejandrohdezma]

Hey @rtyley. No, the grouping functionality just creates grouped PRs with all the updates that fall into the group. So in your case, it should have created a new PR titled AWS Scala dependency updates, which hasn't happened. Would you be able to run it again with the debug option enabled? (Check Run Scala Steward in Debug Mode in here).

[rtyley]

Thanks @alejandrohdezma - I ran the job again with debug enabled:

https://github.com/guardian/scala-steward-public-repos/actions/runs/3901103941/jobs/6662904571

Hyperlinks to particular points of this large log file don't seem to work in the GitHub UI, but I think you could find the relevant bit by searching for Steward guardian/support-frontend in the logs - I've captured the snippet here: https://gist.github.com/rtyley/339372abc49511ce78cd8b210efa9b0d

Unfortunately I don't think it's printed out the aggregate config, and also at that moment in time I don't think there were any new AWS PRs to open.

[alejandrohdezma]

Hey @rtyley, if it isn't fixed, do you mind opening a new issue and cc-ing me? So we can properly track it