Skip to content

Cache documentation improvements #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 31, 2023
Merged

Cache documentation improvements #40

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 30 additions & 21 deletions src/site/markdown/getting-started.md.vm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,58 +18,67 @@
Getting Started
---------------

To on-board incremental Maven you need to complete several steps:
To onboard the build-cache extension you need to complete several steps:

* Declare caching extension in your project (either in `pom.xml` or `.mvn/extensions.xml`)
* Add `maven-build-cache-config.xml` cache config in `.mvn/` (optional) to customize default behavior
* Validate build results and iteratively, adjust config to properly reflect project specifics
* Add `maven-build-cache-config.xml` cache config in `.mvn/` (optional) to customize the default behavior
* Validate build results and iteratively adjust config to reflect project specifics properly
* Setup remote cache (optional)

#[[###]]# Declaring build cache extension

```xml

<extension>
<groupId>org.apache.maven.extensions</groupId>
<artifactId>maven-build-cache-extension</artifactId>
<version>${project.version}</version>
</extension>
```

either in `pom.xml`'s `<project>/<build>/<extensions>` or in `.mvn/extensions.xml`'s `<extensions>`
either in `pom.xml`'s `<project>/<build>/<extensions>` or in `.mvn/extensions.xml`'s `<extensions>`. Using core
extension model (`.mvn/extensions.xml` file) is preferable as it allows better access to maven APIs and could allow
more sophisticated optimizations in the future.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Copy link
Contributor Author

@AlexanderAshitkin AlexanderAshitkin May 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming that you ask to clarify the statement above, here's the rationale - the core extension initialization model allows binding core API classes, like lifecycle listeners. Besides that, using the cache as a build-level extension tempts the user to initialize the cache in different parts of the project subtree, which was never intended and likely tested. The core extension is the best way to use the cache, which leaves us the door to use all the internal maven APIs future and aligns well with the caching nature. It's just right to make it the recommended option. Honestly, I would deprecate the build extension mode because it doesn't solve any particular problem and just increases accidental complexity.

Let's merge this PR to improve documentation in the next iterations.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good.
BUT fyi this issue (https://issues.apache.org/jira/browse/MINVOKER-336) is a real pain. So for projects having some usage of maven-invoker-plugin apart of other modules it is causing some bugs.
This project by example (https://github.com/eclipse/jetty.project/tree/jetty-12.0.x) cannot use .mvn/extensions.xml this otherwise maven plugins its via m-invoker-p are using the cache from top module which is not expected ;)


#[[###]]# Adding build cache config

Copy template config [`maven-build-cache-config.xml`](../resources/maven-build-cache-config.xml)
to [`.mvn/`](https://maven.apache.org/configure.html) directory of your project.
To get overall understanding of build cache machinery, it is recommended to review the config and read comments. In typical
scenario you need to:
To understand the caching machinery, review the config and read the comments. In a typical scenario, you need to:

* Exclude unstable, temporary files or environment specific files
* Add plugins reconciliation rules – add critical plugins parameters to reconciliation
* Configure precise source code files selectors. Though source code locations discovered automatically from project and plugins config,
there might be edge cases.
* Add remote cache location (if remote cache is used)
* Exclude unstable, temporary files or environment-specific files
* Add critical plugins parameters to runtime reconciliation
* Configure precise source code file selectors. Though source code locations are discovered automatically from project
and
plugin configs, there might be edge cases.
* Configure remote cache (if using the remote cache)

#[[###]]# Adjusting build cache config

Having extension run usual command, like `mvn package`. Verify the caching engine is activated:
After configuring the extension, run a usual command, for example, `mvn package`, and verify the caching engine is
activated:

* Check log output - there should be cache related output or initialization error message:
* Check log output - there should be the cache-related output or initialization error message:
```
[INFO] Loading cache configuration from <project dir>/.mvn/maven-build-cache-config.xml
```
* Navigate to your local repo directory - there should be a sibling directory `build-cache` next to the usual
local `repository`.
* Find `buildinfo.xml` in the cache repository for typical module and review it. Ensure that
* expected source code files are present in the build info
* Review all plugings used in the build and add their critical parameters to reconciliation
* Find `buildinfo.xml` in the cache repository for a typical module and review it. Ensure that
* Expected source code files are present in the build info
* Review all plugins used in the build and add their critical parameters to the reconciliation

It is recommended to find the best working trade-off between fairness and cache efficiency. Adding unnecessary rules and
Try to find the best working trade-off between fairness and cache efficiency. Adding unnecessary rules and
checks could reduce both performance and cache efficiency (hit rate).

#[[###]]# Adding caching CI and remote cache

To leverage remote cache feature there should a shared storage provide. Any technology supported
by [Maven Resolver](https://maven.apache.org/resolver/) will suffice. In simplest form it could be a http web server which
supports get/put operations ([Nginx OSS](http://nginx.org/en/) with fs module or any other equivalent).
See [Remote cache setup](remote-cache.html) for detailed description of cache setup.
To leverage the remote cache feature need to provide shared storage. Any technology supported
by [Maven Resolver](https://maven.apache.org/resolver/) will suffice. In the simplest form, it could be any HTTP web
server
supporting get/put operations. Working examples:

* Artifactory (generic repository)
* [Nginx OSS](http://nginx.org/en/) with fs module

See [Remote cache setup](remote-cache.html) for a detailed description of the remote cache setup
46 changes: 21 additions & 25 deletions src/site/markdown/how-to.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,12 @@

## Overview

Cache configuration provides you additional control over incremental Maven behavior. Follow it step by step to
understand how it works and figure out your optimal config
Cache configuration provides you additional control over the build-cache extension behavior. Follow it step-by-step to
understand how it works, and figure out an optimal config

### Minimal config

Absolutely minimal config
Minimal config

```xml
<?xml version="1.0" encoding="UTF-8" ?>
Expand Down Expand Up @@ -58,7 +58,7 @@ Just add `<remote>` section under `<configuration>`

### Adding more file types to input

Add all the project specific source code files in `<glob>`. Scala in this case:
Add all the project-specific source code files in `<glob>`. Scala, in this case:

```xml
<input>
Expand All @@ -70,8 +70,7 @@ Add all the project specific source code files in `<glob>`. Scala in this case:

### Adding source directory for bespoke project layouts

In most of the cases incremental Maven will recognize directories automatically by build introspection. If not, you can
add additional directories with `<include>`. Also you can filter out undesirable dirs and files by using exclude tag
In most cases, the build-cache extension automatically recognizes directories by introspecting the build. When it is not enough, adding additional directories with `<include>` is possible. Also, you can filter out undesirable dirs and files by using exclude tag.

```xml
<input>
Expand All @@ -87,7 +86,7 @@ add additional directories with `<include>`. Also you can filter out undesirable
</input>
```

### Plugin property is env specific and yields different cache key in different environments
### Plugin property is environment-specific and yields different cache keys in different environments

Consider to exclude env specific properties:

Expand All @@ -108,11 +107,11 @@ Consider to exclude env specific properties:
</input>
```

Implications - builds with different `argLine` will have identical key. Validate that is semantically valid.
Implications - builds with different `argLine` will have an identical key. Validate that it is acceptable in terms of artifact equivalency.

### Plugin property points to directory where only subset of files is relevant
### Plugin property points to a directory where only a subset of files is relevant

If plugin configuration property points to `somedir` it will be scanned with default glob. You can tweak it with custom
If the plugin configuration property points to `somedir`, it will be scanned with the default glob. You can tweak it with custom
processing rule

```xml
Expand All @@ -133,9 +132,9 @@ processing rule
</input>
```

### Local repository is not updated because `install` is cached
### Local repository is not updated because the `install` phase is cached

Add `executionControl/runAlways` section
Add `executionControl/runAlways` section:

```xml
<executionControl>
Expand All @@ -161,10 +160,10 @@ Add `executionControl/runAlways` section
</executionControl>
```

### I occasionally cached build with `-DskipTests=true` and tests do not run now
### I occasionally cached build with `-DskipTests=true`, and tests do not run now

If you add command line flags to your build, they do not participate in effective pom - Maven defers final value
resolution to plugin runtime. To invalidate build if filed value is different in runtime, add reconciliation section
If you add command line flags to your build, they do not participate in effective pom - Maven defers the final value
resolution to plugin runtime. To invalidate the build if the filed value is different in runtime, add a reconciliation section
to `executionControl`:

```xml
Expand Down Expand Up @@ -194,27 +193,24 @@ to `executionControl`:
</cache>
```

Please notice `skipValue` attribute. It denotes value which forces skipped execution.
Read `propertyName="skipTests" skipValue="true"` as if property skipTests has value true, plugin will skip execution If
you declare such value incremental Maven will reuse appropriate full-build though technically they are different, but
because full-build is better it is safe to reuse
Please notice the `skipValue` attribute. It captures the value that makes the plugin skip execution. Think of `skipProperty` as follows: if the build started with `-DskipTests=true`, restoring results from a build with completed tests is safe because the local run does not require completed tests. The same logic applies to any other plugin, not just Surefire.

### How to renormalize line endings in working copy after committing .gitattributes (git 2.16+)

Ensure you've committed (and ideally pushed everything) - no changes in working copy. After that:
Ensure you've committed (and ideally pushed everything) - no changes in the working copy. After that:

```shell
# Rewrite objects and update index
# Rewrite objects and update the index
git add --renormalize .
# Commit changes
git commit -m "Normalizing line endings"
# Remove working copy paths from git cache
git commit -m "Normalizing line endings."
# Remove working copy paths from the git cache
git rm --cached -r .
# Refresh with new line endings
git reset --hard
```

### I want to cache interim build and override it later with final version
### I want to cache the interim build and override it later with the final version

Solution: set `-Dmaven.build.cache.remote.save.final=true` to nodes which produce final builds. Such builds will not be overridden
Solution: set `-Dmaven.build.cache.remote.save.final=true` to nodes that produce final builds. Such builds will not be overridden
and eventually will replace all interim builds
Loading