Skip to content

Commit 15b94cc

Browse files
ndimidukndimiduk
committed
HBASE-24081 Provide documentation for running Yetus with HBase
Signed-off-by: stack <stack@apache.org>
1 parent 43d7c2c commit 15b94cc

File tree

1 file changed

+166
-0
lines changed

1 file changed

+166
-0
lines changed

‎dev-support/HOW_TO_YETUS_LOCAL.md‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
# How To Use Apache Yetus with Apache HBase
2+
3+
This document describes how to get up and running with [Apache Yetus][yetus],
4+
as pertains to the development of Apache HBase. Specifically this covers the use
5+
of `test-patch`, of which HBase developers make use for tasks related to code
6+
quality. These are normally run via automation in the foundation's
7+
[Build][builds] infrastructure. They can also be run locally, which is the
8+
subject of this document.
9+
10+
The Yetus project provides its own documentation of `test-patch` in
11+
[Basic PreCommit][yetus-basic-precommit]. By comparison, this document is
12+
intended to be highly abbreviated, hands-on, and focused on the HBase use-case.
13+
See that document for more complete explanations and further details.
14+
15+
## Installation
16+
17+
In order to run Yetus, you'll need to first install Yetus and its dependencies.
18+
This is somewhat simplified when used in Docker mode. Yetus can be retrieved
19+
from a [distribution artifact][yetus-downloads]. Homebrew/Linuxbrew users can
20+
install from the tap, the process for which is also described on the downloads
21+
page.
22+
23+
# Usage Basics
24+
25+
Apache Yetus is comprised of a number of different components. The focus of our
26+
interest is `test-patch`. `test-patch` is a modular system. Many modules depend
27+
on some external tool to provide the underlying functionality. For example, the
28+
`compile` check delegates to a number of provider modules, for example `maven`
29+
or `gradle` for JVM projects. In order to use these modules, those tools must
30+
be installed. Yetus calls these modules "plugins".
31+
32+
To see a list of all plugins available to `test-patch`, use
33+
34+
```shell script
35+
$ test-patch --list-plugins
36+
```
37+
38+
To see a list of all the options available in both the core modules as well as
39+
all the plugins, use
40+
41+
```shell script
42+
$ test-patch --plugins=all --help
43+
```
44+
45+
An invocation of `test-patch` requires use of one or more plugins. Often times,
46+
when the full suite of checks are run, specify the meta-plugin "all". Otherwise,
47+
a limited selection of plugins can be selected using the `--plugins` argument.
48+
49+
## The Workspace, The Patch
50+
51+
`test-patch` operates within a "workspace," a checkout from a source control
52+
repository. It has a number of options pertaining to this workspace, such as
53+
the path to the workspace (`--basedir`) and whether it will permit the presence
54+
of uncommitted changes therein (`--dirty-workspace`).
55+
56+
Onto this workspace, it can optionally apply a change, a.k.a., the "patch" in
57+
"test patch." The patch can come from a number of sources, including a patch
58+
file, a JIRA ID, a Pull Request, &c. Or, explicitly inform Yetus that no patch
59+
file is provided, and the repository should be checked as is, by passing
60+
`--empty-patch`.
61+
62+
## Personalities
63+
64+
`test-patch` is extremely extensible. Even the functionality of its core
65+
modules can be extended or overridden. It allows for this type of
66+
"personalization" by way of "personality" overrides. Yetus ships a number of
67+
these personalities; a pre-packaged personality can be selected via the
68+
`--project` parameter. There is a provided HBase personality in Yetus, however
69+
the HBase project maintains its own within the HBase source repository. Specify
70+
the path to the personality file using `--personality`. The HBase repository
71+
places this file under `dev-support/hbase-personality.sh`.
72+
73+
## Docker mode
74+
75+
Running Yetus in Docker mode simplifies the concerns of dependencies because
76+
the provided `Dockerfile` handles their installation automatically. However,
77+
for MacOS users, there are a number of known issues with running Docker on OSX,
78+
so it may be preferable to instead run outside of the container.
79+
80+
To run in Docker, of course you must install `docker` or some container runtime
81+
equivalent. [Docker Desktop][docker-desktop] is an option for Mac and Windows
82+
users.
83+
84+
When running `test-patch` with HBase in docker mode, two parameters must be
85+
provided. `--docker` activates the `docker` module, enabling the feature.
86+
Additionally, `--dockerfile` points Yetus at HBase's provided `Dockerfile`,
87+
instead of using the one provided out-of-the-box by Yetus. The HBase repository
88+
places this file under `dev-support/docker/Dockerfile`.
89+
90+
## JVM and JAVA_HOME
91+
92+
HBase supports compilation and test execution on a number of different JVM
93+
versions. To facilitate this, the `Dockerfile` installs multiple JVMs. An
94+
invocation of `test-patch` requires additional parameters in order to specify
95+
the target JVM. How you do this is slightly different based on whether you run
96+
with or without Docker mode.
97+
98+
### Setting `JAVA_HOME` Outside of Docker Mode
99+
100+
Simply specify `JAVA_HOME` in the environment in which `test-patch` is launched.
101+
102+
### Setting `JAVA_HOME` Inside of Docker Mode
103+
104+
The docker image JDKs are available under `/usr/lib/jvm`. Symlinks are provided
105+
for each supported major version, i.e., `java-11`. Use the `test-patch` argument
106+
`--java-home`.
107+
108+
# Example: Run SpotBugs
109+
110+
One of the checks supported by Yetus is the static analysis tool
111+
[SpotBugs][spotbugs]. Let's put together all the above and run SpotBugs with
112+
`test-patch`.
113+
114+
The SpotBugs check depends on a SpotBugs installation. This is provided by the
115+
docker file. For running without docker, you'll need to download and unpack the
116+
SpotBugs binary distribution.
117+
118+
Running the SpotBugs check also depends on [Apache Maven][maven], the build
119+
system used by HBase. In order for the check to function, both the `maven`
120+
and `spotbugs` plugins must be specified. If this dependency is omitted, Yetus
121+
will error with a message similar to
122+
123+
```
124+
ERROR: you can't specify maven as the buildtool if you don't enable the plugin.
125+
```
126+
127+
To run just the SpotBugs check, we must explicitly specify `spotbugs` and its
128+
dependency using `--plugins=maven,spotbugs`.
129+
130+
## Without Docker
131+
132+
Putting it all together, without using docker:
133+
134+
```shell script
135+
$ cd /path/to/hbase
136+
$ JAVA_HOME=/path/to/jdk-8 test-patch.sh \
137+
--plugins=maven,spotbugs \
138+
--spotbugs-home=/path/to/spotbugs/
139+
--dirty-workspace \
140+
--empty-patch \
141+
--personality=./dev-support/hbase-personality.sh
142+
```
143+
144+
## With Docker
145+
146+
Putting it all together, using docker:
147+
148+
```shell script
149+
$ cd /path/to/hbase
150+
$ test-patch.sh \
151+
--plugins=maven,spotbugs \
152+
--dirty-workspace \
153+
--empty-patch \
154+
--personality=./dev-support/hbase-personality.sh \
155+
--docker \
156+
--dockerfile=./dev-support/docker/Dockerfile \
157+
--java-home=/usr/lib/jvm/java-8
158+
```
159+
160+
[builds]: https://builds.apache.org
161+
[docker-desktop]: https://www.docker.com/products/docker-desktop
162+
[maven]: https://maven.apache.org
163+
[spotbugs]: https://spotbugs.github.io
164+
[yetus]: https://yetus.apache.org
165+
[yetus-basic-precommit]: https://yetus.apache.org/documentation/0.11.1/precommit-basic/
166+
[yetus-downloads]: https://yetus.apache.org/downloads/

0 commit comments

Comments
 (0)