Skip to content

Commit

Permalink
[Backport 2.x] Reconcile DEVELOPER_GUIDE.md (opensearch-project#6047)
Browse files Browse the repository at this point in the history 
Several changes have not been backported. This commit was created with
the following command:

```
git checkout main DEVELOPER_GUIDE.md
```

Signed-off-by: Andrew Ross <andrross@amazon.com>
  • Loading branch information
@andrross
andrross authored Jan 27, 2023
1 parent 8e8d9aa commit d343657
Showing 1 changed file with 120 additions and 30 deletions.
150 changes: 120 additions & 30 deletions DEVELOPER_GUIDE.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,20 +37,26 @@
- [testImplementation](#testimplementation)
- [Gradle Plugins](#gradle-plugins)
- [Distribution Download Plugin](#distribution-download-plugin)
- [Misc](#misc)
- [git-secrets](#git-secrets)
- [Installation](#installation)
- [Configuration](#configuration)
- [Creating fat-JAR of a Module](#creating-fat-jar-of-a-module)
- [Components](#components)
- [Build libraries & interfaces](#build-libraries--interfaces)
- [Clients & Libraries](#clients--libraries)
- [Plugins](#plugins-1)
- [Indexing & search](#indexing--search)
- [Indexing & Search](#indexing--search)
- [Aggregations](#aggregations)
- [Distributed Framework](#distributed-framework)
- [Submitting Changes](#submitting-changes)
- [Backports](#backports)
- [LineLint](#linelint)
- [Misc](#misc)
- [Git Secrets](#git-secrets)
- [Installation](#installation)
- [Configuration](#configuration)
- [Submitting Changes](#submitting-changes)
- [Backwards Compatibility](#backwards-compatibility)
- [Data](#data)
- [Developer API](#developer-api)
- [User API](#user-api)
- [Experimental Development](#experimental-development)
- [Backports](#backports)
- [LineLint](#linelint)
- [Lucene Snapshots](#lucene-snapshots)
- [Flaky Tests](#flaky-tests)

Expand Down Expand Up @@ -377,38 +383,49 @@ The Distribution Download plugin downloads the latest version of OpenSearch by d
./gradlew integTest -PcustomDistributionUrl="https://ci.opensearch.org/ci/dbc/bundle-build/1.2.0/1127/linux/x64/dist/opensearch-1.2.0-linux-x64.tar.gz"
```

### Creating fat-JAR of a Module

## Misc
A fat-JAR (or an uber-JAR) is the JAR, which contains classes from all the libraries, on which your project depends and, of course, the classes of current project.

### git-secrets
There might be cases where a developer would like to add some custom logic to the code of a module (or multiple modules) and generate a fat-JAR that can be directly used by the dependency management tool. For example, in [#3665](https://github.com/opensearch-project/OpenSearch/pull/3665) a developer wanted to provide a tentative patch as a fat-JAR to a consumer for changes made in the high level REST client.

Security is our top priority. Avoid checking in credentials.
Use [Gradle Shadow plugin](https://imperceptiblethoughts.com/shadow/).
Add the following to the `build.gradle` file of the module for which you want to create the fat-JAR, e.g. `client/rest-high-level/build.gradle`:

#### Installation
Install [awslabs/git-secrets](https://github.com/awslabs/git-secrets) by running the following commands.
```
git clone https://github.com/awslabs/git-secrets.git
cd git-secrets
make install
apply plugin: 'com.github.johnrengelman.shadow'
```

#### Configuration
You can configure git secrets per repository, you need to change the directory to the root of the repository and run the following command.
Run the `shadowJar` command using:
```
git secrets --install
✓ Installed commit-msg hook to .git/hooks/commit-msg
✓ Installed pre-commit hook to .git/hooks/pre-commit
✓ Installed prepare-commit-msg hook to .git/hooks/prepare-commit-msg
./gradlew :client:rest-high-level:shadowJar
```
Then, you need to apply patterns for git-secrets, you can install the AWS standard patterns by running the following command.

This will generate a fat-JAR in the `build/distributions` folder of the module, e.g. .`/client/rest-high-level/build/distributions/opensearch-rest-high-level-client-1.4.0-SNAPSHOT.jar`.

You can further customize your fat-JAR by customising the plugin, More information about shadow plugin can be found [here](https://imperceptiblethoughts.com/shadow/).

To use the generated JAR, install the JAR locally, e.g.
```
git secrets --register-aws
mvn install:install-file -Dfile=src/main/resources/opensearch-rest-high-level-client-1.4.0-SNAPSHOT.jar -DgroupId=org.opensearch.client -DartifactId=opensearch-rest-high-level-client -Dversion=1.4.0-SNAPSHOT -Dpackaging=jar -DgeneratePom=true
```

Refer the installed JAR as any other maven artifact, e.g.

```
<dependency>
<groupId>org.opensearch.client</groupId>
<artifactId>opensearch-rest-high-level-client</artifactId>
<version>1.4.0-SNAPSHOT</version>
</dependency>
```

## Components

As you work in the OpenSearch repo you may notice issues getting labeled with component labels. It's a housekeeping task to help group together similar pieces of work. You can pretty much ignore it, but if you're curious, here's what the different labels mean:

### Build libraries & interfaces

Tasks to make sure the build tasks are useful and packaging and distribution are easy.

Includes:
Expand All @@ -422,6 +439,7 @@ Includes:


### Clients & Libraries

APIs and communication mechanisms for external connections to OpenSearch. This includes the “library” directory in OpenSearch (a set of common functions).

Includes:
Expand All @@ -431,6 +449,7 @@ Includes:
- CLI

### Plugins

Anything touching the plugin infrastructure within core OpenSearch.

Includes:
Expand All @@ -440,7 +459,8 @@ Includes:
- Plugin interfaces


### Indexing & search
### Indexing & Search

The critical path of indexing and search, including: Measure index and search, performance, Improving the performance of indexing and search, ensure synchronization OpenSearch APIs with upstream Lucene change (e.g. new field types, changing doc values and codex).

Includes:
Expand All @@ -451,6 +471,7 @@ Includes:
- DocValues

### Aggregations

Making sure OpenSearch can be used as a compute engine.

Includes:
Expand All @@ -459,26 +480,94 @@ Includes:
- Framework

### Distributed Framework

Work to make sure that OpenSearch can scale in a distributed manner.

Includes:

- Nodes (Master, Data, Compute, Ingest, Discovery, etc.)
- Nodes (Cluster Manager, Data, Compute, Ingest, Discovery, etc.)
- Replication & Merge Policies (Document, Segment level)
- Snapshot/Restore (repositories; S3, Azure, GCP, NFS)
- Translog (e.g., OpenSearch, Kafka, Kinesis)
- Shard Strategies
- Circuit Breakers

## Submitting Changes
## Misc

### Git Secrets

Security is our top priority. Avoid checking in credentials.

#### Installation
Install [awslabs/git-secrets](https://github.com/awslabs/git-secrets) by running the following commands.
```
git clone https://github.com/awslabs/git-secrets.git
cd git-secrets
make install
```

#### Configuration
You can configure git secrets per repository, you need to change the directory to the root of the repository and run the following command.
```
git secrets --install
✓ Installed commit-msg hook to .git/hooks/commit-msg
✓ Installed pre-commit hook to .git/hooks/pre-commit
✓ Installed prepare-commit-msg hook to .git/hooks/prepare-commit-msg
```
Then, you need to apply patterns for git-secrets, you can install the AWS standard patterns by running the following command.
```
git secrets --register-aws
```

### Submitting Changes

See [CONTRIBUTING](CONTRIBUTING.md).

## Backports
### Backwards Compatibility

OpenSearch strives for a smooth and easy upgrade experience that is resilient to data loss and corruption while minimizing downtime and ensuring integration with
external systems does not unexpectedly break.

To provide these guarantees each version must be designed and developed with [forward compatibility](https://en.wikipedia.org/wiki/Forward_compatibility) in mind.
OpenSearch addresses backward and forward compatibility at three different levels: 1. Data, 2. Developer API, 3. User API. These levels and the developer mechanisms
to ensure backwards compatibility are provided below.

#### Data
The data level consists of index and application data file formats. OpenSearch guarantees file formats and indexes are compatible only back to the first release of
the previous major version. If on disk formats or encodings need to be changed (including index data, cluster state, or any other persisted data) developers must
use Version checks accordingly (e.g., `Version.onOrAfter`, `Version.before`) to guarantee backwards compatibility.

#### Developer API
The Developer API consists of interfaces and foundation software implementations that enable external users to develop new OpenSearch features. This includes
obvious components such as the Plugin framework and less obvious components such as REST Action Handlers. When developing a new feature of OpenSearch it is important
to explicitly mark which implementation components may, or may not, be extended by external implementations. For example, all new API classes with `@opensearch.api`
signal that the new component may be extended by an external implementation and therefore provide backwards compatibility guarantees. Similarly, any class explicitly
marked with the `@opensearch.internal` annotation, or not explicitly marked by an annotation should not be extended by external implementation components as it does not
guarantee backwards compatibility and may change at any time. The `@deprecated` annotation should also be added to any `@opensearch.api` classes or methods that are
either changed or planned to be removed across minor versions.

#### User API
The User API consists of integration specifications (e.g., [Query Domain Specific Language](https://opensearch.org/docs/latest/opensearch/query-dsl/index/),
[field mappings](https://opensearch.org/docs/latest/opensearch/mappings/)) and endpoints (e.g., [`_search`](https://opensearch.org/docs/latest/api-reference/search/),
[`_cat`](https://opensearch.org/docs/latest/api-reference/cat/index/)) users rely on to integrate and use OpenSearch. Backwards compatibility is critical to the
User API, therefore OpenSearch commits to using [semantic versioning](https://opensearch.org/blog/what-is-semver/) for all User facing APIs. To support this
developers must leverage `Version` checks for any user facing endpoints or API specifications that change across minor versions. Developers must also inform
users of any changes by adding the `>breaking` label on Pull Requests, adding an entry to the [CHANGELOG](https://github.com/opensearch-project/OpenSearch/blob/main/CHANGELOG.md)
and a log message to the OpenSearch deprecation log files using the `DeprecationLogger`.

#### Experimental Development
Rapidly developing new features often benefit from several release cycles before committing to an official and long term supported (LTS) API. To enable this cycle OpenSearch
uses an Experimental Development process leveraging [Feature Flags](https://featureflags.io/feature-flags/). This allows a feature to be developed using the same process as
a LTS feature but with additional guard rails and communication mechanisms to signal to the users and development community the feature is not yet stable, may change in a future
release, or be removed altogether. Any Developer or User APIs implemented along with the experimental feature should be marked with the `@opensearch.experimental` annotation to
signal the implementation is not subject to LTS and does not follow backwards compatibility guidelines.

### Backports

The Github workflow in [`backport.yml`](.github/workflows/backport.yml) creates backport PRs automatically when the original PR with an appropriate label `backport <backport-branch-name>` is merged to main with the backport workflow run successfully on the PR. For example, if a PR on main needs to be backported to `1.x` branch, add a label `backport 1.x` to the PR and make sure the backport workflow runs on the PR along with other checks. Once this PR is merged to main, the workflow will create a backport PR to the `1.x` branch.

## LineLint
### LineLint

A linter in [`code-hygiene.yml`](.github/workflows/code-hygiene.yml) that validates simple newline and whitespace rules in all sorts of files. It can:
- Recursively check a directory tree for files that do not end in a newline
- Automatically fix these files by adding a newline or trimming extra newlines.
Expand All @@ -494,6 +583,7 @@ Pass a list of files or directories to limit your search.
linelint README.md LICENSE

### Lucene Snapshots

The Github workflow in [lucene-snapshots.yml](.github/workflows/lucene-snapshots.yml) is a Github worfklow executable by maintainers to build a top-down snapshot build of lucene.
These snapshots are available to test compatibility with upcoming changes to Lucene by updating the version at [version.properties](buildsrc/version.properties) with the `version-snapshot-sha` version. Example: `lucene = 10.0.0-snapshot-2e941fc`.

Expand All @@ -505,7 +595,7 @@ If you encounter a build/test failure in CI that is unrelated to the change in y

1. Follow failed CI links, and locate the failing test(s).
2. Copy-paste the failure into a comment of your PR.
3. Search through [issues](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aopen+is%3Aissue+label%3A%22flaky-test%22) using the name of the failed test for whether this is a known flaky test.
3. Search through [issues](https://github.com/opensearch-project/OpenSearch/issues?q=is%3Aopen+is%3Aissue+label%3A%22flaky-test%22) using the name of the failed test for whether this is a known flaky test.
5. If an existing issue is found, paste a link to the known issue in a comment to your PR.
6. If no existing issue is found, open one.
7. Retry CI via the GitHub UX or by pushing an update to your PR.

0 comments on commit d343657

Please sign in to comment.