diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md deleted file mode 100644 index 0ee83ce580..0000000000 --- a/DEVELOPER_GUIDE.md +++ /dev/null @@ -1,309 +0,0 @@ -- [Developer Guide](#developer-guide) - - [Forking and Cloning](#forking-and-cloning) - - [Build Tools](#build-tools) - - [Install Prerequisites](#install-prerequisites) - - [Pyenv](#pyenv) - - [Python 3.9](#python-39) - - [Pipenv](#pipenv) - - [NVM and Node](#nvm-and-node) - - [Yarn](#yarn) - - [Install Dependencies](#install-dependencies) - - [Run Tests](#run-tests) - - [Build OpenSearch](#build-opensearch) - - [Code Linting](#code-linting) - - [Type Checking](#type-checking) - - [Code Coverage](#code-coverage) - - [Pre-Commit Cheatsheet](#pre-commit-cheatsheet) - - [Jenkins Pipelines and Shared Libraries](#jenkins-pipelines-and-shared-libraries) - - [Install Prerequisites](#install-prerequisites-1) - - [Java](#java) - - [Run Tests](#run-tests-1) - - [Regression Tests](#regression-tests) - - [Testing in Jenkins](#testing-in-jenkins) - - [Integ Tests in Jenkins](#integ-tests-in-jenkins) - -# Developer Guide - -## Forking and Cloning - -Fork this repository on GitHub, and clone locally with `git clone`. - -## Build Tools - -This project contains a collection of tools to build, test and release OpenSearch and OpenSearch Dashboards. - -### Install Prerequisites - -#### Pyenv - -Use pyenv to manage multiple versions of Python. This can be installed with [pyenv-installer](https://github.com/pyenv/pyenv-installer) on Linux and MacOS, and [pyenv-win](https://github.com/pyenv-win/pyenv-win#installation) on Windows. - -``` -curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash -``` - -#### Python 3.9 - -Python projects in this repository use Python 3.9. See the [Python Beginners Guide](https://wiki.python.org/moin/BeginnersGuide) if you have never worked with the language. - -``` -$ python3 --version -Python 3.9.17 -``` - -If you are using pyenv. - -``` -pyenv install 3.9.17 # use 3.9.17 on Windows, the latest at the time of writing this -pyenv global 3.9.17 -``` - -#### Pipenv - -This project uses [pipenv](https://pipenv.pypa.io/en/latest/), which is typically installed with `pip install --user pipenv`. Pipenv automatically creates and manages a virtualenv for your projects, as well as adds/removes packages from your `Pipfile` as you install/uninstall packages. It also generates the ever-important `Pipfile.lock`, which is used to produce deterministic builds. - -``` -$ pip install pipenv - -$ pipenv --version -pipenv, version 2023.6.12 -``` - -On Windows, run `pyenv rehash` if `pipenv` cannot be found. This rehashes pyenv shims, creating a `pipenv` file in `/.pyenv/pyenv-win/shims/`. - -#### NVM and Node - -Install [nvm](https://github.com/nvm-sh/nvm/blob/master/README.md) to use the Node version defined in [OpenSearch-Dashboards](https://github.com/opensearch-project/OpenSearch-Dashboards) repository `.nvmrc` file as it is required. - -Use the official [nvm guide](https://github.com/nvm-sh/nvm#installing-and-updating) to install nvm and corresponding Node version. - -#### Yarn - -[Yarn](https://classic.yarnpkg.com/en/docs/install) is required for building and running the OpenSearch Dashboards and plugins - -``` -npm install -g yarn -``` - -### Install Dependencies - -Install dependencies. - -``` -$ pipenv install -Installing dependencies from Pipfile.lock (1f4869)... - 🐍 β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰ 0/0 β€” 00:00:00 -To activate this project's virtualenv, run pipenv shell. -Alternatively, run a command inside the virtualenv with pipenv run. -``` - -### Run Tests - -This project uses [pytest](https://docs.pytest.org/en/6.x/) to ensure Python code quality. See [tests](tests). - -``` -$ pipenv run pytest -2 passed in 02s -``` - -``` -$ ./gradlew test - -> Task :test -BUILD SUCCESSFUL in 7s -3 actionable tasks: 1 executed, 2 up-to-date -``` - -### Build OpenSearch - -Try running `./build.sh`. It should complete and show usage. - -``` -$ ./build.sh -Installing dependencies in . ... -Installing dependencies from Pipfile.lock (41aca1)… - 🐍 β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰ 14/14 β€” 00:00:01 -To activate this project's virtualenv, run the following: - $ pipenv shell -Running ./src/run_build.py ... -usage: build.sh [-h] [-s] [-c COMPONENT] [--keep] manifest -build.sh: error: the following arguments are required: manifest -``` - -### Code Linting - -This project uses a [pre-commit hook](https://pre-commit.com/) for linting Python code and YAML files. - -``` -$ pipenv run pre-commit install -$ pipenv run pre-commit run --all-files -``` -Pre-commit hook will run isort, flake8, mypy and pytest before making a commit. - -This project uses [isort](https://github.com/PyCQA/isort) to ensure that imports are sorted, and [flake8](https://flake8.pycqa.org/en/latest/) to enforce code style. - -``` -$ pipenv run flake8 -./src/assemble_workflow/bundle_recorder.py:30:13: W503 line break before binary operator -``` - -Use `isort .` to fix any sorting order. - -``` -$ pipenv run isort . -Fixing tests/system/test_arch.py -``` - -Use [black](https://black.readthedocs.io/en/stable/) to auto-format your code. - -``` -$ pipenv run black . -All done! ✨ 🍰 ✨ -23 files left unchanged. -``` - -This project uses [yamllint](https://yamllint.readthedocs.io/) to enforce formatting in YAML files. - -Use [yamlfix](https://github.com/lyz-code/yamlfix) to auto-format your YAML files. - -``` -$ git status -s | grep -e "[MA?]\s.*.y[a]*ml" | xargs pipenv run yamlfix -``` - -If your code isn't properly formatted, don't worry, [a CI workflow](./github/workflows/tests.yml) will make sure to remind you. - -### Type Checking - -This project uses [mypy](https://github.com/python/mypy) as an optional static type checker. - -``` -pipenv run mypy . -src/assemble.py:14: error: Cannot find implementation or library stub for module named "assemble_workflow.bundle" -``` - -### Code Coverage - -This project uses [codecov](https://about.codecov.io/) for code coverage. Use `pipenv run codecov` to run codecov locally. - -``` -$ pipenv run coverage run -m pytest -47 passed in 5.89s - -$ pipenv run coverage report -TOTAL 23491 12295 48% -``` - -### Pre-Commit Cheatsheet - -The pre-commit hook checks for imports, type, style and test. - -``` -pipenv run pre-commit run --all-files -``` - -Auto-fix format and sort imports by running. - -``` -git status -s | grep -e "[MA?]\s.*.py" | cut -c4- | xargs pipenv run black -pipenv run isort . -``` - -## Jenkins Pipelines and Shared Libraries - -This project contains [Jenkins pipelines](jenkins) and [Jenkins shared libraries](src/jenkins) that execute the tools that build OpenSearch and OpenSearch Dashboards. - -### Install Prerequisites - -#### Java - -Use Java 11 for Jenkins jobs CI. This means you must have a JDK 11 installed with the environment variable `JAVA_HOME` referencing the path to Java home for your JDK installation, e.g. `JAVA_HOME=/usr/lib/jvm/jdk-11`. Download Java 11 from [here](https://adoptium.net/releases.html?variant=openjdk11). - -### Run Tests - -This project uses [JenkinsPipelineUnit](https://github.com/jenkinsci/JenkinsPipelineUnit) to unit test Jenkins pipelines and shared libraries. See [tests/jenkins](tests/jenkins). - -``` -$ ./gradlew test - -> Task :test -BUILD SUCCESSFUL in 7s -3 actionable tasks: 1 executed, 2 up-to-date -``` - -#### Regression Tests - -Jenkins workflow regression tests typically output a .txt file into [tests/jenkins/jobs](tests/jenkins/jobs). -For example, [TestHello.groovy](tests/jenkins/TestHello.groovy) executes [Hello_Jenkinsfile](tests/jenkins/jobs/Hello_Jenkinsfile) -and outputs [Hello_Jenkinsfile.txt](tests/jenkins/jobs/Hello_Jenkinsfile.txt). If the job execution changes, the regression test will fail. - -- To update the recorded .txt file run `./gradlew test -info -Ppipeline.stack.write=true` or update its value in [gradle.properties](gradle.properties). - -- To run a specific test case, run `./gradlew test -info --tests=TestCaseClassName` - -#### Tests for jenkins job -Each jenkins job should have a test case associated with it. -Eg: [TestSignStandaloneArtifactsJob.groovy](tests/jenkins/TestSignStandaloneArtifactsJob.groovy) -- Save the regression file for the `jenkins-job` in `tests/jenkins/jenkinsjob-regression-files//` -- All tests for jenkins job should extend [BuildPipelineTest.groovy](tests/jenkins/BuildPipelineTest.groovy) -- All tests should have a `setUp()` which is used to set the variables associated with the job -- Add setups for all libraries used in the job using `this.registerLibTester` with appropriate values -(Eg: [TestDataPrepperReleaseArtifacts](tests/jenkins/TestDataPrepperReleaseArtifacts.groovy)) in `setUp()` before `super.setUp()` is called. - -#### Tests for jenkins libraries - -##### Lib Tester -Each jenkins library should have a lib tester associated with it. Eg: [SignArtifactsLibTester](tests/jenkins/lib-testers/SignArtifactsLibTester.groovy) -- Library tester should extend [LibFunctionTester.groovy](tests/jenkins/LibFunctionTester.groovy) -- implement `void configure(helper, bindings)` method which sets up all the variables used in the library - - Note: This will not include the variables set using function arguments -- implement `void libFunctionName()`. This function will contain the name of function. -- implement `void parameterInvariantsAssertions()`. This function will contain assertions verifying the type and -accepted values for the function parameters -- implement `void expectedParametersMatcher()`. This function will match args called in the job to expected values from -the test - -##### Library Test Case -Each jenkins library should have a test case associated with it. Eg: [TestSignArtifacts](tests/jenkins/TestSignArtifacts.groovy)
-- Jenkins' library test should extend [BuildPipelineTest.groovy](tests/jenkins/BuildPipelineTest.groovy) -- Create a dummy job such as [Hello_Jenkinsfile](tests/jenkins/jobs/Hello_Jenkinsfile) to call and test the function - and output [Hello_Jenkinsfile.txt](tests/jenkins/jobs/Hello_Jenkinsfile.txt) -- If using remote libs from [opensearch-build-libraries](https://github.com/opensearch-project/opensearch-build-libraries) repository with tag (ex: 1.0.0), make sure - both the Jenkins Test file as well as the Jenkins Job file are overriding the libs version with the same tag (ex: 1.0.0), or Jacoco test will fail to generate reports. - This would happen if defaultVersion in BuildPipelineTest.groovy (default to 'main') have a different HEAD commit id compares to tag commit id you defined to use. -``` -super.setUp() -...... -helper.registerSharedLibrary( - library().name('jenkins') - .defaultVersion('1.0.0') - .allowOverride(true) - .implicit(true) - .targetPath('vars') - .retriever(gitSource('https://github.com/opensearch-project/opensearch-build-libraries.git')) - .build() -) -``` - -``` -lib = library(identifier: 'jenkins@1.0.4', retriever: modernSCM([ - $class: 'GitSCMSource', - remote: 'https://github.com/opensearch-project/opensearch-build-libraries.git', -])) -``` - -#### Testing in Jenkins -* [Build_OpenSearch_Dashboards_Jenkinsfile](tests/jenkins/jobs/Build_OpenSearch_Dashboards_Jenkinsfile): is similar to [OpenSearch Dashboards Jenkinsfile](jenkins/opensearch-dashboards/Jenkinsfile) w/o notifications. - -Make your code changes in a branch, e.g. `jenkins-changes`, including to any of the above jobs. Create a pipeline in Jenkins with the following settings. - -* GitHub Project: `https://github.com/[your username]/opensearch-build/`. -* Pipeline repository URL: `https://github.com/[your username]/opensearch-build`. -* Branch specifier: `refs/heads/jenkins-changes`. -* Script path: `tests/jenkins/jobs/Build_DryRun_Jenkinsfile` - -You can now iterate by running the job in Jenkins, examining outputs, and pushing updates to GitHub. - -#### Integ Tests in Jenkins -- Opensearch bundle build executes the integration tests for the opensearch as well as plugins. -- To add integ tests for a new plugin, add the plugin in the latest version [test manifest](manifests/2.0.0/opensearch-2.0.0-test.yml). -- The test execution is triggered by the [integtest.sh](scripts/default/integtest.sh). In case a custom implementation is required, plugin owner can add that script in their own repo and [script_finder](src/paths/script_finder.py) will pick that up over the default script. diff --git a/README.md b/README.md deleted file mode 100644 index eb39d1a86a..0000000000 --- a/README.md +++ /dev/null @@ -1,310 +0,0 @@ -OpenSearch Build - -[![python](https://github.com/opensearch-project/opensearch-build/actions/workflows/python-tests.yml/badge.svg)](https://github.com/opensearch-project/opensearch-build/actions/workflows/python-tests.yml) -[![groovy](https://github.com/opensearch-project/opensearch-build/actions/workflows/groovy-tests.yml/badge.svg)](https://github.com/opensearch-project/opensearch-build/actions/workflows/groovy-tests.yml) -[![manifests](https://github.com/opensearch-project/opensearch-build/actions/workflows/manifests.yml/badge.svg)](https://github.com/opensearch-project/opensearch-build/actions/workflows/manifests.yml) -[![codecov](https://codecov.io/gh/opensearch-project/opensearch-build/branch/main/graph/badge.svg?token=03S5XZ80UI)](https://codecov.io/gh/opensearch-project/opensearch-build) - -- [Releasing OpenSearch](#releasing-opensearch) - - [Releases and Versions](#releases-and-versions) - - [Release labels](#release-labels) -- [Onboarding a New Plugin](#onboarding-a-new-plugin) -- [Building and Testing an OpenSearch Distribution](#building-and-testing-an-opensearch-distribution) - - [Building from Source](#building-from-source) - - [Assembling a Distribution](#assembling-a-distribution) - - [Building Patches](#building-patches) - - [Min snapshots](#min-snapshots) - - [CI/CD Environment](#cicd-environment) - - [Build Numbers](#build-numbers) - - [Latest Distribution URL](#latest-distribution-url) - - [Testing the Distribution](#testing-the-distribution) - - [Checking Release Notes](#checking-release-notes) - - [Signing Artifacts](#signing-artifacts) - - [PGP](#pgp) - - [Windows](#windows) - - [Signing RPM artifacts](#signing-rpm-artifacts) -- [Making a Release](#making-a-release) - - [Releasing for Linux](#releasing-for-linux) - - [Releasing for FreeBSD](#releasing-for-freebsd) - - [Releasing for Windows](#releasing-for-windows) - - [Releasing for macOS](#releasing-for-macos) -- [Utilities](#utilities) - - [Checking Out Source](#checking-out-source) - - [Cross-Platform Builds](#cross-platform-builds) - - [Sanity Checking the Bundle](#sanity-checking-the-bundle) - - [Auto-Generating Manifests](#auto-generating-manifests) -- [Deploying Infrastructure](#deploying-infrastructure) -- [Contributing](#contributing) -- [Getting Help](#getting-help) -- [Code of Conduct](#code-of-conduct) -- [Security](#security) -- [License](#license) -- [Copyright](#copyright) - -### Releasing OpenSearch - -Please refer to the [release process document](./RELEASE_PROCESS_OPENSEARCH.md) for detailed information on how to release the OpenSearch and OpenSearch Dashboards software. - - -#### Releases and Versions - -The OpenSearch project releases as versioned distributions of OpenSearch, OpenSearch Dashboards, and the OpenSearch plugins. It [follows semantic versioning](https://opensearch.org/blog/technical-post/2021/08/what-is-semver/). Software, such as Data Prepper, clients, and the Logstash output plugin, are versioned independently of the OpenSearch Project. They also may have independent releases from the main project distributions. The OpenSearch Project may also release software under alpha, beta, release candidate, and generally available labels. The definition of when to use these labels is derived from [the Wikipedia page on Software release lifecycle](https://en.wikipedia.org/wiki/Software_release_life_cycle). Below is the definition of when to use each label. - -#### Release labels: - -* **Alpha** - The code is released with instructions to build. Built distributions of the software may not be available. Some features many not be complete. Additional testing and development work is planned. Distributions will be postfixed with `-alphaX` where "X" is the number of the alpha version (e.g., "2.0-alpha1"). -* **Beta** - Built distributions of the software are available. All features are completed. Additional testing and development work is planned. Distributions will be postfixed with `-betaX` where "X" is the number of the beta version (e.g., "2.0.0-beta1"). -* **Release Candidate** - Built distributions of the software are available. All features are completed. Code is tested and minimal validation remains. At this stage the software is potentially stable and will release unless significant bugs emerge. Distributions will be postfixed with `-rcX` where "X" is the number of the release candidate version (e.g., "2.0.0-rc1"). -* **Generally Available** - Built distributions of the software are available. All features are completed and documented. All testing is completed. Distributions for generally available versions are not postfixed with an additional label (e.g., "2.0.0"). - - -### Onboarding a New Plugin - -Plugin owners can follow the [Onboarding Process](ONBOARDING.md) to onboard their plugins to the release process. - -### Building and Testing an OpenSearch Distribution - -The distribution workflow builds a complete OpenSearch and OpenSearch Dashboards distribution from source. You can currently build 1.0, 1.1, 1.1-SNAPSHOT and 1.2 versions. This system performs a top-down [build](src/build_workflow) of all components required for a specific OpenSearch and OpenSearch Dashboards release, then [assembles](src/assemble_workflow/) a distribution. The input to the system is a manifest that defines the order in which components should be built. All manifests for our current releases are [here](manifests). - -#### Building from Source - -```bash -./build.sh manifests/1.3.0/opensearch-1.3.0.yml -``` - -This builds OpenSearch 1.3.0 from source, placing the output into `./builds/opensearch`. - -See [build workflow](src/build_workflow) for more information. - -#### Assembling a Distribution - -```bash -./assemble.sh builds/opensearch/manifest.yml -``` - -The assembling step takes output from the build step, installs plugins, and assembles a full distribution into the `dist` folder. - -See [assemble workflow](src/assemble_workflow) for more information. - -#### Building Patches - -A patch release contains output from previous versions mixed with new source code. Manifests can mix such references. See [opensearch-1.3.1.yml](https://github.com/opensearch-project/opensearch-build/blob/opensearch-1.3.1/manifests/1.3.1/opensearch-1.3.1.yml) for an example. - -OpenSearch is often released with changes in `opensearch-min`, and no changes to plugins other than a version bump. This can be performed by a solo engineer following [a cookbook](https://github.com/opensearch-project/opensearch-plugins/blob/main/META.md#increment-a-version-in-every-plugin). See also [opensearch-build#1375](https://github.com/opensearch-project/opensearch-build/issues/1375) which aims to automate incrementing versions for the next development iteration. - -#### Min Snapshots - -Snapshots for OpenSearch core/min can be downloaded and used in CI's, local development, etc using below links: - -Linux: -``` -https://artifacts.opensearch.org/snapshots/core/opensearch/-SNAPSHOT/opensearch-min--SNAPSHOT-linux-x64-latest.tar.gz -``` -macOS: -``` -https://artifacts.opensearch.org/snapshots/core/opensearch/-SNAPSHOT/opensearch-min--SNAPSHOT-darwin-x64-latest.tar.gz -``` - -Windows: -``` -https://artifacts.opensearch.org/snapshots/core/opensearch/-SNAPSHOT/opensearch-min--SNAPSHOT-windows-x64-latest.zip -``` - -#### CI/CD Environment - -We build, assemble, and test our artifacts on docker containers. We provide docker files in [docker/ci](docker/ci) folder, and images on [staging docker hub repositories](https://hub.docker.com/r/opensearchstaging/ci-runner/). All Jenkins pipelines can be found in [jenkins](./jenkins/). Jenkins itself is in the process of being made public and its CDK open-sourced. - -See [jenkins](./jenkins) and [docker](./docker) for more information. - -#### Build Numbers - -The distribution URL and the build output manifest include a Jenkins auto-incremented build number. For example, the [manifest](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/5905/linux/x64/rpm/dist/opensearch/manifest.yml) from [OpenSearch build 5905](https://build.ci.opensearch.org/job/distribution-build-opensearch/5905/) contains the following. - -```yml -build: - name: OpenSearch - version: 2.2.0 - platform: linux - architecture: x64 - distribution: rpm - id: '5905' -``` - -#### Latest Distribution URL - -Use the `latest` keyword in the URL to obtain the latest build for a given version. For example `https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/latest/linux/x64/rpm/dist/opensearch/manifest.yml` redirects to [build 5905](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/5905/linux/x64/rpm/dist/opensearch/manifest.yml) at the time of writing this. - -The `latest` keyword is resolved to a specific build number by checking the `index.json` [file](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/index.json). This file has contents such as this. - -``` -{"latest":"5905"} -``` - -The file is updated when a distribution build job is completed for the given product and version (or is created when such distribution job succeeds for the first time). Since one distribution build job consists of multiple stages for different combinations of distribution type, platform and architecture, the `index.json` is only modified once all stages succeed. With this said, the `latest` URL only works when the distribution build job succeeds at least once for the given product and version. - -The resolution logic is implemented in the [CloudFront URL rewriter](https://github.com/opensearch-project/opensearch-ci/tree/main/resources/cf-url-rewriter). -The TTL (time to live) is set to `5 mins` which means that the `latest` URL may need up to 5 mins to get new contents after `index.json` is updated. - -All the artifacts accessible through the regular distribution URL can be accessed by the `latest` URL. This includes both OpenSearch Core, OpenSearch Dashboards Core and their plugins. - -For example, you can download the latest .tar.gz distribution build of OpenSearch 2.2.0 directly at `https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/latest/linux/x64/tar/dist/opensearch/opensearch-2.2.0-linux-x64.tar.gz`, without having to first download and parse the [complete build manifest](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/latest/linux/x64/tar/dist/opensearch/manifest.yml). - -For plugin artifacts, you can also use the `latest` keyword to get the latest plugin artifacts with a known version. E.g. in order to get performance-analyzer x64 tarball artifacts for 2.1.0, you can obtain it with link `https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.1.0/latest/linux/x64/tar/builds/opensearch/plugins/opensearch-performance-analyzer-2.1.0.0.zip`, which will redirect you to `https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.1.0/5757/linux/x64/tar/builds/opensearch/plugins/opensearch-performance-analyzer-2.1.0.0.zip`. - -For bundled artifacts, here are some examples for LINUX and Windows: -* Linux Tar: `https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.4.0/latest/linux/x64/tar/dist/opensearch/opensearch-2.4.0-linux-x64.tar.gz` -* Windows Zip: `https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.4.0/latest/windows/x64/zip/dist/opensearch/opensearch-2.4.0-windows-x64.zip` - -#### Testing the Distribution - -Tests the OpenSearch distribution, including integration, backwards-compatibility and performance tests. - -```bash -./test.sh -``` - -See [src/test_workflow](./src/test_workflow) for more information. - -#### Checking Release Notes - -Workflow to check if the release notes exist or not and shows the latest commit for OpenSearch and Dashboard distributions. - -To run: -```bash -./release_notes.sh check manifests/2.2.0/opensearch-2.2.0.yml --date 2022-07-26 -``` - -See [src/release_notes_workflow](./src/release_notes_workflow) for more information. -#### Signing Artifacts - -For all types of signing within OpenSearch project we use `opensearch-signer-client` (in progress of being open-sourced) which is a wrapper around internal signing system and is only available for authenticated users. The input requires a path to the build manifest or directory containing all the artifacts or a single artifact. - -Usage: - -```bash -./sign.sh builds/opensearch/manifest.yml -``` - -The tool currently supports following platforms for signing. - -##### PGP - -Anything can be signed using PGP signing eg: tarball, any type of file, etc. A `.sig` file will be returned containing the signature. OpenSearch and OpenSearch dashboards distributions, components such as data prepper, etc. as well as maven artifacts are signed using PGP signing. See [this page](https://opensearch.org/verify-signatures.html) for how to verify signatures. - - -##### Windows - -Windows signing can be used to sign windows executables such as `.msi, .msp, .msm, .cab, .dll, .exe, .appx, .appxbundle, .msix, .msixbundle, .sys, .vxd, .ps1, .psm1`, and any PE file that is supported by [Signtool.exe](https://docs.microsoft.com/en-us/dotnet/framework/tools/signtool-exe). Various windows artifacts such as SQL OBDC, opensearch-cli, etc are signed using this method. -Windows code signing uses EV (Extended Validated) code signing certificates. - -| Types of signing/Details | Digest | Cipher | Key Size| -| ------------- |:-------------| :-----| :-----| -| PGP | SHA1 | AES-128 | 2048 | -| Windows | SHA256 | RSA | | -| RPM | SHA512 | RSA | 4096 | - - -#### Signing RPM artifacts - -RPM artifacts are signed via a legacy shell script which uses a [macros template](scripts/pkg/sign_templates/rpmmacros). See [this commit](https://github.com/opensearch-project/opensearch-build/commit/950d55c1ed3f82e98120541fa40ff506338c1059) for more information and [this issue](https://github.com/opensearch-project/opensearch-build/issues/1547) to add RPM artifact signing functionality to the above signing system. Currently we are only signing OpenSearch and OpenSearch dashboards RPM distributions using this method. - -See [src/sign_workflow](./src/sign_workflow) for more information. - - -### Making a Release - -#### Releasing for Linux and Windows - -The Linux / Windows release is managed by a team at Amazon following [this release template](.github/ISSUE_TEMPLATE/release_template.md) (e.g. [opensearch-build#2649](https://github.com/opensearch-project/opensearch-build/issues/2649)). - -#### Releasing for FreeBSD - -The FreeBSD ports and packages for OpenSearch are managed by a community [OpenSearch Team](https://wiki.freebsd.org/OpenSearch) at FreeBSD. When a new release is rolled out, this team will update the port and commit it to the FreeBSD ports tree. Anybody is welcome to help the team by providing patches for [upgrading the ports](https://docs.freebsd.org/en/books/porters-handbook/book/#port-upgrading) following the [FreeBSD Porter's Handbook](https://docs.freebsd.org/en/books/porters-handbook/book/) instructions. - -#### Releasing for macOS - -At this moment there's no official macOS distribution. However, this project does support building and assembling OpenSearch for macOS. See [opensearch-build#37](https://github.com/opensearch-project/opensearch-build/issues/37) and [#38](https://github.com/opensearch-project/opensearch-build/issues/38) for more details. - -### Utilities - -#### Checking Out Source - -The [checkout workflow](src/checkout_workflow) checks out source code for a given manifest for further examination. - -```bash -./checkout.sh manifests/1.3.0/opensearch-1.3.0.yml -``` - -See [src/checkout_workflow](./src/checkout_workflow) for more information. - -#### Cross-Platform Builds - -You can perform cross-platform builds. For example, build and assemble a Windows distribution on macOS. - -```bash -export JAVA_HOME=$(/usr/libexec/java_home) # required by OpenSearch install-plugin during assemble -./build.sh manifests/1.3.0/opensearch-1.3.0.yml --snapshot --platform windows -./assemble.sh builds/opensearch/manifest.yml -``` - -This will produce `dist/opensearch-1.3.0-SNAPSHOT-windows-x64.zip` on Linux and macOS. - -#### Sanity Checking the Bundle - -This workflow runs sanity checks on every component present in the bundle, executed as part of the [manifests workflow](.github/workflows/manifests.yml) in this repository. It ensures that the component GitHub repositories are correct and versions in those components match the OpenSearch version. - -The following example sanity-checks components in the OpenSearch 1.3.0 manifest. - -```bash -./ci.sh manifests/1.3.0/opensearch-1.3.0.yml --snapshot -``` - -See [src/ci_workflow](./src/ci_workflow) for more information. - -#### Auto-Generating Manifests - -The [manifests workflow](src/manifests_workflow) reacts to version increments in OpenSearch and its components by extracting Gradle properties from project branches. When a new version is found, a new input manifest is added to [manifests](manifests), and [a pull request is opened](.github/workflows/manifests.yml) (e.g. [opensearch-build#491](https://github.com/opensearch-project/opensearch-build/pull/491)). - -Show information about existing manifests. - -```bash -./manifests.sh list -``` -Check for updates and create any new manifests. - -```bash -./manifests.sh update -``` - -See [src/manifests_workflow](./src/manifests_workflow) for more information. - - -### Deploying Infrastructure - -Storage and access roles for the OpenSearch release process are codified in a [CDK project](./deployment/README.md). - -## Contributing - -See [developer guide](DEVELOPER_GUIDE.md) and [how to contribute to this project](CONTRIBUTING.md). - -## Getting Help - -If you find a bug, or have a feature request, please don't hesitate to open an issue in this repository. - -For more information, see [project website](https://opensearch.org/) and [documentation](https://docs-beta.opensearch.org/). If you need help and are unsure where to open an issue, try [forums](https://discuss.opendistrocommunity.dev/). - -## Code of Conduct - -This project has adopted the [Amazon Open Source Code of Conduct](CODE_OF_CONDUCT.md). For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq), or contact [opensource-codeofconduct@amazon.com](mailto:opensource-codeofconduct@amazon.com) with any additional questions or comments. - -## Security - -If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public GitHub issue. - -## License - -This project is licensed under the [Apache v2.0 License](LICENSE.txt). - -## Copyright - -Copyright OpenSearch Contributors. See [NOTICE](NOTICE) for details. diff --git a/docker/ci/README.md b/docker/ci/README.md deleted file mode 100644 index 20f2d7d07a..0000000000 --- a/docker/ci/README.md +++ /dev/null @@ -1,26 +0,0 @@ -- [CI CD Environment](#ci-cd-environment) - - [Build CI Runner Docker Image from Dockerfile](#build-ci-runner-docker-image-from-dockerfile) - - -## CI CD Environment - -We build, assemble, and test our artifacts on docker containers. All of our pipelines are using the same docker image for consistency. This folder contains docker files in the [ci](./ci) folder, and images on [staging docker hub repositories](https://hub.docker.com/r/opensearchstaging/ci-runner/). - -### Build CI Runner Docker Image from Dockerfile - -To build the docker image for either x64 or arm64, run the below command on a x64 or arm64 host respectively within the `opensearch-build/docker/ci` folder: - -```bash -./build-image-single-arch.sh -r -v -f -``` -After the build, you can locate an image in your host using the `docker images` command with the following format: `opensearchstaging/:`. - - -If you want to build multi-arch docker image for both x64 and arm64, you can use the below command.\ -Make sure you are running it within the `opensearch-build/docker/ci` folder. - -```bash -./build-image-multi-arch.sh -r -v -f -``` -Docker buildx is a tool utilized for building multi-arch images. It leverages a `moby/buildkit` container to construct these images, combining all the CPU architecture layers into a single entity. Consequently, you can only upload the resulting image to Docker Hub or store it locally as cache. Due to the limitation that your host cannot support multiple CPU architectures, the image will not be visible when running the `docker images` command. - diff --git a/docker/release/README.md b/docker/release/README.md deleted file mode 100644 index a3837b2bc8..0000000000 --- a/docker/release/README.md +++ /dev/null @@ -1,137 +0,0 @@ -### Summary -We support building OpenSearch and OpenSearch Dashboards docker for both [multi-CPU](https://docs.docker.com/desktop/multi-arch/) and single CPU architectures. -Users are welcome to choose either type of the image to build on their local development environment, or directly pulling existing images published on Docker Hub: - -[OpenSearch DockerHub Repository](https://hub.docker.com/r/opensearchproject/opensearch/) - -[OpenSearch-Dashboards DockerHub Repository](https://hub.docker.com/r/opensearchproject/opensearch-dashboards/) - -[OpenSearch ECR Repository](https://gallery.ecr.aws/opensearchproject/opensearch/) - -[OpenSearch-Dashboards ECR Repository](https://gallery.ecr.aws/opensearchproject/opensearch-dashboards/) - -``` -# DockerHub -docker pull opensearchproject/opensearch:latest -docker pull opensearchproject/opensearch-dashboards:latest - -# ECR -docker pull public.ecr.aws/opensearchproject/opensearch:latest -docker pull public.ecr.aws/opensearchproject/opensearch-dashboards:latest -``` - -### Building Docker Images -#### We provide two scripts to build docker images. For single-arch image you need to install just the Docker Engine on your host machine. For multi-arch image (currently support x64/arm64) you need to install Docker Desktop (macOS/Windows) or Docker Buildx (LINUX). - -Install Docker through the official docker webpage: https://docs.docker.com/get-docker/ - - -After installation, verify if you have Docker Engine by running: - ``` - docker build --help - ``` - - -Verify if you have Docker Desktop or Docker Buildx Standalone by running: - ``` - docker buildx --help - ``` - - -You need to run both script within the `opensearch-build/docker/release` folder. Running them - within other path would cause the scripts to fail. - - -The OpenSearch and OpenSearch-Dashboards Docker image is based on [AmazonLinux container images](https://github.com/amazonlinux/container-images) and sourced directly from the official [AmazonLinux ECR Repository](https://gallery.ecr.aws/amazonlinux/amazonlinux/) to get timely updates. - - -#### Build single-arch image with these commands: - * OpenSearch 1.0.0 x64: - ``` - ./build-image-single-arch.sh -v 1.0.0 -f ./dockerfiles/opensearch.al2.dockerfile -p opensearch -a x64 - ``` - * OpenSearch 1.0.0 arm64 with local tarball: - ``` - ./build-image-single-arch.sh -v 1.0.0 -f ./dockerfiles/opensearch.al2.dockerfile -p opensearch -a arm64 -t opensearch-1.0.0.tar.gz - ``` - * OpenSearch-Dashboards 1.0.0 x64: - ``` - ./build-image-single-arch.sh -v 1.0.0 -f ./dockerfiles/opensearch-dashboards.al2.dockerfile -p opensearch-dashboards -a x64 - ``` - * OpenSearch-Dashboards 1.0.0 arm64 with local tarball: - ``` - ./build-image-single-arch.sh -v 1.0.0 -f ./dockerfiles/opensearch-dashboards.al2.dockerfile -p opensearch-dashboards -a arm64 -t opensearch-dashboards-1.0.0.tar.gz - ``` -#### Build multi-arch image with this commands (only support x64 + arm64 in one image for now), the image will be immediately uploaded to a docker registry, so you need to provide docker repo name: - * OpenSearch 1.0.0: - ``` - ./build-image-multi-arch.sh -v 1.0.0 -f ./dockerfiles/opensearch.al2.dockerfile -p opensearch -a "x64,arm64" -r "/:" - ``` - * OpenSearch-Dashboards 1.0.0 with local tarball(s): - ``` - ./build-image-multi-arch.sh -v 1.0.0 -f ./dockerfiles/opensearch-dashboards.al2.dockerfile -p opensearch-dashboards -a "x64,arm64" -r "/:" -t "opensearch-1.0.0.tar.gz,opensearch-dashboards-1.0.0.tar.gz" - ``` - -### Disable Security Plugin, Security Dashboards Plugin, Security Demo Configurations and Related Configurations -(This change is added since OpenSearch/OpenSearch-Dashboards 1.1.0) -There are 3 environment variables available for users to disable security related settings during docker container startup: - -* 2 for OpenSearch: - * __DISABLE_INSTALL_DEMO_CONFIG__: Default to `null`, set to `true` disables running of [install_demo_configuration.sh](https://github.com/opensearch-project/security/blob/1.0.0.0/tools/install_demo_configuration.sh) bundled with Security Plugin, which installs demo certificates and security configurations to OpenSearch. - * __DISABLE_SECURITY_PLUGIN__: Default to `null`, set to `true` disables Security Plugin entirely in OpenSearch by setting `plugins.security.disabled: true` in opensearch.yml. - -* 1 for Dashboards: - * __DISABLE_SECURITY_DASHBOARDS_PLUGIN__: Default to `null`, set to `true` disables Security Dashboards Plugin in OpenSearch-Dashboards by removing securityDashboards plugin folder, remove all related settings in opensearch_dashboards.yml, and set `opensearch.hosts` entry protocol from HTTPS to HTTP. This step is not reversible as the Security Dashboards Plugin is removed in the process. If you want to re-enable security for OpenSearch-Dashboards, you need to start a new container with `DISABLE_SECURITY_DASHBOARDS_PLUGIN` unset, or false. - - -Here are three example scenarios of using above variables: - -#### Scenario 1: Original behavior, install demo certs/configs + enable security on both OpenSearch and OpenSearch-Dashboards: - * OpenSearch: - ``` - $ docker run -it -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" opensearchproject/opensearch:1.1.0 - ``` - * OpenSearch-Dashboards: - ``` - $ docker run -it --network="host" opensearchproject/opensearch-dashboards:1.1.0 - ``` - -#### Scenario 2: No demo certs/configs + disable security on both OpenSearch and OpenSearch-Dashboards: - * OpenSearch: - ``` - $ docker run -it -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "DISABLE_INSTALL_DEMO_CONFIG=true" -e "DISABLE_SECURITY_PLUGIN=true" opensearchproject/opensearch:1.1.0 - ``` - * OpenSearch-Dashboards: - ``` - $ docker run -it --network="host" -e "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" opensearchproject/opensearch-dashboards:1.1.0 - ``` - -#### Scenario 3: No demo certs/configs + enable security on both OpenSearch and OpenSearch-Dashboards (cluster exit with errors due to demo install script is not run. Therefore, no certs/configs are available for Security Plugin. Useful if you want to setup your own certs/configs): - * OpenSearch: - ``` - $ docker run -it -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "DISABLE_INSTALL_DEMO_CONFIG=true" opensearchproject/opensearch:1.1.0 - ``` - * Dashboards: - ``` - $ docker run -it --network="host" -e opensearchproject/opensearch-dashboards:1.1.0 - ``` - -### Disable Performance Analyzer Agent CLI and Related Configurations -(This change is added after OpenSearch 2.4.0 and after OpenSearch 1.3.6) - - * 1 for OpenSearch: - * __DISABLE_PERFORMANCE_ANALYZER_AGENT_CLI__: Default to `null`, set to `true` disables running of [performance-analyzer-agent-cli](https://github.com/opensearch-project/performance-analyzer/blob/2.4/packaging/performance-analyzer-agent-cli) bundled with Performance Analyzer Plugin, which starts the performance analyzer root cause analysis (RCA) agent. - -Here are some example scenarios of using above variables: - -#### Scenario 1: Original behavior, enable Performance Analyzer Agent Cli - * OpenSearch: - ``` - $ docker run -it -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" opensearchproject/opensearch: - ``` - -#### Scenario 2: Disable Performance Analyzer Agent Cli - * OpenSearch: - ``` - $ docker run -it -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "DISABLE_PERFORMANCE_ANALYZER_AGENT_CLI=true" opensearchproject/opensearch: - ``` diff --git a/jenkins/README.md b/jenkins/README.md deleted file mode 100644 index 74771ceb00..0000000000 --- a/jenkins/README.md +++ /dev/null @@ -1,29 +0,0 @@ -- [Jenkinsfiles](#jenkinsfiles) - - [Check for Build](#check-for-build) - - [Add Periodically Run Jobs](#add-periodically-run-jobs) - - [Jenkins Shared Library](#jenkins-shared-library) - -## Jenkinsfiles - -* [OpenSearch](./opensearch/) -* [OpenSearch Dashboards](./opensearch-dashboards/) -* [Docker](./docker/) - - -### Check for Build - -To ensure that builds are trigger as changes come in the [Check-for-build](check-for-build.jenkinsfile) job is setup with a cron schedule to check manifests and run them at the appropriate build job. From in a manifest a stable build version can be created and reference against uploaded versions. If there is an exist match the build will be skipped, if not the job will be triggered in its own build workflow. - -For concurrent builds the Check-for-build job has an internal locks to prevent interruption for the same manifest and target job name, any requests that match an existing lock will be skipped to prevent a backup of queued build jobs. - -#### Add Periodically Run Jobs - -* Set cron to `H/10 * * * *` for jobs that are in active development. -* Set cron to `H 1 * * *` for all other jobs. - -### Jenkins Shared Library - -This project contains a Jenkins global shared library used extensively in the Jenkinsfiles. See [opensearch-build-libraries](https://github.com/opensearch-project/opensearch-build-libraries) for more details on shared libraries - -* [Global Shared Functions](https://github.com/opensearch-project/opensearch-build-libraries/tree/main/vars) -* [Common Classes](https://github.com/opensearch-project/opensearch-build-libraries/tree/main/src/jenkins) diff --git a/scripts/README.md b/scripts/README.md deleted file mode 100644 index 3e17fbe925..0000000000 --- a/scripts/README.md +++ /dev/null @@ -1,74 +0,0 @@ -- [Scripts](#scripts) - - [Run Deployment Script](#run-deployment-script) - - [Requirements](#requirements) - - [Usage](#usage) - -## Scripts - -This folder contains default and custom scripts located by [src/paths/script_finder.py](../src/paths/script_finder.py), and the following scripts which are used in either tar/docker or legacy github actions. - -#### Run Deployment Script - -This is a script to deploy a single node OpenSearch + OpenSearch-Dashboards cluster to a x64 or arm64 host. - -##### Requirements - -* If you are using LINUX host to run dockerd, provision container based on the dockerfile, and execute integTest: - * You must run these commands on the LINUX host for OpenSearch process to run: - ``` - sudo sysctl -w vm.max_map_count=262144 - ulimit -n 65535 - ``` - - * You must make this edit on the LINUX host and restart docker daemon for OpenSearch process to run. - This will also prevent errors during Dashboards integtest (cypress) so that headless chrome will not crash. - * Option 1 (Recommended): - ``` - # Edit /etc/docker/daemon.json with these changes: - { - "default-ulimits": { - "nofile": { - "Hard": 65535, - "Name": "nofile", - "Soft": 65535 - } - }, - "default-shm-size": "1024M" - } - # Then restart docker - sudo systemctl stop docker - sudo systemctl start docker - ``` - * Option 2: - ``` - # If you have /etc/sysconfig/docker on your LINUX host - # Edit /etc/sysconfig/docker to have nofiles to 65535, and default shared memory to 1024MiB - # OPTIONS="--default-ulimit nofile=65535:65535 --default-shm-size 1024000000" - # Then restart docker - sudo systemctl stop docker - sudo systemctl start docker - ``` - -##### Usage - -* If you are using macOS host to run Docker Desktop, provision container based on the dockerfile, and execute integTest: - * You must change docker memory/RAM utilization in resources settings if you use Docker Desktop on macOS. - * Make sure at least 4GiB of RAM is used, so Dashboards process can startup correctly. - -* The deployment script is able to deploy a single node cluster with the specific version of OpenSearch/Dashboards bundle tarball on your host/container: - * You must specify `-v VERSION`, `-t TYPE`, `-a ARCHITECTURE`, `-s ENABLE_SECURITY` - * The script will attempt to download bundled tarball of OpenSearch/Dashboards from `artifacts.opensearch.org` and deploy to a temporary working directory. - * Example: - * Deploy OpenSearch/Dashboards 1.0.0 version, from snapshots channel, CPU arch x64, enable security: - ``` - ./deploy_single_tar_cluster.sh -v 1.0.0 -t snapshots -a x64 -s true - ``` - * Deploy OpenSearch/Dashboards 1.0.0 version, from releases channel, CPU arch arm64, disable security: - ``` - ./deploy_single_tar_cluster.sh -v 1.0.0 -t releases -a arm64 -s false - ``` - * The script allows in process wait and cleanup, meaning a `SIGINT / SIGTERM / SIGEXIT` or any child process throwing `EXIT` signal will result in termination of the - cluster as well as the temporary working directory. - * If you want to run the deployment in detach mode, just run the command with `nohup &`. - - diff --git a/scripts/components/README.md b/scripts/components/README.md deleted file mode 100644 index 1cb850ba7f..0000000000 --- a/scripts/components/README.md +++ /dev/null @@ -1 +0,0 @@ -These scripts override [default scripts](../default/) in bundle workflow for individial components. \ No newline at end of file diff --git a/wiki/CODE_OF_CONDUCT.md b/wiki/CODE_OF_CONDUCT.md deleted file mode 100644 index 7129188428..0000000000 --- a/wiki/CODE_OF_CONDUCT.md +++ /dev/null @@ -1,22 +0,0 @@ -This code of conduct applies to all spaces provided by the OpenSource project including in code, documentation, issue trackers, mailing lists, chat channels, wikis, blogs, social media and any other communication channels used by the project. - -**Our open source communities endeavor to:** - -* Be Inclusive: We are committed to being a community where everyone can join and contribute. This means using inclusive and welcoming language. -* Be Welcoming: We are committed to maintaining a safe space for everyone to be able to contribute. -* Be Respectful: We are committed to encouraging differing viewpoints, accepting constructive criticism and work collaboratively towards decisions that help the project grow. Disrespectful and unacceptable behavior will not be tolerated. -* Be Collaborative: We are committed to supporting what is best for our community and users. When we build anything for the benefit of the project, we should document the work we do and communicate to others on how this affects their work. - -**Our Responsibility. As contributors, members, or bystanders we each individually have the responsibility to behave professionally and respectfully at all times. Disrespectful and unacceptable behaviors include, but are not limited to:** - -* The use of violent threats, abusive, discriminatory, or derogatory language; -* Offensive comments related to gender, gender identity and expression, sexual orientation, disability, mental illness, race, political or religious affiliation; -* Posting of sexually explicit or violent content; -* The use of sexualized language and unwelcome sexual attention or advances; -* Public or private harassment of any kind; -* Publishing private information, such as physical or electronic address, without permission; -* Other conduct which could reasonably be considered inappropriate in a professional setting; -* Advocating for or encouraging any of the above behaviors. -* Enforcement and Reporting Code of Conduct Issues: - -Instances of abusive, harassing, or otherwise unacceptable behavior may be reported. [Contact us](mailto:opensource-codeofconduct@amazon.com). All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. \ No newline at end of file diff --git a/CODE_OF_CONDUCT.md b/wiki/Code-of-Conduct.md similarity index 100% rename from CODE_OF_CONDUCT.md rename to wiki/Code-of-Conduct.md diff --git a/wiki/HOME.md b/wiki/HOME.md deleted file mode 100644 index b158b60900..0000000000 --- a/wiki/HOME.md +++ /dev/null @@ -1,5 +0,0 @@ -## OpenSearch Build - -![build_logo](https://raw.githubusercontent.com/opensearch-project/opensearch-build/main/opensearch_build_image.png) - -This repository hosts the build, test and release scripts for OpenSearch and OpenSearch Dashboard distributions. diff --git a/wiki/Home.md b/wiki/Home.md new file mode 100644 index 0000000000..e31f01f778 --- /dev/null +++ b/wiki/Home.md @@ -0,0 +1,12 @@ +## OpenSearch Build + +![build_logo](https://raw.githubusercontent.com/opensearch-project/opensearch-build/main/opensearch_build_image.png) + +This repository hosts the build, test and release scripts for OpenSearch and OpenSearch Dashboard distributions. + +####Workflows: + [Assemble Workflow](https://github.com/chawinphat/opensearch-build/blob/main/wiki/src/assemble-workflow.md) + [Build Workflow](https://github.com/chawinphat/opensearch-build/blob/main/wiki/src/build-workflow.md) + [Checkout Workflow](https://github.com/chawinphat/opensearch-build/blob/main/wiki/src/checkout-workflow.md) + [Manifests Workflow](https://github.com/chawinphat/opensearch-build/blob/main/wiki/src/manifests-workflow.md) + [Release Notes Workflow](https://github.com/chawinphat/opensearch-build/blob/main/wiki/src/release-notes-workflow.md) diff --git a/wiki/Docker/ci.md b/wiki/docker/ci.md similarity index 100% rename from wiki/Docker/ci.md rename to wiki/docker/ci.md diff --git a/wiki/Docker/release.md b/wiki/docker/release.md similarity index 100% rename from wiki/Docker/release.md rename to wiki/docker/release.md diff --git a/src/assemble_workflow/README.md b/wiki/src/assemble-workflow.md similarity index 100% rename from src/assemble_workflow/README.md rename to wiki/src/assemble-workflow.md diff --git a/wiki/src/assemble_workflow.md b/wiki/src/assemble_workflow.md deleted file mode 100644 index 68b43e8756..0000000000 --- a/wiki/src/assemble_workflow.md +++ /dev/null @@ -1,33 +0,0 @@ -- [Assemble a Distribution](#assemble-a-distribution) - - [Assemble.sh Options](#assemblesh-options) - - [Custom Install Scripts](#custom-install-scripts) - -## Assemble a Distribution - -```bash -./assemble.sh builds/opensearch/manifest.yml -``` - -The assembling step takes output from the build step, installs plugins, and assembles a full distribition into the `dist` folder. The input requires a path to the build manifest and is expected to be inside the `builds` directory that contains `dist`, `maven`, `plugins` and `core-plugins` subdirectories from the build step. - -Artifacts will be created as follows. - -``` -/dist - .tar.gz / .zip / .rpm <- assembled tarball / zip / rpm depending on distribution value in builds/opensearch/manifest.yml, default to tarball if 'distribution' key not found - manifest.yml <- bundle manifest describing versions for the min bundle and all installed plugins and their locations -``` - -### Assemble.sh Options - -The following options are available in `assemble.sh`. - -| name | description | -|--------------------|-------------------------------------------------------------------------| -| -b, --base-url | The base url to download the artifacts. | -| --keep | Do not delete the temporary working directory on both success or error. | -| -v, --verbose | Show more verbose output. | - -### Custom Install Scripts - -You can perform additional plugin install steps by adding an `install.sh` script. By default the tool will look for a script in [scripts/bundle-build/components](../../scripts/bundle-build/components), then default to a noop version implemented in [scripts/default/install.sh](../../scripts/default/install.sh). diff --git a/src/build_workflow/README.md b/wiki/src/build-workflow.md similarity index 100% rename from src/build_workflow/README.md rename to wiki/src/build-workflow.md diff --git a/wiki/src/build_workflow.md b/wiki/src/build_workflow.md deleted file mode 100644 index 6a4b02c751..0000000000 --- a/wiki/src/build_workflow.md +++ /dev/null @@ -1,103 +0,0 @@ -- [Building from Source](#building-from-source) - - [OpenSearch](#opensearch) - - [OpenSearch Dashboards](#opensearch-dashboards) - - [Build Paths](#build-paths) - - [Build.sh Options](#buildsh-options) - - [Custom Build Scripts](#custom-build-scripts) - - [Avoiding Rebuilds](#avoiding-rebuilds) - -## Building from Source - -The build workflow builds components declared in an input manifest from source, or downloads previously built artifacts, in order. All manifests for current releases are in [manifests](../../manifests). Here are some examples. - -| name | description | -|--------------------------------------------------------------------------------------|----------------------------------------------------------------| -| [opensearch-1.0.0.yml](/manifests/1.0.0/opensearch-1.0.0.yml) | Manifest to reproduce 1.0.0 build. | -| [opensearch-1.0.0-maven.yml](/manifests/1.0.0/opensearch-1.0.0-maven.yml) | One-time manifest to build maven artifacts for 1.0 from tags. | -| [opensearch-1.1.0.yml](/manifests/1.1.0/opensearch-1.1.0.yml) | Manifest for 1.1.0, the current version. | -| [opensearch-1.1.1.yml](/manifests/1.1.1/opensearch-1.1.1.yml) | Manifest for 1.1.1, a patch release. | -| [opensearch-1.2.0.yml](/manifests/1.2.0/opensearch-1.2.0.yml) | Manifest for 1.2.0, the next version. | -| [opensearch-2.0.0.yml](/manifests/2.0.0/opensearch-2.0.0.yml) | Manifest for 2.0.0, the next major version of OpenSearch. | -| [opensearch-dashboards-1.1.0.yml](/manifests/1.1.0/opensearch-dashboards-1.1.0.yml) | Manifest for 1.1.0, the next version of OpenSearch Dashboards. | - -The following example builds a snapshot version of OpenSearch 1.2.0. - -```bash -./build.sh manifests/1.2.0/opensearch-1.2.0.yml --snapshot -``` - -While the following builds a snapshot version of OpenSearch-Dashboards 1.2.0. - -```bash -./build.sh manifests/1.2.0/opensearch-dashboards-1.2.0.yml --snapshot -``` - -The [OpenSearch repo](https://github.com/opensearch-project/OpenSearch) is built first, followed by [common-utils](https://github.com/opensearch-project/common-utils), and all declared plugin repositories. These dependencies are published to maven local under `~/.m2`, and subsequent project builds pick those up. - -The [OpenSearch Dashboards repo](https://github.com/opensearch-project/OpenSearch-Dashboards) is built first, followed by all declared plugin repositories. - -All final output is placed into a `builds/opensearch` and `builds/opensearch-dashboards` folder respectively, along with a build output `manifest.yml` that contains output details. - -#### OpenSearch - -The build order first publishes `OpenSearch` followed by `common-utils`, and publishes these artifacts to maven local so that they are available for each component. In order to ensure that the same versions are used, a `-Dopensearch.version` flag is provided to each component's build script that defines which version the component should build against. - -#### OpenSearch Dashboards - -The build order first pulls down `OpenSearch-Dashboards` and then utilizes it to build other components. Currently, building plugins requires having the core repository built first to bootstrap and build the modules utilized by plugins. - -### Build Paths - -To build components we rely on a common entry-point in the form of a `build.sh` script. Within each build script components have the option to place artifacts in a set of directories to be picked up and published on their behalf. These are as follows. - -| name | description | -|--------------------|---------------------------------------------------------------------------------------------------------| -| /dist | Include any distributions, e.g. opensearch or opensearch-dashboards min tarball. | -| /maven | Include any publications that should be pushed to maven | -| /plugins | Where a plugin zip should be placed. If included it will be installed during bundle assembly. | -| /core-plugins | Where plugins shipped from `https://github.com/opensearch-project/OpenSearch` should be placed | -| /bundle | Where the min bundle should be placed when built from `https://github.com/opensearch-project/OpenSearch`| -| /libs | Where any additional libs should be placed that are required during bundle assembly | - -### Build.sh Options - -The following options are available in `build.sh`. - -| name | description | -|-------------------------|----------------------------------------------------------------------------------------| -| -s, --snapshot | Build a snapshot instead of a release artifact, default is `false`. | -| -a, --architecture | Specify architecture to build, default is architecture of build system. | -| -d, --distribution | Specify distribution to build, default is `tar`. | -| -p, --platform | Specify platform to build, default is platform of build system. | -| --component [name ...] | Rebuild a subset of components by name, e.g. `--component common-utils job-scheduler`. | -| --keep | Do not delete the temporary working directory on both success or error. | -| --continue-on-error | Do not fail the bundle build on plugin component failure. | -| -l, --lock | Generate a stable reference manifest. | -| -v, --verbose | Show more verbose output. | - -### Custom Build Scripts - -Each component build relies on a `build.sh` script that is used to prepare bundle artifacts for a particular bundle version that takes two arguments: version and target architecture. By default the tool will look for a script in [scripts/components](../../scripts/components), then in the checked-out repository in `scripts/build.sh`, then default to a Gradle build implemented in [scripts/default/opensearch/build.sh](../../scripts/default/opensearch/build.sh). - -### Avoiding Rebuilds - -Builds can automatically generate a `manifest.lock` file with stable git references (commit IDs) and build options (platform, architecture and snapshot) by specifying `--lock`. The output can then be reused as input manifest after checking against a collection of prior builds. - -```bash -MANIFEST=manifests/1.2.0/opensearch-1.2.0.yml -SHAS=shas - -./build.sh --lock $MANIFEST # generates opensearch-1.2.0.yml.lock - -MANIFEST_SHA=$(sha1sum $MANIFEST.lock | cut -f1 -d' ') # generate a checksum of the stable manifest - -if test -f "$SHAS/$MANIFEST_SHA.lock"; then - echo "Skipping $MANIFEST_SHA" -else - ./build.sh $MANIFEST.lock # rebuild using stable references in .lock - mkdir -p $SHAS - cp $MANIFEST.lock $SHAS/$MANIFEST_SHA.lock # save the stable reference manifest -fi -``` - -The [Jenkins workflows](../../jenkins) in this repository can use this mechanism to avoid rebuilding all of OpenSearch and OpenSearch Dashboards unnecessarily. diff --git a/src/checkout_workflow/README.md b/wiki/src/checkout-workflow.md similarity index 100% rename from src/checkout_workflow/README.md rename to wiki/src/checkout-workflow.md diff --git a/wiki/src/checkout_workflow.md b/wiki/src/checkout_workflow.md deleted file mode 100644 index f7c34d6e23..0000000000 --- a/wiki/src/checkout_workflow.md +++ /dev/null @@ -1,15 +0,0 @@ -- [Checking Out Components](#checking-out-components) - -## Checking Out Components - -This workflow checks out source code for a given manifest for further examination. - -```bash -./checkout.sh manifests/1.2.0/opensearch-1.2.0.yml -``` - -The following options are available. - -| name | description | -|--------------------|-------------------------------------------------------------------------| -| -v, --verbose | Show more verbose output. | diff --git a/src/ci_workflow/README.md b/wiki/src/ci-workflow.md similarity index 100% rename from src/ci_workflow/README.md rename to wiki/src/ci-workflow.md diff --git a/wiki/src/ci_workflow.md b/wiki/src/ci_workflow.md deleted file mode 100644 index 8d28897fea..0000000000 --- a/wiki/src/ci_workflow.md +++ /dev/null @@ -1,47 +0,0 @@ -- [Sanity Testing the Distribution](#sanity-testing-the-distribution) - - [Manifest Checks](#manifest-checks) - - [Ci.sh Options](#cish-options) - -## Sanity Testing the Distribution - -This workflow runs sanity checks on every component present in the bundle, executed as part of the [manifests workflow](../../.github/workflows/manifests.yml) in this repository. It ensures that the component GitHub repositories are correct and versions in those components match the OpenSearch or OpenSearch Dashboards versions. - -To use checks, nest them under `checks` in the manifest. - -```yaml -- name: common-utils - repository: https://github.com/opensearch-project/common-utils.git - ref: main - checks: - - gradle:publish - - gradle:properties:version - - gradle:dependencies:opensearch.version - - gradle:dependencies:opensearch.version: alerting -``` - -### Manifest Checks - -The following checks are available. - -| name | description | -|-----------------------------------------------|-----------------------------------------------------------------------------------| -| gradle:properties:version | Check version of the component. | -| gradle:dependencies:opensearch.version | Check dependency on the correct version of OpenSearch in gradle properties. | -| gradle:publish | Check that publishing to Maven local works, and publish. | -| npm:package:version | Check dependency on the correct version of OpenSearch Dashboards in package.json. | - -The following example sanity-checks components in the the OpenSearch 1.2.0 manifest. - -```bash -./ci.sh manifests/1.2.0/opensearch-1.2.0.yml --snapshot -``` - -### Ci.sh Options - -The following options are available. - -| name | description | -|-------------------------|-------------------------------------------------------------------------------------| -| --component [name ...] | Test a subset of components by name, e.g. `--component common-utils job-scheduler`. | -| --keep | Do not delete the temporary working directory on both success or error. | -| -v, --verbose | Show more verbose output. | diff --git a/src/manifests_workflow/README.md b/wiki/src/manifests-workflows.md similarity index 100% rename from src/manifests_workflow/README.md rename to wiki/src/manifests-workflows.md diff --git a/wiki/src/manifests_workflows.md b/wiki/src/manifests_workflows.md deleted file mode 100644 index f4dee5104d..0000000000 --- a/wiki/src/manifests_workflows.md +++ /dev/null @@ -1,25 +0,0 @@ -- [Auto-Generating Manifests](#auto-generating-manifests) - -## Auto-Generating Manifests - -This workflow reacts to version increments in OpenSearch and its components by extracting Gradle properties from project branches. Currently OpenSearch `main`, and `x.y` branches are checked out one-by-one, published to local maven, and their versions extracted using `./gradlew properties`. When a new version is found, a new input manifest is added to [manifests](../../manifests), and [a pull request is opened](../../.github/workflows/manifests.yml) (e.g. [opensearch-build#491](https://github.com/opensearch-project/opensearch-build/pull/491)). - -Show information about existing manifests. - -```bash -./manifests.sh list -``` - -Check for updates and create any new manifests. - -```bash -./manifests.sh update -``` - -The following options are available. - -| name | description | -|--------------------|-------------------------------------------------------------------------| -| --keep | Do not delete the temporary working directory on both success or error. | -| --type | Only list manifests of a specific type). | -| -v, --verbose | Show more verbose output. | diff --git a/src/release_notes_workflow/README.md b/wiki/src/release-notes-workflow.md similarity index 100% rename from src/release_notes_workflow/README.md rename to wiki/src/release-notes-workflow.md diff --git a/wiki/src/release_notes_workflow.md b/wiki/src/release_notes_workflow.md deleted file mode 100644 index e8a3e66c38..0000000000 --- a/wiki/src/release_notes_workflow.md +++ /dev/null @@ -1,37 +0,0 @@ -#### Components Release Notes Check - -Pulls the latest code to check if the release notes exists and whether new commits have been made based on user passed argument `--date`. Outputs a formated markdown table as follows. - -*Usage* -``` -./release_notes.sh check manifests/3.0.0/opensearch-3.0.0.yml --date 2022-07-26 -``` - -*Sample Output* -``` -# OpenSearch CommitID(after 2022-07-26) & Release Notes info -| Repo | Branch |CommitID|Commit Date|Release Notes| -|-------------------------|------------|--------|-----------|-------------| -|OpenSearch |tags/2.2.0 |b1017fa |2022-08-08 |True | -|common-utils |tags/2.2.0.0|7d53102 |2022-08-04 |False | -|job-scheduler |tags/2.2.0.0|a501307 |2022-08-02 |True | -|ml-commons |tags/2.2.0.0|a7d2695 |2022-08-08 |True | -|performance-analyzer |tags/2.2.0.0|3a75d7d |2022-08-08 |True | -|security |tags/2.2.0.0|8e9e583 |2022-08-08 |True | -|geospatial |tags/2.2.0.0|a71475a |2022-08-04 |True | -|k-NN |tags/2.2.0.0|53185a0 |2022-08-04 |True | -|cross-cluster-replication|tags/2.2.0.0|14d871a |2022-08-05 |False | -``` - -The workflow uses the following arguments: -* `--date`: To check if commit exists after a specific date (in format yyyy-mm-dd, example 2022-07-26). -* `--output`: To dump the output into an `.md` file, example `--output table.md`). - - -The following options are available. - -| name | description | -|--------------------|-------------------------------------------------------------------------| -| --date | Shows commit after a specific date. | -| --output | Saves the table output to user specified file. | -| -v, --verbose | Show more verbose output. | diff --git a/src/report_workflow/README.md b/wiki/src/report-workflow.md similarity index 100% rename from src/report_workflow/README.md rename to wiki/src/report-workflow.md diff --git a/wiki/src/report_workflow.md b/wiki/src/report_workflow.md deleted file mode 100644 index 289b53cacd..0000000000 --- a/wiki/src/report_workflow.md +++ /dev/null @@ -1,67 +0,0 @@ -- [Generate test-report manifest for each test](#generate-test-report-manifest-for-each-test) -- [Guide on test-report manifest from CI](#guide-on-test-report-manifest-from-ci) - -# Generate test-report manifest for each test -As the name specifies, the test report workflow helps to automatically generate a consolidated report of the tests run at distribution level along with commands to reproduce the error and associated failures. - -### How to Use -To use this workflow, execute the following command: -``` -./report.sh --artifact-paths opensearch=<...> opensearch-dashboards=<...> --test-run-id <...> --test-type integ-test --base-path <...> -``` -For example: -``` -./report.sh manifests/2.9.0/opensearch-2.9.0-test.yml -p opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.9.0/8172/linux/x64/tar --test-run-id 5328 --test-type integ-test --base-path https://ci.opensearch.org/ci/dbc/integ-test/2.9.0/8172/linux/x64/tar -``` - -### Available Options -| name | description | -|---------------------------------|--------------------------------------------------------------------------------| -| test-manifest-path | Specify a test manifest path. | -| -p, --artifact-paths | Artifact paths of distributions used for testing. | -| --base-path | Base paths of testing logs. | -| --test-type | Type of tests report generates on. | -| --output-path | Specify the path location for the test-report manifest. | -| --test-run-id | Specify the unique execution id that matches the id of the test. | -| --component | Specify a specific component or components instead of the entire distribution. | -| --verbose | Show more verbose output. | - -# Guide on Test-Report Manifest from CI - -The test-report manifest(`test-report.yml`) generated by the report workflow contains aggregated test results for the components. -This machine-readable manifest schema provides comprehensive test result data. -The test-report manifest is generated each time the CI test workflow runs on Jenkins and is uploaded to our S3 bucket. - -The test-report manifest includes the following attributes: - -| name | description | -|:-------------------------------|:----------------------------------------------------------------------------| -| schema-version | Version of the manifest schema (e.g., '1.0'). | -| name | Name of the product (e.g., OpenSearch, OpenSearch Dashboards). | -| test-run/Command: | Command to trigger the test. | -| test-run/TestType: | Type of test reported by this manifest (e.g., integ-test). | -| test-run/TestManifest: | Location of the test manifest used. | -| test-run/DistributionManifest: | URL or local path of the bundle manifest. | -| test-run/TestID: | Test ID. | -| components/command: | Command to trigger the test for a specific component. | -| components/configs/name: | Configuration name used for testing (e.g., with-security/without-security). | -| components/configs/status: | Status of the test run with this configuration (e.g., PASS/FAIL). | -| components/configs/yml: | URL or local path to the component YAML file with detailed testing logs. | - -## FAQ -### _**How to Reproduce the Testing?**_ - - Use the provided commands from this test-report manifest. Additional details on running the commands can be found [here](https://github.com/opensearch-project/opensearch-build/tree/main/src/test_workflow#testing-a-distribution). -### _**Which Bundle Distribution Were We Using for This Test Run?**_ - - Find the bundle manifest under the DistributionManifest key. It contains information about the bundle artifacts used for testing. -### _**How Do I See the CI Logs of This Test Run?**_ - - Access our Jenkins job: - - For `OpenSearch`: `https://build.ci.opensearch.org/job/<__TestType__>/<__TestID__>/`. e.g. https://build.ci.opensearch.org/job/integ-test/1234/ - - For `OpenSearch Dashboards`: `https://build.ci.opensearch.org/job/<__TestType__>-opensearch-dashboards/<__TestID__>/`. e.g. https://build.ci.opensearch.org/job/integ-test-opensearch-dashboards/1234/ -### _**Where Can I See Detailed Testing Logs or Service Logs (e.g., Local Cluster Logs, Testing Videos)?**_ - - You can find details in the component YAML file under `components/configs/yml` in this test-report manifest. The component YAML file includes local cluster logs, stdout & stderr files, specific testing logs, and Cypress videos. -### _**Why Are Some Component Testing Results Missing or Unavailable?**_ - - This test-report manifest is generated based on component YAML files generated during the testing process. If component YAML files are missing, we won't have details for that testing component. - - Reasons for unavailable component YAML files: - - The testing bundle doesn't include this component, even though it's listed in our TestManifest. Check the DistributionManifest to confirm if your component is built in. - - The test was not run for this component, even though it's included in the TestManifest. Check our Jenkins job to see if the test for your component was executed or not. The way to look into our CI is listed above. [How do I see the CI logs of this test run](#how-do-i-see-the-ci-logs-of-this-test-run) - - The test on this component was running, but the test recorder may not have captured the results correctly to generate the component YAML. This can happen if the testing cluster is not available. Check our Jenkins job for details. The way to look into our CI is listed above. [How do I see the CI logs of this test run](#how-do-i-see-the-ci-logs-of-this-test-run) diff --git a/src/sign_workflow/README.md b/wiki/src/sign-workflow.md similarity index 100% rename from src/sign_workflow/README.md rename to wiki/src/sign-workflow.md diff --git a/wiki/src/sign_workflow.md b/wiki/src/sign_workflow.md deleted file mode 100644 index 676618716b..0000000000 --- a/wiki/src/sign_workflow.md +++ /dev/null @@ -1,22 +0,0 @@ -- [Signing Artifacts](#signing-artifacts) - - [Sign.sh Options](#signsh-options) - -## Signing Artifacts - -The signing step (optional) takes the manifest file created from the build step and signs all its component artifacts using a tool called `opensearch-signer-client` (in progress of being open-sourced). The input requires a path to the build manifest and is expected to be inside the artifacts directory with the same directories mentioned in the build step. - -```bash -./sign.sh builds/opensearch/manifest.yml -``` - -### Sign.sh Options - -The following options are available. - -| name | description | -|---------------|---------------------------------------------------------------------------------------| -| --component | The component name(s) of the component(s) whose artifacts will be signed. | -| --type | The artifact type to be signed. Currently one of 3 options: [plugins, maven, bundle]. | -| -v, --verbose | Show more verbose output. | - -The signed artifacts (.asc) will be found in the same location as the original artifact. diff --git a/src/test_workflow/README.md b/wiki/src/test-workflow.md similarity index 100% rename from src/test_workflow/README.md rename to wiki/src/test-workflow.md diff --git a/wiki/src/test_workflow.md b/wiki/src/test_workflow.md deleted file mode 100644 index 5e7294c730..0000000000 --- a/wiki/src/test_workflow.md +++ /dev/null @@ -1,350 +0,0 @@ -- [Testing a Distribution](#testing-a-distribution) - - [Test.sh Options](#testsh-options) - - [Integration Tests](#integration-tests) - - [Backwards Compatibility Tests](#backwards-compatibility-tests) - - [Performance Tests](#performance-tests) - - [Identifying Regressions in Performance Tests](#identifying-regressions-in-performance-tests) - - [Identifying Regressions in Nightly Performance Tests](#identifying-regressions-in-nightly-performance-tests) - - [Identifying Issues in Longevity Tests](#identifying-issues-in-longevity-tests) - - [Benchmark Tests](#benchmarking-tests) - - [Onboard feature for nightly benchmark runs](#onboard-feature-for-nightly-benchmark-runs) -- [Testing in CI/CD](#testing-in-cicd) - - [Test Workflow (in development)](#test-workflow-in-development) - - [Component-Level Details](#component-level-details) - - [test-orchestrator pipeline](#test-orchestrator-pipeline) - - [integTest job](#integtest-job) - - [bwcTest job](#bwctest-job) - - [perfTest job](#perftest-job) -- [Manifest Files](#manifest-files) -- [Dependency Management](#dependency-management) -- [S3 Permission Model](#s3-permission-model) -- [Appendix](#appendix) - -## Testing a Distribution - -Testing is run via `./test.sh`. - -### Test.sh Options - -The following options are available. - -| name | description | -|------------------------|-------------------------------------------------------------------------| -| test-type | Run tests of a test suite. [integ-test, bwc-test, perf-test] | -| test-manifest-path | Specify a test manifest path. | -| --paths | Location of manifest(s). | -| --test-run-id | Unique identifier for a test run. | -| --component [name ...] | Test a subset of specific components. | -| --keep | Do not delete the temporary working directory on both success or error. | -| -v, --verbose | Show more verbose output. | - -### Integration Tests - -Runs integration tests invoking `run_integ_test.py` in each component from distribution manifest. - -To run integration tests locally, use below command. This pulls down the built bundle and its manifest file, reads all components of the distribution, and runs integration tests against each component. - -Usage: - -```bash -./test.sh integ-test -``` - -For example, build locally and run integration tests. - -```bash -./build.sh manifests/1.3.5/opensearch-1.3.5.yml -./assemble.sh builds/opensearch/manifest.yml -./test.sh integ-test manifests/1.3.5/opensearch-1.3.5-test.yml . # looks for "./builds/opensearch/manifest.yml" and "./dist/opensearch/manifest.yml" -``` - -Or run integration tests against an existing build. - -```bash -./test.sh integ-test manifests/1.3.5/opensearch-1.3.5-test.yml --paths opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.5/5960/linux/x64/tar # looks for https://.../builds/opensearch/manifest.yml and https://.../dist/opensearch/manifest.yml -``` - -To run OpenSearch Dashboards integration tests. - -```bash -./test.sh integ-test manifests/1.3.0/opensearch-dashboards-1.3.0-test.yml --paths opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.5/5960/linux/x64/tar -opensearch-dashboards=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/1.3.5/4056/linux/x64/tar -``` - -To run OpenSearch Dashboards integration tests with local artifacts on different distributions -```bash -./test.sh integ-test manifests/2.0.0/opensearch-dashboards-2.0.0-test.yml --paths opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.0.0-rc1/latest/linux/x64/tar opensearch-dashboards=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.0.0-rc1/latest/linux/x64/tar -./test.sh integ-test manifests/2.0.0/opensearch-dashboards-2.0.0-test.yml --paths opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.0.0-rc1/latest/linux/x64/rpm opensearch-dashboards=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.0.0-rc1/latest/linux/x64/rpm -``` - -:warning: RPM Test requires user to run the `./test.sh` command with sudo permission, as rpm requires root to install and start the service. - - -### Backwards Compatibility Tests - -Runs backward compatibility invoking `run_bwc_test.py` in each component from a distribution manifest. - -Usage: - -```bash -./test.sh bwc-test -``` - -For example, build locally and run BWC tests. - -```bash -./build.sh manifests/1.3.0/opensearch-1.3.0.yml -./assemble.sh builds/opensearch/manifest.yml -./test.sh bwc-test manifests/1.3.0/opensearch-1.3.0-test.yml . # looks for "./builds/opensearch/manifest.yml" and "./dist/opensearch/manifest.yml" -``` - -Or run BWC tests against an existing build. - -```bash -./test.sh bwc-test manifests/1.3.0/opensearch-1.3.0-test.yml --paths opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.0.0-rc1/latest/linux/x64/tar # looks for https://.../builds/opensearch/manifest.yml and https://.../dist/opensearch/manifest.yml -``` - -To run OpenSearch Dashboards BWC tests. - -```bash -./test.sh bwc-test manifests/1.3.0/opensearch-dashboards-1.3.0-test.yml --paths -opensearch-dashboards=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/1.3.5/4056/linux/x64/tar -``` - -#### BWC tests on distribution bundle level - -The BWC tests running on distribution level are using the same framework from OpenSearch. The test cluster is spin up with the `latest` distribution bundle of provided version exclusively when the project is initialized with property `-PcustomDistributionDownloadType=bundle`. In this repo, the test workflow will be enable this gradle property by default.[BWC test script](https://github.com/opensearch-project/opensearch-build/blob/edffab782abf96390b3993ccc92425c63ab77884/scripts/default/bwctest.sh#L38) - -Example distribution bundle URL: `https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.2/latest/linux/x64/tar/dist/opensearch/opensearch-1.3.2-linux-x64.tar.gz` -This feature for BWC testing is supported for distribution versions starting `v1.3.2`. - -On CI level for plugins, security certificates need to be manually imported when spinning up the test cluster as security plugin is included in the distribution bundle. When upgrading the version within the test cluster, `nextNodeToNextVersion` is used for a single node upgrade and `goToNextVersion` is for a full restart upgrade. - -See [anomaly-detection#766](https://github.com/opensearch-project/anomaly-detection/pull/766) or [observability#1366](https://github.com/opensearch-project/observability/pull/1366) for more information. - -### Performance Tests - -TODO: Add instructions for running performance tests with `test.sh` - - -Performance tests from `test.sh` are executed using an internal service which automatically provisions hosts that run [OpenSearch Benchmark](https://github.com/opensearch-project/OpenSearch-Benchmark). Work to open source these internal features is being tracked in [opensearch-benchmark#97](https://github.com/opensearch-project/opensearch-benchmark/issues/97). - -Comparable performance data can be generated by directly using OpenSearch Benchmark, assuming that the same cluster and workload setups are used. More details on the performance testing configuration used for the nightly runs can be found in [OpenSearch#2461](https://github.com/opensearch-project/OpenSearch/issues/2461). - -In addition to the standard performance tests that run on the order of hours, longevity tests are run which load to a cluster for days or weeks. These tests are meant to validate cluster stability over a longer timeframe. -Longevity tests are also executed using OpenSearch Benchmark, using a modified version of the [nyc_taxis workload](https://github.com/opensearch-project/opensearch-benchmark-workloads/tree/main/nyc_taxis) that repeats the schedule for hundreds of iterations. - -#### Identifying Regressions in Performance Tests - -Before trying to identify a performance regression a set of baseline tests should be run, in order to establish expected values for performance metrics and to understand the variance between tests for the same configuration. Performance regressions are primarily determined based on decreased indexing throughput and/or increased query latency. -There is some amount of variance expected between any two tests. Empirically it has been found that generally tests for the same configuration can differ by about 5% of the mean for average indexing throughput and by about 10% of the mean for p90 or p99 query latency. Note that these values may vary depending on the underlying hardware of the cluster and the workload being used. - -If performance metrics for a certain testing configuration consistently fall outside the range created by the expected value for a metric +/- the standard deviation for the metric in the baseline tests then there is likely a performance regression. - -The nightly performance runs use the nyc_taxis workload with 2 warmup and 3 test iterations; tests using this configuration can also use the particular values defined in [this section](#identifying-regressions-in-nightly-performance-tests) for identifying performance regression. - -Additionally, error rates can be indicative of a performance regression. Error rates on the order of 0.01% are acceptable, though higher values are cause for concern. High error rates may point to issues with cluster availability or a change in the logic for processing a specific operation. -For tests using OpenSearch Benchmark with an external OpenSearch cluster configured as the data store, more details on the cause of the errors can be found by searching for the test execution ID in the `benchmark-metrics-*` index of the metrics data store. - - -##### Identifying Regressions in Nightly Performance Tests - -Using the aggregate results from the nightly performance test runs, compare indexing and query metrics to the specifications laid out in the table below. -The data for this table came from tests using OpenSearch 1.2 build #762 ([arm64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.4/762/linux/arm64/dist/opensearch/manifest.yml -)/[x64](https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.4/762/linux/x64/dist/opensearch/manifest.yml)), more details on the test setup can be found in [OpenSearch#2461](https://github.com/opensearch-project/OpenSearch/issues/2461). - -Please keep in mind the following: - -1. Changing the number of iterations or the workload type for a test can drastically change performance characteristics. This table is not necessarily applicable to other workload configurations. -2. StDev% Mean is the standard deviation as a percentage of the mean. It is expected that metrics for a test will be +/- this value relative to the expected value. If the average of several tests consistently falls outside this bound relative to the expected value there may be a performance regression (or improvement). -3. MinMax% Diff is the worst case variance between any two tests with the same configuration. If there is a difference greater than this value than there is likely a performance regression or an issue with the test setup. In general, comparing one off test runs should be avoided if possible. - - -|Instance Type|Security|Expected Indexing Throughput Avg (req/s)|Expected Indexing Error Rate|Indexing StDev% Mean|Indexing MinMax% Diff|Expected Query Latency p90 (ms)|Expected Query Latency p99 (ms)|Expected Query Error Rate|Query StDev% Mean|Query MinMax% Diff| -|---|---|---|---|---|---|---|---|---|---|---| -|m5.xlarge|Enabled|30554|0|~5%|~12%|431|449|0|~10%|~23%| -|m5.xlarge|Disabled|34472|0|~5%|~15%|418|444|0|~10%|~25%| -|m6g.xlarge|Enabled|38625|0|~3%|~8%|497|512|0|~8%|~23| -|m6g.xlarge|Disabled|45447|0|~2%|~3%|470|480|0|~5%|~15%| - -#### Identifying Issues in Longevity Tests - -Longevity tests are long running performance tests meant to measure the stability of a cluster over the course of several days or weeks. -Internal tools provide dashboards for monitoring cluster behavior during these tests. Use the following steps to spot issues in automated longevity tests: - -1. Navigate to the Jenkins build for a longevity test. -2. In the Console Output search for ``` INFO:root:Test can be monitored on ``` -3. Navigate to that link then click the link for "Live Dashboards" -4. Use the table below to monitor metrics for the test: - -|Metric|Health Indicators / Expected Values|Requires investigations / Cause for concerns| -|---|---|---| -|Memory|saw tooth graph|upward trends| -|CPU| |upward trends or rising towards 100%| -|Threadpool|0 rejections|any rejections| -|Indexing Throughput|Consistent rate during each test iteration|downward trends| -|Query Throughput|Varies based on the query being issued|downward trends between iterations| -|Indexing Latency|Consistent during each test iteration|upward trends| -|Query Latency|Varies based on the query being issued|upward trends| - -### Benchmarking Tests - -Runs benchmarking tests on a remote opensource OpenSearch cluster, uses [OpenSearch Benchmark](https://github.com/opensearch-project/OpenSearch-Benchmark). -At a high-level the benchmarking test workflow uses [opensearch-cluster-cdk](https://github.com/opensearch-project/opensearch-cluster-cdk.git) to first set-up an OpenSearch cluster (single/multi-node) and then executes `opensearch-benchmark` to run benchmark test against that cluster. The performance metric that opensearch-benchmark generates during the run are ingested into another OS cluster for further analysis and dashboarding purpose. - -The benchmarking tests will be run nightly and if you have a feature in any released/un-released OpenSearch version that you want to benchmark periodically please create an issue and the team will reach out to you. In case you want to run the benchmarking test locally you can use `opensearch-cluster-cdk` repo to spin up an OS cluster in your personal AWS account and then use `opensearch-benchmark` to run performance test against it. The detailed instructions are available on respective GitHub repositories. - -#### Onboard feature for nightly benchmark runs: - -1. Checkout [opensearch-build](https://github.com/opensearch-project/opensearch-build) repo and open `jenkins/opensearch/benchmark-test.jenkinsfile` file. -2. You will then need add an entry in `parameterizedCron` section of the jenkinsfile. -3. The structure of the `parameterizedCron` section as follows: - 1. Schedule: `H * * *`, edit the `HOUR` section to any hour of the day, 0-24. `H` adds a jitter to the cron to make sure multiple crons are not started together. - 2. BUNDLE_MANIFEST_URL: The distribution manifest URL that contains the artifact details such as tar location, arch, build id, commit-id, etc. - 3. TEST_WORKLOAD: This could be any workload that [opensearch-benchmark-workload](https://github.com/opensearch-project/opensearch-benchmark-workloads) repo provides, if not provided `nyc-taxis` is used as default. - 4. SINGLE_NODE_CLUSTER: Values are `true/false`. Do you want to run the benchmark against a single-node cluster or multi-node. - 5. USE_50_PERCENT_HEAP: Values are `true/false`. Recommended to use 50 percent physical memory as heap. Keep this `true`. - 6. MIN_DISTRIBUTION: Values are `true/false`. If the `BUNDLE_MANIFEST_URL` you provided is for a min/snapshot distribution then set this as `true` else don't provide this parameter. - 7. ADDITIONAL_CONFIG: The configuration that needs to be added to `opensearch.yml` to enable your feature. - 8. USER_TAGS: The metadata that needs to be added to the benchmark metrics ingested in datastore, this helps filter out the metrics for each use-case. Mandatory tags are `run-type:nightly,segrep:,arch:,instance-type:,major-version:<3x|2x>,cluster-config:--` - 9. WORKLOAD_PARAMS: Additional parameters that need to be passed to opensearch-benchmark workload. - 10. To get more information on each parameter and explore more options please visit [here](https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/benchmark-test.jenkinsfile#L140-L247) - -Here's the sample entry for enabling nightly runs for `remote-store` feature -``` -H 9 * * * %BUNDLE_MANIFEST_URL=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.10.0/latest/linux/arm64/tar/dist/opensearch/manifest.yml;TEST_WORKLOAD=http_logs;SINGLE_NODE_CLUSTER=false;DATA_NODE_COUNT=3;USE_50_PERCENT_HEAP=true;ENABLE_REMOTE_STORE=true;CAPTURE_SEGMENT_REPLICATION_STAT=true;USER_TAGS=run-type:nightly,segrep:enabled-with-remote-store,arch:arm64,instance-type:r6g.xlarge,major-version:2x,cluster-config:arm64-r6g.xlarge-3-data-3-shards;ADDITIONAL_CONFIG=opensearch.experimental.feature.remote_store.enabled:true cluster.remote_store.enabled:true opensearch.experimental.feature.segment_replication_experimental.enabled:true cluster.indices.replication.strategy:SEGMENT;WORKLOAD_PARAMS={"number_of_replicas":"2","number_of_shards":"3"} -``` - -Once you have added the configuration in the jenkinsfile please raise the PR and opensearch-infra team will review it. - -## Testing in CI/CD - -The CI/CD infrastructure is divided into two main workflows - `build` and `test`. The `build` workflow automates the process to generate all OpenSearch and OpenSearch Dashboards artifacts, and provide them as distributions to the `test` workflow, which runs exhaustive testing on the artifacts based on the artifact type. The next section talks in detail about the test workflow. - -### Test Workflow (in development) - -The test workflow development is in progress. To see a previously-unfinished prototype design, see [#609](https://github.com/opensearch-project/opensearch-build/pull/609). - -The progress of this design is tracked in meta issue [#123](https://github.com/opensearch-project/opensearch-build/issues/123). - -### Component-Level Details - -#### test-orchestrator pipeline - -This pipeline is in development. To see a previously-unfinished prototype design, see [#423](https://github.com/opensearch-project/opensearch-build/pull/423), [#523](https://github.com/opensearch-project/opensearch-build/pull/523). - -The development of `test-orchestration-pipeline` is tracked by meta issue [#123](https://github.com/opensearch-project/opensearch-build/issues/123) - -#### integTest job - -It is a Jenkins job that runs integration tests on a build artifact. It reads the build artifact composition from the associated manifest files and spins up parallel, independent integrationTest runs for each component built inside the artifact. For instance, if the artifact is a full distribution, which has all OpenSearch plugins, the job will kick off integration test suite for each individual plugin. Each plugin integration tests would run against a dedicated single node cluster, which is created from the built artifact. Once all integration tests complete, this job publishes the test results to an S3 bucket. - -See the integration test [configuration file](jenkins/opensearch/integ-test.jenkinsfile) and related [jenkins job](https://build.ci.opensearch.org/job/integ-test/) - -The development of `integTest` job is tracked by meta issue [#818](https://github.com/opensearch-project/opensearch-build/issues/818) - -#### bwcTest job - -It is a Jenkins job that runs bwc tests on the current version and compatible bwc versions of the artifact. OpenSearch core and each plugin would have their backwards compatibility tests and a `bwctest.sh` which will be used to trigger the bwc tests. Currently, only core and anomaly-detection bwc tests run and the other plugins can be added once they have their bwc tests ready. For example, for anomaly-detection, the job currently runs bwc tests for current version 1.1.0.0 and bwc version 1.13.0.0. - -When the bwc test is triggered for a particular component, the tests set up their own cluster and test the required functionalities in the upgrade paths, for the example above, a multi-node cluster starts with bwc versions of OpenSearch and AD installed on it, one or more nodes are upgraded to the current version of OpenSearch and AD installed on it and backwards compatibility is tested. The plugins would add tests for all bwc versions (similar to OpenSearch core) and they can be triggered from the bwcTest job. - -See the bwc test [configuration file](jenkins/opensearch/bwc-test.jenkinsfile) and related [jenkins job](https://build.ci.opensearch.org/job/bwc-test/) - -The development of the bwc test automation is tracked by meta issue [#90](https://github.com/opensearch-project/opensearch-build/issues/90). - -#### perfTest job - -It is a Jenkins job that runs performance tests on the bundled artifact using [OpenSearch Benchmark](https://github.com/opensearch-project/OpenSearch-Benchmark) (Mensor). It reads the bundle-manifest, config files and spins up a remote cluster with the bundled artifact installed on it. It will run performance test with and without security for specified architecture of the opensearch bundle. The job will kick off the single node cdk that sets up a remote cluster. It will then run the performance tests on those cluster using the mensor APIs from the whitelisted account and remote cluster endpoint(accessible to mensor system). These tests are bundle level tests. Any plugin on-boarding does not need to be a separate process. If the plugin is a part of the bundle, it is already onboarded. - -Once the performance tests completes (usually takes 5-8 hours for nyc_taxis track), it will report the test results and publish a human readable report in S3 bucket. - -See the performance test [configuration file](jenkins/opensearch/perf-test.jenkinsfile) and related [jenkins job](https://build.ci.opensearch.org/job/perf-test/) - -You can download the test results report using below url: - -``` -https://ci.opensearch.org/ci/dbc/perf-test///linux/x64/tar/test-results//perf-test//perf-test.html -``` -You can download the json format for above results using same url and replacing `.html` with `.json` - -Example: -https://ci.opensearch.org/ci/dbc/perf-test/1.3.6/6041/linux/x64/tar/test-results/678/perf-test/without-security/perf-test.html - -_Note: The without security test results might be not present for distribution that lacks the security plugin. As of now we only run performance tests on tarballs._ - -Conversion of Performance Test results to HTML file and JSON file: - -The result conversion of the perf-test is as follows- -After the performance test completes, it will report back the test results as well as the test-id. The results here will be in JSON format where they are converted into a tabular HTML string using a template library which is json2html. They will be then written to a HTML file where the file-name’s format is in .html. Along with generating a HTML file, a JSON file having the raw JSON data will also be generated whose filename’s format is in .json. The reason for creating a JSON file is to give the user the option to view both the JSON data in tabular format in the HTML file and the raw JSON data in the JSON file. The .html as well as .json file generated, will be stored in the path given by the user during the test-suite flow as a command-line args. These will be then taken and published in the S3 bucket. - -The development is tracked by meta issue [#126](https://github.com/opensearch-project/opensearch-build/issues/126) - -## Manifest Files - -Manifest files are configurations for a particular bundle. `test-workflow` uses three types of manifest files to run test suites. - -1. `test-manifest.yml` provides a list of test configurations to run against a given component in the bundle. An example of a configuration would be, integration test `index-management` plugin `with-security` and `without-security`. This manifest file serves as a support matrix config for the testing and should be updated by plugins if new components or test suites are to be added as part of the release workflow. See [here](manifests/1.3.0/opensearch-1.3.0-test.yml) -2. `build-manifest.yml` created by the build-workflow and provides a list of artifacts built as part of the build-workflow. It assists `test-workflow` pull the maven and build dependencies to run the test suites. -3. `bundle-manfest.yml` created by the build-workflow and provides a list of components packaged in a given bundle. It assists `test-workflow` to identify what components should be tested for a given bundle. - -## Dependency Management - -This section talks about how the `test-workflow` gets the dependencies required by plugins for running integration test suite (and will be extended to backward compatibility in future). There are two types of dependencies - - -1. `maven` - maven artifacts required by plugins -2. `build` - assembled zip artifacts required by plugins, e.g. job-scheduler zip required by index-management plugin. - -Plugins depend on a bunch of maven artifacts to successfully run integration tests. Normally the plugin build system pulls these maven artifacts from maven central repository. However, when testing on unreleased candidates, these maven dependencies are not yet available in maven central repo. - -In order to get around this issue, the instrumentation logic in `test-workflow` provides these dependencies into maven local repo, before kicking off the integration test for plugins. The test workflow installs these maven dependencies from the s3 bucket where the build-workflow publishes them during bundle creation phase. Once these dependencies are available in maven local, the plugin build system can use them to run integration tests. - -Similarly, some plugins have dependency on other plugins and require their zip artifacts for running integration tests. Example, `index-management` requires `job-scheduler zip` artifacts. These are referred to as build dependencies and are made available by the `test-workflow` to the plugins before the test is started. - -## S3 Permission Model - -This section defines how the permissions are configured for reading bundle, tarball, maven dependencies etc. from S3 and writing the results log back to s3 once the tests complete. - -The Jenkins infrastructure is setup in an AWS account via cdk. The account provides a `test-orchestrator-role` with permission policy to read and write from s3, see [[1] test-orchestrator-role policy](#appendix) . The `instance-profile` role has `assume-role` permissions on this `test-orchestrator role` which allows the jenkins instance to read and write from required s3 locations. - -## Appendix - -[1] test-orchestrator-role policy - -``` -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "s3:ListBucket" - ], - "Resource": [ - "" - ] - }, - { - "Effect": "Allow", - "Action": [ - "s3:GetObject" - ], - "Resource": [ - "/builds/*" - ] - }, - { - "Action": "s3:GetObject", - "Resource": [ - "/bundles/*" - ], - "Effect": "Allow" - }, - { - "Effect": "Allow", - "Action": "s3:PutObject", - "Resource": "/bundles/*/tests/*" - } - ] -} -``` diff --git a/src/validation_workflow/README.md b/wiki/src/validation-workflow.md similarity index 100% rename from src/validation_workflow/README.md rename to wiki/src/validation-workflow.md diff --git a/wiki/src/validation_workflow.md b/wiki/src/validation_workflow.md deleted file mode 100644 index 8286f51681..0000000000 --- a/wiki/src/validation_workflow.md +++ /dev/null @@ -1,16 +0,0 @@ -#### Automate the Validation during Releases. -The input requires a mandatory version number , optional --distribution types(tar,rpm,yum) and --platform type (currently uses only linux) to automatically download and verify the artifacts. - -*Usage* -``` -./validation.sh --version 2.3.0 --distribution rpm --platform linux -``` -The following options are available. - -| name | description | -|------------------------|---------------------------------------------------------------------| -| version | Accepts a mandatory version number. | -| -d, --distribution | Assigns the distribution type specifed by the user | -| -p, --platform | Determines the platform type of the atrifacts. | -| --verbose | Show more verbose output. | -