-
Notifications
You must be signed in to change notification settings - Fork 37
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
106 changed files
with
7,061 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
name: Havoc Lint | ||
on: | ||
push: | ||
permissions: | ||
contents: read | ||
jobs: | ||
golangci: | ||
defaults: | ||
run: | ||
working-directory: havoc | ||
name: lint | ||
runs-on: ubuntu-latest | ||
steps: | ||
- uses: actions/checkout@v3 | ||
- uses: actions/setup-go@v4 | ||
with: | ||
go-version: '1.20' | ||
cache: true | ||
- name: golangci-lint | ||
uses: golangci/golangci-lint-action@v3 | ||
with: | ||
version: v1.51.2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
on: | ||
release: | ||
types: [created] | ||
|
||
permissions: | ||
contents: write | ||
packages: write | ||
|
||
jobs: | ||
release-linux-amd64: | ||
defaults: | ||
run: | ||
working-directory: havoc | ||
name: release linux/amd64 | ||
runs-on: ubuntu-latest | ||
steps: | ||
- uses: actions/checkout@v3 | ||
- uses: wangyoucao577/go-release-action@v1 | ||
with: | ||
github_token: ${{ secrets.GITHUB_TOKEN }} | ||
goos: linux | ||
goarch: amd64 | ||
project_path: cmd | ||
release-linux-arm64: | ||
defaults: | ||
run: | ||
working-directory: havoc | ||
name: release linux/arm64 | ||
runs-on: ubuntu-latest | ||
steps: | ||
- uses: actions/checkout@v3 | ||
- uses: wangyoucao577/go-release-action@v1 | ||
with: | ||
github_token: ${{ secrets.GITHUB_TOKEN }} | ||
goos: darwin | ||
goarch: arm64 | ||
project_path: cmd |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
name: CLI tests | ||
on: | ||
push: | ||
concurrency: | ||
group: ${{ github.workflow }}-${{ github.ref }} | ||
cancel-in-progress: true | ||
jobs: | ||
test: | ||
defaults: | ||
run: | ||
working-directory: havoc | ||
runs-on: ubuntu-latest | ||
steps: | ||
- uses: actions/checkout@v3 | ||
- name: Setup Go | ||
uses: actions/setup-go@v3 | ||
with: | ||
go-version-file: 'go.mod' | ||
cache: true | ||
- name: Run tests | ||
run: | | ||
make test |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -7,3 +7,4 @@ k8s-test-runner/chart/**/*.yaml | |
node_modules/ | ||
index.yaml | ||
wasp/** | ||
havoc/testdata/** |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
# IDE and environment | ||
.idea/ | ||
.vscode/ | ||
.DS_STORE | ||
|
||
# Binaries for programs and plugins | ||
*.exe | ||
*.exe~ | ||
*.dll | ||
*.so | ||
*.dylib | ||
|
||
# Test binary, built with `go test -c` | ||
*.test | ||
|
||
# Output of the go coverage tool, specifically when used with LiteIDE | ||
*.out | ||
|
||
# Dependency directories | ||
dist/ | ||
vendor/ | ||
node_modules/ | ||
.yarn/ | ||
|
||
# Mercuy server configuration file | ||
config.toml | ||
config.*.toml | ||
|
||
# Personal/secret env vars | ||
.envrc-personal | ||
|
||
# Other | ||
tmp/ | ||
*.log | ||
*.swp | ||
.air.toml | ||
.DS_Store | ||
output.txt | ||
|
||
# Default binary name with go build | ||
main | ||
test.sh | ||
|
||
# Default generated experiments | ||
havoc-experiments | ||
havoc-default | ||
experiments-crib-core | ||
testdata/experiments-test | ||
havoc-monkey-temp-dir | ||
|
||
# General env vars config | ||
.envrc | ||
|
||
# Dumps of configs | ||
config_dump.toml | ||
pods_dump.json | ||
testdata/result/** | ||
testdata/results/** | ||
!testdata/results/.gitkeep |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
The MIT License (MIT) | ||
|
||
Copyright (c) 2024 SmartContract ChainLink, Ltd. | ||
|
||
Permission is hereby granted, free of charge, to any person obtaining a copy | ||
of this software and associated documentation files (the "Software"), to deal | ||
in the Software without restriction, including without limitation the rights | ||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
copies of the Software, and to permit persons to whom the Software is | ||
furnished to do so, subject to the following conditions: | ||
|
||
The above copyright notice and this permission notice shall be included in | ||
all copies or substantial portions of the Software. | ||
|
||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN | ||
THE SOFTWARE. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
.PHONY: test | ||
test: | ||
go test -v -count 1 `go list ./... | grep -v examples` -run TestSmoke | ||
|
||
.PHONY: test_race | ||
test_race: | ||
go test -v -race -count 1 `go list ./... | grep -v examples` -run TestSmoke | ||
|
||
.PHONY: test+cover | ||
test_cover: | ||
go test -v -coverprofile cover.out -count 1 `go list ./... | grep -v examples` -run "TestSmoke|TestAPI" | ||
go tool cover -html cover.out | ||
|
||
.PHONY: install | ||
install: | ||
go install cmd/havoc.go | ||
|
||
.PHONE: build | ||
build: | ||
go build cmd/havoc.go | ||
|
||
.PHONY: lint | ||
lint: | ||
golangci-lint --color=always run -v |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,171 @@ | ||
## Havoc | ||
|
||
_DISCLAIMER_: This software is not even early Alpha, and still in development, use it on your own risk | ||
|
||
Havoc is a tool that introspects your k8s namespace and generates a `ChaosMesh` CRDs suite for you | ||
|
||
You can use havoc as a CLI to quickly test hypothesis or run it in "monkey" mode with your load tests and have Grafana annotations | ||
|
||
### How it works | ||
|
||
![img.png](img.png) | ||
|
||
Havoc generates groups of experiments based on your pods and labels found in namespace | ||
|
||
In order to test your namespace you need to label pods accordingly: | ||
|
||
- `havoc-component-group` is like `app.kubernetes.io/component` (see [recommendation](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/)) but should be set explicitly | ||
- `havoc-network-group` in most cases match `havoc-component-group` but sometimes you need to break network even inside component group, ex. distributed databases | ||
|
||
Example: | ||
|
||
``` | ||
havoc-component-group: node | ||
havoc-network-group: nodes-1 | ||
``` | ||
|
||
Every pod without a group will be marked as `no-group` and experiments will be assigned accordingly | ||
|
||
Single pod experiments: | ||
|
||
- PodFailure | ||
- NetworkChaos (Pod latency) | ||
- Stress (Memory) | ||
- Stress (CPU) | ||
- External service failure (Network partition) | ||
- Blockchain specific experiments | ||
|
||
Group experiments: | ||
|
||
- Group failure | ||
- Group latency | ||
- Group CPU | ||
- Group memory | ||
- Group network partition | ||
- OpenAPI based HTTP experiments | ||
|
||
You can generate default chaos suite by [configuring](havoc.toml) havoc then set `dir` param and add your custom experiments, then run monkey to test your services | ||
|
||
### Why use it? | ||
|
||
#### Without Havoc your workflow is | ||
|
||
- Inspect full rendered deployment of your namespace | ||
- Figure out multiple groups of components you can select by various labels or annotations to form experiments | ||
- If some components are not selectable - ask DevOps guys to change the manifests | ||
- Create set of experiments for each chaos experiment type by hand or copy from other product chaos tests | ||
- Calculate permutations of different groups and calculate composite experiments (network partitioning, latency) | ||
- Create experiment for each API in every OpenAPI spec | ||
- Compose huge ChaosMesh Workflow YAML that fails without proper validation errors if group has no match or label is invalid | ||
- Run the load test, then manually run the chaos suite | ||
- Check experiment logs to debug with kubectl | ||
- Figure out which failures are caused by which experiments | ||
- If you have more than one project, use some templating make experiments work for other projects | ||
|
||
#### With Havoc | ||
|
||
- Have a simple labelling convention for your namespaces, fill 5 vars in `TOML` config | ||
- Run chaos testing with `havoc -c havoc.toml run ${namespace}` | ||
|
||
### Install | ||
|
||
Please use GitHub releases of this repo | ||
Download latest [release](https://github.com/smartcontractkit/havoc/releases) | ||
|
||
You need `kubectl` to available on your machine | ||
|
||
You also need [ChaosMesh](https://chaos-mesh.org/) installed in your `k8s` cluster | ||
|
||
### Grafana integration | ||
|
||
Set env variables | ||
|
||
``` | ||
HAVOC_LOG_LEVEL={warn,info,debug,trace} | ||
GRAFANA_URL="..." | ||
GRAFANA_TOKEN="..." | ||
``` | ||
|
||
Set dashboard names in `havoc.toml` | ||
|
||
``` | ||
[havoc.grafana] | ||
# UIDs of dashboard which should be annotated with chaos experiments metadata | ||
# You can also try to use name as you see it in the top bar of your dashboard but that's not guaranteed to match | ||
dashboard_uids = ["WaspDebug", "e98b5451-12dc-4a8b-9576-2c0b67ddbd0c"] | ||
``` | ||
|
||
### Manual usage | ||
|
||
Generate default experiments for your namespace | ||
|
||
``` | ||
havoc -c havoc.toml generate [namespace] | ||
``` | ||
|
||
Check this [section](havoc.toml) for `ignore_pods` and `ignore_group_labels`, default settings should be reasonable, however, you can tweak them | ||
|
||
This will create `havoc-experiments` dir, then you can choose from recommended experiments | ||
|
||
``` | ||
havoc -c havoc.toml apply | ||
``` | ||
|
||
You can also apply your experiment directly, using absolute or relative path to experiment file | ||
|
||
``` | ||
havoc -c havoc.toml apply ${experiment_file_path} | ||
``` | ||
|
||
### Monkey mode | ||
|
||
You can run havoc as an automated sequential or randomized suite | ||
|
||
``` | ||
havoc -c havoc.toml run [namespace] | ||
``` | ||
|
||
See `[havoc.monkey]` config [here](havoc.toml) | ||
|
||
### Programmatic usage | ||
|
||
See how you can use recommended experiments from code in [examples](examples) | ||
|
||
### Custom experiments | ||
|
||
Havoc is just a generator and a module that reads your `dir = $mydir` from config | ||
|
||
If you wish to add custom experiments written by hand create your custom directory and add experiments | ||
|
||
Experiments will be executed in lexicographic order, however, for custom experiments there are 2 simple rules: | ||
|
||
- directory names must be in | ||
|
||
``` | ||
"external", | ||
"failure", | ||
"latency", | ||
"cpu", | ||
"memory", | ||
"group-failure", | ||
"group-latency", | ||
"group-cpu", | ||
"group-memory", | ||
"group-partition", | ||
"blockchain_rewind_head", | ||
"http" | ||
``` | ||
|
||
- `metadata.name` should be equal to your experiment filename | ||
|
||
When you are using `run` monkey command, if directory is not empty havoc won't automatically generate experiments, so you can extend generated experiments with your custom modifications | ||
|
||
### Developing | ||
|
||
We are using [nix](https://nixos.org/) | ||
|
||
Enter the shell | ||
|
||
``` | ||
nix develop | ||
``` |
Oops, something went wrong.