Skip to content

Commit 552cf9a

Browse files
hughsaundershughsaunders
committed
Initialise bash-lib repo
This commit adds the structure of the libraries and a few funcions. Most of the code in this commit is geared towards making sure this repo stays tested and documented in future. The following things are checked: * All functions are documented * All functions are tested * All scripts are linted See Readme for a list of the functions that are included, and for more information on testing with BATS.
1 parent 1cabfa5 commit 552cf9a

34 files changed

+1513
-2
lines changed

‎.gitmodules‎

Whitespace-only changes.

‎.gittrees‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Git Subtrees
2+
3+
# The advantage of subtrees is that users don't have to care about them - its
4+
# just a single repo. The disadvantage is that git doesn't track the metadata
5+
# as it does for submodules.
6+
7+
# This file provides an enumeration of the subtrees in this repo, and the URLs
8+
# they came from.
9+
10+
# subtree_path remote_url remote_name
11+
test-utils/bats https://github.com/bats-core/bats bats
12+
test-utils/bats-support https://github.com/ztombol/bats-support bats-support
13+
test-utils/bats-assert-1 https://github.com/jasonkarns/bats-assert-1 bats-assert-1

‎Jenkinsfile‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
#!/usr/bin/env groovy
2+
3+
pipeline {
4+
agent { label 'executor-v2' }
5+
6+
options {
7+
timestamps()
8+
buildDiscarder(logRotator(numToKeepStr: '30'))
9+
}
10+
11+
triggers {
12+
cron(getDailyCronString())
13+
}
14+
15+
environment {
16+
BATS_OUTPUT_FORMAT="junit"
17+
}
18+
19+
stages {
20+
stage('Bash Linting') {
21+
environment {
22+
BATS_SUITE="BATS-Lint"
23+
}
24+
steps {
25+
sh './tests-for-this-repo/run-bats-tests tests-for-this-repo/lint.bats'
26+
}
27+
}
28+
29+
stage('Python Linting') {
30+
steps {
31+
sh './tests-for-this-repo/python-lint'
32+
}
33+
}
34+
35+
stage('Bash Tests') {
36+
environment {
37+
BATS_SUITE="BATS-Tests"
38+
}
39+
steps {
40+
sh './tests-for-this-repo/run-bats-tests $(ls tests-for-this-repo/*.bats | grep -v lint.bats)'
41+
}
42+
}
43+
stage('Secrets Leak Check') {
44+
steps {
45+
sh './tests-for-this-repo/run-gitleaks'
46+
}
47+
}
48+
}
49+
50+
post {
51+
always {
52+
junit '*-junit.xml'
53+
cleanupAndNotify(currentBuild.currentResult)
54+
}
55+
}
56+
}

‎README.md‎

Lines changed: 201 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,201 @@
1-
# bashlib
2-
Common bash functions for use in test pipelines
1+
# bash-lib
2+
```
3+
_______________ _______________
4+
.' .' .|
5+
.' .' .' |
6+
.'_______________.'______________ .' |
7+
| ___ _____ ___ || ___ _____ ___ | |
8+
||_=_|__=__|_=_||||_=_|__=__|_=_|| |
9+
______||_____===_____||||_____===_____|| | __________
10+
.' ||_____===_____||||_____===_____|| .' .'|
11+
.' ||_____===_____||||_____===_____|| .' .' |
12+
.'___________|_______________||_______________|.'__________.' |
13+
|.----------.|.-----___-----.||.-----___-----.|| |_____.----------.
14+
|] |||_____________||||_____________||| .' [ |
15+
|| ||.-----___-----.||.-----___-----.||.' | |
16+
|| |||_____________||||_____________|||==========| |
17+
|| ||.-----___-----.||.-----___-----.|| |_____| |
18+
|] o|||_____________||||_____________||| .' [ 'o|
19+
|| ||.-----___-----.||.-----___-----.||.' | |
20+
|| ||| ||||_____________|||==========| |
21+
|| ||| |||.-----___-----.|| |_____| |
22+
|] ||| |||| ||| .' [ |
23+
||__________|||_____________||||_____________|||.'________|__________|
24+
''----------'''------------------------------'''----------''
25+
(o)LGB (o)
26+
```
27+
28+
The place to store functions that are used in pipelines for multiple repos.
29+
30+
Please add whatever is useful to you, but keep it tidy so its still useful to everyone else :)
31+
32+
## Usage
33+
34+
Add bash-lib into your project in the way that best fits your workflow. The only requirement is that you **pin the version of
35+
bash-lib that you use**. This is important so that changes to bash-lib do not have the power to break all projects that use
36+
bash-lib. Your project can then test updates to bash-lib and roll forward periodicly.
37+
38+
Options:
39+
* Add a submodule: they are an easy way to integrate bash-lib and automatically use a single SHA until manually updated. Submodules add a pointer from a mount point in your repo to the external repo (bash-lib), and require workflow changes to ensure that pointer is derferenced during clone, checkout and some other opertaions.
40+
* Add a subtree: This repo uses subtrees to pull in test dependencies. Various helpers for working with subtrees can be found in [git/lib](git/lib). Subtrees copy an external repo into a subdirectory of the host repo, no workflow changes are required. Tools are provided for updating the subtree. Subtrees naturally keep a single version of bash-lib until explicitly updated. Note that subtree merge commits do not rebase well :warning:, so best to keep subtree updates in separate PRs from normal commits.
41+
* Clone bash-lib in your deployment process, bash-lib doesn't have to be within your repo, just needs to be somewhere where your scripts can source [init](init). This is where it's most important that you implement a mechanism to always use the same SHA, as a **clone will track master by default, which is not an allowed use of bash-lib**.
42+
43+
Once you have bash-lib cloned in your project, you source two things:
44+
45+
1. Source `bash-lib/init`. This ensures submodules are initalised and sets the BASH_LIB_DIR env var to the absolute path to the bash-lib dir. This makes it easy to source libraries from other scripts.
46+
2. Source `${BASH_LIB_DIR}/lib-name/lib` for any libraries you are interested in.
47+
48+
You are now ready to use bash-lib functions :)
49+
50+
## Structure
51+
The `/init` script sets up everything required to use the library, most
52+
importantly the `BASH_LIB_DIR` variable which gives the absolute path to the root
53+
of the library and should be used for sourcing the modules.
54+
55+
The repo is organized into libraries, each library is a directory that has a
56+
lib file. Sourcing the lib for a library should expose all the functions
57+
that library offers. The lib file may source or reference other supporting
58+
files within it's directory.
59+
60+
```
61+
.
62+
├── libname
63+
│ ├── lib
64+
│ └── supporting-file
65+
├── init # init script, source this first
66+
├── run-tests # top level test script, executes all tests
67+
├── secrets.yml # secrets required for executing tests
68+
├── test-utils
69+
│ ├── bats # subtree
70+
│ ├── bats-assert-1 # subtree
71+
│ ├── bats-support # subtree
72+
│ ├── lib
73+
│ └── tap2junit
74+
└── tests-for-this-repo
75+
├── filehandling.bats
76+
├── fixtures #
77+
│ └── libname # Dir containing test fixtures for a library
78+
├── tap2junit
79+
├── libname.bats # contains tests for libname/lib
80+
├── python-lint # supporting files for python lint
81+
├── run-bats-tests # script to run bats tests
82+
├── run-gitleaks # script to check for leaked secrets
83+
└── run-python-lint # script to run python lint
84+
```
85+
## Style Guide
86+
Follow the [google shell style guide](https://google.github.io/styleguide/shell.xml#Naming_Conventions).
87+
TL;DR:
88+
1. Use snake_case function and variable names
89+
1. Use `function` when declaring functions.
90+
91+
92+
## Contents
93+
94+
<!-- html table due to markdown's lack of support for lists within tables -->
95+
<table>
96+
<thead>
97+
<tr>
98+
<th>Library</th>
99+
<th>Description</th>
100+
<th>Functions</th>
101+
</tr>
102+
</thead>
103+
<tbody>
104+
<tr>
105+
<td><a href="filehandling/lib">filehandling</a></td>
106+
<td>Functions relating to file and path handling
107+
<td>
108+
<ol>
109+
<li> <b>abs_path</b>: Ensure a path is absolute</li>
110+
</ol>
111+
</td>
112+
</tr>
113+
<tr>
114+
<td><a href="git/lib">git</a></td>
115+
<td>Git helpers</td>
116+
<td>
117+
<ol>
118+
<li><b>repo_root</b>: Find the root of the current git repo.</li>
119+
<li><b>all_files_in_repo</b>: List files tracked by git.</li>
120+
<li>tracked_files_excluding_subtrees: List files tracked by git, but excluding any files that are in paths listed in <code>.gittrees</code>.</li>
121+
<li><b>cat_gittrees</b>: Returns the contents of .gittrees from the top level of the repo, excluding any comments. Fails if .gittrees is not present.</li>
122+
<li><b>ensure_gittrees</b>: Creates git trees with header comments if it doesn't already exist.</li>
123+
<li><b>add_subtree</b>: Adds a subtree to the current repo, and stores the path, remote_url and remote_name in .gittrees so that this subtree can be maniuplated in future.</li>
124+
<li><b>add_gittree_remotes</b>: Read .gittrees and add git remotes for each remote_url and remote_name pair.</li>
125+
<li><b>update_subtree</b>: Changes a subtree to reflect a different ref in the upstream repo. Checks if the new ref is the same as the current ref before proceeding.</li>
126+
<li><b>remote_latest_tag</b>: Returns the symbolic name of the latest tag from a remote.</li>
127+
<li><b>remote_latest_tagged_commit</b>: Returns the SHA of the most recently tagged commit in a remote repo (<code>tag^{}</code>).</li>
128+
<li><b>remote_sha_for_ref</b>: Returns the SHA for a given ref from a named remote.</li>
129+
<li><b>remote_tag_for_sha</b>: Returns the tag corresponding to a SHA from a named remote - if there is one.</li>
130+
<li><b>update_subtrees_to_latest_tag</b>: Read .gittrees and update all listed subtrees to the newest tag from their upstream repo.</li>
131+
</ol>
132+
</td>
133+
</tr>
134+
<td><a href="helpers/lib">helpers</a></td>
135+
<td>Bash scripting helpers</td>
136+
<td>
137+
<ol>
138+
<li><b>die</b>: print message and exit 1</li>
139+
<li><b>spushd/spopd</b>: Safe verisons of pushd & popd that call die if the push/pop fails, they also drop stdout. </li>
140+
</ol>
141+
</td>
142+
</tr>
143+
<tr>
144+
<td><a href="k8s/lib">k8s</a></td>
145+
<td>Utils for connecting to K8s</td>
146+
<td>
147+
<ol>
148+
<li><b>build_gke_image</b>: Build docker image for running kubectl commands against GKE.</li>
149+
<li><b>delete_gke_image</b>: Delete image from GKE.</li>
150+
<li><b>run_docker_gke_command</b>: Run command in gke-utils container, already authenticated to k8s cluster.</li>
151+
</ol>
152+
</td>
153+
</tr>
154+
<tr>
155+
<td><a href="logging/lib">logging</a></td>
156+
<td>Helpers related to login</td>
157+
<td>
158+
<ol>
159+
<li><b>announce</b>: Echo message in ascii banner to distinguish it from other log messages.</li>
160+
</ol>
161+
</td>
162+
</tr>
163+
<tr>
164+
<td><a href="test-utils/lib">test-utils</a></td>
165+
<td>Helpers for executing tests</td>
166+
<td>
167+
<ol>
168+
<li><b>shellcheck_script</b>: Execute shellcheck against a script, uses docker.</li>
169+
<li><b>find_scripts</b>: Find git tracked files with extension.</li>
170+
<li><b>tap2junit</b>: Convert a subset of <a href="http://testanything.org/">TAP</a> to JUnit XML. Retains logs for errors.</li>
171+
</ol>
172+
</td>
173+
</tr>
174+
</tbody>
175+
</table>
176+
177+
## Testing
178+
Tests are written using [BATS](https://github.com/bats-core/bats). Each libould have a `lib-name.bats` file in [tests-for-this-repo](/tests-for-this-repo).
179+
Asserts are provided by [bats-assert-1](https://github.com/jasonkarns/bats-assert-1). The value in these is that they provide useful debugging output when the assertion fails, eg expected x got y.
180+
181+
Example:
182+
```bash
183+
# source support and assert libraries
184+
. "${BASH_LIB_DIR}/test-utils/bats-support/load.bash"
185+
. "${BASH_LIB_DIR}/test-utils/bats-assert-1/load.bash"
186+
187+
# source the library under test
188+
. "${BASH_LIB_DIR}/git/lib"
189+
190+
# define a test that calls a library function
191+
@test "it does the thing" {
192+
some_prep_work
193+
# run is a wrapper that catches failures so that assertsions can be run,
194+
# otherwise the test would immediately fail.
195+
run does_the_thing
196+
assert_success
197+
assert_output "thing done"
198+
}
199+
```
200+
201+
Test fixtures should go in /tests-for-this-repo/[fixtures](tests-for-this-repo/fixtures)/lib-name.

‎filehandling/lib‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
#!/bin/bash
2+
3+
: "${BASH_LIB_DIR:?BASH_LIB_DIR must be set. Please source bash-lib/init before other scripts from bash-lib.}"
4+
. "${BASH_LIB_DIR}/helpers/lib"
5+
6+
#https://stackoverflow.com/a/23002317
7+
function abs_path() {
8+
# generate absolute path from relative path
9+
# $1 : relative filename
10+
# return : absolute path
11+
if [ -d "$1" ]; then
12+
# dir
13+
(spushd "$1"; pwd)
14+
elif [ -f "$1" ]; then
15+
# file
16+
if [[ $1 = /* ]]; then
17+
echo "$1"
18+
elif [[ $1 == */* ]]; then
19+
echo "$(spushd "${1%/*}"; pwd)/${1##*/}"
20+
else
21+
echo "$(pwd)/$1"
22+
fi
23+
fi
24+
}

0 commit comments

Comments
 (0)