Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use asciidoc for the doc to remain readable #36

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
.env*
!.env-sample
tests
.github
.git
Expand Down
7 changes: 7 additions & 0 deletions .env-sample
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_DEFAULT_REGION=us-east-1
AWS_S3_BUCKET=...
AWS_BUCKET="${AWS_S3_BUCKET}"

HTTP=https://${AWS_S3_BUCKET}.s3.${AWS_DEFAULT_REGION}.amazonaws.com
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
.env
.env*
!.env-sample
/tests/.cleanup-pids
/tests/.snapshots-overwrite
/tests/.aws-destinations
39 changes: 39 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
ARG REGISTRY_PATH=docker.io/paritytech

FROM docker.io/library/ubuntu:latest

ARG VCS_REF=master
ARG BUILD_DATE=""
ARG UID=1000
ARG GID=1000
ARG VERSION=0.0.1

LABEL summary="Releng scripts" \
name="${REGISTRY_PATH}/gnupg" \
maintainer="devops-team@parity.io" \
version="${VERSION}" \
description="Releng scripts" \
io.parity.image.vendor="Parity Technologies" \
io.parity.image.source="https://github.com/paritytech/scripts/blob/${VCS_REF}/dockerfiles/releng-scripts/Dockerfile" \
io.parity.image.documentation="https://github.com/paritytech/scripts/blob/${VCS_REF}/dockerfiles/releng-scripts/README.md" \
io.parity.image.revision="${VCS_REF}" \
io.parity.image.created="${BUILD_DATE}"

RUN apt-get update && apt-get install -yq --no-install-recommends ca-certificates bash jq unzip curl && \
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "/tmp/awscliv2.zip" && \
unzip "/tmp/awscliv2.zip" && rm "/tmp/awscliv2.zip" && \
/aws/install && rm -rf /aws && \
apt -yq remove ca-certificates unzip && apt -yq autoremove && \
aws --version

WORKDIR /scripts

COPY . .

RUN set -x \
&& groupadd -g $GID nonroot \
&& useradd -u $UID -g $GID -s /bin/bash -m nonroot

USER nonroot:nonroot

ENTRYPOINT [ "./rs" ]
182 changes: 98 additions & 84 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,90 +1,106 @@
# TOC

- [Introduction](#introduction)
- [Reserving crates](#reserving-crates)
- [Usage](#usage)
- [GitHub Actions](#usage-github-actions)
- [GitLab Jobs](#usage-gitlab-jobs)
- [Locally](#usage-locally)
- [Development](#development)
- [Repository structure](#development-repository-structure)
- [External consumption](#development-repository-structure-external-consumption)
- [Tools](#development-repository-structure-tools)
- [Maintenance](#development-repository-structure-maintenance)
- [Tests](#development-tests)
- [Linters](#development-linters)
- [S3](#development-s3)

# Introduction <a name="introduction"></a>

This repository contains scripts managed and used by
[release-engineering](https://github.com/orgs/paritytech/teams/release-engineering).
# Introduction

This repository contains scripts managed and used by [release-engineering](https://github.com/orgs/paritytech/teams/release-engineering).

See the [Tools wiki page](https://github.com/paritytech/releng-scripts/wiki/Tools#TOC) for information on the functionality provided through this repository.

# Reserving crates <a name="reserving-crates"></a>
# rs

The commands offerered by `rs` can be access via script, GHS, Docker, etc..
Those use cases are described in the documentation.

The following chapters explain what those commands **are** and how to use them.

You may find convenient testing using:

alias rs='docker run --rm -it -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION -e AWS_BUCKET paritytech/releng-scripts'

# `rs version`

Get the version:

docker run --rm -it paritytech/releng-scripts version

output:

0.0.1

# `rs upload`

Uplooad an artifact.

Check the help with `rs upload --help`

rs upload --bucket $AWS_BUCKET custom foo/bar s3 tests/fixtures/foo.txt

1. Go to https://github.com/paritytech/releng-scripts/actions/workflows/reserve-crate.yml
2. Click the "Run workflow" button to access the workflow's form
3. Fill and send the workflow's form. After that a workflow run
([example](https://github.com/paritytech/releng-scripts/actions/runs/3642900863/attempts/1))
will be created; you might need to refresh the page in order to see it.
4. Wait for the workflow run to finish
# `rs download`

# Usage <a name="usage"></a>
Download an artifact.

# `rs delete`

Delete an artifact.

# Reserving crates

1. Go to <https://github.com/paritytech/releng-scripts/actions/workflows/reserve-crate.yml>

2. Click the "Run workflow" button to access the workflow’s form

3. Fill and send the workflow’s form. After that a workflow run
([example](https://github.com/paritytech/releng-scripts/actions/runs/3642900863/attempts/1))
will be created; you might need to refresh the page in order to see it.

4. Wait for the workflow run to finish

# Usage

## Docker

The image is available as `paritytech/releng-scripts` and usable as:

```bash
# Show the help
docker run --rm -it paritytech/releng-scripts
# Show the version
docker run --rm -it paritytech/releng-scripts version
```
# Show the help
docker run --rm -it paritytech/releng-scripts
# Show the version
docker run --rm -it paritytech/releng-scripts version

## GitHub Actions <a name="usage-github-actions"></a>

```yaml
jobs:
upload-artifact:
name: My workflow
runs-on: ubuntu-latest
steps:
- name: First step
run: |
git clone --depth=1 https://github.com/paritytech/releng-scripts
./releng-scripts/foo ...
```
## GitHub Actions

## GitLab Jobs <a name="usage-gitlab-jobs"></a>
jobs:
upload-artifact:
name: My workflow
runs-on: ubuntu-latest
steps:
- name: First step
run: |
git clone --depth=1 https://github.com/paritytech/releng-scripts
./releng-scripts/foo ...

```yaml
job:
script:
- git clone --depth=1 https://github.com/paritytech/releng-scripts
- ./releng-scripts/foo ...
```
## GitLab Jobs

## Locally <a name="usage-locally"></a>
job:
script:
- git clone --depth=1 https://github.com/paritytech/releng-scripts
- ./releng-scripts/foo ...

## Locally

Clone this repository and run the scripts

# Development <a name="development"></a>
# Contributing

## Repository structure <a name="development-repository-structure"></a>
## Repository structure

### External consumption <a name="development-repository-structure-external-consumption"></a>
### External consumption

If a script is meant for external consumption, such as the tools' entrypoints,
then avoid adding file extensions to them since that's more subject to breaking
changes in case we want to change the script's programming language later.
then avoid adding file extensions to them since thats more subject to breaking
changes in case we want to change the scripts programming language later.
Adding the extension is encouraged for files which are not meant for external
consumption, i.e. scripts which are used only internally or are run through some
command runner such as `just`.

Here's an example:
Heres an example:

```
/repository
Expand All @@ -99,24 +115,24 @@ include an extension. On the other hand, `upload.sh`, which corresponds to the
`upload` subcommand of `releng-scripts`, can keep its extension because it's not meant for
external consumption, as it's invoked by `releng-scripts`.

### Tools <a name="development-repository-structure-tools"></a>
### Tools

Tools are organized with the following hierarchy:

- Their entrypoints are located at the root of the repository for
ease-of-external-consumption's sake.
- Their entrypoints are located at the root of the repository for
ease-of-external-consumptions sake.

Avoid including the extension to those files because that's more subject to
breaking changes in case we want to change the tool's programming language
later.
Avoid including the extension to those files because that's more subject to
breaking changes in case we want to change the tool's programming language
later.

Please maintain an entry to the tools in `./justfile` for ease-of-use's sake.
Please maintain an entry to the tools in `./justfile` for ease-of-use's sake.

- In case the tool has subcommands, they are located at `./cmd/$TOOL/$SUBCOMMAND`
- In case the tool has subcommands, they are located at `./cmd/$TOOL/$SUBCOMMAND`

This is to avoid noisy handling of too many commands within a single file.
This is to avoid noisy handling of too many commands within a single file.

Here's an example:
Heres an example:

```
/repository
Expand All @@ -129,37 +145,35 @@ Here's an example:
`releng-scripts` is the tool's entrypoint and `upload.sh` corresponds to the `upload`
subcommand of `releng-scripts`.

### Maintenance <a name="development-repository-structure-maintenance"></a>
### Maintenance

The `./tasks` directory groups scripts for tasks related to project maintenance,
such as running linters and tests.

Please maintain an entry to those scripts in `./justfile` for ease-of-use's sake.
Please maintain an entry to those scripts in `./justfile` for ease-of-uses sake.

## Tests <a name="development-tests"></a>
## Tests

Run the test: `just tests`

Update the snapshots: `just tests --update`

Delete stale snapshots: `just tests --delete-stale-snapshots`

## Linters <a name="development-linters"></a>
## Linters

`just linters`

## S3 <a name="development-s3"></a>
## S3

For testing out the S3 backend you can use https://github.com/adobe/S3Mock.
For testing out the S3 backend you can use [S3Mock](https://github.com/adobe/S3Mock).

Set up the following environment variables:

```
export AWS_ACCESS_KEY_ID=1234567890
export AWS_SECRET_ACCESS_KEY=valid-test-key-ref
export AWS_DEFAULT_REGION=us-east-1
export AWS_BUCKET=test
```
export AWS_ACCESS_KEY_ID=1234567890
export AWS_SECRET_ACCESS_KEY=valid-test-key-ref
export AWS_DEFAULT_REGION=us-east-1
export AWS_BUCKET=test

Then start S3Mock:

Expand Down
21 changes: 21 additions & 0 deletions README_src.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
:toc:

== Introduction

This repository contains scripts managed and used by https://github.com/orgs/paritytech/teams/release-engineering[release-engineering].

See the https://github.com/paritytech/releng-scripts/wiki/Tools#TOC[Tools wiki page] for information on the functionality provided through this repository.

include::doc/rs.adoc[]

include::doc/crate-reservation.adoc[leveloffset=+1]

include::doc/usage.adoc[leveloffset=+1]

include::doc/contributing.adoc[leveloffset=+1]

== License

----
include::LICENSE[]
----
8 changes: 6 additions & 2 deletions cmd/releng-scripts/upload.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
#!/usr/bin/env bash

# requirements:
# jq, aws

set -Eeu -o pipefail
shopt -s inherit_errexit

Expand Down Expand Up @@ -45,7 +48,7 @@ upload_to_s3() {
"${backend_upload_args[@]}"
"${forwarded_backend_args[@]}"
--
"$file"
"${file}"
"$destination"
)

Expand Down Expand Up @@ -83,6 +86,7 @@ upload_to_s3() {
;;
*)
log info "Checking metadata for destination $destination"
echo "$response"
if echo -n "$response" | jq -e; then
# Valid JSON response for metadata, which means that the file exists
log error "File already exists: $destination"
Expand Down Expand Up @@ -222,7 +226,7 @@ main() {
local remote_destination="${upload_dir:+$upload_dir/}$filename"

if [ ! "${DRY_RUN:-}" ]; then
log "Uploading file: $file"
log "Uploading file: $file to bucket $bucket"
log "Upload destination: $remote_destination"
fi

Expand Down
Loading
Loading