Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up builds by fixing github action cache and cargo build cache #332

Merged
merged 1 commit into from
Feb 4, 2022
Merged

Speed up builds by fixing github action cache and cargo build cache #332

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
224 changes: 133 additions & 91 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,22 @@ jobs:
container:
image: timescaledev/rust-pgx:latest
env:
# Must keep in sync with `path: target` in cache entry below
CARGO_TARGET_DIR_NAME: target
# TODO Why? Cargo default is to pass `-C incremental` to rustc; why don't we want that?
# https://doc.rust-lang.org/rustc/codegen-options/index.html#incremental
# Well turning it off takes the extension target size down from 3G to 2G...
CARGO_INCREMENTAL: 0
# TODO Why? If we're concerned about trouble fetching crates, why not
# just fetch them once at the time we select a dependency?
# Errors fetching crates are probably rare enough that we don't see the
# need to bother, but then why not just let the build fail?
CARGO_NET_RETRY: 10
# TODO What reads this? It's not listed on
# https://doc.rust-lang.org/cargo/reference/environment-variables.html
CI: 1
RUST_BACKTRACE: short
# TODO We don't seem to run rustup, nor does it seem like we should.
RUSTUP_MAX_RETRIES: 10

steps:
Expand All @@ -30,41 +42,55 @@ jobs:
with:
ref: ${{ github.event.pull_request.head.sha }}

# ~postgres/.pgx/config.toml comes in on the image.
# Setting HOME=~postgres (which is /home/postgres) in cargo's environment
# would work, but sticking with this approach for now.
- name: chown Repository
run: chown -R postgres .

# CARGO_HOME is /usr/local/cargo which looks like something we'd need to
# stay root to update, but the parts we need to update (.crates.toml,
# .crates2.json, .package-cache, git, registry) are already owned by
# `postgres` in the image.
- name: Cache cargo directories
uses: actions/cache@v2
with:
path: |
~/.cargo/registry
~/.cargo/git
key: ${{ runner.os }}-cargo-clippy-cargo
restore-keys: |
${{ runner.os }}-cargo-build-target-clippy-cargo-0
/usr/local/cargo/registry
/usr/local/cargo/git
key: ${{ runner.os }}-clippy-cargo-1.5
# TODO Commented out because we have no working keys yet.
# At release time, we should increment the number above and put the old below.
# Not commented-out from that point on, of course.
#restore-keys: ${{ runner.os }}-clippy-cargo-PREVIOUS

- name: Cache cargo target dir
uses: actions/cache@v2
with:
path: target
key: ${{ runner.os }}-cargo-build-target-clippy-target
restore-keys: |
${{ runner.os }}-cargo-build-target-clippy-target-0
key: ${{ runner.os }}-clippy-target-1.5
#restore-keys: ${{ runner.os }}-clippy-target-PREVIOUS

- name: Run Clippy
run: sudo -HEsu postgres sh -c "/usr/local/cargo/bin/cargo clippy --workspace --features 'pg14 pg_test' -- -D warnings"
# Github captures stdout and stderr separately and then intermingles them
# in the wrong order. We don't actually care to distinguish, so redirect
# stderr to stdout so we get the proper order.
run: su postgres -c 'sh tools/build clippy 2>&1'

test12:
name: Test PG 12
runs-on: ubuntu-latest
container:
image: timescaledev/rust-pgx:latest
env:
CARGO_INCREMENTAL: 0
CARGO_NET_RETRY: 10
CI: 1
RUST_BACKTRACE: short
RUSTUP_MAX_RETRIES: 10

env:
PGVERSION: 12
CARGO_TARGET_DIR_NAME: target
CARGO_INCREMENTAL: 0
CARGO_NET_RETRY: 10
CI: 1
RUST_BACKTRACE: short
RUSTUP_MAX_RETRIES: 10

steps:
- name: Checkout Repository
Expand All @@ -79,50 +105,44 @@ jobs:
uses: actions/cache@v2
with:
path: |
~/.cargo/registry
~/.cargo/git
key: ${{ runner.os }}-cargo-pg12-cargo
restore-keys: |
${{ runner.os }}-cargo-pg12-3
${{ runner.os }}-cargo-pg12-2
${{ runner.os }}-cargo-pg12
/usr/local/cargo/registry
/usr/local/cargo/git
key: ${{ runner.os }}-test-pg${{ env.PGVERSION }}-cargo-1.5
#restore-keys: ${{ runner.os }}-test-pg${{ env.PGVERSION }}-cargo-PREVIOUS

- name: Cache cargo target dir
uses: actions/cache@v2
with:
path: target
key: ${{ runner.os }}-cargo-build-target-pg12-4
restore-keys: |
${{ runner.os }}-cargo-build-target-pg12-3
${{ runner.os }}-cargo-build-target-pg12-2
${{ runner.os }}-cargo-build-target-pg12
key: ${{ runner.os }}-test-pg${{ env.PGVERSION }}-target-1.5
#restore-keys: ${{ runner.os }}-test-pg${{ env.PGVERSION }}-target-PREVIOUS

- name: Run PG 12 Tests
run: sudo -HEsu postgres sh -c "/usr/local/cargo/bin/cargo test --workspace --features 'pg12 pg_test'"
- name: Run pgx tests
run: su postgres -c 'sh tools/build -pg$PGVERSION test-extension 2>&1'

# TODO post-install + doc test adds a good 90s. We could probably put
# them together run serially in a separate job to shave that 90s off the
# long pole.
Comment on lines +123 to +125
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

post-install is testing update scripts, which should be safe to just do once. doc tests are mostly testing continuous aggregates, which I think we want to do on every version 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't looked into the post-install scripts yet. Are you saying they're the same as "update tests"? If so, I misunderstood the conversation yesterday and may have written some incorrect details in #335. In any case, can you add some more details about the post-install, update, and upgrade tests to #335?

- name: Run post-install tests
run: |
sudo -HEsu postgres sh -c "/usr/local/cargo/bin/cargo pgx stop pg12 && /usr/local/cargo/bin/cargo pgx start pg12"
RUST_BACKTRACE=short cargo run --manifest-path ./tools/post-install/Cargo.toml /home/postgres/.pgx/12.9/pgx-install/bin/pg_config
cargo run --manifest-path ./tools/testrunner/Cargo.toml -- -h localhost -p 28812

- name: Run Doc Tests
run: |
sudo -HEsu postgres sh -c "/usr/local/cargo/bin/cargo pgx stop pg12 && /usr/local/cargo/bin/cargo pgx start pg12"
sql-doctester -h localhost -s "CREATE EXTENSION timescaledb; CREATE EXTENSION timescaledb_toolkit; SET SESSION TIMEZONE TO 'UTC'" -p 28812 docs
run: su postgres -c 'sh tools/build -pg$PGVERSION test-post-install 2>&1'

- name: Run doc tests
run: su postgres -c 'sh tools/build -pg$PGVERSION test-doc 2>&1'

test13:
name: Test PG 13
runs-on: ubuntu-latest
container:
image: timescaledev/rust-pgx:latest
env:
CARGO_INCREMENTAL: 0
CARGO_NET_RETRY: 10
CI: 1
RUST_BACKTRACE: short
RUSTUP_MAX_RETRIES: 10

env:
PGVERSION: 13
CARGO_TARGET_DIR_NAME: target
CARGO_INCREMENTAL: 0
CARGO_NET_RETRY: 10
CI: 1
RUST_BACKTRACE: short
RUSTUP_MAX_RETRIES: 10

steps:
- name: Checkout Repository
Expand All @@ -137,44 +157,83 @@ jobs:
uses: actions/cache@v2
with:
path: |
~/.cargo/registry
~/.cargo/git
key: ${{ runner.os }}-cargo-build-cargo-pg13
restore-keys: |
${{ runner.os }}-cargo-build-target-pg13-3
${{ runner.os }}-cargo-build-target-pg13-2
${{ runner.os }}-cargo-pg13
/usr/local/cargo/registry
/usr/local/cargo/git
key: ${{ runner.os }}-test-pg${{ env.PGVERSION }}-cargo-1.5
#restore-keys: ${{ runner.os }}-test-pg${{ env.PGVERSION }}-cargo-PREVIOUS

- name: Cache cargo target dir
uses: actions/cache@v2
with:
path: target
key: ${{ runner.os }}-cargo-build-target-pg13-4
restore-keys: |
${{ runner.os }}-cargo-build-target-pg13-3
${{ runner.os }}-cargo-build-target-pg13-2
${{ runner.os }}-cargo-build-target-pg13
key: ${{ runner.os }}-test-pg${{ env.PGVERSION }}-target-1.5
#restore-keys: ${{ runner.os }}-test-pg${{ env.PGVERSION }}-target-PREVIOUS

- name: Run PG 13 Tests
run: sudo -HEsu postgres sh -c "/usr/local/cargo/bin/cargo test --workspace --features 'pg13 pg_test'"
- name: Run pgx tests
run: su postgres -c "sh tools/build -pg$PGVERSION test-extension 2>&1"

- name: Run post-install tests
run: |
sudo -HEsu postgres sh -c "/usr/local/cargo/bin/cargo pgx stop pg13 && /usr/local/cargo/bin/cargo pgx start pg13"
RUST_BACKTRACE=short cargo run --manifest-path ./tools/post-install/Cargo.toml /home/postgres/.pgx/13.5/pgx-install/bin/pg_config
cargo run --manifest-path ./tools/testrunner/Cargo.toml -- -h localhost -p 28813
run: su postgres -c 'sh tools/build -pg$PGVERSION test-post-install 2>&1'

- name: Run Doc Tests
run: |
sudo -HEsu postgres sh -c "/usr/local/cargo/bin/cargo pgx stop pg13 && /usr/local/cargo/bin/cargo pgx start pg13"
sql-doctester -h localhost -s "CREATE EXTENSION timescaledb; CREATE EXTENSION timescaledb_toolkit; SET SESSION TIMEZONE TO 'UTC'" -p 28813 docs
- name: Run doc tests
run: su postgres -c 'sh tools/build -pg$PGVERSION test-doc 2>&1'

test14:
name: Test PG 14
runs-on: ubuntu-latest
container:
image: timescaledev/rust-pgx:latest

env:
PGVERSION: 14
CARGO_TARGET_DIR_NAME: target
CARGO_INCREMENTAL: 0
CARGO_NET_RETRY: 10
CI: 1
RUST_BACKTRACE: short
RUSTUP_MAX_RETRIES: 10

steps:
- name: Checkout Repository
uses: actions/checkout@v2
with:
ref: ${{ github.event.pull_request.head.sha }}

- name: chown Repository
run: chown -R postgres .

- name: Cache cargo directories
uses: actions/cache@v2
with:
path: |
/usr/local/cargo/registry
/usr/local/cargo/git
key: ${{ runner.os }}-test-pg${{ env.PGVERSION }}-cargo-1.5
#restore-keys: ${{ runner.os }}-test-pg${{ env.PGVERSION }}-cargo-PREVIOUS

- name: Cache cargo target dir
uses: actions/cache@v2
with:
path: target
key: ${{ runner.os }}-test-pg${{ env.PGVERSION }}-target-1.5
#restore-keys: ${{ runner.os }}-test-pg${{ env.PGVERSION }}-target-PREVIOUS

- name: Run pgx tests
run: su postgres -c "sh tools/build -pg$PGVERSION test-extension 2>&1"

- name: Run post-install tests
run: su postgres -c 'sh tools/build -pg$PGVERSION test-post-install 2>&1'

- name: Run doc tests
run: su postgres -c 'sh tools/build -pg$PGVERSION test-doc 2>&1'

testcrates:
name: Test Crates
runs-on: ubuntu-latest
container:
image: timescaledev/rust-pgx:latest
env:
CARGO_TARGET_DIR_NAME: target
CARGO_INCREMENTAL: 0
CARGO_NET_RETRY: 10
CI: 1
Expand All @@ -194,34 +253,17 @@ jobs:
uses: actions/cache@v2
with:
path: |
~/.cargo/registry
~/.cargo/git
key: ${{ runner.os }}-cargo-build-cargo-pg14
restore-keys: |
${{ runner.os }}-cargo-build-target-pg14-3
${{ runner.os }}-cargo-build-target-pg14-2
${{ runner.os }}-cargo-pg14
/usr/local/cargo/registry
/usr/local/cargo/git
key: ${{ runner.os }}-test-crates-cargo-1.5
#restore-keys: ${{ runner.os }}-test-crates-cargo-PREVIOUS

- name: Cache cargo target dir
uses: actions/cache@v2
with:
path: target
key: ${{ runner.os }}-cargo-build-target-pg14-4
restore-keys: |
${{ runner.os }}-cargo-build-target-pg14-3
${{ runner.os }}-cargo-build-target-pg14-2
${{ runner.os }}-cargo-build-target-pg14
key: ${{ runner.os }}-test-crates-target-1.5
#restore-keys: ${{ runner.os }}-test-crates-target-PREVIOUS

- name: Run PG 14 Tests
run: sudo -HEsu postgres sh -c "/usr/local/cargo/bin/cargo test --workspace --features 'pg14 pg_test'"

- name: Run post-install tests
run: |
sudo -HEsu postgres sh -c "/usr/local/cargo/bin/cargo pgx stop pg14 && /usr/local/cargo/bin/cargo pgx start pg14"
RUST_BACKTRACE=short cargo run --manifest-path ./tools/post-install/Cargo.toml /home/postgres/.pgx/14.1/pgx-install/bin/pg_config
cargo run --manifest-path ./tools/testrunner/Cargo.toml -- -h localhost -p 28814

- name: Run Doc Tests
run: |
sudo -HEsu postgres sh -c "/usr/local/cargo/bin/cargo pgx stop pg14 && /usr/local/cargo/bin/cargo pgx start pg14"
sql-doctester -h localhost -s "CREATE EXTENSION timescaledb; CREATE EXTENSION timescaledb_toolkit; SET SESSION TIMEZONE TO 'UTC'" -p 28814 docs
- name: Run Crates Tests
run: su postgres -c 'sh tools/build test-crates 2>&1'
Loading