Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: streamline the build and test process #51

Draft
wants to merge 78 commits into
base: master
Choose a base branch
from
Draft

Conversation

bassco
Copy link
Contributor

@bassco bassco commented Nov 6, 2023

Addresses some of the issues, and more, raised in #11

  • Create preview images in Pull Requests
  • Use conventional commits for release automation
  • Add OCI labels to docker images
  • Fix the README to enable local tests - compose is missing and so is the Makefile
  • Fix the preview image creation step
  • Delete preview container image and artifacts on merge or PR closing
  • verify the renovate settings
  • add a conventional commit labeler to the pull requests
  • bump the toml version and references during release creation and tagging
  • add mac silicon preview and release images for local development? Stretch goal
  • added benchmark notes and samples script to run benchmark
  • add a cargo deny github action to catch renovate version bumps that are not useful
  • fix builds when features are changed - inconsistent code behaviour causes error
  • fix the failing e2e exif test when features awc_client is enabled
  • build dali and dali-awc in the preview image so that both versions can be used in tests

This is the error we recevied when we tried to cargo build --features aws_client

error[E0428]: the name `client` is defined multiple times
  --> src/commons/http.rs:66:1
   |
10 | pub mod client {
   | -------------- previous definition of the module `client` here
...
66 | pub mod client {
   | ^^^^^^^^^^^^^^ `client` redefined here
   |
   = note: `client` must be defined only once in the type namespace of this module

error: features `crate/hyper_client` and `crate/awc` are mutually exclusive
 --> src/main.rs:4:1
  |
4 | compile_error!("features `crate/hyper_client` and `crate/awc` are mutually exclusive");
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This is caused because cargo treats features as additive. Because hyper_client was set as a default; it was also set when we tried the build the awc_client feature. The test to see which feature to enable had to be changed. To enable the default http_server, we tested the negated setting of features awc_client.

Cargo allows you to reference a crate from a local or remote git repository. By changing the Cargo.toml to a remote branch, I was able to determine which method is panicking.

The Cargo.toml change:

diff --git Cargo.toml Cargo.toml
index a52b08d..5103c5d 100644
--- Cargo.toml
+++ Cargo.toml
@@ -41,7 +41,10 @@ serde_qs = {version = "0.12.0", features=["actix4"]}
 config = "0.13.3"
 hyper = { version = "0.14.27", features=["full"], optional = true }
 hyper-timeout = { version = "0.4.1", optional = true }
-libvips = "1.6.1"
+# or use a git path or remote repo and branch
+# libvips = {git = "file:///absolute/path/to/git/repo", rev = "12345"}
+libvips = {git = "https://github.com/olxgroup-oss/libvips-rust-bindings.git", branch = "make-error-messages-explicit"}
+# libvips = "1.5.0"
 rexif = "0.7.3"
 lazy_static = "1.4.0"
 cfg-if = "1.0.0"

Running the tests produced.

$  ./scripts/dali-tests-runner.sh awc_client

e8f44d88da0e873de851d806ab60b3f4604ad9c9946ce3a8984887f3e866ad1a
Connection to localhost port 9000 [tcp/cslistener] succeeded!
    Finished dev [unoptimized + debuginfo] target(s) in 0.28s
     Running `target/debug/dali`
Connection to localhost port 8080 [tcp/http-alt] succeeded!
    Finished test [unoptimized + debuginfo] target(s) in 0.28s
     Running unittests src/main.rs (target/debug/deps/dali-d0d0bec70bf6e467)

<REDACTED_TEST_DATA_OUTPUT_FOR_BREVITY>

test test_get_exif_watermark ... FAILED

failures:

---- test_get_exif_watermark stdout ----
URL: <REDACTED_FOR_BREVITY>
Response:
ClientResponse HTTP/1.1 200 OK
  headers:
    "content-length": "21"
    "content-type": "text/plain; charset=utf-8"
    "date": "Sun, 12 Nov 2023 21:26:24 GMT"

thread 'test_get_exif_watermark' panicked at tests/utils/mod.rs:110:58:
Unable to read image from dali: InitializationError("VipsImage:new_from_buffer - Could not initialise VipsImage")
.....

bassco and others added 30 commits November 4, 2023 11:08
* Add renovate.json

* chore: configure renovate settings

* tests: fix musl version

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Andrew Basson <andrew.basson@gmail.com>
* fix(deps): update rust crate chrono to 0.4.31

* fix(sec): removed chrono because of security advisor

Signed-off-by: Gabriel Melillo <gabriel@melillo.me>

* ci(deps): enable cache and fix dockerfile

* build(ci): install posix compliant tar

* build(ci): combine dependency install steps

* chore: update cargo deps from default branch

* ci: disable releases during maintenance tasks

---------

Signed-off-by: Gabriel Melillo <gabriel@melillo.me>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Gabriel Melillo <gabriel@melillo.me>
Co-authored-by: Andrew Basson <andrew.basson@gmail.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Bumps [num-bigint](https://github.com/rust-num/num-bigint) from 0.4.2 to 0.4.3.
- [Release notes](https://github.com/rust-num/num-bigint/releases)
- [Changelog](https://github.com/rust-num/num-bigint/blob/master/RELEASES.md)
- [Commits](rust-num/num-bigint@num-bigint-0.4.2...num-bigint-0.4.3)

---
updated-dependencies:
- dependency-name: num-bigint
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [regex](https://github.com/rust-lang/regex) from 1.5.4 to 1.5.6.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](rust-lang/regex@1.5.4...1.5.6)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(deps): update rust crate config to 0.13.3

* fix(deps): update config to 0.13.3

Signed-off-by: Gabriel Melillo <gabriel@melillo.me>

---------

Signed-off-by: Gabriel Melillo <gabriel@melillo.me>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Gabriel Melillo <gabriel@melillo.me>
* fix(deps): update rust crate hyper to 0.14.27

* ci(tests): add openssl to test workflow

* fix(deps): upgrade many deps and fixed code errors

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Andrew Basson <andrew.basson@gmail.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* master:
  chore: release-drafter be explicit with default branch name
  chore: release-drafter use default branch for comparison [skip-changelog]
  chore: release-drafter template add comparison [skip-changelog]
  ci(docs): update release previewer template
  ci(docs): add release previewer (#52)
.dockerignore Outdated Show resolved Hide resolved
.releaserc.json Outdated Show resolved Hide resolved
* cargo install cargo-criterion
* cargo install cargo-deny
* added tokio as a dependency for the benchmarks to compile
* downgraded serde and serde_derive to 1.0.190
* added benchmark script to test different features
* added banchmarking notes
* initialised cargo deny and added deny.toml to repo
* cargo deny needs to be added to the renovate job
  to stop bad version bumps from creating duplicate versions and to also
  start the process of license checking. Additionally, cargo deny can
  be used for advisery alerting
* fixed some code paths that were not used when using the `awc_client`
  feature.
* added dali-bench.sh script to enable the compiling and testing of the
  different http servers: the default hyper_client vs awc_client.
* added the first baseline test results
* Cargo features are additive in nature and default features are
included when specifying additional features. There is no ability to
remove a default. Our old logic exposed this bug when we tried to
compile with  the `awc_client` feature. We had feature `hyper_client`
as the default to make it easy to build without specifying which http
server implementation to use; which inadvertently set both features to
enabled, when we choose the alternative, `awc_client`. When choosing
feature = `awc_client`; `hyper_client` is also present, as it is the
default. Therefore, we evaluate the not `awc_client` feature to enable
the default behaviour.
* add benchmark reports showing awc_client is superior to hyper_client
* benchmark script now takes a features parameter
* benchmark script opens browser when completed
* master:
  fix(deps): update rust crate env_logger to 0.10.1
  fix(deps): update serde monorepo to 1.0.192
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants