Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky CI test #263

Merged
merged 6 commits into from
Apr 20, 2021
Merged

Fix flaky CI test #263

merged 6 commits into from
Apr 20, 2021

Conversation

cmichi
Copy link
Collaborator

@cmichi cmichi commented Apr 19, 2021

It's hunting season for #234 🏹!

I found it!

The issue was that we use

CARGO_TARGET_DIR: "/ci-cache/${CI_PROJECT_NAME}/targets/${CI_COMMIT_REF_NAME}/${CI_JOB_NAME}"

in our CI configuration. This made all tests build to the same target directory. Since we have a number of tests which build a contract called new_project this made our tmp_dir's obsolete and resulted in overlapping contracts build to e.g. /ci-cache/cargo-contract/targets/263/test/ink/new_project.contract.

The way I fixed it now is by appending a hash of a contract's manifest_path to the target_directory (for tests this path contains the tmp_dir path). This results in the paths now being e.g. /ci-cache/cargo-contract/targets/263/test/0xd82f746d702f636172676f2d636f6e74726163742e746573742e5479444c634b2f6e65775f70726f6a6563742f436172676f2e746f6d6c/ink/new_project.contract.

I'm not super-happy with that solution, since it makes caching obsolete for the contract build in those particular tests (since the hash is unique each time a test is run, due to tmp_dir being included in it).

I have a better idea, but it's a bit more involved:

  • We could use the full test path (e.g. cmd::metadata::tests::generate_metadata) for the hash instead.
  • This would be achieved by writing a proc. macro (could also be called #[test]), which wraps around the Rust #[test] macro.
  • The macro would then extract the module_path! and function_name! and put those into a thread-local variable.
  • This thread-local variable would be used for hashing.

The disadvantage is that we would require something like use crate::test; for the macro to become available for tests. But we would then again have proper caching.

But: The CI hasn't really slowed down with the current fix in this PR ‒ a run of the test stage still takes 4-5 minutes. So maybe the current fix is already good enough. Though it fills up the cache and I'm not sure when it is cleared.

Wdyt?

@cmichi cmichi changed the title Add debug output Fix flaky CI test Apr 19, 2021
@cmichi cmichi force-pushed the cmichi-debug-flaky-test branch 4 times, most recently from da19415 to 7f802c8 Compare April 20, 2021 07:49
@cmichi cmichi force-pushed the cmichi-debug-flaky-test branch from 7f802c8 to 5c81fee Compare April 20, 2021 07:52
@ascjones
Copy link
Collaborator

An alternative solution would be to add an additional with_new_contract_project function which auto generates a unique name for the contract project and maybe runs new::exec expecting success, and passing the resulting ManifestPath to the caller.

The unique name could either be a hash of the tmp dir as you have done or you could even use an atomic counter and append that to the name e.g. new_project1, new_project2 which would mean unique names within a CI run, but would get overwritten on new runs rather than creating endless new artifacts.

@cmichi cmichi requested a review from ascjones April 20, 2021 11:25
@cmichi cmichi marked this pull request as ready for review April 20, 2021 11:25
src/util.rs Outdated Show resolved Hide resolved
src/util.rs Outdated Show resolved Hide resolved
Copy link
Collaborator

@ascjones ascjones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cmichi cmichi merged commit 307de23 into master Apr 20, 2021
@cmichi cmichi deleted the cmichi-debug-flaky-test branch April 20, 2021 13:19
@ascjones ascjones mentioned this pull request Apr 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants