-
-
Notifications
You must be signed in to change notification settings - Fork 236
Refactor integration tests #3064
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
✅ PR OK, no changes in deprecations or warnings Total deprecations: 0 Total warnings: 0 Build statistics: statistics (-before, +after)
executable size=5265672 bin/dub
-rough build time=60s
+rough build time=58s Full build output
|
604aa21
to
62d6498
Compare
CI looking good, outside of buildkite which I don't have access to |
|
Simply removing that line and changing whatever was used to invoke the test runner with Although, that test is one of the more intrusive ones. It used to overwrite I did get a few failures when running the testsuite as part of a Gentoo build and one of it seems to be a genuine (minor) bug in the vibe json code but I need more time to make sure it's an actual bug, because I can only reproduce it with a locally built ldc2, and not with dmd, gdc or the upstream releases of ldc2. |
The dlang/ci error is https://github.com/dlang/ci/blob/dc95e2fe07e163e159ae3b5b2bf1efc374fef818/buildkite/build_project.sh#L180-L184 Can get around it by adding |
Oh, so that's where it was.
buildkite doesn't look like it's running the integration tests, so what's the point in removing those unused test files? Should buildkite be running the full testsuite? |
In regards to the bug: dub/source/dub/internal/vibecompat/data/json.d Line 1452 in ad3f246
Json.this(BigInt)
zeroFields and then initBigInt dub/source/dub/internal/vibecompat/data/json.d Line 1264 in ad3f246
BigInt.opAssign that, in turn, calls BigUint.opAssign . BigUint , however, has an invariant :https://github.com/dlang/phobos/blob/2d8ae67b396f5cc5f4633695c1c5ac67d2bf448e/std/internal/math/biguintcore.d#L240-L244: pure invariant()
{
assert( data.length >= 1 && (data.length == 1 || data[$-1] != 0 ),
"Invariant requires data to not empty or zero");
} which then fails because the data array has been zero-ed out by
You can hit this if the stdlib has been built without
This shouldn't be causing any problems though, since the bigint would be in a valid state after |
See dlang/dub#3064 (comment) 1. It does nothing 2. It causes the run to fail when the tests are renamed/removed Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
62d6498
to
55ec9c7
Compare
Incorporated #3066 and changed I've also updated the tests' |
As a small benchmark, doing two consecutive runs, the first with a clean repo and no
|
55ec9c7
to
523afd7
Compare
Changed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would like to get this in soon (DConf hackathon perhaps?). Why the move the new_test though ?
// Find the backend version of the compiler, not the dmd FE. | ||
// Specificically, for gdmd-14 this function should return | ||
// 14.X.Y not 2.108.Z |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why ? I don't think that makes a lot of sense.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See #3066 (comment).
For short:
ldmd
uses the version ofldc
, not the dlang frontend version- I've only seen people depend on major versions of gcc, at most a minor version if they need a bug fix. Having versions like
2.108.1
,2.108.0-rc.1
etc. all map togdc-14
seems like poor design.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah it's already merged. I guess this needs a rebase then ?
I still think there's something wrong here - either you have something in front of LDC / GDC that pretends to be DMD, or you don't. Having something semi-transparent will just shift the problem somewhere else. But I think we can move ahead for now as it's a really minor change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure why this PR showed changed that were already in master, but a rebase does seem to have fixed it.
I still think there's something wrong here - either you have something in front of LDC / GDC that pretends to be DMD, or you don't. Having something semi-transparent will just shift the problem somewhere else. But I think we can move ahead for now as it's a really minor change.
Ok, let me try to explain my reasoning again. issue1531-toolchain-requirements didn't work with gdmd. The reasoning was that a line such as toolchainRequirements gdc="~>14"
would work with gdc-14
but not with gdmd-14
with the error of Error Installed gdc-2.108.1 does not comply with a compiler requirement: gdc ~>14.0
. To fix this I either had to change to test to the D frontend version (up to the patch version) or I had to change dub to extract (what I think is) the correct version.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. I assume frontend="~>2.108"
still works for both GDC and GDMD ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes
I feared that trying to do it in place would make it way harder to do atomic commits for easier review and I also worried that I would miss a dependency on one of the tests. Putting everything in another directory helped with safely porting tests one by one. This does leave me with having to put everything back into tests. I'm fine doing it in 1 big commit but I think I can manage rebasing each commit to leave everything in |
W.r.t. atomic commits, I don't think this is the kind of thing that can be reviewed commit-by-commit anyway. If it was I would prefer if it was split into multiple pull requests, but I don't think it's a good use of yours or my time. |
523afd7
to
35a3386
Compare
@the-horo : LMK when this is ready for the next round of review (waiting for files to be moved back). |
35a3386
to
ff16742
Compare
Have at it ;) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pretty hard to review the whole thing (my browser slows down when I try to scroll down...) but made a few observations.
.github/workflows/main.yml
Outdated
- { dc: ldc-master, do_test: true } | ||
# Test on ARM64 | ||
- { os: macOS-14, dc: ldc-latest, do_test: true } | ||
- { os: ubuntu-24.04, dc: gdc-13, do_test: false } # ice when building tests |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a link to the ICE ?
dub test --compiler=${DC} -c library-nonet | ||
testLibraryNonet=1 | ||
if [[ ${DC} =~ gdc|gdmd ]]; then | ||
# ICE with gdc-14 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also link so that the next guy can tell at a glance if it has been fixed ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this only fails in CI, so maybe fixed with more recent gdc-14
version-filters/version-filters | ||
version-spec/newfoo/foo-test-application | ||
version-spec/oldfoo/foo-test-application | ||
*/* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's fine for this PR as the state of things wasn't great but we should really find a better solution. This */*
is going to be very surprising for users and quite error prone.
@@ -1,36 +0,0 @@ | |||
/+ dub.sdl: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why does this need to change ? I don't see it as an improvement to have to go from one self contained test file to having the heavier package baggage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, the total amount of lines doesn't really change between having the recipe be part or not be part of the source file. On top of this most of the .script.d
tests still required a separate directory to do their tests, so you might as well put the test code in there as well.
I originally wanted to support both the dub-project and the dub-script tests, the unittest runner still carrying the architecture for using multiple types of runners. On the way I decided to just drop the idea, we don't gain anything for having single-file tests compared to using a simple project layout, outside of having to have addition logic in the runner.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that's my main disagreement. Tests in Dub are quite hard to write, and inline recipe make it easier.
@@ -0,0 +1,21 @@ | |||
import std.file : exists, remove; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should really combine the init-fail and init-fail-json tests, but this can be done in another PR.
bool must_be_run_alone = false; | ||
} | ||
|
||
TestConfig parseConfig(string content, ErrorSink errorSink) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of this just use Configy ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't do a lot of D programming, my only knowledge of the D ecosystem is what comes with the stdlib. And, one thing that I like about my implementation is that it copes with <array_setting> = <scalar_value>
which configy
doesn't seem to support.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The argument for Configy was that it's what Dub is already using, and it does what you're doing here. It can support the syntax you describe as you have the ability to implement any parsing logic you want. But not a blocker.
@@ -0,0 +1,46 @@ | |||
import run_unittest.log; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing module decl
test/run_unittest/source/app.d
Outdated
return 0; | ||
} | ||
|
||
auto testDir = buildNormalizedPath(__FILE_FULL_PATH__, "..", "..", ".."); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use dirName
instead.
@@ -0,0 +1,197 @@ | |||
module run_unittest.log; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is fine for now, but why was std.logger
not an option ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No good argument against it, it just doesn't look like std.logger
is able to do what want do to. That being print messages to console and/or to a file and prefix them with the test's name.
} | ||
|
||
if (dcBackendGuess != thisDc) { | ||
sink.error("The DC environment is not the same backend as the D compiler"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice error handling 👍
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
ldc2 on alpine is configured to use the lld linker, however, the lld package is not pulled in as a dependency. Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
c7f70e1
to
50b329a
Compare
Yup, the github webui is a pain with so many changes |
Any way we can break it down to smaller (but probably not small) PRs ? |
Yeah, I think I can do that. |
Main takeaways
gdc
.sh
tests with.d
programs (in order to run on windows)It took me a while but I think it was worth it.
I've kept the commits to be one test each and the changes are staged to a new directory (
new_tests
). I don't expect this to be the final form if this PR is merged so I'll keep this as a draft but I hope that this would make reviewing easier.From my notes while working on this:
frameworks
wasn't testing anything (Add ability to specify frameworks to link against #3061 (comment))issue2046
wasn't runissue2650-deprecated-modules
wasn't runissue103
is missing a test case (https://github.com/dlang/dub/pull/1177/files#diff-a6b4cf5ac30d4b153861d8ad368ab9cb2aebf25ee28482a6ec201a827ee25a98)ddox
sometimes decided to ask for confirmation to overwrite files from stdin on windows (making the test stuck). Maybe it had to do with my setup in which I was making changes to how it was run so maybe it's unlikely that a real user would actually hit it. Regardless, I thinkdub
should be preventing the spawned process for having access tostdin