Skip to content

Conversation

the-horo
Copy link
Contributor

Main takeaways

  • make the testsuite work with gdc
  • have one test runner that runs all the (possible) tests on as many platforms
  • replace the .sh tests with .d programs (in order to run on windows)
  • run tests in parallel
  • make the test output prettier

It took me a while but I think it was worth it.

I've kept the commits to be one test each and the changes are staged to a new directory (new_tests). I don't expect this to be the final form if this PR is merged so I'll keep this as a draft but I hope that this would make reviewing easier.

From my notes while working on this:

@github-actions
Copy link

github-actions bot commented Jul 29, 2025

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 0

Total warnings: 0

Build statistics:

 statistics (-before, +after)
 executable size=5265672 bin/dub
-rough build time=60s
+rough build time=58s
Full build output
DUB version 1.40.0, built on Jun  7 2025
LDC - the LLVM D compiler (1.41.0):
  based on DMD v2.111.0 and LLVM 20.1.5
  built with LDC - the LLVM D compiler (1.41.0)
  Default target: x86_64-unknown-linux-gnu
  Host CPU: znver3
  http://dlang.org - http://wiki.dlang.org/LDC


  Registered Targets:
    aarch64     - AArch64 (little endian)
    aarch64_32  - AArch64 (little endian ILP32)
    aarch64_be  - AArch64 (big endian)
    amdgcn      - AMD GCN GPUs
    arm         - ARM
    arm64       - ARM64 (little endian)
    arm64_32    - ARM64 (little endian ILP32)
    armeb       - ARM (big endian)
    avr         - Atmel AVR Microcontroller
    bpf         - BPF (host endian)
    bpfeb       - BPF (big endian)
    bpfel       - BPF (little endian)
    hexagon     - Hexagon
    lanai       - Lanai
    loongarch32 - 32-bit LoongArch
    loongarch64 - 64-bit LoongArch
    mips        - MIPS (32-bit big endian)
    mips64      - MIPS (64-bit big endian)
    mips64el    - MIPS (64-bit little endian)
    mipsel      - MIPS (32-bit little endian)
    msp430      - MSP430 [experimental]
    nvptx       - NVIDIA PTX 32-bit
    nvptx64     - NVIDIA PTX 64-bit
    ppc32       - PowerPC 32
    ppc32le     - PowerPC 32 LE
    ppc64       - PowerPC 64
    ppc64le     - PowerPC 64 LE
    r600        - AMD GPUs HD2XXX-HD6XXX
    riscv32     - 32-bit RISC-V
    riscv64     - 64-bit RISC-V
    sparc       - Sparc
    sparcel     - Sparc LE
    sparcv9     - Sparc V9
    spirv       - SPIR-V Logical
    spirv32     - SPIR-V 32-bit
    spirv64     - SPIR-V 64-bit
    systemz     - SystemZ
    thumb       - Thumb
    thumbeb     - Thumb (big endian)
    ve          - VE
    wasm32      - WebAssembly 32-bit
    wasm64      - WebAssembly 64-bit
    x86         - 32-bit X86: Pentium-Pro and above
    x86-64      - 64-bit X86: EM64T and AMD64
    xcore       - XCore
    xtensa      - Xtensa 32
   Upgrading project in /home/runner/work/dub/dub/
    Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.41.0/x64/ldc2-1.41.0-linux-x86_64/bin/ldc2 for x86_64.
    Building dub 1.39.0-rc.1+commit.266.gcbaa0039: building configuration [application]
     Linking dub
STAT:statistics (-before, +after)
STAT:executable size=5265672 bin/dub
STAT:rough build time=58s

@the-horo the-horo force-pushed the run-unittest-ci branch 3 times, most recently from 604aa21 to 62d6498 Compare July 30, 2025 15:55
@the-horo the-horo marked this pull request as ready for review July 30, 2025 20:00
@the-horo
Copy link
Contributor Author

CI looking good, outside of buildkite which I don't have access to

@thewilsonator
Copy link
Contributor

rm: cannot remove 'test/issue895-local-configuration.sh': No such file or directory

@the-horo
Copy link
Contributor Author

rm: cannot remove 'test/issue895-local-configuration.sh': No such file or directory

Simply removing that line and changing whatever was used to invoke the test runner with dub run --root test/run_unittest should be good enough. I am trying to get the testsuite to run on as many platforms as it can and I don't want tests that fail on certain setups. It would be best if the tests worked but, if they really can't, they have full control to skip themselves, rather than requiring a user to remove the test directory.

Although, that test is one of the more intrusive ones. It used to overwrite ./etc/dub/settings.json but now it modified a scoped dub.settings.json so it should be safer. It also created a dummy ldc2 named file which can be picked up when invoking dub normally which can pose issues. This is why the test has must_be_run_alone = true, so it can't affect other tests.


I did get a few failures when running the testsuite as part of a Gentoo build and one of it seems to be a genuine (minor) bug in the vibe json code but I need more time to make sure it's an actual bug, because I can only reproduce it with a locally built ldc2, and not with dmd, gdc or the upstream releases of ldc2.

@Geod24
Copy link
Member

Geod24 commented Jul 31, 2025

The dlang/ci error is https://github.com/dlang/ci/blob/dc95e2fe07e163e159ae3b5b2bf1efc374fef818/buildkite/build_project.sh#L180-L184

Can get around it by adding -f and removing it once the PR is merged.

@the-horo
Copy link
Contributor Author

The dlang/ci error is https://github.com/dlang/ci/blob/dc95e2fe07e163e159ae3b5b2bf1efc374fef818/buildkite/build_project.sh#L180-L184

Oh, so that's where it was.

Can get around it by adding -f and removing it once the PR is merged.

buildkite doesn't look like it's running the integration tests, so what's the point in removing those unused test files? Should buildkite be running the full testsuite?

@the-horo
Copy link
Contributor Author

a genuine (minor) bug in the vibe json code

In regards to the bug:

assert(parseJsonString("17559991181826658461") == Json(BigInt(17559991181826658461UL)));
is calling Json.this(BigInt)
this(BigInt v) nothrow @trusted { zeroFields; m_type = Type.bigInt; initBigInt(); m_bigInt = v; }
which calls zeroFields and then initBigInt
m_bigInt = BigInt.init;
which calls BigInt.opAssign that, in turn, calls BigUint.opAssign. BigUint, however, has an invariant:
https://github.com/dlang/phobos/blob/2d8ae67b396f5cc5f4633695c1c5ac67d2bf448e/std/internal/math/biguintcore.d#L240-L244:

    pure invariant()
    {
        assert( data.length >= 1 && (data.length == 1 || data[$-1] != 0 ),
                "Invariant requires data to not empty or zero");
    }

which then fails because the data array has been zero-ed out by zeroFields. As a stack trace:

(gdb) bt
#0  0x00007ffff7f19c20 in _d_assert_msg () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#1  0x00007ffff7c2feae in std.internal.math.biguintcore.BigUint.__invariant() const () from /usr/lib/ldc2/1.40/lib64/libphobos2-ldc-shared.so.110
#2  0x0000555555917f29 in std.internal.math.biguintcore.BigUint.opAssign!(void).opAssign(std.internal.math.biguintcore.BigUint) ()
#3  0x000055555590a60f in std.bigint.BigInt.opAssign!(std.bigint.BigInt).opAssign(std.bigint.BigInt) ()
#4  0x000055555590a191 in dub.internal.vibecompat.data.json.Json.initBigInt() ()
#5  0x000055555590a0f9 in dub.internal.vibecompat.data.json.Json.this(std.bigint.BigInt) ()
#6  0x000055555590e565 in dub.internal.vibecompat.data.json.__unittest_L1450_C7() ()
#7  0x000055555592827e in dub.internal.vibecompat.data.json.__unittest ()
#8  0x00007ffff7f3b7d8 in ?? () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#9  0x00007ffff7f60607 in ?? () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#10 0x00007ffff7f65c98 in rt.sections_elf_shared.DSO.opApply(scope int(ref rt.sections_elf_shared.DSO) delegate) ()
   from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#11 0x00007ffff7f605ac in rt.minfo.moduleinfos_apply(scope int(immutable(object.ModuleInfo*)) delegate) () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#12 0x00007ffff7f4e4ff in object.ModuleInfo.opApply(scope int(object.ModuleInfo*) delegate) () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#13 0x00007ffff7f3b5c8 in runModuleUnitTests () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#14 0x00007ffff7f58bbb in rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#15 0x00007ffff7f589d4 in _d_run_main2 () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#16 0x00007ffff7f587ed in _d_run_main () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#17 0x0000555555bdbea2 in main ()
#18 0x00007ffff774c22e in ?? () from /usr/lib64/libc.so.6
#19 0x00007ffff774c2e9 in __libc_start_main () from /usr/lib64/libc.so.6
#20 0x00005555555d4bc5 in _start ()

You can hit this if the stdlib has been built without -release, which my ldc2 built was, but you can also hit this if you do:

$ DFLAGS='-unittest -link-defaultlib-debug -g' dub test --compiler=ldc2

This shouldn't be causing any problems though, since the bigint would be in a valid state after opAssign and the fact that it's invalid as per the invariant doesn't actually break any code. It still needs to be fixed not to trigger the failure though

the-horo added a commit to the-horo/dlang-ci that referenced this pull request Jul 31, 2025
See dlang/dub#3064 (comment)

1. It does nothing
2. It causes the run to fail when the tests are renamed/removed

Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
@the-horo
Copy link
Contributor Author

the-horo commented Aug 2, 2025

Incorporated #3066 and changed issue1531 to use it since I got it to fail on one of my containers where gdc-14 was based on dmd 2.108.1 rather than 2.108.0

I've also updated the tests' README

@the-horo
Copy link
Contributor Author

the-horo commented Aug 3, 2025

As a small benchmark, doing two consecutive runs, the first with a clean repo and no ~/.dub, the second with all the artifacts from the first, with the old and the new test runner, using dmd:

run # run_unittest -j8 run-unittest.d
1st, clean 5m27.496s 6m45.855s
2nd, cache 2m43.358s 4m21.038s

@the-horo
Copy link
Contributor Author

the-horo commented Aug 9, 2025

Changed run_unittest to compile with -preview=in and -preview=nosharedaccess

Copy link
Member

@Geod24 Geod24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would like to get this in soon (DConf hackathon perhaps?). Why the move the new_test though ?

Comment on lines 104 to 106
// Find the backend version of the compiler, not the dmd FE.
// Specificically, for gdmd-14 this function should return
// 14.X.Y not 2.108.Z
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why ? I don't think that makes a lot of sense.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #3066 (comment).

For short:

  • ldmd uses the version of ldc, not the dlang frontend version
  • I've only seen people depend on major versions of gcc, at most a minor version if they need a bug fix. Having versions like 2.108.1, 2.108.0-rc.1 etc. all map to gdc-14 seems like poor design.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah it's already merged. I guess this needs a rebase then ?
I still think there's something wrong here - either you have something in front of LDC / GDC that pretends to be DMD, or you don't. Having something semi-transparent will just shift the problem somewhere else. But I think we can move ahead for now as it's a really minor change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why this PR showed changed that were already in master, but a rebase does seem to have fixed it.

I still think there's something wrong here - either you have something in front of LDC / GDC that pretends to be DMD, or you don't. Having something semi-transparent will just shift the problem somewhere else. But I think we can move ahead for now as it's a really minor change.

Ok, let me try to explain my reasoning again. issue1531-toolchain-requirements didn't work with gdmd. The reasoning was that a line such as toolchainRequirements gdc="~>14" would work with gdc-14 but not with gdmd-14 with the error of Error Installed gdc-2.108.1 does not comply with a compiler requirement: gdc ~>14.0. To fix this I either had to change to test to the D frontend version (up to the patch version) or I had to change dub to extract (what I think is) the correct version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I assume frontend="~>2.108" still works for both GDC and GDMD ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

@the-horo
Copy link
Contributor Author

Would like to get this in soon (DConf hackathon perhaps?). Why the move the new_test though ?

I feared that trying to do it in place would make it way harder to do atomic commits for easier review and I also worried that I would miss a dependency on one of the tests. Putting everything in another directory helped with safely porting tests one by one.

This does leave me with having to put everything back into tests. I'm fine doing it in 1 big commit but I think I can manage rebasing each commit to leave everything in /test, if you want it.

@Geod24
Copy link
Member

Geod24 commented Aug 21, 2025

W.r.t. atomic commits, I don't think this is the kind of thing that can be reviewed commit-by-commit anyway. If it was I would prefer if it was split into multiple pull requests, but I don't think it's a good use of yours or my time.

@Geod24
Copy link
Member

Geod24 commented Aug 22, 2025

@the-horo : LMK when this is ready for the next round of review (waiting for files to be moved back).

@the-horo
Copy link
Contributor Author

@the-horo : LMK when this is ready for the next round of review (waiting for files to be moved back).

Have at it ;)

Copy link
Member

@Geod24 Geod24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty hard to review the whole thing (my browser slows down when I try to scroll down...) but made a few observations.

- { dc: ldc-master, do_test: true }
# Test on ARM64
- { os: macOS-14, dc: ldc-latest, do_test: true }
- { os: ubuntu-24.04, dc: gdc-13, do_test: false } # ice when building tests
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a link to the ICE ?

dub test --compiler=${DC} -c library-nonet
testLibraryNonet=1
if [[ ${DC} =~ gdc|gdmd ]]; then
# ICE with gdc-14
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also link so that the next guy can tell at a glance if it has been fixed ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this only fails in CI, so maybe fixed with more recent gdc-14

version-filters/version-filters
version-spec/newfoo/foo-test-application
version-spec/oldfoo/foo-test-application
*/*
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's fine for this PR as the state of things wasn't great but we should really find a better solution. This */* is going to be very surprising for users and quite error prone.

@@ -1,36 +0,0 @@
/+ dub.sdl:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this need to change ? I don't see it as an improvement to have to go from one self contained test file to having the heavier package baggage.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, the total amount of lines doesn't really change between having the recipe be part or not be part of the source file. On top of this most of the .script.d tests still required a separate directory to do their tests, so you might as well put the test code in there as well.

I originally wanted to support both the dub-project and the dub-script tests, the unittest runner still carrying the architecture for using multiple types of runners. On the way I decided to just drop the idea, we don't gain anything for having single-file tests compared to using a simple project layout, outside of having to have addition logic in the runner.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's my main disagreement. Tests in Dub are quite hard to write, and inline recipe make it easier.

@@ -0,0 +1,21 @@
import std.file : exists, remove;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should really combine the init-fail and init-fail-json tests, but this can be done in another PR.

bool must_be_run_alone = false;
}

TestConfig parseConfig(string content, ErrorSink errorSink) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of this just use Configy ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't do a lot of D programming, my only knowledge of the D ecosystem is what comes with the stdlib. And, one thing that I like about my implementation is that it copes with <array_setting> = <scalar_value> which configy doesn't seem to support.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The argument for Configy was that it's what Dub is already using, and it does what you're doing here. It can support the syntax you describe as you have the ability to implement any parsing logic you want. But not a blocker.

@@ -0,0 +1,46 @@
import run_unittest.log;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing module decl

return 0;
}

auto testDir = buildNormalizedPath(__FILE_FULL_PATH__, "..", "..", "..");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use dirName instead.

@@ -0,0 +1,197 @@
module run_unittest.log;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine for now, but why was std.logger not an option ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No good argument against it, it just doesn't look like std.logger is able to do what want do to. That being print messages to console and/or to a file and prefix them with the test's name.

}

if (dcBackendGuess != thisDc) {
sink.error("The DC environment is not the same backend as the D compiler");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice error handling 👍

Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
ldc2 on alpine is configured to use the lld linker, however, the lld
package is not pulled in as a dependency.

Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
Signed-off-by: Andrei Horodniceanu <a.horodniceanu@proton.me>
@the-horo
Copy link
Contributor Author

Pretty hard to review the whole thing (my browser slows down when I try to scroll down...) but made a few observations.

Yup, the github webui is a pain with so many changes

@Geod24
Copy link
Member

Geod24 commented Sep 1, 2025

Yup, the github webui is a pain with so many changes

Any way we can break it down to smaller (but probably not small) PRs ?
I tried to go with marking files as "reviewed" but I can't even scroll when trying to link a file that was removed with the package introducing the code, so I would have to pull this blindly (or review from the CLI, not a great experience either).

@the-horo
Copy link
Contributor Author

the-horo commented Sep 2, 2025

Yup, the github webui is a pain with so many changes

Any way we can break it down to smaller (but probably not small) PRs ? I tried to go with marking files as "reviewed" but I can't even scroll when trying to link a file that was removed with the package introducing the code, so I would have to pull this blindly (or review from the CLI, not a great experience either).

Yeah, I think I can do that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants