Add "project" mode for faster code coverage and allocation tracking

[IanButterworth]

Adds a project option to --code-coverage and --track-allocation for just tracking the active project module and its submodules.

For instance with code coverage:

$ pwd
/usr/Foo.jl

$ julia --code-coverage=project --project -e 'using Foo; Foo.bar()`

after which coverage files will be generated within the Foo module and its submodules, but no other imported modules.

The idea is to increase the efficiency of code coverage and allocation tracking given that the current most narrow option is user which is any non-Core module, including stdlibs.

The julia-runtest github action defaults to having code coverage on which Pkg sets to user mode, so is currently spending the majority of code-coverage tracking time in modules that won't be evaluated for code coverage by the coverage tooling that just evaluates coverage files the active repo.

Benchmarks

A favorable case, where the project code is small, but dep code is large
src/Foo.jl

module Foo
import PNGFiles
function bar(n = 10000)
    for _ in 1:3
        @time Threads.@threads for tid in 1:Threads.nthreads()
            f = "$(tid)_img.png"
            for _ in 1:n
                PNGFiles.save(f, rand(UInt8, 100, 100))
                PNGFiles.load(f)
            end
        end
    end
end
end # module Foo

Code Coverage

/Foo$ julia --project --code-coverage=all -e "using Foo; Foo.bar(100)"
 16.068388 seconds (1.71 M allocations: 152.005 MiB, 0.67% gc time, 4.32% compilation time)
 15.516894 seconds (33.76 k allocations: 65.759 MiB)
 15.424748 seconds (33.77 k allocations: 65.759 MiB)
... --code-coverage=user
  0.533590 seconds (2.02 M allocations: 170.098 MiB, 90.94% compilation time)
  0.068086 seconds (33.76 k allocations: 65.758 MiB)
  0.206111 seconds (33.76 k allocations: 65.758 MiB, 64.23% gc time)

all is a lot slower than the others, so compare the others with more iterations

/Foo$ julia --project --code-coverage=user -e "using Foo; Foo.bar(10000)"
  7.720869 seconds (5.24 M allocations: 6.517 GiB, 2.93% gc time, 6.13% compilation time)
  6.893507 seconds (3.36 M allocations: 6.421 GiB, 0.33% gc time)
  6.883390 seconds (3.36 M allocations: 6.421 GiB, 0.43% gc time)
... --code-coverage=project
  3.343359 seconds (5.35 M allocations: 6.523 GiB, 8.20% gc time, 15.61% compilation time)
  2.751018 seconds (3.36 M allocations: 6.421 GiB, 3.41% gc time)
  2.848022 seconds (3.36 M allocations: 6.421 GiB, 3.25% gc time)
... --code-coverage=none
  3.423068 seconds (5.03 M allocations: 6.504 GiB, 6.62% gc time, 12.47% compilation time)
  2.760789 seconds (3.36 M allocations: 6.421 GiB, 1.10% gc time)
  2.765776 seconds (3.36 M allocations: 6.421 GiB, 1.05% gc time)

So project is ~2x faster than user, and the .cov files for src/Foo.jl are near-identical for user and project (might code-cov iterators not be thread-safe?)

In real-world testing, it might be hard to see this in above CI variability.

Here I ran OrdinaryDiffEq's test suite on 5f6b8a1 (before this PR) and 27f01f2 (this PR)

A few testsets as examples:

before  -> after
6m54.2s -> 6m43.3s
6m01.6s -> 6m48.2s
55.7s   -> 1m03.9s
8m31.9s -> 7m23.3s

Tracking Allocations

/Foo$ julia --project --track-allocation=all -e "using Foo; Foo.bar(100)"
 38.192262 seconds (2.02 M allocations: 169.261 MiB, 2.38% compilation time)
 37.371601 seconds (33.76 k allocations: 65.759 MiB)
 37.923109 seconds (33.76 k allocations: 65.759 MiB, 0.33% gc time)
... --track-allocation=user
  1.041746 seconds (1.61 M allocations: 146.495 MiB, 10.20% gc time, 67.78% compilation time)
  0.240931 seconds (33.76 k allocations: 65.759 MiB)
  0.248995 seconds (33.77 k allocations: 65.759 MiB)

again, all is a lot slower than the others, so compare the others with more iterations

/Foo$ julia --project --track-allocation=user -e "using Foo; Foo.bar(10000)"
 24.162710 seconds (5.14 M allocations: 6.510 GiB, 0.95% gc time, 3.02% compilation time)
 22.962427 seconds (3.36 M allocations: 6.421 GiB, 0.09% gc time)
 23.114932 seconds (3.36 M allocations: 6.421 GiB, 0.13% gc time)
... --track-allocation=project
  3.487994 seconds (4.90 M allocations: 6.498 GiB, 6.20% gc time, 11.88% compilation time)
  2.909793 seconds (3.36 M allocations: 6.421 GiB, 1.08% gc time)
  2.908083 seconds (3.36 M allocations: 6.421 GiB, 0.99% gc time)
... --track-allocation=none
  3.239760 seconds (5.04 M allocations: 6.505 GiB, 6.70% gc time, 12.27% compilation time)
  2.817924 seconds (3.36 M allocations: 6.421 GiB, 1.00% gc time)
  2.806771 seconds (3.36 M allocations: 6.421 GiB, 0.92% gc time)

So project is ~8x faster than user, and the .mem files for src/Foo.jl are identical for user and project

Todo

• Fix tests
• Add News
• Revert the Pkg branch change. (Pkg does an odd thing of testing the project code from a temporary env, so the actual "active project" for this purpose needs a setting that Pkg can override)

Closes #41626
May close #36142

[IanButterworth]

a couple of different packages that they want to get code coverage for at the same time.

That's possible here via

push!(Base.explicit_project_dirs, dir1, dir2)

[IanButterworth]

I think as long as people can directly set --code-coverage=user in julia-runtest to override the Pkg setting, this will work in 99% of cases out of the box, and also be adjustable for edge cases.. I think.

[IanButterworth]

@JeffBezanson do you think this is a good idea? If so, I'd be happy to fix up tests etc.

[JeffBezanson]

This should probably not be added, since it's not an intrinsic property of a module but can change based on context.

[IanButterworth]

Triage suggested simplifying this instead with a --code-coverage=@ModuleName api and checking Base.moduleroot(mod) against that

[IanButterworth]

Replaced by #44359