CI: Intermittent codegen test failure

[Keno]

compiler/codegen                          (3) \|         failed at 2022-07-09T10:31:22.148
  | Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-6.0/build/default-macmini-x64-6-0/julialang/julia-master/julia-dc87e60444/share/julia/test/compiler/codegen.jl:272
  | Expression: was_gced

E.g. https://buildkite.com/julialang/julia-master/builds/13664#0181e302-9427-4d2f-9615-6e4aca77740b

[giordano]

I think we started seeing this with the upgrade to llvm 14 #45195, but I couldn't reproduce locally.

[George9000]

Similar, intermittent issue, after build of f71b839 (#7307)

Error in testset compiler/codegen:
Error During Test at /Volumes/ssd500/julia/test/compiler/codegen.jl:697
  Test threw exception
  Expression: readchomp(`$pfx/bin/$(Base.julia_exename()) -e 'print("no codegen!\n")'`) == "no codegen!"

Failed build:

Build with codegen failure:

julia> versioninfo()
Julia Version 1.9.0-DEV.1679
Commit f71b8398e6* (2022-10-27 16:23 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.6.0)
  CPU: 10 × Apple M1 Max
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 5 on 8 virtual cores
Environment:
  JULIA_NUM_THREADS = 4,1
  JULIA_EDITOR = Emacs
  JULIA_PKG_DEVDIR = /Users/foo/Documents/julia/dev
  
------------------
Test Summary:                                     |     Pass  Error  Broken     Total      Time
  Overall                                         | 40778074      1  352702  41130777  16m59.3s
    LinearAlgebra/bunchkaufman                    |     5689                     5689     33.4s
    LinearAlgebra/eigen                           |      951                      951     46.7s
    LinearAlgebra/qr                              |     4705                     4705     57.2s
    LinearAlgebra/lapack                          |      802                      802     17.3s
    LinearAlgebra/svd                             |      606                      606     32.5s
    LinearAlgebra/tridiag                         |     1601                     1601     25.8s
    LinearAlgebra/schur                           |      496                      496   1m25.2s
    LinearAlgebra/special                         |     2990                     2990   1m45.6s
    LinearAlgebra/cholesky                        |     2511                     2511     42.8s
    LinearAlgebra/generic                         |      632                      632     16.5s
    LinearAlgebra/lu                              |     1378                     1378     58.5s
    LinearAlgebra/dense                           |     8561                     8561   2m25.0s
    LinearAlgebra/bidiag                          |     4792                     4792   1m42.9s
    LinearAlgebra/lq                              |     3098                     3098     27.4s
    LinearAlgebra/hessenberg                      |      632                      632     28.0s
    LinearAlgebra/uniformscaling                  |      446                      446     30.9s
    LinearAlgebra/matmul                          |     1486                     1486   2m53.9s
    LinearAlgebra/givens                          |     1991                     1991      4.1s
    LinearAlgebra/pinv                            |      500                      500      5.1s
    LinearAlgebra/ldlt                            |        8                        8      0.4s
    LinearAlgebra/factorization                   |       80             16        96      2.1s
    ambiguous                                     |      108              2       110      2.7s
    compiler/datastructures                       |      283                      283      1.9s
    LinearAlgebra/adjtrans                        |      475                      475     15.8s
    compiler/effects                              |      545              1       546      1.0s
    compiler/validation                           |       28                       28      0.2s
    compiler/ssair                                |      129                      129      4.3s
    LinearAlgebra/blas                            |     1629                     1629     26.1s
    compiler/inference                            |     1249              2      1251      7.9s
    LinearAlgebra/diagonal                        |     2919                     2919   2m08.9s
    compiler/irpasses                             |      249              4       253      2.5s
    compiler/contextual                           |       12                       12      1.1s
    compiler/inline                               |      269              3       272      3.2s
    LinearAlgebra/structuredbroadcast             |      670                      670     24.6s
    compiler/codegen                              |      175      1               176      8.3s
    compiler/EscapeAnalysis/interprocedural       |       32              4        36      5.7s
    compiler/EscapeAnalysis/local                 |      347             21       368      8.4s
    compiler/AbstractInterpreter                  |       24                       24     11.2s
    strings/search                                |      876                      876      3.0s
    strings/io                                    |    12764                    12764      1.7s
    unicode/utf8                                  |       19                       19      0.1s
    strings/basic                                 |    87695                    87695      6.0s
    strings/types                                 |  2302691                  2302691      1.7s
    worlds                                        |      102                      102      0.9s
    keywordargs                                   |      151                      151      0.9s
    strings/util                                  |     1152                     1152      6.2s
    atomics                                       |     3448                     3448      7.3s
    char                                          |     1639                     1639      1.2s
    triplequote                                   |       29                       29      0.2s
    intrinsics                                    |      309                      309      1.3s
    subtype                                       |   337907             42    337949      8.4s
    hashing                                       |    12519                    12519      7.5s
    iobuffer                                      |      209                      209      0.7s
    staged                                        |       65                       65      2.9s
    dict                                          |   144450                   144450     13.0s
    numbers                                       |  1584800              1   1584801     33.5s
    tuple                                         |      670                      670      4.8s
    offsetarray                                   |      511              3       514     19.9s
    core                                          |  8445953              3   8445956     50.6s
    reduce                                        |     8598                     8598     11.0s
    LinearAlgebra/symmetric                       |     2853                     2853   2m37.5s
    simdloop                                      |      240                      240      1.0s
    vecelement                                    |      678                      678      3.8s
    intfuncs                                      |   227943                   227943     15.4s
    rational                                      |    98695              2     98697     14.4s
    copy                                          |      556                      556      1.9s
    LinearAlgebra/triangular                      |    37908                    37908   4m45.1s
    fastmath                                      |      946                      946      3.6s
    arrayops                                      |     2118              5      2123     58.4s
    functional                                    |       98                       98      3.3s
    reducedim                                     |     1225              6      1231     42.9s
    ordering                                      |       37                       37      2.2s
    path                                          |      373             12       385      0.6s
    operators                                     |    13055                    13055      4.3s
    parse                                         |    16098                    16098      2.4s
    math                                          |  1909883                  1909883     22.0s
    ccall                                         |   165130              1    165131     13.4s
    gmp                                           |     2445                     2445      2.9s
    loading                                       |   152662                   152662     15.0s
    backtrace                                     |       39              1        40      1.4s
    exceptions                                    |       70                       70      1.8s
    file                                          |     1074                     1074      5.5s
    spawn                                         |      252              4       256     14.6s
    version                                       |     2453                     2453      0.5s
    namedtuple                                    |      218                      218      1.5s
    abstractarray                                 |    55998          24806     80804   1m11.4s
    subarray                                      |   459022                   459022   2m14.3s
    mpfr                                          |     1137              1      1138      8.8s
    floatapprox                                   |       49                       49      0.7s
    iterators                                     |    86615                    86615     49.3s
    reflection                                    |      434                      434      3.9s
    regex                                         |      142                      142      1.0s
    read                                          |     3812                     3812     17.8s
    sysinfo                                       |        4                        4      0.1s
    complex                                       |     8480              2      8482      7.5s
    env                                           |       97                       97      0.3s
    rounding                                      |   150016                   150016      2.8s
    combinatorics                                 |      170                      170      3.5s
    mod2pi                                        |       80                       80      0.2s
    euler                                         |       12                       12      0.5s
    client                                        |        5                        5      0.8s
    errorshow                                     |      247                      247      4.0s
    sets                                          |     3629              1      3630     14.2s
    goto                                          |       19                       19      0.0s
    llvmcall                                      |       19                       19      0.4s
    show                                          |   128981              8    128989     19.9s
    llvmcall2                                     |       20                       20      0.1s
    some                                          |       72                       72      0.5s
    ryu                                           |    31215                    31215      1.0s
    meta                                          |       69                       69      0.8s
    float16                                       |   762093                   762093     26.4s
    stacktraces                                   |       48                       48      1.2s
    docs                                          |      239                      239      2.5s
    broadcast                                     |      525                      525     38.6s
    binaryplatforms                               |      341                      341      3.3s
    enums                                         |      100                      100      3.1s
    ranges                                        | 12110921         327683  12438604     31.3s
    atexit                                        |       40                       40      5.1s
    interpreter                                   |        3                        3      1.4s
    bitset                                        |      200                      200      1.7s
    checked                                       |     1239                     1239      5.6s
    int                                           |   736188                   736188      7.8s
    error                                         |       83                       83      1.5s
    boundscheck                                   |                              None      5.0s
    osutils                                       |       43                       43      0.0s
    cartesian                                     |      349              3       352      4.4s
    bitarray                                      |   945160                   945160   1m51.1s
    iostream                                      |       50                       50      0.9s
    specificity                                   |      175                      175      0.1s
    secretbuffer                                  |       30                       30      1.0s
    sorting                                       |    25284              9     25293   1m20.5s
    misc                                          |  1282242                  1282242     25.0s
    syntax                                        |     1662              1      1663      5.4s
    corelogging                                   |      235                      235      2.7s
    smallarrayshrink                              |       36                       36      0.2s
    opaque_closure                                |       68              1        69      0.6s
    filesystem                                    |        6                        6      0.3s
    reinterpretarray                              |      421                      421     11.1s
    SparseArrays/allowscalar                      |        9                        9      0.2s
    asyncmap                                      |      307                      307      9.0s
    SparseArrays/ambiguous                        |        1                        1      3.6s
    channels                                      |      256                      256     18.8s
    SparseArrays/fixed                            |       58                       58      4.8s
    missing                                       |      574              1       575     16.1s
    SparseArrays/cholmod                          |      611                      611     19.3s
    download                                      |                              None     31.4s
    SparseArrays/linalg_solvers                   |      364                      364      9.1s
    SparseArrays/issues                           |      337                      337     32.0s
    SparseArrays/linalg                           |     4622                     4622     42.8s
    SparseArrays/spqr                             |       98                       98      5.2s
    SparseArrays/sparsematrix_ops                 |      663                      663     29.3s
    SparseArrays/higherorderfns                   |     7172              1      7173     57.4s
    LibGit2/online                                |       10                       10      1.9s
    Dates/accessors                               |  7723858                  7723858      4.1s
    Dates/adjusters                               |     3149                     3149      3.2s
    Dates/query                                   |     1004                     1004      0.8s
    Dates/periods                                 |      953                      953     11.9s
    SparseArrays/umfpack                          |      345                      345     30.6s
    Dates/rounding                                |      315                      315      1.7s
    Dates/types                                   |      232                      232      1.1s
    Dates/ranges                                  |   350639                   350639     12.7s
    Dates/io                                      |      332                      332      8.0s
    Dates/conversions                             |      161                      161      0.7s
    Dates/arithmetic                              |      385                      385      4.9s
    Artifacts                                     |     1453                     1453      3.4s
    ArgTools                                      |      180                      180      5.2s
    CRC32c                                        |      664                      664      0.4s
    CompilerSupportLibraries_jll                  |        4                        4      0.0s
    Base64                                        |     2031                     2031      1.5s
    cmdlineargs                                   |      267              5       272   2m37.5s
    Future                                        |                              None      0.0s
    GMP_jll                                       |        1                        1      0.4s
    LibGit2/libgit2                               |      758                      758   1m08.8s
    LLD_jll                                       |        2                        2      0.3s
    LLVMLibUnwind_jll                             |        7                        7      0.1s
    LazyArtifacts                                 |        4                        4      1.4s
    LibCURL                                       |        6                        6      1.2s
    LibCURL_jll                                   |        1                        1      0.0s
    LibGit2_jll                                   |        2                        2      0.6s
    LibSSH2_jll                                   |                              None      0.0s
    LibUV_jll                                     |        1                        1      0.2s
    LibUnwind_jll                                 |                              None      0.1s
    Libdl                                         |       80              1        81      0.5s
    Logging                                       |       40                       40      0.7s
    MPFR_jll                                      |        1                        1      0.2s
    InteractiveUtils                              |      297                      297      8.3s
    MbedTLS_jll                                   |        1                        1      0.0s
    Markdown                                      |      257                      257      2.4s
    MozillaCACerts_jll                            |        1                        1      0.0s
    NetworkOptions                                |     3518                     3518      0.6s
    OpenBLAS_jll                                  |        1                        1      0.0s
    OpenLibm_jll                                  |        1                        1      0.2s
    PCRE2_jll                                     |        2                        2      0.2s
    Mmap                                          |      140                      140     13.2s
    floatfuncs                                    |      232                      232   2m53.5s
    SparseArrays/sparsematrix_constructors_indexing |     1839                     1839   2m07.5s
    Printf                                        |     1017                     1017      7.3s
    SparseArrays/sparsevector                     |    10507              4     10511   2m07.5s
    Downloads                                     |      238                      238   1m07.9s
    REPL                                          |     1579              3      1582     18.8s
    Serialization                                 |      131              1       132      3.5s
    Sockets                                       |      172                      172      5.5s
    SuiteSparse                                   |                              None      0.1s
    SuiteSparse_jll                               |        1                        1      0.5s
    TOML                                          |      415              8       423      6.3s
    Random                                        |   204840                   204840     32.8s
    LinearAlgebra/addmul                          |     5454                     5454   6m54.1s
    UUIDs                                         |     1029                     1029      0.2s
    Unicode                                       |      795                      795      2.5s
    Zlib_jll                                      |        1                        1      0.2s
    dSFMT_jll                                     |        1                        1      0.2s
    libLLVM_jll                                   |        1                        1      0.1s
    libblastrampoline_jll                         |        1                        1      0.0s
    nghttp2_jll                                   |        1                        1      0.0s
    p7zip_jll                                     |        1                        1      0.0s
    Profile                                       |      116                      116     48.7s
    Tar                                           |     3188             11      3199     17.2s
    Statistics                                    |      810                      810     31.2s
    SHA                                           |      107                      107     42.0s
    Test                                          |      458             17       475     21.8s
    FileWatching                                  |      614                      614   1m51.6s
    Pkg                                           |     3018              2      3020   8m02.3s
    precompile                                    |      217                      217     18.0s
    SharedArrays                                  |      114                      114     11.9s
    threads                                       |       40                       40   1m21.4s
    Distributed                                   |       15                       15   1m59.9s
    stress                                        |      118                      118      7.4s
    FAILURE

The global RNG seed was 0xc4b7d971e261fcd99b03aa16712d1b56.

Error in testset compiler/codegen:
Error During Test at /Volumes/ssd500/julia/test/compiler/codegen.jl:697
  Test threw exception
  Expression: readchomp(`$pfx/bin/$(Base.julia_exename()) -e 'print("no codegen!\n")'`) == "no codegen!"
  failed process: Process(`/var/folders/mh/d010xh0j5ss8yws8rknh568c0000gn/T/jl_h7uCdH/bin/julia -e 'print("no codegen!\n")'`, ProcessExited(1)) [1]

  Stacktrace:
   [1] pipeline_error
     @ ./process.jl:565 [inlined]
   [2] read(cmd::Cmd)
     @ Base ./process.jl:449
   [3] read(cmd::Cmd, #unused#::Type{String})
     @ Base ./process.jl:458
   [4] readchomp
     @ ./io.jl:961 [inlined]
   [5] macro expansion
     @ /Volumes/ssd500/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:477 [inlined]
   [6] (::Main.Test39Main_compiler_codegen.var"#36#38")(pfx::String)
     @ Main.Test39Main_compiler_codegen /Volumes/ssd500/julia/test/compiler/codegen.jl:697
ERROR: LoadError: Test run finished with errors
in expression starting at /Volumes/ssd500/julia/test/runtests.jl:93
gmake[1]: *** [Makefile:25: all] Error 1
gmake: *** [Makefile:571: testall] Error 2




% gmake test-compiler                                                                                                                                      14:42
:
:/Volumes/ssd500/julia
:
Whitespace check found no issues.
    JULIA test/compiler
Running parallel tests with:
  nworkers() = 8
  nthreads() = 4
  Sys.CPU_THREADS = 8
  Sys.total_memory() = 64.000 GiB
  Sys.free_memory() = 39.524 GiB

Test                                (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
compiler/validation                      (5) |        started at 2022-10-27T14:46:12.450
compiler/ssair                           (6) |        started at 2022-10-27T14:46:12.513
compiler/datastructures                  (2) |        started at 2022-10-27T14:46:12.537
compiler/inference                       (3) |        started at 2022-10-27T14:46:12.537
compiler/effects                         (4) |        started at 2022-10-27T14:46:12.537
compiler/inline                          (9) |        started at 2022-10-27T14:46:12.538
compiler/irpasses                        (7) |        started at 2022-10-27T14:46:12.538
compiler/codegen                         (8) |        started at 2022-10-27T14:46:12.538
compiler/validation                      (5) |     0.37 |   0.00 |  1.2 |      34.64 |   309.03
compiler/contextual                      (5) |        started at 2022-10-27T14:46:13.044
compiler/effects                         (4) |     1.45 |   0.02 |  1.1 |     138.15 |   323.69
compiler/AbstractInterpreter             (4) |        started at 2022-10-27T14:46:14.050
compiler/contextual                      (5) |     1.34 |   0.02 |  1.3 |      69.58 |   316.45
compiler/EscapeAnalysis/local            (5) |        started at 2022-10-27T14:46:14.388
compiler/datastructures                  (2) |     2.61 |   0.02 |  0.9 |     224.45 |   322.66
compiler/EscapeAnalysis/interprocedural  (2) |        started at 2022-10-27T14:46:15.213
compiler/irpasses                        (7) |     3.24 |   0.04 |  1.3 |     332.63 |   360.78
compiler/inline                          (9) |     3.57 |   0.09 |  2.5 |     251.08 |   340.11
compiler/ssair                           (6) |     4.92 |   0.04 |  0.8 |     337.96 |   354.66
      From worker 8:	ERROR: ERROR: fatal: error thrown and no exception handler available.
      From worker 8:	ErrorException("`ccall` requires the compiler")
      From worker 8:	enable_finalizers at gcutils.jl:124
      From worker 8:	unlock at lock.jl:177
      From worker 8:	unlock at stream.jl:284
      From worker 8:	unlock at show.jl:397
      From worker 8:	print at strings/io.jl:49
      From worker 8:	println at strings/io.jl:75
      From worker 8:	#show_full_backtrace#923 at errorshow.jl:591
      From worker 8:	show_full_backtrace at errorshow.jl:587
      From worker 8:	show_backtrace at errorshow.jl:791
      From worker 8:	#showerror#900 at errorshow.jl:90
      From worker 8:	showerror at errorshow.jl:86
      From worker 8:	show_exception_stack at errorshow.jl:894
      From worker 8:	display_error at client.jl:111
compiler/EscapeAnalysis/interprocedural  (2) |     4.90 |   0.12 |  2.4 |     474.39 |   388.22
compiler/codegen                         (8) |         failed at 2022-10-27T14:46:20.248
Error During Test at /Volumes/ssd500/julia/test/compiler/codegen.jl:697
  Test threw exception
  Expression: readchomp(`$pfx/bin/$(Base.julia_exename()) -e 'print("no codegen!\n")'`) == "no codegen!"
  failed process: Process(`/var/folders/mh/d010xh0j5ss8yws8rknh568c0000gn/T/jl_J7KW7x/bin/julia -e 'print("no codegen!\n")'`, ProcessExited(1)) [1]

  Stacktrace:
   [1] pipeline_error
     @ ./process.jl:565 [inlined]
   [2] read(cmd::Cmd)
     @ Base ./process.jl:449
   [3] read(cmd::Cmd, #unused#::Type{String})
     @ Base ./process.jl:458
   [4] readchomp
     @ ./io.jl:961 [inlined]
   [5] macro expansion
     @ /Volumes/ssd500/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:477 [inlined]
   [6] (::Main.Test88Main_compiler_codegen.var"#36#38")(pfx::String)
     @ Main.Test88Main_compiler_codegen /Volumes/ssd500/julia/test/compiler/codegen.jl:697

compiler/inference                       (3) |     8.13 |   0.12 |  1.4 |     947.03 |   415.88
compiler/EscapeAnalysis/local            (5) |     7.83 |   0.15 |  1.9 |     719.95 |   430.77
compiler/AbstractInterpreter             (4) |     9.73 |   0.18 |  1.8 |     748.89 |   485.28

Test Summary:                             | Pass  Error  Broken  Total   Time
  Overall                                 | 3355      1      35   3391  12.5s
    compiler/validation                   |   28                    28   1.3s
    compiler/effects                      |  545              1    546   2.4s
    compiler/contextual                   |   12                    12   1.3s
    compiler/datastructures               |  296                   296   3.6s
    compiler/irpasses                     |  249              4    253   4.1s
    compiler/inline                       |  269              3    272   4.5s
    compiler/ssair                        |  129                   129   5.8s
    compiler/EscapeAnalysis/interprocedural |   32              4     36   4.9s
    compiler/codegen                      |  175      1            176   8.6s
    compiler/inference                    | 1249              2   1251   9.1s
    compiler/EscapeAnalysis/local         |  347             21    368   7.8s
    compiler/AbstractInterpreter          |   24                    24   9.7s
    FAILURE

The global RNG seed was 0xd462b65b268765e0c639da95a099d6e2.

Error in testset compiler/codegen:
Error During Test at /Volumes/ssd500/julia/test/compiler/codegen.jl:697
  Test threw exception
  Expression: readchomp(`$pfx/bin/$(Base.julia_exename()) -e 'print("no codegen!\n")'`) == "no codegen!"
  failed process: Process(`/var/folders/mh/d010xh0j5ss8yws8rknh568c0000gn/T/jl_J7KW7x/bin/julia -e 'print("no codegen!\n")'`, ProcessExited(1)) [1]

  Stacktrace:
   [1] pipeline_error
     @ ./process.jl:565 [inlined]
   [2] read(cmd::Cmd)
     @ Base ./process.jl:449
   [3] read(cmd::Cmd, #unused#::Type{String})
     @ Base ./process.jl:458
   [4] readchomp
     @ ./io.jl:961 [inlined]
   [5] macro expansion
     @ /Volumes/ssd500/julia/usr/share/julia/stdlib/v1.9/Test/src/Test.jl:477 [inlined]
   [6] (::Main.Test88Main_compiler_codegen.var"#36#38")(pfx::String)
     @ Main.Test88Main_compiler_codegen /Volumes/ssd500/julia/test/compiler/codegen.jl:697
ERROR: LoadError: Test run finished with errors
in expression starting at /Volumes/ssd500/julia/test/runtests.jl:93
gmake[1]: *** [Makefile:25: compiler] Error 1
gmake: *** [Makefile:578: test-compiler] Error 2

Successful build (did have isolated depot path):

Details

julia> versioninfo()
Julia Version 1.9.0-DEV.1679
Commit f71b8398e6 (2022-10-27 16:23 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.6.0)
  CPU: 10 × Apple M1 Max
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 5 on 8 virtual cores
Environment:
  JULIA_NUM_THREADS = 4,1
  JULIA_DEPOT_PATH = /Volumes/ssd500/juliabuildtemp
  
------------------

Test Summary:                     | Pass  Total  Time
Threads.Conditions must be locked |    5      5  0.0s
Test Summary:       | Pass  Total  Time
IntrusiveLinkedList |   72     72  0.1s
Test Summary:        | Pass  Total  Time
@spawn interpolation |   10     10  0.3s
Test Summary:        | Pass  Total  Time
@async interpolation |    8      8  0.0s
Test Summary:                   | Pass  Total  Time
spawn interpolation: macrocalls |    2      2  1.7s
Test Summary: | Pass  Total  Time
Co-schedule   |    1      1  0.0s
Test Summary:                 | Pass  Total  Time
Threads.foreach(f, ::Channel) |    7      7  1.2s
Test Summary:     | Pass  Total  Time
Task-local random |    4      4  0.1s
Test Summary: | Pass  Total  Time
Racy `@spawn` |    1      1  0.1s
Test Summary:   | Pass  Total  Time
package loading |    1      1  0.2s
Test Summary:                     | Pass  Total  Time
Threads.Conditions must be locked |    5      5  0.0s
Test Summary:       | Pass  Total  Time
IntrusiveLinkedList |   72     72  0.1s
Test Summary:        | Pass  Total  Time
@spawn interpolation |   10     10  0.3s
Test Summary:        | Pass  Total  Time
@async interpolation |    8      8  0.0s
Test Summary:                   | Pass  Total  Time
spawn interpolation: macrocalls |    2      2  1.6s
Test Summary: | Pass  Total  Time
Co-schedule   |    1      1  0.0s
Test Summary:                 | Pass  Total  Time
Threads.foreach(f, ::Channel) |    7      7  1.2s
Test Summary:     | Pass  Total  Time
Task-local random |    4      4  0.0s
Test Summary: | Pass  Total  Time
Racy `@spawn` |    1      1  0.1s
Test Summary:   | Pass  Total  Time
package loading |    1      1  0.1s
Test Summary:                     | Pass  Total  Time
Threads.Conditions must be locked |    5      5  0.0s
Test Summary:       | Pass  Total  Time
IntrusiveLinkedList |   72     72  0.1s
Test Summary:        | Pass  Total  Time
@spawn interpolation |   10     10  0.3s
Test Summary:        | Pass  Total  Time
@async interpolation |    8      8  0.0s
Test Summary:                   | Pass  Total  Time
spawn interpolation: macrocalls |    2      2  1.6s
Test Summary: | Pass  Total  Time
Co-schedule   |    1      1  0.0s
Test Summary:                 | Pass  Total  Time
Threads.foreach(f, ::Channel) |    7      7  1.4s
Test Summary:     | Pass  Total  Time
Task-local random |    4      4  0.0s
Test Summary: | Pass  Total  Time
Racy `@spawn` |    1      1  0.1s
Test Summary:   | Pass  Total  Time
package loading |    1      1  0.1s
Test Summary:                     | Pass  Total  Time
Threads.Conditions must be locked |    5      5  0.0s
Test Summary:       | Pass  Total  Time
IntrusiveLinkedList |   72     72  0.1s
Test Summary:        | Pass  Total  Time
@spawn interpolation |   10     10  0.3s
Test Summary:        | Pass  Total  Time
@async interpolation |    8      8  0.0s
Test Summary:                   | Pass  Total  Time
spawn interpolation: macrocalls |    2      2  1.7s
Test Summary: | Pass  Total  Time
Co-schedule   |    1      1  0.0s
Test Summary:                 | Pass  Total  Time
Threads.foreach(f, ::Channel) |    7      7  1.4s
Test Summary:     | Pass  Total  Time
Task-local random |    4      4  0.0s
Test Summary: | Pass  Total  Time
Racy `@spawn` |    1      1  0.1s
Test Summary:   | Pass  Total  Time
package loading |    1      1  0.1s
threads                                          (1) |    82.47 |   0.01 |  0.0 |      36.06 |   485.78
Distributed                                      (1) |        started at 2022-10-27T17:12:21.848
┌ Warning: Skipping a thread-unsafe test because `Threads.maxthreadid() > 1`
│   Threads.maxthreadid() = 5
└ @ Main /Volumes/ssd500/julia/usr/share/julia/stdlib/v1.9/Distributed/test/distributed_exec.jl:161
┌ Warning: Skipping a thread-unsafe test because `Threads.maxthreadid() > 1`
│   Threads.maxthreadid() = 5
└ @ Main /Volumes/ssd500/julia/usr/share/julia/stdlib/v1.9/Distributed/test/distributed_exec.jl:161
[ Info: SO_REUSEPORT is unsupported, skipping reuseport tests
      From worker 28:
      From worker 28:	[16349] signal (15): Terminated: 15
      From worker 28:	in expression starting at none:0
      From worker 28:	__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 28:	unknown function (ip: 0x0)
      From worker 28:	__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 28:	unknown function (ip: 0x0)
      From worker 28:	__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 28:	unknown function (ip: 0x0)
      From worker 28:	__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 28:	unknown function (ip: 0x0)
      From worker 28:	kevent at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 28:	unknown function (ip: 0x0)
      From worker 28:	Allocations: 3252889 (Pool: 3250230; Big: 2659); GC: 5
┌ Warning: rmprocs: process 1 not removed
└ @ Distributed /Volumes/ssd500/julia/usr/share/julia/stdlib/v1.9/Distributed/src/cluster.jl:1049
Distributed                                      (1) |   118.70 |   0.01 |  0.0 |      31.85 |   487.53
stress                                           (1) |        started at 2022-10-27T17:14:20.544
(libuv) kqueue(): Too many open files
stress                                           (1) |     7.54 |   0.01 |  0.1 |      44.58 |   488.31

Test Summary: |     Pass  Broken     Total      Time
  Overall     | 40769966  352702  41122668  19m54.4s
    SUCCESS

[giordano]

The error looks completely different?

[George9000]

Commonality is the intermittent failure in codegen. Perhaps this is completely unrelated, but I wondered if there was a higher level issue producing the intermittent nature of both errors.

[vtjnash]

I haven't seen this one again