Julia tests don't pass

[timholy]

Here's a status list for progress with JuliaInterpreter, running Julia's own test suite. It's organized by the number of passes, failures (ideally should be 0), errors (ideally 0), broken (these are not JuliaInterpreter's problem), and aborted blocks (tests that took too long, given the settings). Some tests error outside a @test (marked by "X" in the table below) and others cause Julia itself to exit (marked by ☠️)
The tests below were run on a multiprocessor server from the Linux command line with

$ JULIA_CPU_THREADS=8 julia --startup-file=no juliatests.jl --nstmts 1000000 --skip compiler

The --nstmts 1000000 allows you to control the maximum number of interpreter statements per lowered block of code; tests that require more than this are marked as being "aborted." The default setting is 10000 (10^4). The higher you make this number, in general the more tests that should finish, but of course also the longer the suite will take to run. On my laptop, running with 2 worker processes the entire suite takes less than 5 minutes to complete using the default settings.

The remaining arguments are the same as given to Julia's own test/runtests.jl: you can either provide a list of tests you want to run (e.g., julia --startup-file=no juliatests.jl ambiguous), or you can list some to skip (here, all the compiler/* tests). "Blank" indicates that one is running all the tests, so the line above runs everything except those in compiler/*.

The key point of having a status list is that it allows us to discover issues with JuliaInterpreter; consequently, the next step is to use these results to fix those problems. Help is very much wanted! Here are good ways to help out:

• (moderate) investigate failures and file an issue with a MWE. Highest priority should probably go to ones that caused errors or process exit (note: with the possible exceptions of channels, worlds, and arrayops, it appears that most such errors are due to a single cause, MWE of char crash #28; deleting this block and rebuilding Julia fixes them) (EDIT: all of these appear to be fixed now). Then would be error that occurs outside of tests (the Xs), errors that occur inside a @test (those marked as Errors by the test suite), failures, and of lowest priority the aborted blocks. Note that aborted blocks can lead to test failures due to repeating work (see Compiled resumers #44), so many of these may go away if you increase nstmts. However, note that aborted blocks could indicate that the interpreter has incorrectly gotten itself stuck in an infinite loop (yes, the author has seen that happen), and as a consequence it's possible that some of these too are actually errors.
• (hard) fix the bugs.

A good way to get started is to pick one test that's exhibiting problems, and uncomment these lines. Then, the easiest way to dive into this is to run tests in a REPL session, e.g.,

include("utils.jl")
const juliadir = dirname(dirname(Sys.BINDIR))
const testdir = joinpath(juliadir, "test")
configure_test()
nstmts = 10000
run_test_by_eval("ambiguous", joinpath(testdir, "ambiguous.jl"), nstmts)  # replace ambiguous with whatever test you want to run

from within JuliaInterpreter's test/ directory. If you get failures, make sure you first check whether they go away if you increase nstmts (typically by 10x or more).

When you see test errors, the expression printed right above it is the one causing the problem. Go into the source code and copy the relevant source lines into a quote block. Once you have a minimal expression ex that triggers a problem, do this:

modexs, _ = JuliaInterpreter.split_expressions(m, ex)
for modex in modexs
    frame = JuliaInterpreter.prepare_thunk(modex)
    nstmtsleft = nstmts
    while true
        ret, nstmtsleft = evaluate_limited!(frame, nstmtsleft, true)
        if isa(ret, Aborted)
            run_compiled(frame)
            break
        elseif isa(ret, Some)
            break
        end
    end
end

where m is the module you want to execute this in. You may want to do

module JuliaTests
using Test, Random
end
m = JuliaTests

to isolate the tests from your current session.

To diagnose problems in greater detail, uncommenting these lines can be a great first start.

Without further ado, here's the current list (note the time of the run to determine how current this is):

Julia Version 1.1.1-pre.0
Commit a84cf6f56c (2019-01-22 04:33 UTC)
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.1 (ORCJIT, haswell)
Test run at: 2019-02-26T11:36:49.456

Maximum number of statements per lowered expression: 1000000

Test file	Passes	Fails	Errors	Broken	Aborted blocks
ambiguous	62	0	0	2	0
subarray	281	0	0	0	1
strings/basic	87293	0	0	0	3
strings/search	522	0	0	0	0
strings/util	449	0	0	0	0
strings/io	12749	0	0	0	1
strings/types	2302688	0	0	0	3
unicode/utf8	19	0	0	0	0
core	X	X	X	X	X
worlds	X	X	X	X	X
keywordargs	126	0	1	0	0
numbers	1387242	0	0	0	9
subtype	X	X	X	X	X
char	1522	0	0	0	0
triplequote	28	0	0	0	0
intrinsics	44	0	0	0	0
dict	X	X	X	X	X
hashing	X	X	X	X	X
iobuffer	200	0	0	0	0
staged	55	5	0	0	0
offsetarray	341	11	0	0	1
arrayops	1833	0	0	0	7
tuple	483	0	1	0	0
reduce	292	0	0	0	2
reducedim	689	0	0	0	1
abstractarray	1791	0	0	0	1
intfuncs	4410	0	0	0	0
simdloop	X	X	X	X	X
vecelement	X	X	X	X	X
rational	97522	0	0	0	2
bitarray	897826	0	0	0	9
copy	511	0	1	0	1
math	X	X	X	X	X
fastmath	907	3	3	0	0
functional	95	0	0	0	0
iterators	1555	0	0	0	2
operators	12922	0	0	0	1
path	274	0	0	12	2
ccall	X	X	X	X	X
parse	10303	0	0	0	1
loading	2272	289	4	0	9
bigint	2156	0	0	0	4
sorting	4864	0	0	0	4
spawn	X	X	X	X	X
backtrace	5	9	12	1	0
exceptions	27	19	6	0	0
file	X	X	X	X	X
read	X	X	X	X	X
version	2468	0	0	0	1
namedtuple	152	0	8	1	0
mpfr	932	0	0	0	0
broadcast	418	0	5	0	2
complex	8250	0	0	2	1
floatapprox	49	0	0	0	0
reflection	X	X	X	X	X
regex	29	0	0	0	0
float16	124	0	0	0	0
combinatorics	98	0	0	0	1
sysinfo	2	0	0	0	0
env	53	0	0	0	0
rounding	112720	0	0	0	2
ranges	12109069	2	0	327755	7
mod2pi	80	0	0	0	0
euler	12	0	0	0	5
show	X	X	X	X	X
errorshow	X	X	X	X	X
sets	773	0	0	1	1
goto	X	X	X	X	X
llvmcall	X	X	X	X	X
llvmcall2	6	0	0	0	0
grisu	683	1	0	0	1
some	64	0	0	0	0
meta	X	X	X	X	X
stacktraces	X	X	X	X	X
docs	X	X	X	X	X
misc	X	X	X	X	X
threads	X	X	X	X	X
enums	88	0	0	0	0
cmdlineargs	X	X	X	X	X
int	10727	0	0	0	0
checked	1219	0	0	0	0
bitset	192	0	0	0	0
floatfuncs	134	0	0	0	1
boundscheck	X	X	X	X	X
error	30	0	0	0	0
cartesian	7	0	0	0	0
osutils	42	0	0	0	0
channels	X	X	X	X	X
iostream	6	0	2	0	0
secretbuffer	16	0	0	0	0
specificity	X	X	X	X	X
reinterpretarray	118	0	0	0	1
syntax	X	X	X	X	X
logging	117	2	0	0	0
missing	406	0	0	1	1
asyncmap	292	0	0	0	0
SparseArrays/higherorderfns	7000	79	0	73	7
SparseArrays/sparse	2184	0	0	0	19
SparseArrays/sparsevector	9921	0	0	0	5
Pkg/resolve	182	0	0	0	3
LinearAlgebra/triangular	33194	0	0	0	2
LinearAlgebra/qr	3120	0	0	0	1
LinearAlgebra/dense	7720	0	0	0	7
LinearAlgebra/matmul	711	0	0	0	3
LinearAlgebra/schur	390	0	0	0	1
LinearAlgebra/special	1068	0	0	0	3
LinearAlgebra/eigen	406	0	0	0	2
LinearAlgebra/bunchkaufman	5145	0	0	0	1
LinearAlgebra/svd	412	0	0	0	1
LinearAlgebra/lapack	778	2	0	0	3
LinearAlgebra/tridiag	1222	0	0	0	2
LinearAlgebra/bidiag	1946	0	0	0	1
LinearAlgebra/diagonal	1607	0	0	0	2
LinearAlgebra/cholesky	2194	0	0	0	1
LinearAlgebra/lu	1191	0	0	0	3
LinearAlgebra/symmetric	1982	0	0	0	1
LinearAlgebra/generic	430	0	0	0	3
LinearAlgebra/uniformscaling	338	0	0	0	0
LinearAlgebra/lq	1253	0	0	0	1
LinearAlgebra/hessenberg	40	0	0	0	0
LinearAlgebra/blas	628	0	0	0	1
LinearAlgebra/adjtrans	253	0	0	0	1
LinearAlgebra/pinv	288	0	0	0	1
LinearAlgebra/givens	1840	0	0	0	1
LinearAlgebra/structuredbroadcast	408	0	0	0	2
LibGit2/libgit2	219	2	59	1	0
Dates/accessors	X	X	X	X	X
Dates/adjusters	X	X	X	X	X
Dates/query	988	0	0	0	0
Dates/periods	681	0	0	0	0
Dates/ranges	349123	0	0	0	5
Dates/rounding	296	0	0	0	0
Dates/types	171	0	0	0	0
Dates/io	258	0	0	0	1
Dates/arithmetic	318	0	0	0	0
Dates/conversions	160	0	0	0	0
Base64	1015	0	0	0	1
CRC32c	658	0	6	0	0
DelimitedFiles	80	0	1	0	1
FileWatching	X	X	X	X	X
Future	0	0	0	0	0
InteractiveUtils	104	3	2	0	4
Libdl	X	X	X	X	X
Logging	35	1	0	0	1
Markdown	232	0	0	0	0
Mmap	131	0	0	0	1
Printf	701	38	0	0	0
Profile	10	0	0	0	2
REPL	990	0	0	5	0
Random	203081	4	0	0	7
SHA	X	X	X	X	X
Serialization	105	1	1	0	0
Sockets	X	X	X	X	X
Statistics	606	0	0	0	4
SuiteSparse	770	0	0	0	0
Test	X	X	X	X	X
UUIDs	22	0	0	0	0
Unicode	752	0	0	0	0

[KristofferC]

For + string/basic we get stuck at

ex = quote
            let b, n
                for T = (UInt8, Int8, UInt16, Int16, UInt32, Int32, UInt64, Int64, UInt128, Int128, BigInt), b = 2:62, _ = 1:10
                    n = if T != BigInt
                            rand(T)
                        else
                            BigInt(rand(Int128))
                        end
                end
            end
        end
end

I think we will have some problem finishing tests that do this type of combinatorial testing.

[timholy]

Thanks! Agreed this is an issue. Plans here: #12 (comment)

[timholy]

The numbers seem substantially better on a relatively recent build of Julia's master branch. I found that julia 1.0.x doesn't even finish, at least on an earlier version the script (though I didn't investigate why).

I also ran with a much higher nstmts threshold, so this is much more comprehensive than what I first posted. And it wasn't even that slow, maybe 10 minutes on an 8-worker run.

[timholy]

Now that #24 is merged, OP updated to provide helpful links about how to debug problems.

[KristofferC]

One worker for me seems to get stuck at something and consume a pretty hefty amount of memory

I'll try figure out what expression it is getting stuck at.

[KristofferC]

With the stdlibs now running, it seems the REPL tests deadlock. Perhaps this is something to do with the tasks in the REPL tests (which are notorious for deadlocking if something goes wrong). In this case the nstmts don't save us so perhaps another option is to give a timer to abort a test if it has been running too long?