Making Unit Tests Work!

[zaskar9]

This issue is intended to track the current state of the unit tests. Below is a summary of all tests that need work. They fall into the Unsupported, Failed, and Unexpectedly Passed categories. For every test, there is a brief summary of the problem or a description of the feature that is missing.

Unsupported

• loop_1.mod: LOOP and EXIT statement missing → infinite loop
• module_unterminated.mod: no terminating dot at end of module → infinite loop
• procedure_missing_return.mod: no error on missing/unreachable RETURN

Failed

• arithmetic_12.mod: data type BYTE missing
• array_5.mod: data type PROCEDURE missing
• case_1.mod: CASE statement missing
• loop_2.mod: LOOP statement missing
• system_6.mod: problem in Out.Real (NaN instead of INF/-INF)
• system_7.mod: problem with signedness of 64-bit hexadecimal values
• fail_on_extra_semicolon.mod: expected fail

Unexpectedly Passed

• arithmetic_3.mod: should trigger integer overflow trap
• arithmetic_5.mod: divisor cannot be negative (cf. O07.8.2.2)
• arithmetic_6.mod: divisor cannot be negative (cf. O07.8.2.2)
• arithmetic_9.mod: should trigger integer overflow trap
• arithmetic_10.mod: should trigger division by zero trap
• arithmetic_11.mod: Oberon has no implicit type conversions, should be disabled by default
• arithmetic_15.mod: should trigger division by zero trap
• array_6.mod: should trigger out-of-bounds trap
• procedure_4.mod: Oberon-07 scoping rules not enforced: intermediate declarations cannot be accessed in nested procedures
• if_with_trailing_end.mod: parser does not complain about superfluous END in IF-ELSIF-ELSE statement

Overall State

  Unsupported        :  2
  Passed             : 68
  Expectedly Failed  :  7
  Failed             :  5
  Unexpectedly Passed:  9

[zaskar9]

Latest version of oberon-lang includes fixes to the scanner and parser to address the following issues.

• module_unterminated.mod: no terminating dot at end of module → infinite loop
• fail_on_extra_semicolon.mod: expected fail
• if_with_trailing_end.mod: parser does not complain about superfluous END in IF-ELSIF-ELSE statement

[tenko]

Result MSYS2:

Failed Tests (7):
Oberon :: codegen/arithmetic_12.mod
Oberon :: codegen/array_5.mod
Oberon :: codegen/case_1.mod
Oberon :: codegen/const_long_hex.mod
Oberon :: codegen/const_longint.mod
Oberon :: codegen/loop_2.mod
Oberon :: codegen/system_6.mod

Unexpectedly Passed Tests (7):
Oberon :: codegen/arithmetic_11.mod
Oberon :: codegen/arithmetic_15.mod
Oberon :: codegen/arithmetic_3.mod
Oberon :: codegen/arithmetic_5.mod
Oberon :: codegen/arithmetic_6.mod
Oberon :: codegen/arithmetic_9.mod
Oberon :: codegen/procedure_4.mod

Unsupported : 2
Passed : 66
Expectedly Failed : 9
Failed : 7
Unexpectedly Passed: 7

codegen/const_long_hex.mod and codegen/const_longint.mod fails for me.
I am guessing this is related to use of long which is 32bit on my platform.

[zaskar9]

I just pushed two commits to the master branch that update Scanner, AST, Sema and CodeGen to use fixed-width integers (i.e., int64_t, int32_t, and int16_t for LONGINT, INTEGER, and SHORTINT as well as uint8_t for CHAR). Does that improve portability? It surely makes things more consistent.

[tenko]

codegen/const_long_hex.mod is now fixed.

codegen/const_longint.mod still failed.

I added the following to rt_out_int :

#ifdef MINGW32
sprintf(buf, "%lld", i);
#else
sprintf(buf, "%" PRId64, i);
#endif

This fixes the problem.
Probably the macro PRId64 is not correct on my platform.

[zaskar9]

Since I currently don't have access to my Windows virtual machine, due to Easter travel, I was wondering if you could try the following. If you build the compiler in DEBUG mode and run it with the -v flag, it will print out a lot (like really a lot) of diagnostic information that will help me to localize the problem.

For example, if I run const_longint.mod in DEBUG it prints (among other things) the module how the compiler "sees" it. On my machine this look as shown below.

MODULE ConstLongInt(*Scope:0*);
IMPORT Out; 

   CONST longintmax(*Scope:1*) = 9223372036854775807(*L*)(*Type:LONGINT*);
         longintmin(*Scope:1*) = -9223372036854775808(*L*)(*Type:LONGINT*);

   PROCEDURE Test(*Scope:1*)();
   BEGIN
      Out.Long(9223372036854775807(*L*), 0(*S*));
      Out.Ln();
      Out.Long(-9223372036854775808(*L*), 0(*S*));
      Out.Ln()
   END Test;

BEGIN
   Test()
END ConstLongInt.

The first thing to make sure is that all constant literals are annotated by the correct type (S = SHORTINT, I = INTEGER, L = LONGINT, etc). If this is not the case, the problem most likely resides in the Scanner code assigns types to literals. If all literals are correctly typed, the problem will be in Out.Long().

As the compiler can do more and more, I am currently trying to implement more parts of Out and Texts in Oberon rather than in C. So, if the problem is in the output procedure, it hopefully resolves itself "automatically", but if it is in the Scanner, I will put it on the to-do list.

[tenko]

No problem. Output MSYS2:

MODULE ConstLongInt(Scope:0);
IMPORT Out;

CONST longintmax(Scope:1) = 9223372036854775807(L)(Type:LONGINT);
longintmin(Scope:1) = -9223372036854775808(L)(Type:LONGINT);

PROCEDURE Test(Scope:1)();
BEGIN
Out.Long(9223372036854775807(L), 0(S));
Out.Ln();
Out.Long(-9223372036854775808(L), 0(S));
Out.Ln()
END Test;

BEGIN
Test()
END ConstLongInt.

The Scanner is then OK.

Sidenote on VM images: The NativeOberon image is available at. This
works suprisingly well and really fun to explore. Happy easter!

[zaskar9]

On my Windows machine (VMware, Windows 11, x86_64), I have a strange problem that obfuscates a clear view of the status of the unit tests. The issue is related to some modules failing in JIT mode due to unresolved symbols. Using the MSYS/Clang64 toolchain the missing symbol is ___chkstk_ms, while using the MSVC toolchain the missing symbols are __security_cookie and __security_check_cookie. Both symbols are related to the stack protection mechanisms of the respective toolchains.

I checked the assembly generated by oberon-lang and calls to these symbols are indeed emitted if the optimization level is set to -O0, otherwise the calls are (probably) optimized away by a (yet to be identified) optimization pass. Since there is no option to run optimization in JIT mode, these errors cannot be avoided for certain modules for the time being. I have not yet fully understood under which circumstances these calls are emitted. The sets of modules affected by the problem are similar but not the same in the two toolchains.

I'm now trying to figure out if and how I can load the libraries into the JIT session that contain these symbols or whether it makes sense to run the optimization pass that removes these calls also in JIT mode.

[tenko]

I found this problem also and did not find any easy option to turn this off.

It seems that on un-optimized Windows build stack protection functions are added by default.
These functions reside in the static library libclang_rt.builtins-x86_64.a as far as I can tell for MSYS2 & LLVM.
For MSVC it is probably enough with the ucrtbase.dll (Only Windows 10 and later. msvcrt for earlier versions).

In order to solve this I always linked in this libraries on the Windows platform found in these commits 1 and 2.
The latter commit I placed the static library alongside the .exe file so that upgrading CLANG MSYS2 will not
break the installation.

An other option could be just to define some dummy functions covering this and add to JIT symbol table.

I understand from reading some LLVM mailing lists that the long-term solution being worked on is a separate
runtime library for the JIT functionlality. This runtime library is still pending, so we might just choose to install a
temporary solution as referenced above.

[zaskar9]

Interesting! Do you have a link to the mailing list discussion?

[tenko]

I did not find the post, but I believe this issue is relevant : 60034

Edit : Turns out there is an option to link and export the symbols in the .exe directly: 1 2
This is maybe the cleanest solution as it will be resolved at compile time and can be just inserted with
#ifdefs into the C++ code.

[tenko]

Confirm the fix for the stack protection issue is working for me.
The result for unittests is:

Failed Tests (8):
Oberon :: codegen/arithmetic_12.mod
Oberon :: codegen/arithmetic_16.mod
Oberon :: codegen/array_5.mod
Oberon :: codegen/case_1.mod
Oberon :: codegen/const_1.mod
Oberon :: codegen/loop_2.mod
Oberon :: codegen/system_6.mod
Oberon :: codegen/system_7.mod

Unexpectedly Passed Tests (7):
Oberon :: codegen/arithmetic_11.mod
Oberon :: codegen/arithmetic_15.mod
Oberon :: codegen/arithmetic_3.mod
Oberon :: codegen/arithmetic_5.mod
Oberon :: codegen/arithmetic_6.mod
Oberon :: codegen/arithmetic_9.mod
Oberon :: codegen/procedure_4.mod

Testing Time: 22.87s
Unsupported : 2
Passed : 65
Expectedly Failed : 9
Failed : 8
Unexpectedly Passed: 7

codegen/system_7.mod fails with output -1.0E+00 against CHECK: -1.00E+00.
This is probably due to update to Out module and similar on other platforms?

codegen/system_6.mod fails with NaN output where INF and -INF where expected for REAL.

[zaskar9]

Interesting! These results are very different from what I see on the platforms available to me (including MSYS2/Clang64). Have you rebuilt liboberon before running the tests?

[tenko]

Sorry. Stale version of liboberon.

Result is:

Failed Tests (4):
Oberon :: codegen/arithmetic_12.mod
Oberon :: codegen/array_5.mod
Oberon :: codegen/case_1.mod
Oberon :: codegen/loop_2.mod

Unexpectedly Passed Tests (7):
Oberon :: codegen/arithmetic_11.mod
Oberon :: codegen/arithmetic_15.mod
Oberon :: codegen/arithmetic_3.mod
Oberon :: codegen/arithmetic_5.mod
Oberon :: codegen/arithmetic_6.mod
Oberon :: codegen/arithmetic_9.mod
Oberon :: codegen/procedure_4.mod

Testing Time: 29.32s
Unsupported : 2
Passed : 69
Expectedly Failed : 9
Failed : 4
Unexpectedly Passed: 7