Unit Tests for v86

[nepx]

I'm in the process of writing my own x86 emulator which uses dynamic recompilation, but I've run into a huge bottleneck: testing. It can translate code well, but I'm afraid that my emulator will have errors that will come back to bite me in the form of weird errors that takes loads of time to debug.

I've seen the tests/ folder, but it doesn't provide any true CPU testing. The QEMU test is cool, but the problem is that I need to be able to boot Linux first...

Besides manually looking at each opcode description and checking for accuracy, are there any other CPU correctness tests out there?

I've googled "Testing CPU Emulators", and I've seen an interesting CPU testing scheme that sounds cool but requires protected mode...

[copy]

I don't think there's a test suite out there, at least I couldn't find one when I started this project in 2011.

An alternative approach I have considered (but have not found the time to implement) is to use to use the host cpu as an oracle, i.e. assemble a program that loads input registers and prints output registers of a single instruction. You can then use random inputs and compare the result from your emulator and the oracle.

You can also use this emulator as an oracle. cpu.js (plus its dependencies) is relatively stand-alone and free of side-effects. It's not documented, but shouldn't be too hard to use. I can't guarantee correctness of course, but that should at least bring you to boot Linux.

As a hint, read about how paging and segmentation work before implementing instructions. Otherwise you might need to do quite some refactoring later.

By the way, is your project going to be open-source?

[nepx]

Yes! It is going to be open source. I just want a project that can do something before I publish it.
I have actually used the CPU as a test "oracle" based on a paper called "Testing CPU Emulators". It worked well, and it has worked really well for me. Basically, I wrote a node.js program that generated random instructions with certain constraints. After it generates a few tests, then it injects it into an executable I wrote in C and executes it. The executable prints out the register/eflags state and compares it to my CPU. The only problem is that it only works on Windows.

I wrote a replacement version using NASM (I am not a fan of GCC inline assembly), but for some reason it doesn't capture stdout/stderr.

Old Fuzzer
fuzzer.js
fuzzeroutline.c
New Fuzzer
fuzzer.asm

Also, this emulator is supposed to be a process-level emulator for Linux/Windows, because I want to make an extremely simple way for people to package programs for the web. So that means that I don't need to worry about the GDT or paging (I do need to implement segmentation, but just enough for modify_ldt to work).

I haven't thought of using cpu.js as a reference, but sounds like an interesting idea!

[nepx]

What I'm really curious about is: is there something like a "test suite" for v86 besides the QEMU usermode test? Something covering exceptions, paging, segmentation, instructions, etc.

That would be helpful with both accuracy and regression testing and for emulator developers.

[copy]

@nepx We don't have tests like that for v86. If you write some, I'd love to adapt them for also testing this project.

[nepx]

The KVM testsuite looks like it can be useful. As it says on their website:

The unit tests are tiny guest operating systems that generally execute only tens of lines of C and assembler test code in order to obtain its PASS/FAIL result.

I also saw tests/perf, could that be useful too?

[copy]

The KVM testsuite looks like it can be useful. As it says on their website:

These look quite useful, I might go ahead and run them (depending on #95).

I also saw tests/perf, could that be useful too?

These were used for testing performance of decoding and the interpreter loop, not correctness.

[nepx]

Oh oops -- I didn't look at the code that closely for tests/perf.

Another intresting way is to generate code "on the fly" and test it. I found a good implementation (in Java) for another emulator (here)[https://svn.code.sf.net/p/boxedwine/code/trunk/boxedwine/java/src_test/wine/emulation/testCPU.java].

But it would be cool if there were docs on how to correctly debug the emulator at an instruction level (with disassembly and what-not) and the original tests used to debug v86 (if there were any).

[copy]

if there were any

There weren't any that aren't in the repository.

By the way, I integrated kvm-unit-test and they were quite helpful when implementing the APIC. I'll add them to CI.

[copy]

v86 now has unit tests written in assembly that test single instructions. See tests/nasm/run.js. The results are compared locally using gdb. We'll extend them in the future to cover more instructions.