Tester unification

[ducky64]

Currently, we have three tester interfaces:

• Circuit ("synthesizable") tester, the only tester included in chisel3 (everything else lives in a separate repo). The testdriver, like the DUT, is synthesizable hardware and is written in Chisel.
• Peek-poke tester, where Scala drives the device-under-test through peek, poke, and check commands. Obviously non-synthesizable, since it's possible to peek values, do some compute in Scala-land, and poke the results back.
• Advanced tester, like the peek-poke tester, but provides parallel actions with a limited amount of interfaces (like Decoupled).

Of course, all these testers have their problems and advantages (some fundamental, some engineering):

• Neither peek-poke nor advanced tester are available in the chisel3 repository. You have to include a separate dependency. Minor usability nit, though I think many like Python's approach of "batteries included".
• Circuit tester is great for lightweight or concurrent tests, but sequencing events in time usually requires the test writer to manually write a state machine. Not fun.
• Circuit tester is synthesizable, meaning that full testbenches can be deployed on FPGA (and generally are self-contained). Apparently some people actually do this.
• Circuit tester doesn't support any backends other than Verilator, and Verilator has a high compilation cost. The chisel3 standard tests usually take several minutes to run, probably a good percentage of that time is compiling design through Verilator.
• It's difficult to specify concurrency (see Better tester concurrency specifications? #547) in the peek-poke tester, and advanced tester only allows limited concurrency models. The circuit tester has concurrency native from its circuit nature.

The goal here is to unify all the different testers into a single tester included in Chisel3.

Reasoning:

• support all the different backends (mainly for performance reasons: chisel3 tests may run faster through the FIRRterpreter)
• unify the test specification model (which may be a hybrid approach, as detailed below)
• include it in base chisel3 (tests aren't optional for engineers serious about hardware design).

Some hybrid approaches have been suggested:

• If synthesizability isn't useful, then it's possible to write the circuit tester programming model in the peek-poke tester framework. Only the step command needs to be added to the peek-poke tester.
• It is also possible to translate series of step, check, and pokes to synthesizable hardware, having Chisel generate the state machine. There would be no poke support, since the testbench would be standalone and not have a Scala testdriver.
  • Several attempts have been made before with systems like these, for whatever reason they have never gained traction.
  • This may make testvector literal specification a non-issue, since all test constructs live in Chisel-land and can seamlessly use Chisel literals and future features like Bundle literals.

@jackkoenig @jackbackrack @sdtwigg @dt27182 @chick @azidar @aswaterman

[donggyukim]

As a heavy user & developer of peekpoke testers 1~3 years ago, I strongly disagree with this suggestion unless the following issues are resolved:

• The main reason I stopped using peekpoke testers is it's extremely hard to use in the development mode. Once you have a submodule of chisel-testers in your project, you need to local-publish chisel, firrtl, and firrrtl-interpreters altogether. Don't tell me I need to local-publish them with this change. This sounds like adding additional dependencies on chisel, which will worsen Workspace Issues: Compile errors with firrtl master #499.
• For the above reason, I switched all peek-poke testers to hw testers to cut out the dependencies.(e.g. No more chisel testers ucb-bar/riscv-mini#2) Never tell me I should change testers again with this change.

[ducky64]

So there are two separate goals for this issue:

1. Determine the underlying programming model (or a hybrid approach). Synthesizable circuit-based, Scala-driven peek-poke, synthesized check-poke, etc...
2. Determine the implementation strategy, include packaging strategy. I would advocate for everything to be included in chisel3 instead of the current chisel-testers and dependency hell we have right now.

So having a peek-poke programming model isn't mutually exclusive with not having dependency hell, and we should make sure to discuss the pros and cons of each issue separately.

[jackkoenig]

You're right that there are two separate goals, but I think @donggyukim is right in that fixing the packaging strategy should be super high priority because the current system doesn't really work. Unifying the repos won't fix the problem because this problem affects anyone who wishes to create a repo with their own Chisel utils or custom Firrtl passes (eg. https://github.com/ucb-bar/barstools).

To the best of my knowledge, sbt does not have an easy way of doing development with local versions (as we have discussed in #499). I recently found a couple of StackOverflow posts that might be helpful:

• http://stackoverflow.com/questions/14351940/setting-up-sbt-environment-to-hack-on-multiple-libraries-at-once
• http://stackoverflow.com/questions/14623138/how-to-toggle-between-project-and-library-dependencies-in-sbt

[shunshou]

I'd be kind of upset if you entirely remove the peek-poke setup. It's super easy to get started with and mimics how some people would write (non-synthesizable) test benches.

And I agree that the packaging strategy right now is a huge nightmare. Every time I want to update something, I hate to go through 5 different dependencies and all publish-local and hope that I'm on compatible versions.

I spent a good hour last night just updating firrtl and everything else and hoping that no dependency broke. (and then I ran into the problem of barstools still not actually compiling...).

[dt27182]

+1 on the sbt packaging issues

It seems like there are 2 options for getting everything that we what (easy concurrent test drivers and easy sequencing of events in time).

One option is to stick with the scala based software test model and implement some sort of event driven scheduling system. This option provides the full power of scala in writing the driver logic, but writing a event scheduler seems like a lot of work and a step back to verilog ...

The other options is to synthesize peek (check?), poke, step into Chisel state machines. This option provides good backwards compatibility with the existing hardware based tests, so people don't have to rewrite the tests yet again, which is a pretty big pro. However, there is less power in how to sequence the peek, poke, steps and there might be issues with running very large test vectors with this system since everything has to be synthesized into hardware.

I think I prefer the synthesized peek, poke, steps approach. Very large software driven tests probably don't need to concurrent drivers since they are whole system tests, so they can probably be written against the verilator produced C++. Also, don't we already have the SteppedHWIOTester? Is nobody using that anymore?

[ducky64]

@shunshou when you want to write non-synthesizable testbenches, is it because you want to be able to control things in timesteps, or because you need to do some compute in Scala? The first could be solved in a synthesizable test specification model by having a framework to generate a state machine from check / poke / step commands, while the second is pretty much incompatible with a synthesizable framework.

One issue with synthesizable testbenches would be the delay model to use, since a common programming model is waiting until some condition (like signal valid) is true. A solution might be just wait(expr), which steps until expr === true.B.

It also seems like everyone here hates the packaging setup. What does everyone want to see there? A mega-repo of chisel3, FIRRTL, FIRRterpreter, and chisel-testers? Or some other split of those repositories?

It seems the SteppedHWIOTester is still there, but not much work has happened on it. Which I think is a good reason we should unifying testers...

[dt27182]

Mega repo with git submodules seems like a logical solution. What was the reason for moving away from that model before? I know rocket had some issues with it. What where they?

[jackkoenig]

@dt27182 The project repo with git submodules is still the goal, the problem comes from dependencies between these submodules and sbt's lack of support for specifying that you want to use a local build instead of the library dependency. I'm trying to figure out how to work around this (see the StackOverflow links above), but I'm pretty sure it's impossible to do 100% with sbt, so we can look into using some sort of setup script or possibly environment variables to workaround.

[shunshou]

@ducky64 for both reasons. Synthesizable tb makes sense, but only after I do sufficient debug with peek/poke. It's like I personally prefer to peek and stare at stuff before I convert everything into expects.

[dt27182]

@jackkoenig
Sbt does support using local builds via sbt subprojects right? (I think this is what we were using before) Is it just that sbt doesn't allow a way to switch between subprojects and library dependencies?

And what is the reason we want to publish the subparts as sbt libraries?

[ducky64]

@shunshou do you think the peek-for-debug could be replaced with a cycle-dependent printf?

[jackkoenig]

@ducky64 a single mega repo is a one time solution that doesn't fix the fundamental problem. Anyone who wants to create a chisel library that uses an sbt library dependency in the common case but can also be used in development mode will run into the same problem.

@dt27182 That is correct, and I think you've hit the nail on the head.

Should we be publishing the subparts all separately? We currently have a 1-to-1 mapping of git repos to sbt libraries. Perhaps it would be better to publish everything together again. The repos can stay separate and instead the chisel-release repo publishes them as a single library.

[jackkoenig]

Publishing together still doesn't solve the problem of separate chisel libraries though. I have an idea that might work:

Libraries like barstools have two sbt projects in them: barstools and barstools-dev. barstools has a library dependency on chisel3 and whatever else. barstools-dev is kind of a raw project that is intended to be used as a git submodule and have a parent project handle it's dependencies. Does this make sense? I'm kind of just spitballing here.

[shunshou]

@ducky64 does the printf thing work w/ Verilator?

[jackkoenig]

@shunshou Yes it does! And checkout the chisel3 wiki page about printing (if you don't know about Chisel's custom string interpolator for printf).