Verilator option for regressions?

[steveri]

THE CHALLENGE:

Can we provide verilator as an option for CGRA regression testing, as an alternative to existing vcs etc.

EXECUTIVE SUMMARY of progress so far:

• It turns out that the verilog "clocking" construct, used in our existing testbench, is not supported by Verilator.
• My plan is to see if we can remove this construct from our testbench; opposing suggestions for a better plan are more than welcome!

DETAILS

• Installed latest version of verilator

% verilator --version
Verilator 5.029 devel rev v5.028-160-gc05c48aaf

• Ran pointwise benchmark to build a populated testbench /aha/garnet/tests/test_app
• Built a bare-bones verilator testbench CGRA.cpp
• Tried it out

VFILES='
  /aha/garnet/global_buffer/header/global_buffer_param.svh
  /aha/garnet/global_buffer/header/glb.svh
  /aha/garnet/global_controller/header/glc.svh
  /aha/garnet/garnet.v
  /aha/garnet/global_buffer/systemRDL/output/glb_pio.sv
  /aha/garnet/global_buffer/systemRDL/output/glb_jrdl_decode.sv
  /aha/garnet/global_buffer/systemRDL/output/glb_jrdl_logic.sv
  /aha/garnet/global_controller/systemRDL/output/glc_pio.sv
  /aha/garnet/global_controller/systemRDL/output/glc_jrdl_decode.sv
  /aha/garnet/global_controller/systemRDL/output/glc_jrdl_logic.sv
  /aha/garnet/genesis_verif/*.sv
'
CW1=/cad/cadence/GENUS_19.10.000_lnx86/share/synth/lib/chipware/sim/verilog/CW/
CW2=/cad/cadence/GENUS_19.10.000_lnx86/share/synth/lib/chipware/sim/verilog/CWTECH/
verilator -Wno-fatal -Wall --cc CGRA.cpp --exe \
  /aha/garnet/global_buffer/header/global_buffer_param.svh \
  tb/top.sv \
  $VFILES \
  -F tb/tb_cgra.f \
  -y $CW1 -y $CW2 \
  libcgra.so

• Result:

    %Error-UNSUPPORTED: tb/axil_ifc.sv:102:20: Unsupported: Modport clocking
      102 |     modport driver(clocking cbd);

OPTIONS GOING FORWARD

• fix verilator (not likely, see Support Modport Clocking  verilator/verilator#4555)
• write and maintain a separate full-testbench for verilator
• rewrite existing testbench to not use clocking construct
• your name here?

RESOURCES

• verilator "clocking" issue, see Support Modport Clocking  verilator/verilator#4555
• https://www.doulos.com/knowhow/systemverilog/systemverilog-tutorials/systemverilog-clocking-tutorial/
• https://www.chipverify.com/systemverilog/systemverilog-clocking-blocks

TODO

• I will make a garnet verilator branch containing my experiments including e.g. CGRA.cpp
• I will see what might be involved in rewriting our existing testbench without clocking

[kalhankoul96]

Looks good Steve, let me know if you are able to figure it out.

[steveri]

Here's a status update of how things are going so far...

Executive summary

I removed the clocking constructs and made a bunch of other changes (details below). And now the first phase of verilator simulation runs successfully.

Details

Verilator simulation occurs in three stages:

1. translate verilog source files into c++ equivalents
2. compile the c++ to make a simulator executable
3. run the simulator

Phase 1 now completes without error. Here's how that happened:

I removed all the clocking structures without regard to adjusting the timing, just to see if I could get Verilator to compile a simulator. This means that, if/when the simulator is running, it will probably get wrong answers until the timing is corrected again.

Old verilator did not understand timing delays, so I upgraded verilator to v5.

Gcc9 did not understand coroutines, so I upgraded gcc to gcc-10.

I turned off a BUNCH of warnings that will need to be revisited later

I made several changes w.r.t. directory $GARNET/tests/test_app:

• removed clocking signals from five testbench files

tests/test_app/tb/axil_driver.sv
tests/test_app/tb/axil_ifc.sv
tests/test_app/tb/environment.sv
tests/test_app/tb/proc_driver.sv
tests/test_app/tb/proc_ifc.sv

• rewrote non-compliant $timeformat calls in tb/environment.sv
• fixed(?) incorrect nonblocking assignments in tb/top.sv initial block
• made a local cache vfiles of all the verilog files needed for simulation
• made a reference script bin/verilator.sh for using vfiles etc. to build a verilator simulator
• everything is checked into the garnet repo 'verilator' branch as commit eb9ad82be; in theory, people can check out that commit and use verilator.sh to duplicate my progress so far (although they would possibly have to upgrade verilator and gcc first, see above)

What's next

• C++ compilation of the simulator has errors, these need to be squashed.

• Then we can try running the simulator to see what happens.

[steveri]

Status update: Stage 2 works, verilator produces a simulator executable without error. Dunno yet if it actually works.

[kalhankoul96]

Awesome! If the applications aren't working, let me know. I can help debug/look at waveforms.

[steveri]

Awesome! If the applications aren't working, let me know. I can help debug/look at waveforms.

Thanks! I hope to take you up on that soon. For now, no waveforms, the simulator is kind of a brick so far :(

[steveri]

Got a waveform: 8ns of clock going up and down. Hooray!

Next steps:

• clean up and check in progress so far
• do a complete run (not just 8ns) and see what happens
• THEN I will probably need help debugging the waveforms

[kalhankoul96]

Great, awesome Steve. Let me know what help you need.

[steveri]

Update:

• Simulation goes 210 ns without seg fault, believe it or not this is progress.
• It goes at a rate of about 2ns of sim time for every 1s of wall time. That's pretty terrible, but maybe it will be faster after the debug phase is over.
• This (below) is how far it gets in 210ns:

t=  100ns
Monitor initialization success
...
=====After Optimization=====
[ITER CTRL - loop 0] extent: 4094, cycle stride: 1, data stride: 2
Configuration of flush signal crossbar is updated to 0x0
[APP0-pointwise] group_start: 0
[APP0-pointwise] glb mapping success
Mapping kernel 0 Succeed
...
t=  119ns
Turn on interrupt enable registers
t=  210ns
- S i m u l a t i o n   R e p o r t: Verilator 5.028 2024-08-21
- Verilator: end at 210ns; walltime 100.779 s; speed 2.114 ns/s
- Verilator: cpu 99.561 s on 1 threads; alloced 512 MB

NEXT

• check everything in to verilator branch
• see what happens after 2000ns
• figure out how long it will take to do the whole thing maybe

[steveri]

Progress report:

Someone (Kalhan I think) suggested that, to make Verilator work properly, we might have to replace all the object-oriented (class) code in our testbench with non-class-based equivalents. This seems to work, whereas other things I have tried have not worked, so I am going down that path. The class-based code comprises 1564 lines of verilog, while my translated equivalent (so far) totals about 500 lines, so maybe that means I'm about 1/3 of the way through. Continuing down this path means this translation phase should be done within a week or two, with any luck.

Also: I'm not sure why, but suddenly the verilated code is running much much faster than before. Earlier I reported 2ns of sim time for 1s of wall time, i.e. about 2 Hz; now it seems to be zipping through 500ns of simulation in well under a second, so, better than 500Hz at least...

[steveri]

The verilator testbench seems to be working now, producing what appears to be all the right signals to and from my stubbed-out garnet.v DUT.

Next steps:

• see if test bench works with the REAL garnet.v instead of pipecleaner stub
• see if test bench works with VCS as well as Verilator.