Add support for VCK5000 over PCIe

[LorenzoSun-V]

I only have a VCK5000 board but you built and test the project on VCK190. Will it work correctly if I follow your workflow on VCK5000?

[stephenneuendorffer]

Hi Lorenzo... Most of the flow would be the same for a VCK5000. The main difference would be the base system design, which would probably have to be modified slightly. Our intention is to do this at some point, but it hasn't happened yet.

[LorenzoSun-V]

Thanks for your reply!
I have another question. If I develop a kernel based on mlir-aie tools on VCK190 and it could run on VCK190 correctly. Could this kernel also run correctly on VCK5000 without redevelopment?
All in all, what I want to ask can be summarized as following:

1. I'm going to develop my own kernel if I buy a VCK190 and run your test files successfully following your documents. After my kernel can be run successfully on VCK190, do I need to redevelop the kernel on VCK5000? Or I can reuse what I develop on VCK190 and run the kernel correctly on VCK5000.
2. If I still use VCK5000 to develop following your documents, what does "base system design" mean? I only use VCK5000 as an accelerator card like U50. Can I burn an OS like petalinux VCK190 using into VCK5000 so that I can run your test files in your project on VCK5000.

I'm looking forward to reciving your reply ASAP. :-D

[stephenneuendorffer]

1. Basically yes. They have the same device, so anything that is internal to the device works the same way. Interfacing is different: The VCK190 has 3 memory banks, and a mix of DDR4 and LPDDR4 memory. The VCK5000 has 4 memory banks, all DDR4. This can affect the design that you would want to build and some VCK5000 designs architected for 4 memory banks would simply not work on the VCK190
2. We use a simple FPGA design to configure the NOC, enabling the AIEngine processors to access external memory. This design would have to be different for the VCK5000. Another difference is that the VCK5000 does not have an on-board SD card, as it is designed to be configured over PCIe. Today we use a Petalinux kernel with an ubuntu rootfs that boots from the SD card. For VCK5000, one approach would be to run the control software on the X86 host, instead of the on-board ARM processor.

[stephenneuendorffer]

If you're interested prototyping the PCIe path for VCK5000, we're happy to provide some support to getting things up and running.

[elliottbinder]

Hello -- I also have a VCK5000 I'd like to target. How should the installation instructions change? I'm currently unsure what I should be doing to provide the sysroot directory when building MLIR-AIE.

[stephenneuendorffer]

@elliottbinder Probably the easiest way to bring up the VCK5000 would be to avoid cross-compiling the mlir-aie tools. Instead, they would run on x86 and configure the AIE device over PCIe. This would remove the need for a sysroot. I believe with the new libXAIE v2 that this should work, but we have not tested it. We are currently bringing up libXAIEv2 on ARM first.

[elliottbinder]

Is omitting the VitisSysroot definition in the build instructions sufficient? Building with ninja proceeded fine, but I'm getting this error when testing with check-aie:

[0/1] Running the aie regression tests
llvm-lit: /home/elliott/software/llvm-project/llvm/utils/lit/lit/TestingConfig.py:102: fatal: unable to parse config file '/home/elliott/software/mlir-aie/test/lit.cfg.py', traceback: Traceback (most recent call last):
  File "/home/elliott/software/llvm-project/llvm/utils/lit/lit/TestingConfig.py", line 91, in load_from_path
    exec(compile(data, path, 'exec'), cfg_globals, None)
  File "/home/elliott/software/mlir-aie/test/lit.cfg.py", line 84, in <module>
    result = subprocess.run([os.path.join(config.peano_tools_dir, 'llc'),'-mtriple=aie','--version'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
  File "/usr/lib/python3.8/subprocess.py", line 493, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '<unset>/bin/llc'

It looks like this could be because peano isn't installed, but I don't see that listed as a prerequisite.

[stephenneuendorffer]

@elliottbinder Sorry about that... I've checked in a fix.

[elliottbinder]

Your fix resolved that issue, thanks! I'm able to get through the building step, but cmake is saying

-- Could NOT find LibXAIE (missing: XILINX_XAIE_INCLUDE_DIR XILINX_XAIE_LIBS) 

which, according to another issue, looks like it might be related to the sysroot -- is that right? What can I do to locate or install LibXAIE?
I was still able to build and run through ninja check-aie. Here's a summary from the tests:

  Unsupported: 14
  Passed     : 85
  Unresolved : 14
  Failed     : 25

[hanchenye]

Hi @elliottbinder, first I believe the failed test cases are due to the missing of LibXAIE. Basically you need to follow the instructions here to build the vck190_bare platform @stephenneuendorffer mentioned. You will find the sysroot at platforms/vck190_bare/petalinux/sysroot/sysroots/aarch64-xilinx-linux once you build up the platform.

[stephenneuendorffer]

If you're cross-compiling, hanchen is correct. If you're just compiling to run on X86, then you can get libXAIE sources here: https://github.com/Xilinx/embeddedsw/tree/master/XilinxProcessorIPLib/drivers/aiengine. After compiling the you can point cmake to the right location with -DLibXAIE_ROOT=...

[elliottbinder]

I tried making aiengine, but got a missing file error on #include <metal/alloc.h>. Making aienginev2 proceeded without any errors.
Trying to build MLIR-AIE again, I passed in the src directory of aienginev2 to cmake with the define on LibXAIE_DIR as you suggested, but cmake is saying it wasn't used. I also tried passing in the aienginev2 directory via the variables cmake is saying are missing (XILINX_XAIE_INCLUDE_DIR, XILINX_XAIE_LIBS). Cmake no longer complains, but it doesn't seem to make a difference in the test results.

[LorenzoSun-V]

Hi @elliottbinder, I also get stuck on building AIE tools. Are you willing to share your workflow after cloning mlir-aie projects? I downloaded https://github.com/Xilinx/embeddedsw and changed the instruction of building AIE tools: I replaced "-DVitisSysroot=${SYSROOT}" with "-DLibXAIE_DIR=${aienginev2_path}" and I also encountered missing of head file.
In addition, loud is the noise from fans of VCK5000. Do you have any solutions? I want to disconnect the 8-pin interface which supplies electricity for fans but I'm not sure if it will affect its normal work.