[QBox/RISC-V] Does QBOX RISCV64 cpu instance support vector extension?

[donghyun-kang-dn]

Main Issue: illegal_instruction occurs when accessing vlenb CSR even after mstatus.VS is confirmed set

0. luafile for platform-vp

platform = {
    moduletype = "Container";
    quantum_ns = 1000000;

    -- Router for memory map
    -- log level
    -- 0: None
    -- 1: Error
    -- 2: Warning
    -- 3: Info
    -- 4: Debug
    -- 5: Verbose ?
    -- 6: Trace
    router = {
        moduletype = "router";
        log_level = 6; -- TRACE
    };

    -- QEMU Instance Manager
    qemu_inst_mgr = {
        moduletype = "QemuInstanceManager";
    };

    -- QEMU Instance (RISC-V64, single core)
    qemu_inst = {
        moduletype = "QemuInstance";
        args = {"&platform.qemu_inst_mgr", "RISCV64"};
        qemu_args = "-cpu rv64,v=on,vlen=512";
        tcg_mode = "MULTI";
        sync_policy = "multithread-unconstrained";
        log_level = 6;
    };

    -- Boot ROM at 0x0 - Bootrom that jumps to main firmware
    -- Reset signal
    reset = {
        moduletype = "sifive_test",
        args = {"&platform.qemu_inst"},
        target_socket = {address = 0x100000, size = 0x1000, bind = "&router.initiator_socket"};
        log_level = 6;
    };

    bootrom_0 = {
        moduletype = "gs_memory";
        target_socket = {address = 0x0, size = 0x2000, bind = "&router.initiator_socket"};
        shared_memory = false;
        reset = {bind = "&reset.reset"};
    };

    -- Main RAM at 0x80000000 - Main firmware
    ram_0 = {
        moduletype = "gs_memory";
        target_socket = {address = 0x80000000, size = 0x8000000, bind = "&router.initiator_socket"};
        shared_memory = true;
        reset = {bind = "&reset.reset"};
    };

    -- Loader module to load binaries
    loader = {
        moduletype = "loader";
        initiator_socket = {bind = "&router.target_socket"};
        -- {bin_file = bootrom_path, address = 0x0};
        {bin_file = bootrom_path, address = 0x1000};
        {bin_file = fw_path, address = 0x80000000};
        reset = {bind = "&reset.reset"};
    };

    gpex_0 = {
        moduletype = "qemu_gpex";
        args = {"&platform.qemu_inst"};
        bus_master = {bind = "&router.target_socket"};
        pio_iface = { address = 0x3000000, size = 0x10000, bind= "&router.initiator_socket"};
        mmio_iface = { address = 0x40000000, size = 0x40000000, bind= "&router.initiator_socket" };
        ecam_iface = { address = 0x30000000, size = 0x10000000, bind= "&router.initiator_socket" };
        mmio_iface_high = { address = 0x400000000, size = 0x400000000, bind= "&router.initiator_socket" },
        irq_out_0 = {bind = "&plic_0.irq_in_32"};
        irq_out_1 = {bind = "&plic_0.irq_in_33"};
        irq_out_2 = {bind = "&plic_0.irq_in_34"};
        irq_out_3 = {bind = "&plic_0.irq_in_35"};
    };

    global_peripheral_initiator_riscv_0 = {
        moduletype = "global_peripheral_initiator",
        args = {"&platform.qemu_inst", "&platform.cpu_0"},
        global_initiator = {bind = "&router.target_socket"},
    };

    -- PLIC (Platform Level Interrupt Controller)
    plic_0 = {
        moduletype = "plic_sifive",
        args = {"&platform.qemu_inst"},
        mem    = {address=0x0c000000, size=0x600000, bind = "&router.initiator_socket"},
        num_sources = 96,
        num_priorities = 7,
        priority_base = 0x0,
        pending_base = 0x1000,
        enable_base = 0x2000,
        enable_stride = 0x80,
        context_base =  0x200000,
        context_stride = 0x1000,
        aperture_size = 0x600000,
        hart_config = "MS";
    };

    -- CLINT (Core Local Interruptor) - Timer
    -- clint_0 = {
    --     moduletype = "riscv_aclint_mtimer";
    --     args = {"&platform.qemu_inst"};
    --     mem = {address = 0x2000000, size = 0x10000, bind = "&router.initiator_socket"};
    --     timecmp_base = 0x0;
    --     time_base = 0xbff8;
    --     provide_rdtime = true;
    --     aperture_size = 0x10000;
    --     num_harts = 1;
    -- };

    -- UART0 (16550 compatible)
    uart_0 =  {
        moduletype = "uart_16550",
        args = {"&platform.qemu_inst"},
        mem = {address= 0x10000000, size=0x100, bind = "&router.initiator_socket"},
        irq_out={bind = "&plic_0.irq_in_10"},
        regshift=2,
        baudbase=3686400;
    };


    -- RTC (Real Time Clock) - removed for simplicity
    -- rtc_0 = {
    --     moduletype = "goldfish_rtc";
    --     args = {"&platform.qemu_inst"};
    --     mem = {address = 0x101000, size = 0x1000, bind = "&router.initiator_socket"};
    --     irq = {bind = "&cpu_0.irq_in_1"};
    -- };

    -- Flash memory (4MB)
    flash_0 = {
        moduletype = "gs_memory";
        target_socket = {address = 0x20000000, size = 0x400000, bind = "&router.initiator_socket"};
        shared_memory = false;
    };

    -- RISC-V64 CPU
    cpu_0 = {
        moduletype = "cpu_riscv64";
        args = {"&platform.qemu_inst", 0};
        mem = {bind = "&router.target_socket"};
        reset = {bind = "&reset.reset"};
    };

}

1. Summary

We are encountering an illegal_instruction (cause 2) exception when trying to initialize the RISC-V 'V' (Vector) extension in our firmware running on the QEMU/QBox platform.

The core of the problem is a contradiction:

1. Our firmware successfully sets the mstatus.VS bits to 'Initial' (0b01).
2. Our firmware successfully reads back mstatus and confirms the VS bits are set.
3. Despite this, the very next attempt to read a vector CSR (vlenb) immediately fails with an illegal_instruction exception.

2. Firmware Initialization Logic

Our firmware follows the standard procedure to enable the vector unit:

1. It first checks misa (via rvv_is_available()) to confirm the 'V' extension is present. This check passes.
2. It then reads mstatus, sets the FS bits (13-14) to 'Initial' (0b01), and writes it back. This succeeds.
3. It then reads mstatus again, sets the VS bits (9-10) to 'Initial' (0b01), and writes it back. This also succeeds.
4. We added a verification step in our C code to double-check the VS bits immediately after setting them:

   // Verify VS field is set
   mstatus = read_csr(CSR_MSTATUS);
   if ((mstatus & MSTATUS_VS_MASK) == MSTATUS_VS_INITIAL) {
       // OK
   } else {
       return RVV_INIT_FAILED; // Failsafe
   }

   This check passes, confirming that the CPU reports mstatus.VS as 'Initial'.

3. The Crash

Immediately after this verification passes, the firmware attempts to read vlenb:

// This C code is executed:
uint32_t vlenb = read_csr(CSR_VLENB);

This compiles to the following assembly instruction:

0x80000774:  c22027f3   csrrs a5,vlenb,zero

When this instruction executes, we get an illegal_instruction trap.

4. Confusing Log Entry

Oddly, the trap log we captured shows an epc that does not match the vlenb instruction itself, which is confusing. This may be a separate issue, but it's the log we see when the vlenb read is attempted.

riscv_cpu_do_interrupt: hart:0, async:0, cause:0000000000000002, epc:0x0000000080001168, tval:0x00000000cc07f057, desc=illegal_instruction

5. Questions and Hypothesis

Main Question: Why would an access to vlenb (or any vector CSR) cause an illegal_instruction after the mstatus.VS bits have been successfully set and verified as 'Initial' (0b01)?

Hypothesis: We suspect this might be a libQEMU configuration issue rather than a firmware logic error. Is it possible that the QEMU instance is being launched without the v=true flag (e.g., -cpu rv64,v=true)?

• I found that the same firmware binary works on the standard QEMU (10.1), virt with --kernel loader. So I think it's weird..
• Do I have to build libqemu with additional options for this purpose?

[androm3da]

Below are some potential causes of that illegal instruction exception (from target/riscv/csr.c).

I'm going to try and make a reference test case that demonstrates vector unit usage. I'll share it once I have it working.

    /* ensure the CSR extension is enabled */
    if (!riscv_cpu_cfg(env)->ext_zicsr) {
        return RISCV_EXCP_ILLEGAL_INST;
    }

    /* ensure CSR is implemented by checking predicate */
    if (!csr_ops[csrno].predicate) {
        return RISCV_EXCP_ILLEGAL_INST;
    }

    /* privileged spec version check */
    if (env->priv_ver < csr_min_priv) {
        return RISCV_EXCP_ILLEGAL_INST;
    }

    /* read / write check */
    if (write && read_only) {
        return RISCV_EXCP_ILLEGAL_INST;
    }

[donghyun-kang-dn]

Thanks for your quick response.

For more contexts, here's the firmware I compiled & executed on platform-vp with the provided option.
However, I was working on libqemu v9.1.21 w/ qbox 4.7.0 ... Cause libqemu 10.1 was not available when I tried this.

• I'll check it on the latest version (10.1 & 5.0.0) and let you know soon..
• I have also tried other qemu options - privilege, zicsr=on, smarteen... , but they were neither working.
• I compiled the following firmware w/ a simple entry.S file that setups the basic environment (stack, mstatus, ...)

#include "riscv_io.h" // simple wrapper 
#include "riscv_rvv.h" // simple wrapper
#include <stdint.h>
#include <stddef.h>

#define TEST_SIZE 8

// __attribute__((section(".data"))) const size_t test_size = 64;  // 64 bytes per element
__attribute__((section(".data"))) volatile uint32_t in_data_1[TEST_SIZE];
__attribute__((section(".data"))) volatile uint32_t in_data_2[TEST_SIZE];
__attribute__((section(".data"))) volatile uint32_t output_data[TEST_SIZE];
__attribute__((section(".data"))) volatile uint32_t expected_data[TEST_SIZE];

// VLUT AXI vector operations using RVV
void vlut_axi_rvv_vector_copy_simple(uint32_t* data_1, uint32_t* data_2, size_t size) {
    uart_puts("  [RVV-Simple] Vector write to VLUT AXI memory\n");
    
    // Load data from source
    vuint32m1_t v_data = rvv_vle32_v_u32m1(data_1, size);
    rvv_vse32_v_u32m1(data_2, v_data, size);
}

void vlut_axi_rvv_vector_mul_simple(uint32_t* data_1, uint32_t* data_2, uint32_t* output_data, size_t size) {
    // Load from VLUT AXI memory
    vuint32m1_t v_data_1 = rvv_vle32_v_u32m1(data_1, size);
    vuint32m1_t v_data_2 = rvv_vle32_v_u32m1(data_2, size);
    
    // Multiply
    vuint32m1_t v_output = rvv_vmul_vv_u32m1(v_data_1, v_data_2, size);

    rvv_vse32_v_u32m1(output_data, v_output, size);
    
    uart_puts("  [RVV-Simple] Vector mul completed\n");
}

// VLUT AXI vector operations using RVV
void vlut_axi_rvv_vector_copy(uint32_t* data_1, uint32_t* data_2, size_t size) {
    uart_puts("  [RVV] Vector write to VLUT AXI memory\n");
    
    size_t processed = 0;
    while (processed < size) {
        // Calculate elements to process in this iteration
        size_t elements = (size - processed > RVV_VLEN_512_ELEMENTS) ? 
                         RVV_VLEN_512_ELEMENTS : (size - processed);
        
        // Set vector length
        size_t vl = rvv_setvl_e32m4(elements);
        // size_t vl = size;
        
        // Load data from source
        vuint32m1_t v_data = rvv_vle32_v_u32m1(&data_1[processed], vl);
        
        // Store to VLUT AXI memory
        // rvv_vse32_v_u32m1((uint32_t*)(VLUT_AXI_MEMORY_ADDR), v_data, vl);
        rvv_vse32_v_u32m1(&data_2[processed], v_data, vl);
        
        processed += vl;
    }
    
    uart_puts("  [RVV] Vector copy completed\n");
}

void vlut_axi_rvv_vector_mul(uint32_t* data_1, uint32_t* data_2, uint32_t* output_data, size_t size) {
    uart_puts("  [RVV] data1 x data2\n");
    
    size_t processed = 0;
    while (processed < size) {
        // Calculate elements to process in this iteration
        size_t elements = (size - processed > RVV_VLEN_512_ELEMENTS) ? 
                         RVV_VLEN_512_ELEMENTS : (size - processed);
        
        // Set vector length
        size_t vl = rvv_setvl_e32m4(elements);
        // size_t vl = size;
        
        // Load from VLUT AXI memory
        vuint32m1_t v_data_1 = rvv_vle32_v_u32m1(&data_1[processed], vl);
        vuint32m1_t v_data_2 = rvv_vle32_v_u32m1(&data_2[processed], vl);
        
        // Multiply
        vuint32m1_t v_output = rvv_vmul_vv_u32m1(v_data_1, v_data_2, vl);
        
        // Store to read buffer
        rvv_vse32_v_u32m1(&output_data[processed], v_output, vl);
        
        processed += vl;
    }
    
    uart_puts("  [RVV] Vector mul completed\n");
}

bool vlut_axi_rvv_verify(uint32_t* original_data, uint32_t* read_data, size_t size) {
    uart_puts("  [RVV] Vector verification\n");
    
    bool all_match = true;
    
    // Simple scalar verification
    for (size_t i = 0; i < size; i++) {
        // uart_puts("expected: ");
        // uart_put_hex(original_data[i]);
        // uart_puts(", read: ");
        // uart_put_hex(read_data[i]);
        // uart_putc('\n');
        if (read_data[i] != original_data[i]) {
            all_match = false;
            uart_puts("  [RVV] Mismatch at element ");
            // uart_put_hex(i);
            uart_putc('\n');
        }
    }
    
    uart_puts("  [RVV] Vector verification ");
    uart_puts(all_match ? "PASSED" : "FAILED");
    uart_putc('\n');
    
    return all_match;
}


int main(void) {
    uint64_t mstatus;
    uart_puts("RVV Test\n");
    // Initialize RVV
    // uart_puts("RVV Initialization\n");
    // rvv_init_result_t rvv_result = rvv_init();
    // if (rvv_result) {
    //     uart_puts("Fail\n");
    //     return 1;
    // }

    // fence
    // asm volatile("fence" ::: "memory");
    // asm volatile("fence iorw,iorw" ::: "memory");
    // asm volatile("fence iorw,iorw" ::: "memory");

    // rvv_print_status(0);
    mstatus = read_csr(CSR_MSTATUS);
    if ((mstatus & MSTATUS_FS_VS_INITIAL) != MSTATUS_FS_VS_INITIAL) { // MSTATUS_FS_VS_INITIAL = (1 << 9 | 1 << 13) == 0x2200 
        uart_puts("RVV Init Failed\n");
        return 1;
    }
    uart_puts("RVV Init Success\n");
    uart_put_hex(mstatus);
    uart_putc('\n');
    // print_all_csrs();

    uart_puts("Initializing test data...\n");

    for (size_t i = 0; i < TEST_SIZE; i++) {
        in_data_1[i] = i;
        expected_data[i] = i * i;
    }
    
    uart_puts("Test Data Move\n");
    
    vlut_axi_rvv_vector_copy_simple((uint32_t*)in_data_1, (uint32_t*)in_data_2, TEST_SIZE);
    
    uart_puts("Test Data Verify\n");
    bool verify_result_move = vlut_axi_rvv_verify((uint32_t*)in_data_1, (uint32_t*)in_data_2, TEST_SIZE);

    uart_puts("Test Data Mul\n");
    vlut_axi_rvv_vector_mul_simple((uint32_t*)in_data_1, (uint32_t*)in_data_2, (uint32_t*)output_data, TEST_SIZE);
    
    uart_puts("Test Data Verify\n");
    bool verify_result_mul = vlut_axi_rvv_verify((uint32_t*)expected_data, (uint32_t*)output_data, TEST_SIZE);

    if(verify_result_move) {
        uart_puts("Test Move PASSED!\n");
    } else {
        uart_puts("Test Move FAILED!\n");
    }

    if(verify_result_mul) {
        uart_puts("Test Mul PASSED!\n");
    } else {
        uart_puts("Test Mul FAILED!\n");
    }
    
    mstatus = read_csr(CSR_MSTATUS);
    uart_put_hex(mstatus);
    uart_putc('\n');

    return 0;
}

QEMU log (10.1, official)

• full tests, there is no error in the trace file

Running vsearch_fw in QEMU...
RVV Test
RVV Init Success
0x0000000A00002200
Initializing test data...
Test Data Move
  [RVV] Vector write to VLUT AXI memory
  [RVV] Vector copy completed
Test Data Verify
  [RVV] Vector verification
  [RVV] Vector verification PASSED
Test Data Mul
  [RVV] data1 x data2
  [RVV] Vector mul completed
Test Data Verify
  [RVV] Vector verification
  [RVV] Vector verification PASSED
Test Move PASSED!
Test Mul PASSED!
0x8000000A00002600

QBox log

• While it shows lots of illegal instruction errors
• w/ only _simple tests - normal test doesn't work
• It looks that the value of CSR(mstatus) is identical in both cases

VV Test
RRVV Init Success
0RxVV Init Success
00x0000000A00002200
0I
nitializing Initializing test data...
TTest Data Move
Test Data Verify
  [RVV-Simple] Vector write to VLUT AXI memory
Test Data Verify
  [RVV] Vector verification
  [RVV] Vector verification PASSED
Test Data Mul
 [RVV-Simple] Vector mul completed
Test Data Verify
  [RVV] Vector verification
  [RVV] Vector verification PASSED
Test Move PASSED!
Test Mul PASSED!
0x8000000A00002600
Terminated

• I also found that CSR error occurs for all vector-extension related CSRs...

[donghyun-kang-dn]

I found it works well on v5.0.0.

• However, for some unknown reason, it conflicts with the -d in_asm option in this version, so I was unable to check the QEMU assembly trace.
• Without asm trace, I just observed that there were no separate issues with the -d int option set and the main vector test (previously, only the simple version worked) now passes. I think this indicates that the vector extension is well supported.

Thanks for your support.