We currently use ONNX Runtime for our human/pet/vehicle object detection. However, it is certainly not the best fit for what we're working on. It solely contributes to half the size of our rootfs in the Secluso OS image (summing up to roughly ~300MB) and drags in dependencies such as python into the image, which nothing else in our image uses. Replacing this would be a massive improvement.
We incorporated ONNX Runtime a year ago due to there being no Rust alternatives that were fast enough alternatives. However, as of just last month, a new alternative has emerged that is fast enough to compete with it.
Therefore, now that this exists, we can work on incorporating it into the codebase. This will involve some optimization to make it work better on Raspberry Pi. While it's much faster than other Rust alternatives, it still takes 2100ms to run inference, whereas ONNX Runtime only takes 1400ms. I talked to the author and he has given me some tips on what to investigate within the Flex library to improve this number. Therefore, part of this implementing this will be investigating and contributing back to the Burn library to improve performance on Raspberry Pi.
We currently use ONNX Runtime for our human/pet/vehicle object detection. However, it is certainly not the best fit for what we're working on. It solely contributes to half the size of our rootfs in the Secluso OS image (summing up to roughly ~300MB) and drags in dependencies such as python into the image, which nothing else in our image uses. Replacing this would be a massive improvement.
We incorporated ONNX Runtime a year ago due to there being no Rust alternatives that were fast enough alternatives. However, as of just last month, a new alternative has emerged that is fast enough to compete with it.
Therefore, now that this exists, we can work on incorporating it into the codebase. This will involve some optimization to make it work better on Raspberry Pi. While it's much faster than other Rust alternatives, it still takes 2100ms to run inference, whereas ONNX Runtime only takes 1400ms. I talked to the author and he has given me some tips on what to investigate within the Flex library to improve this number. Therefore, part of this implementing this will be investigating and contributing back to the Burn library to improve performance on Raspberry Pi.