[Feature Request] Decrease the installation time of TorchOpt on GPU machine

[Benjamin-eecs]

Motivation

It may take 3-4min to install TorchOpt, seems unnecessary for such small package

Solution

Maybe optimize the cmakefile.

Checklist

• I have checked that there is no similar issue in the repo (required)