FlyCo

Perception & Predictor

✍️ Authors: Chen Feng and Guiyong Zheng

This workspace provides the perception and prediction side of FlyCo, including prompt-driven 2D target tracking, 3D target clustering, and mesh prediction for downstream planning and visualization.

🛠️ Installation

Prerequisite:

• ROS Noetic (Ubuntu 20.04)
• CUDA-capable GPU
• Conda / Miniconda
• PyTorch matching your CUDA driver

Create and activate the Python environment:

  conda create -n flyco python=3.8 -y
  conda activate flyco

Install PyTorch following the official command for your CUDA version. For example, for CUDA 11.8:

  pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

Install ROS and Python dependencies:

  sudo apt update
  sudo apt install python3-catkin-tools python3-rosdep ros-noetic-cv-bridge ros-noetic-pcl-ros ros-noetic-image-transport ros-noetic-tf2-ros

  cd ${YOUR_PERCEPTION_PREDICTOR_PATH}
  pip install -r src/sam_tracking/requirements.txt
  pip install -r src/mesh_predict/requirements.txt

Install SAM2:

  cd ${YOUR_PERCEPTION_PREDICTOR_PATH}/src/sam_tracking/thirdparty/sam2-realtime
  pip install -e .

Download the SAM2 tracking checkpoints:

  cd ${YOUR_PERCEPTION_PREDICTOR_PATH}/src/sam_tracking
  bash ckpt/download_ckpt.sh

Prepare mesh prediction checkpoints under:

  ${YOUR_PERCEPTION_PREDICTOR_PATH}/src/mesh_predict/resources/checkpoints/

The mesh prediction checkpoints are available from Google Drive.

Expected local files are documented in src/mesh_predict/resources/checkpoints/README.md. The runtime can also resolve custom checkpoint paths from:

• MESH_PREDICT_MESH_CKPT
• MESH_PREDICT_NKSR_CKPT

Build the ROS workspace:

  cd ${YOUR_PERCEPTION_PREDICTOR_PATH}
  source /opt/ros/noetic/setup.bash
  catkin_make -DCMAKE_BUILD_TYPE=Release -DPYTHON_EXECUTABLE=$CONDA_PREFIX/bin/python
  source devel/setup.bash

If you prefer catkin tools:

  cd ${YOUR_PERCEPTION_PREDICTOR_PATH}
  source /opt/ros/noetic/setup.bash
  catkin config -DCMAKE_BUILD_TYPE=Release -DPYTHON_EXECUTABLE=$CONDA_PREFIX/bin/python
  catkin build --cmake-args -Wno-dev
  source devel/setup.bash

⚖️ License

This software is released under the PolyForm Noncommercial License 1.0.0. Third-party components retain their original licenses.