DM-Dog-RL

中文说明

DM-Dog-RL 是一个基于强化学习的达妙四足机器狗行走项目，包含训练代码和真机部署代码。

项目结构

• train/：Isaac Sim / Isaac Lab 强化学习训练与微调代码，当前主要任务是 AnymalC-Flat。
• deploy/rl_ws/：ROS Noetic 真机部署工作空间，用于在 NUC 上运行导出的 ONNX 策略。
• video/：当前硬件条件下的真机行走视频。

核心工作与项目贡献

本项目的重点不是简单复现已有代码，而是围绕达妙四足机器狗硬件，完成了一条从强化学习训练、策略导出、ROS 真机部署到实机调试优化的闭环。主要工作包括：

• 基于 RoboParty / Isaac Lab 训练框架，创建了面向达妙真机的仿真训练环境，并修正 URDF 模型、关节命名、关节顺序、默认姿态、动作尺度和观测构造。
• 设计并迭代了面向真机部署的奖励项和约束，包括抬腿高度、身体姿态、后腿高度一致性、足端滑移、电机速度余量和速度-扭矩包络。
• 构建 Isaac Sim 训练、MuJoCo sim2sim 验证、ONNX 策略导出和 ROS C++ 控制器部署链路，使策略在真机部署前可以先经过跨仿真环境检查。
• 在部署侧实现自动站立、遥控器进入 policy、前后左右 / yaw 控制、两段式趴下 / 阻尼 / 电机失能，以及面向真机调试的日志输出。
• 基于真机日志和视频反复分析策略目标、关节跟踪误差、抬腿高度和身体姿态，并根据达妙电机真实性能进行多轮训练与部署参数优化。

这些改动体现的是把强化学习步态从仿真模型落到真实达妙机器狗上的应用能力：不仅训练出策略，还让策略能够通过实际 ROS 控制链路在真机上运行、调试和迭代。

运行环境

• 仿真 / 训练：Isaac Sim 5.1.0，Python 3.11
• 真机部署：Ubuntu 20.04，ROS Noetic
• 真机硬件：达妙电机四足机器狗硬件，达妙电机执行层

参考代码仓库

本项目不是完全从零开始写的，而是在已有开源代码基础上完成了达妙真机对应仿真训练环境构建、URDF 模型修正、策略导出、真机部署和 sim-to-real 调试。

• RoboParty 训练代码：https://github.com/Roboparty/roboto_origin/tree/main/modules/roboparty_train
• 达妙 OpenDog 开源资料：https://gitee.com/kit-miao/open-dog
• Isaac Lab：https://github.com/isaac-sim/IsaacLab
• RSL-RL：https://github.com/leggedrobotics/rsl_rl

训练代码主要基于 RoboParty 的 roboparty_train 进行二次开发，创建了达妙真机对应的 anymal_c 仿真训练环境，修正了 URDF 模型和关节映射，并增加了真机微调相关奖励、策略导出、MuJoCo sim2sim 验证和调试信息。部署代码围绕达妙电机四足机器狗硬件重新整理，当前使用 ROS 控制链路加载 ONNX 强化学习策略。

电机性能与当前瓶颈

当前真机使用的是达妙 DM-J6248P / DM6248P 电机。官方产品页：https://www.damiaokeji.com/index.php?c=show&id=110

代码中和当前调试相关的电机参数如下：

• 电机真实关键性能参数：峰值扭矩 97 Nm，24V 空载速度约 6.28 rad/s（60 rpm）。这是判断当前真机性能瓶颈时最重要的速度-扭矩参数。
• 部署层达妙库里 DM6248P 的 Q_MAX = 12.566 rad、DQ_MAX = 20 rad/s、TAU_MAX = 120 Nm 是协议/命令量程，不代表当前 24V 真机工况下可以持续达到这些动态性能。
• 训练中已经加入电机速度余量惩罚和速度-扭矩包络约束：关节速度接近空载速度时，可用扭矩会按速度-扭矩曲线下降。

因此，当前版本不是一个“策略还没调好”的初版，而是在现有电机和执行链路条件下，经过多轮真机和仿真微调后的稳定部署版本。策略已经能在仿真中稳定给出合理步态，真机也能达到约 3 cm 抬腿高度；剩余主要问题是电机 / 执行层动态性能余量不足，尤其后腿 KFE/HFE 在摆腿和落脚时需要较快响应，但实机关节跟踪存在延迟，导致身体摇晃和后退不够稳定。

当前真机效果

仿真中，机器狗整体平均抬腿高度约 3.1 cm，峰值约 4.2 cm，符合训练时设置的目标抬腿高度。真机中，四条腿实际抬腿高度大致也在 3 cm 左右，能正常抬腿。

目前真机可以通过遥控器控制前进、左右移动；后退也可以实现，但不能把摇杆一次推到底，需要轻推、松手、再轻推，后退才更稳定。当前主要问题是真机电机 / 执行层跟踪偏慢，也就是电机速度跟不上策略给出的目标关节位置。日志里后腿 KFE/HFE 跟踪误差更明显，导致后腿摆腿和落脚时机不够准，进而带来身体摇晃。

基于现在硬件条件录制的真机行走视频放在 video/ 目录下。

快速部署

把 deploy/rl_ws 放到 NUC 的 ~/rl_ws，第一次使用时编译：

cd ~/rl_ws
catkin init
catkin config -DCMAKE_BUILD_TYPE=RelWithDebInfo
catkin build
source devel/setup.bash

每次运行机器狗前，先初始化 CAN，再启动控制器：

cd ~/rl_ws/src/legged_damiao_hw/5.15
bash ./can.sh

cd ~/rl_ws
source devel/setup.bash
roslaunch rl_dog_controller rl_dog_hw.launch

启动后机器狗会自动站立。站稳后按 button0 进入 policy 模式，然后用摇杆控制运动。

遥控器映射：

• axis 5：前进 / 后退，linear.x，范围 [-0.2, 0.5] m/s
• axis 4：左右平移，linear.y，范围 [-0.1, 0.1] m/s
• axis 2：yaw 转向，angular.z，范围 [-0.2, 0.2] rad/s
• button0：进入 RL policy 模式
• button3：两段式安全停止。第一次按下趴下，第二次按下进入阻尼并失能电机。

更多训练命令见 train/README.md，更多真机部署命令见 deploy/README.md。

安全提示

真机测试前建议先用支架或人工保护支撑机器狗。自由行走前确认 button3 两段式趴下 / 阻尼 / 失能流程正常。

English

DM-Dog-RL is a reinforcement-learning locomotion project for a Damiao-motor quadruped robot. It contains both the training stack and the real-robot ROS deployment workspace.

Project Structure

• train/: Isaac Sim / Isaac Lab training and fine-tuning code. The main task used for the current real robot is AnymalC-Flat.
• deploy/rl_ws/: ROS Noetic deployment workspace for running the exported ONNX policy on the NUC.
• video/: real-robot walking videos recorded with the current hardware setup.

Project Contributions

This project is not a simple code reproduction. It builds a complete reinforcement-learning-to-real-robot workflow for a Damiao-motor quadruped, covering training, policy export, ROS deployment, and real-robot debugging. The main project-specific work includes:

• Building a Damiao-real-robot-oriented simulation training environment on top of RoboParty / Isaac Lab, including URDF model corrections, joint naming, joint order, default pose, action scale, and observation layout.
• Designing and iterating real-deployment-oriented rewards and constraints for swing-foot height, body attitude, rear-leg height consistency, foot slip, motor speed margin, and the speed-torque envelope.
• Building a full validation and deployment chain from Isaac Sim training to MuJoCo sim2sim verification, ONNX policy export, and a ROS C++ real-robot controller, so the policy can be checked in a second simulator before hardware deployment.
• Implementing deployment features such as automatic standing, joystick-triggered policy mode, forward/backward/side/yaw commands, two-stage lie-down / damping / motor-disable behavior, and real-robot debug logging.
• Iteratively analyzing real logs and videos for policy targets, joint tracking error, swing-foot height, and body attitude, then tuning both training and deployment parameters around the real Damiao motor performance.

The value of this repository is the application work needed to move an RL gait from simulation onto a real Damiao quadruped: the policy is trained, exported, deployed through ROS, tested on hardware, debugged, and iteratively improved.

Environment

• Simulation / training: Isaac Sim 5.1.0, Python 3.11
• Real-robot deployment: Ubuntu 20.04, ROS Noetic
• Hardware: Damiao-motor quadruped hardware

Reference Repositories

This repository is an integration and adaptation project, not a fully from-scratch stack.

• RoboParty training code: https://github.com/Roboparty/roboto_origin/tree/main/modules/roboparty_train
• Damiao OpenDog materials: https://gitee.com/kit-miao/open-dog
• Isaac Lab: https://github.com/isaac-sim/IsaacLab
• RSL-RL: https://github.com/leggedrobotics/rsl_rl

The training code is developed on top of RoboParty roboparty_train to create a Damiao-real-robot-oriented anymal_c simulation environment, including URDF model corrections, joint mapping, real-machine fine-tuning rewards, policy export, MuJoCo sim2sim verification, and sim-to-real debugging. The deployment code is reorganized around the Damiao-motor quadruped hardware and uses a ROS control chain to run the exported ONNX RL policy.

Motor Performance And Current Bottleneck

The real robot uses Damiao DM-J6248P / DM6248P motors. Official product page: https://www.damiaokeji.com/index.php?c=show&id=110

Motor parameters used by the current code and debugging workflow:

• Real key motor performance parameters: peak torque 97 Nm, 24V no-load speed about 6.28 rad/s (60 rpm). These are the most important speed-torque parameters for understanding the current real-robot bottleneck.
• The deployment-side Damiao DM6248P values Q_MAX = 12.566 rad, DQ_MAX = 20 rad/s, and TAU_MAX = 120 Nm are protocol/command ranges, not evidence that the motor can continuously deliver that dynamic performance under the current 24V real-robot operating condition.
• Training already includes motor speed-margin penalties and a speed-torque envelope: as joint speed approaches no-load speed, available torque decreases according to the speed-torque curve.

This version is therefore not an early untuned policy. It is a real-robot deployment version tuned through multiple simulation and hardware iterations around the current actuator and execution-chain constraints. The policy produces reasonable gaits in simulation and the real robot reaches roughly 3 cm swing-foot height. The remaining bottleneck is limited actuator / execution-layer dynamic margin, especially around the rear-leg KFE/HFE joints, where swing and touchdown require fast tracking. The physical joints lag behind the target positions, causing body shaking and less stable backward walking.

Current Real-Robot Status

In simulation, the robot's average effective swing-foot height is about 3.1 cm, with peaks around 4.2 cm, matching the training target. On the real robot, all four legs also lift to roughly 3 cm on average.

The real robot can currently move forward and sideways through the joystick. Backward walking also works, but it is more stable to apply short backward stick inputs instead of pushing the stick fully backward and holding it. The main remaining issue is slow tracking in the real motor / execution layer: the policy sends target joint positions, but the physical joints respond with delay. Logs show larger tracking errors around the rear-leg KFE/HFE joints, which makes rear-leg swing and touchdown timing less accurate and causes body shaking.

Real-robot walking videos under the current hardware conditions are stored in video/.

Quick Deploy

Copy deploy/rl_ws to the NUC as ~/rl_ws, then build once:

cd ~/rl_ws
catkin init
catkin config -DCMAKE_BUILD_TYPE=RelWithDebInfo
catkin build
source devel/setup.bash

Before each run, initialize CAN and start the controller:

cd ~/rl_ws/src/legged_damiao_hw/5.15
bash ./can.sh

cd ~/rl_ws
source devel/setup.bash
roslaunch rl_dog_controller rl_dog_hw.launch

After launch, the robot stands automatically. Press button0 after it is stable to enter policy mode.

Joystick mapping:

• axis 5: forward/backward, linear.x, range [-0.2, 0.5] m/s
• axis 4: left/right, linear.y, range [-0.1, 0.1] m/s
• axis 2: yaw, angular.z, range [-0.2, 0.2] rad/s
• button0: enter RL policy mode
• button3: two-stage safe stop. First press: lie down. Second press: damping mode and motor disable.

See train/README.md for training commands and deploy/README.md for real-robot deployment commands.

Safety

Test with the robot supported before free walking. Confirm that the button3 two-stage lie-down / damping / motor-disable sequence works before free walking.