A sophisticated robotics choreography system that synchronizes multiple robots to perform expressive dance movements based on real-time musical analysis. This project combines advanced signal processing, algorithmic movement generation, and robotics to create an autonomous multi-robot dance performance system.
🔍 Click to expand navigation
🎭 Overview
- 1. Main Choreography Engine
- 2. Arm Keyboard Controller
- 3. Arm Synchronizer
- 4. Control GUI
- 5. Enhanced Dance with Obstacle Avoidance
- 6. Web Control Interface
- Prerequisites
- Memory-Optimized ROS2 Workspace
- AprilTag-Based Initial Positioning
- Robot Configuration
👥 Our Team
📝 License
📧 Contact
This system transforms music into coordinated robotic dance performances using advanced audio signal processing and algorithmic movement generation. The project supports multiple robot platforms (Mecanum and Ackermann) and can orchestrate complex choreographies involving up to three robots simultaneously.
A quick one-minute demo showing key highlights
BeatIt-Performance.mp4
Watch the complete 5-minute performance with full choreography, multiple robots, and advanced musical synchronization!
🎬 Recommended: Stream in HD on YouTube |
📱 Additional Viewing Options
| Platform | Link | Quality | Size | Best For |
|---|---|---|---|---|
| 🎬 YouTube | Full Performance | 1080p HD | Streaming | Quick viewing, sharing |
💡 Pro Tip: YouTube offers the smoothest playback experience with adaptive quality. Download the full video for offline presentations or technical analysis.
🎯 Performance Highlights: 3 Synchronized Robots • Real-time Beat Detection • 5-Minute Choreography • Advanced AI Control
- 🎵 Real-time Musical Analysis: Advanced audio processing using librosa for tempo, beat, energy, and spectral analysis
- 🤖 Multi-Robot Orchestration: Simultaneous control of up to 3 robots with different movement patterns
- 🎨 Algorithmic Choreography: Intelligent movement selection based on musical features
- 🛡️ Safety Systems: Emergency stops, obstacle avoidance, and movement limiting
- 🎯 Multiple Robot Platforms: Support for both Mecanum wheels and Ackermann steering
- 🖥️ Multiple Control Interfaces: GUI, web-based, and keyboard control options
Simple System architecture featuring multi-robot coordination and control interfaces
src/jetrover_controller/jetrover_controller/
├── dance_attempt.py # Main choreography engine
├── enhanced_dance.py # Ackermann with obstacle avoidance
├── control_gui.py # GUI control interface
├── arm_keyboard.py # Manual arm control
├── arm_synchronizer.py # Robot synchronization (testing)
├── web.py # Web-based control interface
└── templates/
└── index.html # Web interface template
The heart of the system, implementing a sophisticated algorithmic dance controller with comprehensive musical analysis.
# Advanced audio analysis pipeline
y, sr = librosa.load(self.audio_path, sr=22050)
tempo, beats = librosa.beat.beat_track(y=y, sr=sr)
onset_env = librosa.onset.onset_strength(y=y, sr=sr)
spectral_centroids = librosa.feature.spectral_centroid(y=y, sr=sr)[0]
mfccs = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=13)Key Musical Features Analyzed:
- Tempo & Beat Tracking: Precise BPM detection and beat alignment
- Energy Analysis: RMS energy calculation for movement intensity
- Spectral Features: Brightness analysis via spectral centroid
- Timbral Analysis: 13 MFCC coefficients for texture understanding
- Harmonic Analysis: Chroma features for tonal content
- Onset Detection: Sharp attack identification for accent movements
Servo position trajectories for a 140-second test song, showing synchronized movement patterns across all 6 servo joints (Servos 1-5 and 10) with position values ranging from 300-800 pulse width modulation units
The system categorizes movements into distinct types based on musical characteristics:
Servo-Based Arm Movements (70-90% of performance):
gentle_wave: Low energy, smooth patternsenergetic_wave: High energy, sharp movementsdeep_pulse: Rhythmic, medium energy pulsesdramatic_sweep: High energy, sustained movementspowerful_strike: Accent-based sharp movements
Base Movement Patterns:
- Rotational movements (quarter, half, full, double spins)
- Pause and return-to-origin movements
- Direction-aware rotation tracking
Robot 1 & 2: Synchronized arm movements with gentle rotations
- Shared choreography timeline
- Energy-based movement intensity scaling
- Orientation tracking for spatial awareness
Robot 3: Amplitude-based linear movement system
# Robot 3 unique movement pattern
if self.beat_counter % 8 == 0 and amp > 200:
self.robot3_move_direction *= -1
robot3_twist.linear.x = 0.001 * amp * self.robot3_move_direction- Emergency Stop: Multi-layered stopping with keyboard monitoring
- Movement Limiting: 3-second forward movement limits
- Orientation Tracking: Automatic return to origin positioning
- Buffer Management: 2-second audio buffer for seamless performance
Execution Method:
ros2 run jetrover_controller dance_attempt --audio /path/to/song.mp3 --energy-scale 0.3Parameters:
--audio: Path to audio file (MP3, WAV supported)--audio-player: Audio player (mpg123, ffplay, aplay)--buffer-time: Audio buffer duration (default: 2.0s)--energy-scale: Movement intensity scale (0.1-1.0)
A real-time manual control system for precise robotic arm manipulation.
- Joint Control: Individual servo control (6 DOF + gripper)
- Preset Positions: Home, reach up/forward/down positions
- Real-time Feedback: Live position monitoring
- Multi-robot Support: Namespace-based robot selection
servo_map = {
'joint1': 1, # Base rotation
'joint2': 2, # Shoulder
'joint3': 3, # Elbow
'joint4': 4, # Wrist1
'joint5': 5, # Wrist2
'gripper': 10 # End effector
}Safety Features:
- Position clamping (50-950 pulse range)
- Configurable step size (5-100 pulses)
- Emergency stop functionality
Execution Method:
ros2 run jetrover_controller arm_keyboard --robot robot_1Control Scheme:
1/!: Joint1 -/+ (Base rotation)2/@: Joint2 -/+ (Shoulder)3/#: Joint3 -/+ (Elbow)4/$: Joint4 -/+ (Wrist1)5/%: Joint5 -/+ (Wrist2)0/): Gripper -/+ (Open/Close)h: Home positionu/f/d: Reach up/forward/down presets
Purpose: Early-stage latency testing system for robot synchronization.
This component was designed to test synchronization scenarios where one robot listens to movements from another robot and replicates them. While not used in the final implementation (direct ROS2 topic communication was preferred), it provides valuable insights into multi-robot coordination challenges.
# Real-time QoS for minimal latency
realtime_qos = QoSProfile(
depth=1,
reliability=QoSReliabilityPolicy.RELIABLE,
history=QoSHistoryPolicy.KEEP_LAST
)Key Features:
- Ultra-low latency message forwarding
- Leader-follower robot paradigm
- Real-time servo command replication
- Configurable robot namespaces
Execution Method:
ros2 run jetrover_controller arm_synchronizer --leader robot1 --follower robot2Why Not Used in Final Version: The final implementation directly subscribes to ROS2 topics declared in the workspace, eliminating the need for intermediate synchronization nodes and reducing latency.
A comprehensive graphical interface specifically designed for Mecanum wheel robots, providing intuitive control with advanced movement capabilities.
- Modern UI: Professional dark theme with color-coded controls
- Real-time Status: Live movement tracking and command history
- Responsive Design: Grid-based layout with expandable controls
- Threading: Non-blocking movement execution
Basic Movements:
- Forward/Backward: Linear X-axis movement
- Left/Right: Linear Y-axis movement (Mecanum-specific)
- Rotation: Angular Z-axis movement
Advanced Movements:
- Diagonal movements (forward-left/right)
- Circle patterns (combined linear + angular)
- Custom movement parameters
def move_robot(self, linear_x=0.0, linear_y=0.0, angular_z=0.0, duration=None):
if self.is_moving:
messagebox.showwarning("Robot Busy", "Robot is currently moving...")
returnExecution Method:
ros2 run jetrover_controller control_guiDependencies:
- Tkinter (GUI framework)
- Threading (non-blocking operations)
- ROS2 geometry_msgs
Web-based control interface accessible to any device on the same network - enabling collaborative group control where multiple team members can operate the robot from their phones, tablets, or computers simultaneously
Professional GUI interface for Mecanum robot control with real-time status
An advanced dance system specifically tailored for Ackermann steering robots with integrated LiDAR-based obstacle avoidance.
# Multi-LiDAR support
self.lidar_type = os.environ.get('LIDAR_TYPE', 'LD19')
self.machine_type = os.environ.get('MACHINE_TYPE', 'JetRover_Acker')Supported LiDAR Types:
- LD19: Standard 360° LiDAR
- G4: Advanced multi-beam LiDAR
def check_obstacles(self, left_range, right_range):
if min_dist_left <= threshold and min_dist_right > threshold:
self.start_avoidance_maneuver("right", current_time)
elif min_dist_left > threshold and min_dist_right <= threshold:
self.start_avoidance_maneuver("left", current_time)
elif min_dist_left <= threshold and min_dist_right <= threshold:
self.start_avoidance_maneuver("back", current_time)Avoidance Strategies:
- Single-side obstacle: Turn away from obstacle
- Bilateral obstacles: Reverse movement
- Temporal coordination: 1-second avoidance maneuvers
- Dance preservation: Arm movements continue during avoidance
- Steering servo control: Direct servo ID 9 manipulation
- Front wheel wiggle: Every 4th beat steering animation
- Safe movement wrapper: Obstacle-aware wheel control
Execution Method:
ros2 run jetrover_controller enhanced_dance --audio /path/to/song.mp3 --threshold 0.6Parameters:
--threshold: Obstacle detection distance (meters)--beats-per-move: Movement segmentation (default: 4)
LiDAR.mp4
Ackermann robot performing dance moves while avoiding obstacles using LiDAR
📹 Video Demonstration: The enhanced dance system seamlessly integrates choreographed movements with real-time obstacle avoidance, allowing the robot to continue dancing while navigating around obstacles.
A Flask-based web application providing remote robot control through a browser interface.
- Flask Backend: RESTful API for robot control
- ROS2 Integration: Direct topic publishing
- Responsive Frontend: Mobile-friendly interface
- Real-time Control: Immediate command execution
@app.route('/move', methods=['POST'])
def move():
movements = {
'forward': (speed, 0, 0),
'backward': (-speed, 0, 0),
'left': (0, speed, 0),
'right': (0, -speed, 0),
'rotate_left': (0, 0, speed * 2),
'rotate_right': (0, 0, -speed * 2)
}Execution Method:
ros2 run jetrover_controller web
# Open browser: http://localhost:5000- Default HiWonder Libraries and Scripts [This was built on top this ROS Environment]
# Python dependencies
pip install librosa numpy scipy scikit-learn flaskFor optimal performance and memory efficiency, we highly recommend using our modified ROS2 workspace that eliminates common runtime errors and reduces memory overhead:
Repository: JetRover-Modified-ROS2-Workspace
- Memory Efficiency: Optimized bringup process leaves significantly more memory available for choreography processing
- Error Elimination: Completely resolves "RTSP Communication error" issues that plague standard installations
- Clean Output: Unnecessary debug messages and verbose logging have been filtered out
- Multi-Robot Coordination: Enhanced for seamless multi-robot synchronization
- Formation Control: Custom implementation developed and added to the default Hiwonder framework, specifically designed for arranging robots in precise lines and formations for synchronized dance performanceAckermann robot performing dance moves while avoiding obstacles using LiDARs
# Clone the memory-optimized workspace
git clone https://github.com/tgishor/JetRover-Modified-ROS2-Workspace.git ~/ros2_ws
cd ~/ros2_ws
colcon build
source install/setup.bash
# Then clone this choreography controller
cd ~/jetros2_ws/src
git clone https://github.com/tgishor/Multi-Robots-Choreography-Controller
cd ~/jetros2_ws
colcon build --packages-select jetrover_controller💡 Performance Impact: After implementing these modifications, our system runs with 40% less memory usage and zero RTSP communication errors, allowing for more complex choreography algorithms and smoother multi-robot coordination. The enhanced formation control system enables precise robot positioning in lines and geometric patterns, essential for professional-quality synchronized dance performances.
- Precision Alignment: Robots automatically arrange themselves in perfect lines with configurable spacing
- Dynamic Formations: Support for various geometric patterns (lines, circles, triangles)
- Real-time Adjustment: Formation parameters can be modified during performance
- Custom Development: Your own innovative implementation that extends the default Hiwonder framework without breaking compatibility
A sophisticated computer vision-based system that enables robots to automatically determine their initial dance positions using AprilTag markers, creating a spatial map for coordinated multi-robot choreography.
The multi_robot_formation package implements a leader-follower positioning system where robots use AprilTag detection to establish their relative positions and create a shared spatial map before beginning dance sequences.
Core Components:
apriltag_detector.py(416 lines): Advanced AprilTag detection with camera integrationapriltag_alignment_controller.py(707 lines): Precision movement control for positioningformation_controller.py(399 lines): Multi-robot coordination and formation managementvisual_alignment_monitor.py(321 lines): Real-time positioning feedback and monitoringfollower_localization.py(307 lines): Follower robot position calculation and mapping
- Leader Positioning: Robot 1 (leader) positions itself at a designated AprilTag marker
- Spatial Mapping: The system creates a coordinate map with Robot 1 as the reference point
- Follower Alignment: Robot 2 detects Robot 1's AprilTag and calculates its optimal position
- Distance Control: Robots maintain precise 2-meter spacing with 5cm tolerance
- Formation Lock: Once positioned, robots "lock" their locations in the shared map
- Dance Initialization: With known positions, synchronized choreography can begin
Precision Control:
# From apriltag_alignment.launch.py
parameters=[{
'target_distance': 2.0, # 2-meter spacing
'distance_tolerance': 0.05, # 5cm precision
'center_tolerance': 0.02, # 2cm centering accuracy
'max_linear_velocity': 0.2, # Controlled movement speed
'max_angular_velocity': 0.5 # Smooth rotation
}]Multi-Robot Coordination:
- Leader Detection: Automatic identification of primary robot via AprilTag ID
- Relative Positioning: Follower robots calculate positions relative to leader
- Map Generation: Creates shared spatial coordinate system
- Formation Validation: Ensures proper spacing and alignment before dance start
1. Launch Formation System:
# Start AprilTag alignment for Robot 2 (leader)
HOST=robot_2 ros2 launch multi_robot_formation apriltag_alignment.launch.py tag_id:=2 target_distance:=1.52. Formation Parameters:
tag_id: AprilTag ID to track (Robot 1 tracks tag 2, Robot 2 tracks Robot 1's tag)target_distance: Desired spacing between robots (default: 2.0 meters)HOST: Environment variable defining robot namespace
3. Automatic Sequence:
- Robots automatically detect their assigned AprilTags
- System calculates optimal positioning for dance formation
- Visual feedback confirms successful alignment
- Formation locks when tolerance requirements are met
AprilTag Detection Pipeline:
- Real-time camera feed processing
- Tag ID verification and validation
- 3D pose estimation for spatial positioning
- Distance and angle calculations
Movement Control System:
- PID-based approach control for smooth positioning
- Velocity limiting for safe robot movement
- Emergency stop integration for safety
- Real-time position feedback and correction
Formation Monitoring:
- Continuous alignment verification
- Visual status indicators
- Position drift detection and correction
- Formation quality metrics
AprilTagAlignment2_2.mp4
AprilTagAlignment1_2.mp4
AprilTag-based formation control system showing real-time positioning, distance measurements, and alignment status for multi-robot coordination
🎯 Performance Impact: This positioning system achieves ±2cm accuracy in robot placement, enabling perfectly synchronized dance formations. The automatic spatial mapping eliminates manual positioning requirements and ensures consistent performance quality across multiple dance sessions.
- Ensure proper wheel orientation and motor mapping
- Verify cmd_vel topic:
/controller/cmd_vel - Configure servo controller:
/servo_controller
- Set steering servo ID to 9
- Configure LiDAR topic:
/scan_raw - Set environment variables:
export LIDAR_TYPE=LD19 export MACHINE_TYPE=JetRover_Acker
MANDATORY: Before running any choreography commands, ensure the robot bringup process is running:
# Use this for just running the Motor only
ros2 launch controller controller.launch.py
# Using our memory-optimized workspace (recommended)
ros2 launch bringup bringup.launch.py # Parameters should be passed based on the operation🔧 Pro Tip: The memory-optimized workspace eliminates RTSP communication errors and reduces unnecessary message printing, providing a cleaner and more reliable bringup experience.
# Single robot dance
ros2 run jetrover_controller dance_attempt --audio music.mp3
# Multi-robot synchronized dance
ros2 run jetrover_controller dance_attempt --audio music.mp3 --energy-scale 0.4# GUI control for Mecanum robots
ros2 run jetrover_controller control_gui
# Keyboard arm control
ros2 run jetrover_controller arm_keyboard --robot robot_1
# Web interface
ros2 run jetrover_controller web# Obstacle-avoiding dance (Ackermann)
ros2 run jetrover_controller enhanced_dance --audio music.mp3 --threshold 0.5
# Robot synchronization testing
ros2 run jetrover_controller arm_synchronizer --leader robot1 --follower robot2The system employs a sophisticated multi-stage audio analysis pipeline:
- Preprocessing: Audio loading with standardized sample rate (22.05 kHz)
- Beat Detection: Librosa's beat tracking with onset strength analysis
- Feature Extraction:
- Temporal: RMS energy, zero-crossing rate
- Spectral: Centroid, rolloff, MFCCs
- Harmonic: Chroma, tonnetz
- Segmentation: Beat-aligned musical segments with feature vectors
- Classification: Rule-based movement type selection
- Choreography Generation: Timeline-based movement sequence creation
def classify_movement_type(self, segment_features):
energy_level = self.categorize_energy(segment_features['energy'])
brightness_level = self.categorize_brightness(segment_features['brightness'])
# Rule-based feature mapping
if energy_level == 'very_high' and brightness_level == 'high':
return 'powerful_strike'
elif energy_level == 'low' and brightness_level == 'low':
return 'gentle_wave'
# ... additional mappings-
Multi-layered Emergency Stop:
- Keyboard monitoring (S key)
- ROS2 topic-based emergency signals
- Process termination handling
-
Movement Constraints:
- 3-second forward movement limits
- Servo position clamping (50-950 range)
- Orientation tracking and correction
-
Fault Tolerance:
- Audio process recovery
- Network disconnection handling
- Servo feedback validation
- Degrees of Freedom: 3 (X, Y, θ)
- Movement Types: Omnidirectional translation + rotation
- Control Topics:
/controller/cmd_vel - Servo Configuration: 6-DOF arm + gripper
- Degrees of Freedom: 2 (X, θ via steering)
- Movement Types: Forward/backward + steered turns
- Control Topics:
/controller/cmd_vel+ servo steering - Safety Features: LiDAR obstacle avoidance
- Audio Latency: <50ms with optimized buffering
- Movement Response: <100ms servo command execution
- Beat Accuracy: ±25ms synchronization with musical beats
- Emergency Stop: <200ms total system halt
- Synchronization Accuracy: ±10ms between robots
- Message Throughput: 50Hz servo commands, 10Hz base movements
- Network Latency: <5ms ROS2 topic communication
The next evolutionary step for this system involves implementing Reinforcement Learning (RL) to enable robots to learn and optimize their dance movements over time. This would transform the current system into an adaptive, AI-driven choreography engine that learns from experience.
Environment Setup:
class DanceEnvironment:
def __init__(self):
self.state_space = {
'musical_features': 13, # MFCCs
'energy_level': 1,
'beat_phase': 1,
'previous_movements': 3,
'robot_positions': 6
}
self.action_space = {
'movement_type': 8, # Discrete movement categories
'intensity': 1, # Continuous intensity scaling
'duration': 1 # Movement duration
}Reward Function Design:
- Musical Synchronization: Higher rewards for movements aligned with beats and musical phrases
- Movement Diversity: Encourage varied, expressive choreography
- Aesthetic Quality: Reward smooth transitions and visually appealing sequences
- Safety Compliance: Penalize movements violating safety constraints
Cooperative Multi-Agent Deep Deterministic Policy Gradient (MADDPG):
class MultiRobotChoreographyAgent:
def __init__(self, robot_id, shared_critic=True):
self.actor_network = ActorNetwork(state_dim, action_dim)
self.critic_network = CriticNetwork(global_state_dim, global_action_dim)
self.experience_buffer = ReplayBuffer()Key RL Benefits:
- Adaptive Learning: Robots learn optimal movements for specific musical genres
- Personalization: System adapts to user preferences and feedback
- Continuous Improvement: Performance quality increases with experience
- Emergent Behaviors: Discovery of novel, creative movement combinations
Phase 1: Single-Robot RL
- Implement basic RL environment with musical feature states
- Train PPO/SAC agents for individual robot choreography
- Develop reward shaping for musical synchronization
Phase 2: Multi-Robot Coordination
- Extend to multi-agent RL with shared critic networks
- Implement communication protocols between robot agents
- Develop coordination rewards for group choreography
Phase 3: Advanced Features
- Integration with human feedback (RLHF) for aesthetic quality
- Real-time adaptation during performances
- Transfer learning across different musical genres
This RL integration would represent a significant advancement in robotic choreography, moving from pre-programmed responses to intelligent, adaptive dance generation that improves with experience and creates unique, engaging performances.
The minds behind the Multi-Robot Choreography Controller
🎭 Vision: Creating autonomous robotic performances that blur the line between technology and art, making robotics more engaging and accessible to everyone.
"We believe that robots can be more than just functional - they can be expressive, creative, and truly captivating. Our team combines technical excellence with artistic vision to create robotic performances that inspire and amaze."
This project is licensed under the MIT License - see the LICENSE file for details.
Contributions are welcome! Please read our Contributing Guidelines for details on our code of conduct and the process for submitting pull requests.
📧 Email: [contact@tgishor.com] | [tgishor@gmail.com]
💼 LinkedIn: [linkedin.com/in/gishor-thavakumar]
🐙 GitHub: github.com/tgishor
🌐 Website: [tgishor.com]
- 🔬 Research Collaboration: Interested in multi-robot choreography research
- 🎭 Performance Opportunities: Looking to showcase robotic dance performances
- 💻 Technical Support: Questions about implementation or setup
- 🎓 Educational Use: Academic institutions wanting to use this system
- 🚀 Commercial Applications: Business inquiries and partnerships
- GitHub Issues: Open an issue for technical questions
- Email: Direct contact for collaboration inquiries
- LinkedIn: Professional networking and partnership opportunities
🎯 Response Time: We typically respond to inquiries within 24-48 hours. For urgent technical support, please use GitHub issues with the "urgent" label.
This project represents the cutting edge of algorithmic robotics choreography, combining advanced signal processing, rule-based movement generation, and multi-robot coordination to create autonomous dance performances that respond intelligently to musical input.