A browser-based Whitney-style audiovisual instrument controlled by hand tracking.
- Live demo: not published yet
- Local demo: serve the folder locally and open it in a browser with camera access
- Media to add: screenshot or GIF showing the Whitney disc, hand skeleton, and control panel
Hands × Whitney Music Box combines a Whitney-style rotating-disc visual system with Web Audio synthesis and MediaPipe hand tracking. The left hand controls sound behavior, while the right hand controls rotation and motion. The result is a small audiovisual instrument where movement, geometry, and tone generation are connected in real time.
The project is part of Electro-Dig's broader exploration of electronic music archaeology, mathematical music, and embodied browser instruments.
| Input | Effect |
|---|---|
| Left hand height | Transposes the musical output across roughly -12 to +12 semitones |
| Left fist | Cycles sound presets such as piano, guitar, pluck, and bell |
| Right hand tilt | Controls rotation direction and speed |
| Wheel mode | Uses circular hand movement as clockwise / counter-clockwise control |
| Control panel | Adjusts music mode, scale mode, motion, colors, hand lines, Whitney lines, and mirroring |
- Whitney-style rotating disc visualization
- Sound triggers mapped to visual zero-crossing events
- MediaPipe hand tracking for two-hand control
- Tone.js / Web Audio synthesis
- Multiple sound presets
- Multiple scale modes, including classic, harmonic, microtonal, and sci-fi reversed mappings
- Motion and color modes for visual exploration
- Bilingual UI controls in the app
- JavaScript / HTML / CSS
- Paper.js for vector drawing
- MediaPipe Tasks Vision for hand tracking
- Tone.js and Web Audio for sound generation
- Browser camera APIs
Because this project uses ES modules and camera access, run it from a local web server instead of opening index.html directly.
python -m http.server 8000Then open:
http://localhost:8000
Most modern browsers allow camera access on localhost. If camera access fails, check browser permissions and make sure no other app is using the camera.
The app requires camera permission for hand tracking. Camera frames are used locally in the browser for MediaPipe hand landmark detection. The project does not intentionally upload camera video to a backend.
Prototype / active experiment.
Recommended next improvements:
- Publish a live demo
- Add screenshots and a short interaction GIF
- Add a lightweight
package.jsonwith a dev server script - Document the visual modes and scale modes with examples
Inspired by Whitney-style mathematical animation, browser creative coding, and the use of hand gestures as musical control surfaces.
No license has been specified yet.
一个由手势控制的浏览器端 Whitney 风格视听乐器原型。
- 在线演示：暂未发布
- 本地演示：本地启动静态服务器后，在浏览器中打开并允许摄像头权限
- 建议补充素材：展示 Whitney 圆盘、手部骨架与控制面板的截图或 GIF
Hands × Whitney Music Box 把 Whitney 风格的旋转圆盘视觉系统、Web Audio 声音合成和 MediaPipe 手势识别连接在一起。左手控制声音，右手控制旋转与运动，让身体动作、几何图形和声音生成形成实时关联。
这个项目属于 Electro-Dig 对电子音乐考古、数学音乐和身体化浏览器乐器的系列探索。
| 输入 | 效果 |
|---|---|
| 左手高度 | 在大约 -12 到 +12 半音范围内控制移调 |
| 左手握拳 | 切换 piano / guitar / pluck / bell 等音色预设 |
| 右手倾斜 | 控制旋转方向和速度 |
| Wheel mode | 用手的圆周运动控制顺时针 / 逆时针旋转 |
| 控制面板 | 调整音乐模式、音阶模式、运动方式、颜色、手势线、Whitney 连线和镜像 |
- Whitney 风格旋转圆盘视觉
- 视觉零点穿越事件触发声音
- 使用 MediaPipe 进行双手追踪
- 使用 Tone.js / Web Audio 进行声音合成
- 多种音色预设
- 多种音阶模式，包括 classic、harmonic、microtonal 和 sci-fi reversed 映射
- 可切换的运动模式与颜色模式
- 应用内置中英双语 UI 控制
- JavaScript / HTML / CSS
- Paper.js 矢量绘图
- MediaPipe Tasks Vision 手势识别
- Tone.js 与 Web Audio 声音生成
- 浏览器摄像头 API
由于项目使用 ES modules 和摄像头权限，请通过本地服务器运行，而不是直接打开 index.html。
python -m http.server 8000然后打开：
http://localhost:8000
多数现代浏览器允许 localhost 使用摄像头。如果无法访问摄像头，请检查浏览器权限，并确认没有其他应用占用摄像头。
项目需要摄像头权限来识别手势。摄像头画面只用于浏览器本地的 MediaPipe 手部关键点检测；项目本身不会主动把摄像头视频上传到后端。
原型 / 持续实验中。
建议下一步：
- 发布在线 demo
- 增加截图和交互 GIF
- 添加轻量级
package.json与 dev server 脚本 - 用例子说明各种视觉模式与音阶模式
项目灵感来自 Whitney 风格数学动画、浏览器 creative coding，以及将手势作为音乐控制界面的实验。
暂未指定 license。