Skip to content

Commit

Permalink
hand tracking progress
Browse files Browse the repository at this point in the history
  • Loading branch information
jakzo committed Aug 23, 2024
1 parent de38a17 commit d6c12dc
Show file tree
Hide file tree
Showing 29 changed files with 1,107 additions and 952 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
/projects/*/*/thunderstore/build/
generated-*
MelonPreferences.cfg


## Ignore Visual Studio temporary files, build results, and
Expand Down
32 changes: 19 additions & 13 deletions projects/Bonelab/HandTracking/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,8 @@ Adds support for hand tracking.

- This mod is for the **patch 4 Quest standalone** version of the game
- Install **Melon Loader 0.5.7** via Lemon Loader
- Add the hand tracking permission to the game APK using [Quest Patcher](https://github.com/Lauriethefish/QuestPatcher)
- Add the hand tracking permission to the game APK using [my fork of Quest Patcher](https://github.com/jakzo/QuestPatcher/releases)
- **IMPORTANT: This will RESET YOUR GAME and you will LOSE YOUR SAVE AND MODS so back them up first**
- Until [my PR to add the "mod loader: none" option](https://github.com/Lauriethefish/QuestPatcher/pull/188) is merged, you will need to download [my fork of Quest Patcher](https://github.com/jakzo/QuestPatcher/releases) instead of the official one
- Go to the "Tools & Options" tab -> click the "Change App" button -> select `com.StressLevelZero.BONELAB`
- Go to the "Patching" tab -> **select `None` for the mod loader** (you should have already installed Melon Loader)
- Click "Patching Options" -> **scroll to "Hand Tracking Type" -> select `V2`**
Expand All @@ -22,24 +21,31 @@ Adds support for hand tracking.
## Usage

- Click on UIs by pinching
- Open the in-game menu by doing the standard Quest hand tracking menu gesture (left hand open, palm towards your face then touch the tips of your index finger and thumb)
- Walk by moving your hands up and down alternately in a running motion
- Toggle the in-game menu by doing the standard Quest hand tracking menu gesture (left hand open, palm towards your face then touch the tips of your index finger and thumb) then normal pinch to select an item
- Walk by moving your head away from the center of your play space
- Use the Oculus recenter gesture if needed (right hand open, palm towards your face then touch the tips of your index finger and thumb for a few seconds)
- Run by running on the spot in real life (the mod detects this by your head bobbing up and down)
- To stop running, step back to the center of your play space
- Alternatively you can run by moving your hands up and down alternately in a running motion
- Grab by curling middle, ring and pinky fingers into a fist (same effect as pressing trigger and grip on controller)
- Force pull objects by forming a fist with these three fingers then flicking your wrist (in any direction)
- Held items like guns are triggered by curling the index finger

## Tips

Hand tracking's accuracy is _very_ limited. Don't expect everything to work perfectly. I did put in effort to make it less frustrating by taking into account the tracking confidence and assuming certain things when the hands are out of view, but it won't always do what you want if the headset can't see your hands/fingers. Here are some tips to help make your experience smoother:
Hand tracking's accuracy has several limitations and can be a frustrating experience if you don't know about these and how to work around them. I did put in effort to take into account the tracking confidence and assuming certain things when the hands are out of view, but it won't always do what you want if the headset can't see your hands/fingers. Here are some tips to help make your experience smoother:

- When running, make sure the headset has a clear view of your hands by either:
- Holding your hands a bit higher and further forwards than normal (so they are not too close to the headset)
- Looking downwards
- When running, instead of pointing your hands forwards in a fist, open them and have your palms facing you
- Don't hold guns with two hands while aiming (headset gets confused and will make your in-game hands do weird things because your IRL hands are overlapping)
- Don't hold guns too close to your face while aiming (hand will lose tracking if too close to headset)
- While throwing things, climbing or any other action, try not to bring your hands too close to the headset or out of your eye sight
- **When running using your hands:**
- Make sure the headset has a clear view of your hands by either:
- Holding your hands a bit higher and further forwards than normal (so they are not too close to the headset)
- Looking downwards
- Instead of pointing your hands forwards in a fist, open them and have your palms facing you
- **When using guns:**
- Don't hold them with two hands while aiming (headset gets confused and will make your in-game hands do weird things because your IRL hands are overlapping)
- Don't hold them too close to your face while aiming (hand will lose tracking if too close to headset)
- **While throwing things, climbing or any other action:**
- Try not to bring your hands too close to the headset or out of your eye sight

## Fun facts

SLZ has already added a bunch of hand tracking code. By just adding the hand tracking permission to the APK, `MarrowGame.xr.HandLeft/Right` will start tracking the hand position! Finger poses are not updated because the code is missing from the `OculusHandActionMap`, however they do actually have finger pose and gesture code in the generic `HandActionMap.ProcessesHand` method. I tried to lean on their work and get it working but there were too many gaps and it ended being easier to reimplement everything myself.
SLZ has already added a bunch of hand tracking code. By just adding the hand tracking permission to the APK, `MarrowGame.xr.HandLeft/Right` will start tracking the hand position! Finger poses are not updated because the code is missing from the `OculusHandActionMap`, however they do actually have finger pose and gesture code in the generic `HandActionMap.ProcessesHand` method. I tried to lean on their work and get it working but there were too many gaps and it ended up being easier to reimplement everything myself.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
set -eux
cd "$(dirname $0)"
cd "$(dirname $0)/.."

dotnet build ./Project.csproj

Expand All @@ -19,4 +19,4 @@ adb shell am start -n com.StressLevelZero.BONELAB/com.unity3d.player.UnityPlayer
# adb pull /sdcard/Android/data/com.StressLevelZero.BONELAB/files/melonloader/etc/managed/Assembly-CSharp.dll
# adb pull /sdcard/Android/data/com.StressLevelZero.BONELAB/files/melonloader/etc/managed/SLZ.Marrow.dll

# adb logcat -v time MelonLoader:D CRASH:D Mono:W mono:D mono-rt:D Zygote:D A64_HOOK:V DEBUG:D Binder:D AndroidRuntime:D "*:S"
# adb logcat -v time MelonLoader:D "*:S"
34 changes: 34 additions & 0 deletions projects/Bonelab/HandTracking/scripts/set-pref.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/bin/bash
set -eu
cd "$(dirname $0)/.."

NAME="$1"
VALUE="$2"

VALUE_ESCAPED=$(printf '%s\n' "$VALUE" | sed -e 's/[&/\]/\\&/g')

FILENAME="MelonPreferences.cfg"
PREFS_PATH="/sdcard/Android/data/com.StressLevelZero.BONELAB/files/UserData/$FILENAME"
CATEGORY="HandTracking"
LINE_TO_INSERT="$NAME = $VALUE_ESCAPED"

adb pull "$PREFS_PATH"

if grep -q "^$NAME = .*\$" "$FILENAME"; then
sed -i '' "s/^$NAME = .*\$/$LINE_TO_INSERT/" "$FILENAME"
elif grep -q "^\[$CATEGORY\]$" "$FILENAME"; then
sed -i '' -e "/^\[$CATEGORY\]$/a\\
$LINE_TO_INSERT" "$FILENAME"
else
echo "Error: No category with name \"$CATEGORY\" found in $FILENAME"
exit 1
fi

adb shell run-as com.StressLevelZero.BONELAB sh -c "
cat > '$PREFS_PATH' <<'EOF'
$(cat "$FILENAME")
EOF
"
rm "$FILENAME"

./scripts/build-and-start.sh
22 changes: 22 additions & 0 deletions projects/Bonelab/HandTracking/scripts/vid.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
- Tracking quality
- Sleeves block tracking/make hand movement choppy
- Hand mapping
- Different bones in Oculus hand tracking API vs Bonelab
- Movement
- Hand running
- Nimsony head bobbing
- Jumping
- Force pull
- Vehicles
- UI
- Inventory
- Guns
- Aiming is hard to be accurate with, but not unusable
- Two handed gun aiming is hard
- Hard for cameras to see trigger finger when aiming
- Added weapon rotation offset to help the headset see the trigger finger better while aiming down sights
- Struggling with auto sight
- Minor tweaks
- Noose auto-attach
- Future improvements
- Wife tried grabbing avatar selector with fingers (make different types of grips allow different hand shapes to grip?)
130 changes: 130 additions & 0 deletions projects/Bonelab/HandTracking/src/AutoSight.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
using System;
using UnityEngine;
using SLZ.Bonelab;
using SLZ.Interaction;
using Sst.Utilities;
using SLZ.Rig;
using HarmonyLib;
using SLZ.VRMK;

namespace Sst.HandTracking;

public class AutoSight {
private const float ROTATION_SIMILARITY_ACTIVATION_THRESHOLD = 0.95f;
private const float ROTATION_SIMILARITY_DEACTIVATION_THRESHOLD = 0.9f;
private const float ROTATION_FACTOR = 0.1f;
// Sights have a slightly different offset depending on the gun but finding
// the specific value per gun is a lot of manual effort and won't work for
// modded guns whereas hardcoding it works well enough
private static Vector3 SIGHT_OFFSET = new Vector3(0f, 0.03f, 0f);

public HandTracker Tracker;
public HandTracker OtherTracker;
public InteractableHost Weapon;
public bool IsActive;
public Quaternion TargetHandRotation;

public AutoSight(HandTracker tracker, HandTracker otherTracker) {
Tracker = tracker;
OtherTracker = otherTracker;
}

public void UpdateHand() {
// TODO: More efficient way to get held gun and cache all this stuff?
var hand = Tracker.GetPhysicalHand();
if (hand == null)
return;

var host = hand?.AttachedReceiver?.TryCast<TargetGrip>()
?.Host?.TryCast<InteractableHost>();
var gun = host?.GetComponent<Gun>();
if (gun?.firePointTransform == null || host?.Rb == null)
return;

var eye = LevelHooks.RigManager.controllerRig.TryCast<OpenControllerRig>()
?.cameras?[0];
if (eye == null)
return;

var sightRot = gun.firePointTransform.rotation;
var sightPos = gun.firePointTransform.position + sightRot * SIGHT_OFFSET;

var targetRotation =
Quaternion.LookRotation(sightPos - eye.transform.position);
var rotationSimilarity = Quaternion.Dot(targetRotation, sightRot);

if (IsActive) {
if (rotationSimilarity < ROTATION_SIMILARITY_DEACTIVATION_THRESHOLD) {
Dbg.Log("Auto-sight deactivated");
IsActive = false;
return;
}
} else if (rotationSimilarity >= ROTATION_SIMILARITY_ACTIVATION_THRESHOLD) {
Dbg.Log("Auto-sight activated");
IsActive = true;
} else {
return;
}

var handToSightPos = sightPos - host.Rb.transform.position -
hand.joint.connectedAnchor + hand.joint.anchor;
var handToSightRot = sightRot *
Quaternion.Inverse(host.Rb.transform.rotation) *
Quaternion.Inverse(hand.joint.targetRotation);
var sightToHandRot =
sightRot * host.Rb.transform.rotation * hand.joint.targetRotation;

TargetHandRotation = targetRotation * sightToHandRot;
}

// TODO: Get this working
// [HarmonyPatch(
// typeof(GameWorldSkeletonRig),
// nameof(GameWorldSkeletonRig.OnFixedUpdate)
// )]
// internal static class GameWorldSkeletonRig_OnFixedUpdate {
// // [HarmonyPrefix]
// // private static bool Prefix(GameWorldSkeletonRig __instance) {
// [HarmonyPostfix]
// private static void Postfix(GameWorldSkeletonRig __instance) {
// __instance.m_handLf.localPosition =
// new Vector3(Mathf.Sin(Time.time) * 0.25f, 1.5f, 0.2f);
// __instance.m_handLf.localRotation = Quaternion.identity;
// // __instance.m_elbowLf.localPosition = new Vector3(0.2f, 1.2f, 0.2f);
// // __instance.m_elbowLf.localRotation = Quaternion.identity;
// // __instance.bodyPose = __instance._basePose;
// // return false;
// }
// }

// [HarmonyPatch(
// typeof(GameWorldSkeletonRig), nameof(GameWorldSkeletonRig.OnUpdate)
// )]
// internal static class GameWorldSkeletonRig_OnUpdate {
// [HarmonyPostfix]
// private static void Postfix(GameWorldSkeletonRig __instance) {
// __instance.m_handLf.localPosition =
// new Vector3(Mathf.Sin(Time.time) * 0.25f, 1.5f, 0.2f);
// __instance.m_handLf.localRotation = Quaternion.identity;
// }
// }

// [HarmonyPatch(typeof(PhysHand), nameof(PhysHand.UpdateArmTargets))]
// internal static class PhysHand_UpdateArmTargets {
// [HarmonyPrefix]
// private static bool Prefix(PhysHand __instance) {
// var tracker =
// Mod.Instance.GetTrackerOfHand(__instance.hand.handedness);
// tracker.AutoSight.UpdateHand();
// if (!tracker.AutoSight.IsActive)
// return;

// tracker.LogSpam("maxTor", maxTor);
// var delta = Quaternion.Inverse(targetRotation) *
// tracker.AutoSight.TargetHandRotation;

// targetRotation = tracker.AutoSight.TargetHandRotation *
// Quaternion.Slerp(Quaternion.identity, delta, ROTATION_FACTOR);
// }
// }
}
Loading

0 comments on commit d6c12dc

Please sign in to comment.