Fix triangle-triangle collision detection for stability and deep penetration handling

[Copilot]

Overview

This PR addresses two critical issues in the checkTriangleTriangleOverlap method that affect the stability and correctness of triangle-triangle collision detection in mesh simulations.

Problem 1: Inconsistent Separation Direction

When two triangles contact face-to-face (especially vertically), the Separating Axis Theorem (SAT) algorithm could oscillate between choosing face normals or edge-edge cross products as the separation axis. This led to:

• Inconsistent contact normals between frames
• Physics instability and jitter
• Sudden changes between small and large penetration depths

Example scenario: Two triangular mesh faces approaching each other vertically would sometimes report separation along the face normal (small overlap) and other times along an edge-edge axis (potentially large overlap), causing the solver to flip back and forth.

Problem 2: Missing Deep Penetration Case

The algorithm did not handle the case where all 3 vertices of one triangle are on the opposite side of the other triangle's plane. This represents a deeply penetrated configuration where the triangle is considered completely inside the other mesh, but the standard SAT approach would treat it as a regular contact with potentially incorrect contact information.

Solution

Stability Fix (Issue 1)

Introduced a 5% bias factor that gives face normals priority during axis selection when their overlap is comparable to edge-edge overlaps:

const T2 faceNormalBias = T2(1.05);
T2 biasedOverlap = overlap * faceNormalBias;
if (biasedOverlap < minOverlap) {
    minOverlap = overlap;  // Store actual overlap, not biased
    // ...
}

Key design choice: The bias is only applied during axis selection; the actual overlap depth stored remains unbiased to maintain physics accuracy.

Deep Penetration Handling (Issue 2)

Added early detection and specialized handling before running the full SAT algorithm:

1. Detection: Check if all vertices of one triangle have negative signed distance to the other triangle's plane

2. Contact Computation:

   • Use the face normal as the separation direction (consistent and stable)
   • Project incident triangle vertices onto the reference triangle's plane
   • Use the existing snap_to_face() utility to check which projections fall inside the reference triangle
   • Average the valid projections that are "head on" facing the reference triangle
   • Offset the contact point by half the penetration depth (following the convention in tri_plane_penetration)
   • Fall back to averaging all vertices if none project inside

3. Early Return: Avoids unnecessary SAT calculations, improving performance in deeply penetrated scenarios

Changes

Modified File: src/kernel/DEMCollisionKernels.cuh

• Function: checkTriangleTriangleOverlap()
• Lines added: 96
• Lines modified: 4

Compatibility

• ✅ No API changes - function signature remains identical
• ✅ Backward compatible with existing code
• ✅ Only internal collision detection behavior modified
• ✅ Uses existing helper functions (snap_to_face)

Expected Benefits

1. Improved Stability: More consistent contact normals and separation directions in face-to-face contacts
2. Better Physics: Correct handling of deeply interpenetrated triangular meshes
3. Reduced Jitter: More stable forces when meshes are in close proximity
4. Performance: Early return optimization in deep penetration cases
5. Robustness: Graceful fallback handling for edge cases

Testing Recommendations

• Run DEMdemo_MeshCollide to verify basic mesh-mesh collision behavior
• Test with various mesh orientations (vertical, horizontal, angled)
• Test with different penetration depths (shallow to deep)
• Verify long-term stability in production simulations with meshes

Related

This addresses stability and correctness issues observed in mesh-mesh collision detection, particularly in scenarios with complex mesh interactions or significant mesh overlap.

Original prompt

I need you to take a deeper look into the checkTriangleTriangleOverlap method and fix potential problems. 1. Right now, it seems that the separation direction is not that consistent or stable. If two triangles get in contact vertically to each other, then the separation direction could be along the direction they approach, or sometimes it can be along the direction parallel to one triangle (with a big penetration depth, as in this case they could be considered suddenly deeply penetrated). This makes the physics not very stable. 2. I need you to implement another case, where all 3 points of a triangle in "on the other side" of the plane spanned by the other triangle (in this case the triangle is considered completed inside the other mesh, thus still in contact). The contact point and penetration in this case can be estimated by the average of the nodes whose projection can be inside the other triangle (aka "head on" facing the other triangle when projected onto it).

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.