Add face tracking to your Needle Engine projects with ease.
This project contains the sourcecode for the facefilter package as well as an example Unity project (see quickstart below for how to get started without an editor)
- Blendshape mesh face tracking
- Texture face tracking with google/mediapipe canonical or procreate texture layouts (Use the
FaceMeshTextureclass) - Video face tracking: Play a video as a face texture (Use the
FaceMeshVideoclass) - Custom shader face meshes: Use custom materials on your face mesh (Use the
FaceMeshCustomShaderclass) - Tracking for multiple faces at once (with smoothing)
- Can be used with Unity to create filters, animations, materials...
- Example with Blendshapes on Stackblitz
- Example with Sunglasses on Stackblitz
- Example with Texture Mesh on Stackblitz
- Example with ShaderToy on Stackblitz
- Example without a bundler on Stackblitz
- Example with HTML only 2D filter on Stackblitz (or github)
- Example with HTML only 3D filter on Stackblitz (or github)
- Demo Video
Run npm i @needle-tools/facefilter in your web project
Then see the code below or full examples:
- Clone this repository
- Open the Unity project at
Unity FaceFilter Example - Open the Example scene in Unity and click play
Note: The Unity project uses Needle Engine 4.4 alpha.
See full examples in /examples/html/
face-filter |
URL to either a image or model file. Supported formats: glTF, GLB, FBX, OBJ or 2D jpeg, jpg, png, webp |
face-filter-mask |
(optional, 2D only) URL to image file that will be used to mask out the face filter texture |
face-filter-layout |
(optional, 2D only) Either procreate, mediapipe or canonical. Default mediapipe |
face-filter-scale |
(optional, 3D only) Apply scale to the 3D face filter model (e.g. face-filter-scale=".5") |
face-filter-offset |
(optional, 3D only) Offset the 3D face filter model (e.g. face-filter-offset="0.0, 0.1, 0.1") |
face-filter-max-faces |
(optional) How many faces should be tracked automatically. Default: 1 |
face-filter-show-video |
(optional) Should the camera videofeed be rendered in the background? Default: true. Can be set to 0 to hide the videofeed in the background. |
face-filter-video-selector |
(optional) HTML selector for a HTMLVideoElement. Useful if you want to render provide your own video element on the website elsewhere. If none is provided a hidden video element will be created automatically. |
<!DOCTYPE html>
<html>
<head>
<script type="importmap">
{
"imports": {
"three": "https://cdn.jsdelivr.net/npm/@needle-tools/engine@4.4.0-alpha.5/dist/three.min.js",
"@needle-tools/engine": "https://cdn.jsdelivr.net/npm/@needle-tools/engine@4.4.0-alpha.5/dist/needle-engine.min.js",
"@needle-tools/facefilter": "https://cdn.jsdelivr.net/npm/@needle-tools/facefilter/dist/facefilter.min.js"
}
}
</script>
<script type="module" src="https://cdn.jsdelivr.net/npm/@needle-tools/facefilter/dist/facefilter.min.js"></script>
</head>
<body style="margin:0; padding:0;">
<needle-engine
background-color="#ffffdd"
face-filter="https://cdn.needle.tools/static/facefilter/facemask-template-procreate.webp"
face-filter-mask="https://cdn.needle.tools/static/facefilter/facemask-occlusion-procreate.webp"
face-filter-layout="procreate"
>
</needle-engine>
</body>
</html>Open 2D Example – Open 3D Example
import { onStart } from '@needle-tools/engine';
import { FaceMeshTexture, NeedleFaceFilterTrackingManager } from '@needle-tools/facefilter';
onStart(context => {
const scene = context.scene;
// Create a face filter tracking manager and add it to the scene
const filtermanager = new NeedleFaceFilterTrackingManager();
filtermanager.createMenuButton = true;
scene.addComponent(filtermanager);
// Creating a filter
const filter = new FaceMeshTexture({
layout: 'procreate', // we support both the google/mediapipe canonical layout and procreate/arkit layouts
texture: {
url: './assets/crocodile.webp', // provide a URL to the texture
// texture: <your texture> // alternatively you can assign an existing texture directly
},
});
// Activate one of your filters
filtermanager.activateFilter(filter);
});import { onStart } from '@needle-tools/engine';
import { FaceMeshTexture, NeedleFaceFilterTrackingManager } from '@needle-tools/facefilter';
onStart(context => {
const scene = context.scene;
// Create a face filter tracking manager and add it to the scene
const filtermanager = new NeedleFaceFilterTrackingManager();
filtermanager.createMenuButton = false;
scene.addComponent(filtermanager);
// Creating a filter using a GLB/glTF URL model that has blendshapes
const filter = await FaceFilterRoot.create('https://cloud.needle.tools/-/assets/Z23hmXBZWllze-ZWllze/file', {
scale: 0.5,
offset: { x: 0, y: 0.01, z: 0 },
});
if (filter) filtermanager.activateFilter(filter);
});