Head horizontal rotation.

[MariasStory]

Is your feature request related to a problem? Please describe.
In your examples the face detection is broken when the head is rotated horizontally.

Describe the solution you'd like
The in plane face rotation should be properly detected.

Describe alternatives you've considered
There is no good alternative. Some random samples are:
https://github.com/auduno/clmtrackr
https://github.com/YadiraF/PRNet
https://github.com/cmusatyalab/openface
https://github.com/MarekKowalski/DeepAlignmentNetwork

Additional context
The rotation detection should not hinter accuracy or performance much. It also could be a separate pre-processing step.

[xavierjs]

There are 3 rotation axis, X,Y and Z, so what do you mean by rotated horizontally ?
Do you have try to change the neural net ? There is one model which is specialized for wide angles (read the readme.md). The link to the boilerplate is here: https://jeeliz.com/demos/faceFilter/demos/threejs/cubeNNCwideAngles/

The rotation is an output from the neural net, then it is stabilized. There is no "rotation detection step".

[MariasStory]

Hi @xavierjs,
What I mean is that the when I look into the camera and turn my head ~90 degrees, the detection is not working anymore.

BZW: Do you have something similar to: https://github.com/justadudewhohacks/face-api.js ?

[xavierjs]

Hi,

This is normal because the neural network is trained for clamped angles values.
For the default neural net (NNC.json), these values are:

40 degrees for rotX
60 for rotY (Y pointing up, like in OpenGL)
15 for rotZ

on the wideAngles NN, these values are:
45 degrees for rotX
64 for rotY (Y pointing up, like in OpenGL)
35 for rotZ

So 90° is too wide, it won't work. I regularly push the limits of these neural networks.

I don't have anything similar to https://github.com/justadudewhohacks/face-api.js but I don't really understand why you are submitting all the links to other face detection and tracking libraries. Nobody forces you to use FaceFilter if you think another one is better ;) .

[MariasStory]

Hi @xavierjs, please, don't get me wrong. I do like your solution and find the examples impressive. I also appreciate that you do open source, good work. I also find that you are quite skilled 👍.
I just want to make sure that I understand the solution as compared to another ones.
At the moment, I don't really know the status and maturity of the solution, although the examples are impresive :-). I still would like to try it and contribute, if possible. I hope that you'll take my comments as my best effort to help ;-).

[MariasStory]

Related to the question:
I was wandering if it is possible to improve the rotated face detection by using some kind of simple pre-processing, so that the neural network would see more or less aligned face image.

[xavierjs]

Hi @MariasStory MariasStory,

Yes, I would be possible to rotate the detection window according to the output of the neural network, that's a good idea, I will do some trials I add it to my pipeline (but I have quite a lot of work now so I cannot garantee I will do it soon).

I did not dig vey much the problem of face rotation along Z because I though it was not a real problem (when used with a desktop computer, or a laptop, or a static camera in a kiosk for example the head rotation aroung Z is always small). But this is true that with a mobile device we often have wide angles...

We are working on a landmark based library, which will be very similar to FaceFilter in term of interface and integration but which will detect landmarks on the face. We will release it on a separate github. It is not ready yet but the result are very encouraging. We will also release new and more optimized neuron network models for this library, so it is still improving :).

[xavierjs]

Hi,

I have added the feature to the lib.
You can do unlimited Z rotation now :).
You only have to set followZRot parameter to true when initializing FaceFilter API.
I have enabled it on the glasses VTO demo here:
https://github.com/jeeliz/jeelizFaceFilter/tree/master/demos/threejs/VTO

But for the detection the face still have to be aligned on the screen vertical axis (with a tolerance of +/- 30 degrees, depending on the neural net).
Then for the tracking the user can rotate its head around the depth axis as much as he wants.

[xavierjs]

@andrewdeutsch I think this improvement may interest you :)
You asked me, I remember about the limitation for the Z rotation. And also for a better face positionning.

[andrewdeutsch]

This is great. Thanks for the heads up!

On Fri, Nov 16, 2018 at 2:59 PM Bourry Xavier ***@***.***> wrote: @andrewdeutsch <https://github.com/andrewdeutsch> I think this improvement may interest you :) You asked me, I remember about the limitation for the Z rotation. And also for a better face positionning. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABHRai7K1SjvBcYKzmy2JmI1p2tZo0Xwks5uvxkOgaJpZM4W6Omv> .

-- 917.482.1004 https://www.somewonderstuff.com

[xavierjs]

I close the issue since the feature seems to work well. Don't mind to re-open it otherwise :)