Landmark detection training

[csoiram]

I was looking at the code and couldn't find any mention to CelebA dataset used by the paper for landmark detection training. Is this being trained?

[Seanlinx]

No, landmark localization is not implemented in the code.

[dtivger]

@Seanlinx the bounding box provided by the celebA dataset is seemingly larger than the wider face's , which include more context information around the face . How can I change the celebA's data to fit the wider face's? Another more a question , how can I save the training progress's information in log file and save the log file?

[Seanlinx]

@dtivger CelebA faces only contribute to landmark localization task so the bounding box is useless. Use logging.basicConfig() to specify the log file name in example/train.py

[so-as]

@Seanlinx on the basis of your work, how can I change the code to add landmarks localization?

[Seanlinx]

@so-as That would be lots of things to do.
Firstly you should crop, resize and save the faces from celebA dataset, and generate imglist file which contains these faces. The format should be [path to image] [cls_label] [landmark_label]
An example would be:
12/landmark/0 -2 0.42 0.70 0.53 0.52 0.78 0.38 0.35 0.61 0.69 0.65
Where '-2' indicates it's a landmark face, you should change its cls_label to '-1' or the softmax layer won't work. Then modify load_annotation() and append_flipped_images() in imdb.py to deal with landmark faces.
In minibatch.py, two additional labels are needed.
'raw_label': same as cls_label except landmark faces are assigned '-2'
'landmark_target': normalized landmark coordinates
Finally in symbol.py, add a fc layer after prelu4 with 10 hidden nodes to predict landmark, and add a sibling branch to do landmark regression.
cls_label is feed to softmaxoutput layer, landmark_target is for landmark regression layer, and raw_label is for negativemining layer so that we can only propagated gradients for landmark faces to landmark regression layer.

[so-as]

@Seanlinx thank you. I'll try

[so-as]

@Seanlinx I know the label '-2' indicates it's a landmark face. But I did not understand you should change its cls_label to '-1' or the softmax layer won't work. the landmark face cls_label is '2', why change it to '-1'. '-1' is the part face label

[Seanlinx]

"-1" is the default value of label that would be ignored in SoftmaxOutput layer if use_ignore=True. And landmark face should be ignored in classification task. @so-as

[so-as]

@Seanlinx so, Why not just assign the class label '-1' to the landmark face when create the landmark face dataset ? Why first assign '-2' , then change it to '-1' ?

[Seanlinx]

@so-as Then how could you distinguish them from part faces

[so-as]

@Seanlinx you means first read the landmark dataset annotations, then change the cls_label to '-1'?

[so-as]

@Seanlinx I didn't understand when to change the landmark face's label

[Seanlinx]

@so-as That's not the point. I mean you should provide two kinds of label, one is 'raw_label' that contains '-2', the other is 'cls_label' which only contains '0', '1' and '-1'

[so-as]

@ @Seanlinx I should change landmark face class label to '-1', so it can have no contribution to the classification weight updating in the back propagation. So I think I need to focus on the landmark face class label when to change to '-1'.

[AlphaQi]

@so-as Hello , Did you implemented landmark localization training ?

[so-as]

@AlphaQi no, I am sorry

[AlphaQi]

@so-as Ok, it doesn't matter. I'll try

[Cv9527]

@AlphaQi Did you implemented landmark localization training ?

[Cv9527]

That would be lots of things to do.
Firstly you should crop, resize and save the faces from celebA dataset, and generate imglist file which contains these faces. The format should be [path to image] [cls_label] [landmark_label]
An example would be:
12/landmark/0 -2 0.42 0.70 0.53 0.52 0.78 0.38 0.35 0.61 0.69 0.65
Where '-2' indicates it's a landmark face, you should change its cls_label to '-1' or the softmax layer won't work. Then modify load_annotation() and append_flipped_images() in imdb.py to deal with landmark faces.
In minibatch.py, two additional labels are needed.
'raw_label': same as cls_label except landmark faces are assigned '-2'
'landmark_target': normalized landmark coordinates
Finally in symbol.py, add a fc layer after prelu4 with 10 hidden nodes to predict landmark, and add a sibling branch to do landmark regression.
cls_label is feed to softmaxoutput layer, landmark_target is for landmark regression layer, and raw_label is for negativemining layer so that we can only propagated gradients for landmark faces to landmark regression layer.

@Seanlinx since label =1,0,-1,-2, we can distinguish cls, bbox, and landmark, 'raw_label' is unnecessary?

[Seanlinx]

@Cv9527 No. raw_label = 1, 0, -1, -2, which is used to distinguish cls, bbox and landmark in negativemining layer. Since softmax_output layer only accpet label value in [1, 0, -1](-1 is the default value of ignored label), you need to create 'cls_label' that replace -2 with -1 so that landmark faces won't contribute to classification task.

[Cv9527]

@Seanlinx ok，I get the points. so, raw_label is needed to transfer to negativemining layer.

modify symbol.py as picture, is right?

[Seanlinx]

@Cv9527 Right, but 'label' is not needed in negativemining layer since you've got 'raw_label'.

[Cv9527]

@Seanlinx thanks, label is replaced by raw_label. however, since pts_pred and pts_target are transfer to negativemining layer, I have no idea how to modify infer_shape(self, in_shape) in negativemining.py, could you give some advices.

@mx.operator.register("negativemining")
class NegativeMiningProp(mx.operator.CustomOpProp):
def init(self):
super(NegativeMiningProp, self).init(need_top_grad=False)

def list_arguments(self):
    return ['cls_prob', 'bbox_pred', 'pts_pred', 'label', 'bbox_target', 'pts_target']

def list_outputs(self):
    return ['cls_out', 'bbox_out','pts_out', 'cls_keep', 'bbox_keep','pts_keep']

def infer_shape(self, in_shape):
    keep_shape = (in_shape[0][0], )
    return in_shape, [in_shape[0], in_shape[1], keep_shape, keep_shape]

def create_operator(self, ctx, shapes, dtypes):
    return NegativeMiningOperator()

[Seanlinx]

@Cv9527
return in_shape, [in_shape[0], in_shape[1], keep_shape, keep_shape, keep_shape]

[Cv9527]

@Seanlinx infer_shape encounter errors, parameters are not right. how to fix it?

[Seanlinx]

@Cv9527
Sorry, it should be
return in_shape, [in_shape[0], in_shape[1], in_shape[2], keep_shape, keep_shape, keep_shape]

[Cv9527]

@Seanlinx thank you. It seems need to add a function to calculate landmarks loss in metric.py, right?

[Seanlinx]

@Cv9527 Yes, and it should be almost the same as the one for bbox loss

[Cv9527]

@Seanlinx I got a problem when train Rnet with landmark. the loss is nan. how can I solve it?

[kuaikuaikim]

@Seanlinx How we normalized landmark coordinates.
x_i : the i landmark x coordinate. y_i: the i landmark y coordinate
x_tb: the left top bounding box x coordinate. y_tb: the left top bounding box y coordinate.
x_iregression = (x_i-x_tb)/box_width and y_iregression(y_i-y_tb)/box_height.
This is my conception. Is that right?

[Seanlinx]

@quickj That's right if the 'bounding box' you mentioned refers to the cropped image rather than the bbox annotation provided by the data set

[kuaikuaikim]

@Seanlinx Thanks. Normalized by the box_size not the image_size accurately. I have changed the formula .

[pribadihcr]

@quickj @Seanlinx
How about if I want to rotate the detected face, retrain the landmark since the mtcnn sometimes not good for rotation image (e.g: 90 degree)

[AITTSMD]

hi, I trained MTCNN using tensorflow.The result is okay.Please refer https://github.com/AITTSMD/MTCNN-Tensorflow

[HaoLiuHust]

@Cv9527 have you finish the work?