Keypoints class/ops

[bhack]

Describe the feature and the current behavior/state.
Add keypoint class utils
Relevant information

• Are you willing to contribute it (yes/no):
  no
• Are you willing to maintain it going forward? (yes/no):
  probably
• Is there a relevant academic paper? (if so, where):
  Yes but it is a too popular topic to point any specific paper.
• Is there already an implementation in another framework? (if so, where):
  Yes in official Facebook Pytorch detectron2
  https://github.com/facebookresearch/detectron2/blob/master/detectron2/structures/keypoints.py
• Was it part of tf.contrib? (if so, where):
  No we had some sparse keypoints utils/ops in model
  https://github.com/tensorflow/models/blob/master/research/object_detection/predictors/heads/keypoint_head.py
  https://github.com/tensorflow/models/blob/master/research/object_detection/core/keypoint_ops.py
  Which API type would this fall under (layer, metric, optimizer, etc.)
  keypoints
  Who will benefit with this feature?
  Many applications like human pose, hande pose, face landmarks, 3d objects cuboid etc..
  Any other info.
  In Mediapipe there are many pre/postprocessing operations over keypoints that rely on Opencv keypoints with the Tensorflow lite inference. Probably some of these operations could put in graph if we can handle keypoints in Tensorflow directly.

[bhack]

Just an additional reference ticket: tensorflow/models#2366

[bhack]

/cc @jiuqiant

[bhack]

We have some Google heatmap/keypoints operations also in https://github.com/google-research/google-research/blob/master/video_structure/ops.py
/cc @mjlm

[bhack]

We have another new (duplicated?) Keypoint/Heatmap utils in the model garden at https://github.com/tensorflow/models/blob/master/research/object_detection/utils/target_assigner_utils.py#L44

@seanpmorgan @karmel Can you ask internally if they can contribute these general utils and components in Addons instead of embedding inside the model garden repositories?

I think this is in the spirit of the approved RFC so that the users could build their own projects without reinveting the whell or be on hold about Addons potential contributions cause we need to worry about duplicates and continously monitoring the garden.
I think that sharing more common/recurrent/standard components Is a positive goal for the while ecosystem and It will speed-up the library adoption by third party developers.

We could understand If the original internal author doesn't want to maintain that code and we are ok with this but we could evalauted to maintain the code ourself if he open a PR/ISSUW on Addons we have a specific flag in the Template if he don't want to maintain the contribution.

[mjlm]

Hi, I'm the author of https://github.com/google-research/google-research/blob/master/video_structure/ops.py and was CCd on this thread. I'm generally happy to move some of this to a shared component garden if it's not a huge amount of work. However, I know nothing about TensorFlow Addons and would need a bit more guidance on what I should do.

…

On Thu, May 14, 2020 at 11:21 AM bhack ***@***.***> wrote: We have other new (duplicated?) Keypoint/Heatmap utils in the model garden at https://github.com/tensorflow/models/blob/master/research/object_detection/utils/target_assigner_utils.py#L44 @seanpmorgan <https://github.com/seanpmorgan> @karmel <https://github.com/karmel> Can you ask internally if they can contribute these general utils and components in Addons instead of embedding inside the model garden repositories? I think this is in the spirit of the approved RFC <https://github.com/tensorflow/community/blame/master/rfcs/20190802-model-garden-redesign.md#L60-L61> so that the users could build their own projects without reinveting the whell or be on hold about Addons potential contributions cause we need to worry about duplicates and continously monitoring the garden. I think that sharing more common/recurrent/standard components Is a positive goal for the while ecosystem and It will speed-up the library adoption by third party developers. We could understand If the original internal author doesn't want to maintain that code and we are ok with this but we could evalauted to maintain the code ourself if he open a PR/ISSUW on Addons we have a specific flag in the Template if he don't want to maintain the contribution. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1366 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJKYVPBE54ZDT2YP5H5TFTRROZ2FANCNFSM4LSAOXWQ> .

[bhack]

@mjlm Thanks. It would be really great and I am interested if we could collect some generally computer vision/cv models general utils/ops here.

We have seen that there was some internal plan about a computer vision modelling library modeling library for computer vision but we don't know details about this plan.
See tensorflow/models#8278 (comment)

I let @seanpmorgan and @karmel to coordinate more on this.

I am available to support/co-maintain some of these components if we could have something coherent that will not be invalidated with a more or less upstream duplicate that is rolling around the corner.

[bhack]

@tanzhenyu Do you think that you want features like these ones migrated in the brand new keras-cv repo?

[tanzhenyu]

This is generally in our roadmap. What's the timeline for this? keras-cv needs to go through public RFC for APIs that we propose

[bhack]

The general goal here is just to not have very short-lifetime efforts that will be superseded soon by another official repo.
@mjlm Seems interested to port some of his shared components form google-research but as you see I am not the only one to be a little bit confused:

I'm generally happy to move some of this to a
shared component garden if it's not a huge amount of work.

I think this was the general direction cause this kind of utils was embedded "as a service" of the specific model and not designed as reusable components.

So if we agree that keras-cv is the direction to accumulate this is kind of reusable components and deduplicating what it is already in the model garden Vision I think we could go in that direction.

So at this point I will ask you what's the timeline for this in keras-cv? 😄

[tanzhenyu]

The general goal here is just to not have very short-lifetime efforts that will be superseded soon by another official repo.
@mjlm Seems interested to port some of his shared components form google-research but as you see I am not the only one to be a little bit confused:

I'm generally happy to move some of this to a
shared component garden if it's not a huge amount of work.

I think this was the general direction cause this kind of utils was embedded "as a service" of the specific model and not designed as reusable components.

So if we agree that keras-cv is the direction to accumulate this is kind of reusable components and deduplicating what it is already in the model garden Vision I think we could go in that direction.

So at this point I will ask you what's the timeline for this in keras-cv? 😄

That's a fair question :-)

We target to provide keras_cv as a repo that provides modular and reusable API. We have made initial plans for object detection models and instance segmentation models. For keypoint estimation the API is still TBD.

[seanpmorgan]

Thanks for the update... please keep us posted with information/timelines as they become available. In either case we will not be providing this functionality in Addons as it would be duplicated.