From 8ac5c99e4713b516e1d6e3d6b99cd3d7cf3a825f Mon Sep 17 00:00:00 2001 From: Vasil Khalidov Date: Wed, 25 Jul 2018 13:04:15 +0200 Subject: [PATCH] added doc files for PoseTrack DensePose --- .../2018_PoseTrack_DensePose/data_format.md | 117 ++++++++++++++++++ .../2018_PoseTrack_DensePose/evaluation.md | 75 +++++++++++ challenge/2018_PoseTrack_DensePose/readme.md | 87 +++++++++++++ .../results_format.md | 42 +++++++ challenge/2018_PoseTrack_DensePose/upload.md | 67 ++++++++++ 5 files changed, 388 insertions(+) create mode 100644 challenge/2018_PoseTrack_DensePose/data_format.md create mode 100644 challenge/2018_PoseTrack_DensePose/evaluation.md create mode 100644 challenge/2018_PoseTrack_DensePose/readme.md create mode 100644 challenge/2018_PoseTrack_DensePose/results_format.md create mode 100644 challenge/2018_PoseTrack_DensePose/upload.md diff --git a/challenge/2018_PoseTrack_DensePose/data_format.md b/challenge/2018_PoseTrack_DensePose/data_format.md new file mode 100644 index 0000000..14756be --- /dev/null +++ b/challenge/2018_PoseTrack_DensePose/data_format.md @@ -0,0 +1,117 @@ +# Data Format + +The annotations are stored in [JSON](http://json.org/). Please note that +[COCO API](https://github.com/facebookresearch/DensePose/blob/master/detectron/datasets/densepose_cocoeval.py) +can be used to access and manipulate all annotations. + +The annotations file structure is outlined below: +``` +{ + "images" : [image], + "annotations" : [annotation], + "categories" : [category] +} + +image { + "id" : int, + "width" : int, + "height" : int, + "file_name" : str, + "has_no_densepose": True or False, + "is_labeled": True or False, + "vid_id": int, + "frame_id": int, + "nframes": int, + "ignore_regions_x": [[int]], + "ignore_regions_y": [[int]] +} + +annotation { + "area": float, + "bbox": [x, y, width, height], + "category_id": int, + "dp_I": [float], + "dp_U": [float], + "dp_V": [float], + "dp_masks": [dp_mask], + "dp_x": [float], + "dp_y": [float], + "id": int, + "image_id": int, + "iscrowd": 0 or 1, + "keypoints": [float], + "segmentation": RLE or [polygon], + "track_id": int +} + +category { + "id" : int, + "name" : str, + "supercategory" : str, + "keypoints": [str], + "skeleton": [edge] +} + +dp_mask { + "counts": str, + "size": [int, int] +} +``` + +Annotation data consists of 3 fields: `images`, `annotations` and `categories`. + +The `images` field contains a list of image data entries with image ID (`id`), +image size (`width`, `height`), image file name (`file_name`), video ID +(`vid_id`), image frame ID in the video (`frame_id`) and the total number +of frames in the video sequence (`nframes`). +The flag `is_labeled` indicates indicate whether the image was annotated, +`has_no_densepose` is set to `True` if the corresponding DensePose data should +not be considered in the evaluation. If an image is meant to be used in +the evaluation, it either has no entry for `has_no_densepose`, or has it set to +`False`. For more fine-grained control over evaluation, image regions +that should be excluded from the evaluation might be specified through +`ignore_regions_x` and `ignore_regions_y`. +Each region is given as two arrays of integer `x` and `y` coordinates, which +correspond to polygon vertex coordinates. + +The `annotations` field contains a list of annotations. +Each DensePose annotation contains a series of fields, including the category +id and segmentation mask of the person. The segmentation format depends on +whether the instance represents a single object (`iscrowd=0` in which case +polygons are used) or a collection of objects (`iscrowd=1` in which case RLE +is used). Note that a single object (`iscrowd=0`) may require multiple polygons, +for example if occluded. Crowd annotations (`iscrowd=1`) are used to label large +groups of objects (e.g. a crowd of people). In addition, an enclosing bounding +box and `track_id` are provided for each person (box coordinates are measured from +the top left image corner and are 0-indexed). + +DensePose annotations are stored in `dp_*` fields: + +*Annotated masks*: + +* `dp_masks`: RLE encoded dense masks. All part masks are of size 256x256. +They correspond to 14 semantically meaningful parts of the body: `Torso`, +`Right Hand`, `Left Hand`, `Left Foot`, `Right Foot`, `Upper Leg Right`, +`Upper Leg Left`, `Lower Leg Right`, `Lower Leg Left`, `Upper Arm Left`, +`Upper Arm Right`, `Lower Arm Left`, `Lower Arm Right`, `Head`; + +*Annotated points*: + +* `dp_x`, `dp_y`: spatial coordinates of collected points on the image. +The coordinates are scaled such that the bounding box size is 256x256; +* `dp_I`: The patch index that indicates which of the 24 surface patches the +point is on. Patches correspond to the body parts described above. Some +body parts are split into 2 patches: `1, 2 = Torso`, `3 = Right Hand`, +`4 = Left Hand`, `5 = Left Foot`, `6 = Right Foot`, `7, 9 = Upper Leg Right`, +`8, 10 = Upper Leg Left`, `11, 13 = Lower Leg Right`, `12, 14 = Lower Leg Left`, +`15, 17 = Upper Arm Left`, `16, 18 = Upper Arm Right`, `19, 21 = Lower Arm Left`, +`20, 22 = Lower Arm Right`, `23, 24 = Head`; +* `dp_U`, `dp_V`: Coordinates in the UV space. Each surface patch has a +separate 2D parameterization. + +The categories field of the annotation structure stores the mapping of category +id to category and supercategory names. It also has two fields: "keypoints", +which is a length `k` array of keypoint names, and "skeleton", which defines +connectivity via a list of keypoint edge pairs and is used for visualization. + + diff --git a/challenge/2018_PoseTrack_DensePose/evaluation.md b/challenge/2018_PoseTrack_DensePose/evaluation.md new file mode 100644 index 0000000..255a459 --- /dev/null +++ b/challenge/2018_PoseTrack_DensePose/evaluation.md @@ -0,0 +1,75 @@ +# PoseTrack DensePose Evaluation + +This page describes the DensePose evaluation metrics. The +evaluation code provided here can be used to obtain results on the publicly +available PoseTrack DensePose validation set. It computes multiple metrics +described below. To obtain results on the PoseTrack DensePose test set, +for which ground-truth annotations are hidden, generated results must be +uploaded to the +[evaluation server](https://competitions.codalab.org/competitions/19650). +The exact same evaluation code, +described below, is used to evaluate results on the test set. + +## Evaluation Overview + +The multi-person PoseTrack DensePose task involves simultaneous person detection, +segmentation and +estimation of correspondences between image pixels that belong to a human body +and a template 3D model through time. +DensePose evaluation mimics the evaluation metrics +used for [object detection](http://cocodataset.org/#detection-eval) and +[keypoint estimation](http://cocodataset.org/#keypoints-eval) in the COCO +challenge, namely average precision (AP) and average recall (AR) and their +variants. + +At the heart of these metrics is a similarity measure between ground truth +objects and predicted objects. In the case of object detection, +*Intersection over Union* (IoU) serves as this similarity measure (for both +boxes and segments). Thesholding the IoU defines matches between the ground +truth and predicted objects and allows computing precision-recall curves. +In the case of keypoint detection *Object Keypoint Similarity* (OKS) is used. + +To adopt AP/AR for dense correspondence, we define an analogous similarity +measure called *Geodesic Point Similarity* (GPS) which plays the same role +as IoU for object detection and OKS for keypoint estimation. + +## Geodesic Point Similarity + +The geodesic point similarity (GPS) is based on geodesic distances on the template mesh between the collected groundtruth points and estimated surface coordinates for the same image points as follows: + + + + +where is the geodesic distance between estimated +( ) and groundtruth +() +human body surface points and + +is a per-part normalization factor, defined as the mean geodesic distance between points on the part. + +## Metrics + +The following metrics are used to characterize the performance of a dense pose +estimation algorithm on COCO: + +*Average Precision* +``` +AP % AP averaged over GPS values 0.5 : 0.05 : 0.95 (primary challenge metric) +AP-50 % AP at GPS=0.5 (loose metric) +AP-75 % AP at GPS=0.75 (strict metric) +AP-m % AP for medium detections: 32² < area < 96² +AP-l % AP for large detections: area > 96² +``` + +## Evaluation Code + +Evaluation code is available on the +[DensePose](https://github.com/facebookresearch/DensePose/) github, +see [densepose_cocoeval.py](https://github.com/facebookresearch/DensePose/blob/master/detectron/datasets/densepose_cocoeval.py). +Before running the evaluation code, please prepare your results in the format +described on the [results](results_format.md) format page. +The geodesic distances are pre-computed on a subsampled version of the SMPL +model to allow faster evaluation. Geodesic distances are computed after +finding the closest vertices to the estimated UV values in the subsampled mesh. + diff --git a/challenge/2018_PoseTrack_DensePose/readme.md b/challenge/2018_PoseTrack_DensePose/readme.md new file mode 100644 index 0000000..472964d --- /dev/null +++ b/challenge/2018_PoseTrack_DensePose/readme.md @@ -0,0 +1,87 @@ +# ECCV 2018 PoseTrack DensePose Task + +![PoseTrack DensePose Splash Image](https://posetrack.net/workshops/eccv2018/assets/images/densepose-posetrack_examples.jpg) + +## Overview + +The PoseTrack DensePose Task requires dense estimation of human pose through time +in challenging, uncontrolled conditions. The task involves processing video frames +to simultaneously detect people, segment their bodies and map all image pixels +that belong to a human body to the 3D surface of the body. For full details on +this task please see the [evaluation](evaluation.md) page. + +This task is part of the +[PoseTrack Challenge Workshop](https://posetrack.net/workshops/eccv2018/) +at ECCV 2018. For further details about the workshop please +visit the workshop page. Please also see the related PoseTrack +[Articulated Human Pose Estimation and Tracking](https://posetrack.net/workshops/eccv2018/#challenges) +and +[3D Human Pose Estimation](https://posetrack.net/workshops/eccv2018/#challenges) +tasks. + +The PoseTrack DensePose train, validation, and test sets, containing more +than 5,000 images +and 27,000 person instances labeled with DensePose annotations are available +for [download](https://posetrack.net/users/download.php). +Annotations on +[train](https://www.dropbox.com/s/tpbaemzvlojo2iz/densepose_only_posetrack_train2017.json?dl=1) +and +[val](https://www.dropbox.com/s/43h43s0t3hkuogr/densepose_only_posetrack_val2017.json?dl=1) +with over 13,000 people are publicly available. + +Evaluation server for the 2018 task is +[open](https://competitions.codalab.org/competitions/19650). + +## Dates + +[]() | []() +---- | ----- +**August 18, 2018** | Submission deadline (23:59 PST) +September 2, 2018 | Challenge winners notified +September 8, 2018 | Winners present at ECCV 2018 Workshop + +## Organizers + +Riza Alp Güler (INRIA, CentraleSupélec) + +Natalia Neverova (Facebook AI Research) + +Iasonas Kokkinos (Facebook AI Research) + +## Task Guidelines + +Participants are recommended but not restricted to train +their algorithms on PoseTrack DensePose train and val sets. +The [download](https://posetrack.net/users/download.php) page has +links to the image data. When participating in this task, +please specify any and all external data used for training +in the "method description" when uploading results to the +evaluation server. **Listing external data used is mandatory.** +We emphasize that any form of **annotation or use of the test sets +for supervised or unsupervised training is strictly forbidden**. +A more thorough explanation of all these +details is available on the +[upload](upload.md) page, +please be sure to review it carefully prior to participating. +Results in the [correct format](results_format.md) must be +[uploaded](upload.md) to the +[evaluation server](https://competitions.codalab.org/competitions/19650). +The [evaluation](evaluation.md) page lists detailed information +regarding how results will be evaluated. Challenge participants +with the most successful and innovative methods will be invited +to present at the workshop. + +## Tools and Instructions + +We provide extensive API support for the images, annotations, +and evaluation code. To download the COCO DensePose API, +please visit our +[GitHub repository](https://github.com/facebookresearch/DensePose/). +Due to the complexity of this task, the process of participating +may not seem simple. To help, we provide explanations and +instructions for each step of the process: +[download](https://posetrack.net/users/download.php), +[data format](data_format.md), +[results format](results_format.md), +[upload](upload.md) and [evaluation](evaluation.md) pages. + diff --git a/challenge/2018_PoseTrack_DensePose/results_format.md b/challenge/2018_PoseTrack_DensePose/results_format.md new file mode 100644 index 0000000..bd3acf9 --- /dev/null +++ b/challenge/2018_PoseTrack_DensePose/results_format.md @@ -0,0 +1,42 @@ +# Results Format + +This page describes the results format used by PoseTrack DensePose evaluation +procedure. The results format mimics the annotation format detailed on +the [data format](data_format.md) page. Please review the annotation +format before proceeding. + +Each algorithmically generated result is stored separately in its own +result struct. This singleton result struct must contain the id of the +image from which the result was generated (a single image will typically +have multiple associated results). Results for the whole dataset are +aggregated in a single array. Finally, this entire result struct array +is stored to disk as a single JSON file (saved via +[gason](https://github.com/cocodataset/cocoapi/blob/master/MatlabAPI/gason.m) +in Matlab or [json.dump](https://docs.python.org/2/library/json.html) in Python). + +Example result JSON files are available in +[example results](example_results.json). + +The data struct for each of the result types is described below. The format +of the individual fields below (`category_id`, `bbox`, etc.) is the same as +for the annotation (for details see the [data format](data_format.md) page). +Bounding box coordinates `bbox` are floats measured from the top left image +corner (and are 0-indexed). We recommend rounding coordinates to the nearest +tenth of a pixel to reduce the resulting JSON file size. The dense estimates +of patch indices and coordinates in the UV space for the specified bounding +box are stored in `uv_shape` and `uv_data` fields. +`uv_shape` contains the shape of `uv_data` array, it should be of size +`(3, height, width)`, where `height` and `width` should match the bounding box +size. `uv_data` should contain PNG-compressed patch indices and U and V +coordinates scaled to the range `0-255`. + +An example of code that generates results in the form of a `pkl` file can +be found in +[json_dataset_evaluator.py](https://github.com/facebookresearch/DensePose/blob/master/detectron/datasets/json_dataset_evaluator.py). +We also provide an [example script](../encode_results_for_competition.py) to convert +DensePose estimation results stored in a `pkl` file into a PNG-compressed +JSON file. + + + + diff --git a/challenge/2018_PoseTrack_DensePose/upload.md b/challenge/2018_PoseTrack_DensePose/upload.md new file mode 100644 index 0000000..0f0bd05 --- /dev/null +++ b/challenge/2018_PoseTrack_DensePose/upload.md @@ -0,0 +1,67 @@ +# Upload Results to Evaluation Server + +This page describes the upload instructions for submitting results to the +evaluation servers for the PoseTrack DensePose challenge. Submitting results allows +you to participate in the challenges and compare results to the +state-of-the-art on the public leaderboards. Note that you can obtain results +on val by running the +[evaluation code](https://github.com/facebookresearch/DensePose/blob/master/detectron/datasets/densepose_cocoeval.py) +locally. One can also take advantage of the +[vkhalidov/densepose-codalab](https://hub.docker.com/r/vkhalidov/densepose-codalab/) +docker image which was tailored specifically for evaluation. +Submitting to the evaluation server provides results on the val and +test sets. We now give detailed instructions for submitting to the evaluation +server: + +1. Create an account on CodaLab. This will allow you to participate in +PoseTrack DensePose challenge. + +2. Carefully review the [guidelines](readme.md) for +entering the PoseTrack DensePose challenge and using the test sets. +We emphasize that any form of **annotation or use of the test sets +for supervised or unsupervised training is strictly forbidden**. + +3. Prepare a JSON file containing your results in the correct +[results format](results_format.md). + +4. File naming: the JSON file should be named `posetrack_[subset]_[alg]_results.json`. +Replace `[subset]` with the subset you are using (`val` or `test`), +and `[alg]` with your algorithm name. Finally, place the JSON +file into a zip file named `posetrack_[subset]_[alg]_results.zip`. + +5. To submit your zipped result file to the PoseTrack DensePose Challenge, click on +the “Participate” tab on the +[CodaLab evaluation server](https://competitions.codalab.org/competitions/19650) page. +When you select “Submit / View Results” on the left panel, you will be able to choose +the subset. Please fill in the required fields and click “Submit”. A pop-up will +prompt you to select the results zip file for upload. After the file is uploaded +the evaluation server will begin processing. To view the status of your submission +please select “Refresh Status”. Please be patient, the evaluation may take quite +some time to complete (from ~20m to a few hours). If the status of your submission +is “Failed” please check your file is named correctly and has the right format. + +6. Please enter submission information into Codalab. The most important fields +are "Team name", "Method description", and "Publication URL", which are used +to populate the leaderboard. Additionally, under "user setting" in the +upper right, please add "Team members". There have been issues with the +"Method Description", we may collect these via email if necessary. These +settings are not intuitive, but we have no control of the Codalab website. +For the "Method description" we encourage participants to give detailed +method information that will help +the award committee invite participants with the most innovative methods. +**Listing external data used is mandatory.** You may also consider giving some +basic performance breakdowns on test subset (e.g., single model versus +ensemble results), runtime, or any other information you think may be pertinent +to highlight the novelty or efficacy of your method. + +7. After you submit your results to the test eval server, you can control +whether your results are publicly posted to the CodaLab leaderboard. To toggle +the public visibility of your results please select either “post to leaderboard” +or “remove from leaderboard”. Only one result can be published to the leaderboard +at any time. + +8. After evaluation is complete and the server shows a status of “Finished”, +you will have the option to download your evaluation results by selecting +“Download evaluation output from scoring step.” The zip file will contain the +score file `scores.txt`. +