feat(shape_estimation): add ml shape estimation #7860

kaancolak · 2024-07-05T01:28:22Z

Description

Pointnet-based vehicle shape estimation module.

Choosing the estimation method is optional in the config file.

How was this PR tested?

Notes for reviewers

Test using the given ONNX file, no need to any other change

Interface changes

None.

Effects on system behavior

None.

github-actions · 2024-07-05T01:28:42Z

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

You've checked our contribution guidelines.
Your PR follows our pull request guidelines.
All required CI checks pass before marking the PR ready for review.

yukkysaito · 2024-07-05T01:51:04Z

@kaancolak This is really amazing! Could you please describe the mechanism and node parameters in the README? If there is a learning method, please also include the user's learning procedure.

kaancolak · 2024-07-05T10:55:39Z

Thank you @yukkysaito -san, I will prepare documentation about model details and also how to train the model/convert to ONNX.

kaancolak · 2024-07-16T06:03:44Z

@yukkysaito @YoshiRi @miursh

I updated the readme file that added retraining/deploying guidelines. Please check it out and feel free to share your thoughts.

This is the training repository, maybe we can move it under Autoware foundation :
https://github.com/kaancolak/bbox_estimator

perception/shape_estimation/README.md

kaancolak · 2024-07-19T00:36:49Z

@yukkysaito @YoshiRi @miursh

Could you review or assign a proper reviewer?

YoshiRi · 2024-07-19T09:50:31Z

Hi. @kaancolak
Thanks for your PR.

Test with sample rosbag

I built and checked your ml based shape estimation with sample rosbag and confirmed it works.

how to test
- Let detection_by_tracker_node to published cluster with class label (need slight source code change)
- use that output to check ml based shape estimation

sample objects after fitting is like following:

title	figure
normal vehicle
trucks with I-shaped cluster(Not L-shape)
trailer with L-shape cluster

Discussion

I have several opinion and question.

Based on the current CMakeLists setup, it seems that the build process requires a CUDA environment. However, there are situations where we might want to build and run without a CUDA environment. To accommodate this, we should either use #ifdef to conditionally compile or separate the node that uses ML.
The shape_estimator is also called in other places, such as detection_by_tracker, so we should aim to unify the interface for both ML and non-ML shape_estimator calls as much as possible. It might be better for the non-ML side to accept a list of objects as input instead of individual objects. However, for this PR, it might be acceptable to ignore this issue for now.
Just a pure question, the ML estimator is designed for vehicle classes, but what kind of output does it produce if the input belongs to other classes?

perception/shape_estimation/CMakeLists.txt

perception/shape_estimation/config/shape_estimation_ml.param.yaml

YoshiRi · 2024-07-19T10:09:04Z

perception/shape_estimation/include/autoware/shape_estimation/tensorrt_shape_estimator.hpp

+
+  ~TrtShapeEstimator();
+
+  bool inference(const DetectedObjectsWithFeature & input, DetectedObjectsWithFeature & output);


[just a comment]
I think it's better to share interface between ml and non-ml shape estimation in the future update.

I completely agree with that. It will make the package better as a design.

Signed-off-by: Kaan Çolak <kaancolak95@gmail.com>

kaancolak · 2024-07-27T08:57:38Z

@YoshiRi -san, thank you for your review. Sorry, I haven't been looked for last week.

Like your comment, I separated the cuda and non-cuda parts of the code that are controlled by cuda and cuda library founding flags.

Just a pure question, the ML estimator is designed for vehicle classes, but what kind of output does it produce if the input belongs to other classes?

The model takes one hot vector and point cloud as input. Yet if we feed different objects than vehicle labels, it will probably return the closest vehicle shape to the fed point cloud. The model could be easily extended for other classes according to the training readme file.

YoshiRi

LGTM and worked with my local environment.

YoshiRi · 2024-07-29T07:35:05Z

perception/autoware_shape_estimation/CMakeLists.txt

-  lib/corrector/utils.cpp
-  lib/corrector/no_corrector.cpp
+set(${PROJECT_NAME}_SOURCES
+        lib/shape_estimator.cpp


[minor comment]
Do you have any intention to add these indent?

thank you warning, I fixed the indent

YoshiRi · 2024-07-29T07:37:23Z

perception/autoware_shape_estimation/README.md

+- object size classification result
+- object size residuals
+
+### Training ML Based Shape Estimation Model


[minor comment]
It may better to separate training document with README for future.

Signed-off-by: Kaan Çolak <kaancolak95@gmail.com>

Shin-kyoto · 2024-08-01T16:47:19Z

perception/autoware_shape_estimation/lib/tensorrt_shape_estimator.cpp

+      }
+
+      int iter_count = static_cast<int>(input_pc_size) / point_size_of_cloud;
+      int remainer_count = static_cast<int>(input_pc_size) % point_size_of_cloud;


Suggested change

int remainer_count = static_cast<int>(input_pc_size) % point_size_of_cloud;

int remainder_count = static_cast<int>(input_pc_size) % point_size_of_cloud;

Spell check failed, so can you change remainer to another word? I think reminder is better.
https://github.com/autowarefoundation/autoware.universe/actions/runs/10196767728/job/28208175986?pr=7860

YoshiRi · 2024-08-02T06:13:55Z

Rerun CI to check builds. Please wait for a while.

codecov · 2024-08-02T06:19:55Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 5.50%. Comparing base (3d849e9) to head (16a98d9).
Report is 973 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #7860       +/-   ##
==========================================
- Coverage   15.09%   5.50%    -9.59%     
==========================================
  Files        1967     109     -1858     
  Lines      135941    4558   -131383     
  Branches    42122     567    -41555     
==========================================
- Hits        20520     251    -20269     
+ Misses      92700    4256    -88444     
+ Partials    22721      51    -22670

Flag	Coverage Δ
differential	`5.50% <ø> (?)`
total	`?`

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

) * feat(shape_estimation): add ml shape estimation Signed-off-by: Kaan Çolak <kaancolak95@gmail.com> * style(pre-commit): autofix * feat(shape_estimation): fix exceed objects Signed-off-by: Kaan Çolak <kaancolak95@gmail.com> * style(pre-commit): autofix * feat(shape_estimation): fix indent Signed-off-by: Kaan Çolak <kaancolak95@gmail.com> --------- Signed-off-by: Kaan Çolak <kaancolak95@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

kaancolak requested a review from miursh July 5, 2024 01:28

kaancolak self-assigned this Jul 5, 2024

kaancolak requested review from YoshiRi and yukkysaito as code owners July 5, 2024 01:28

github-actions bot added the component:perception Advanced sensor data processing and environment understanding. (auto-assigned) label Jul 5, 2024

kaancolak linked an issue Jul 5, 2024 that may be closed by this pull request

Add deep learning based 3D bouding box shape estimation algorithm #7106

Closed

3 tasks

kaancolak marked this pull request as draft July 5, 2024 23:36

github-actions bot added the type:documentation Creating or refining documentation. (auto-assigned) label Jul 12, 2024

kaancolak force-pushed the feat/add-deep-learning-based-shape-estimator branch 3 times, most recently from fd59254 to 460fb6e Compare July 16, 2024 04:47

kaancolak marked this pull request as ready for review July 16, 2024 06:03

yukkysaito reviewed Jul 16, 2024

View reviewed changes

perception/shape_estimation/README.md Outdated Show resolved Hide resolved

kaancolak force-pushed the feat/add-deep-learning-based-shape-estimator branch from b31afb7 to a13a441 Compare July 16, 2024 06:39

kaancolak requested a review from yukkysaito July 16, 2024 07:56

YoshiRi reviewed Jul 19, 2024

View reviewed changes

feat(shape_estimation): add ml shape estimation

d1213d9

Signed-off-by: Kaan Çolak <kaancolak95@gmail.com>

kaancolak force-pushed the feat/add-deep-learning-based-shape-estimator branch from a13a441 to d1213d9 Compare July 27, 2024 08:47

style(pre-commit): autofix

f126776

YoshiRi approved these changes Jul 29, 2024

View reviewed changes

kaancolak and others added 3 commits July 29, 2024 19:49

feat(shape_estimation): fix exceed objects

2f43482

Signed-off-by: Kaan Çolak <kaancolak95@gmail.com>

style(pre-commit): autofix

56d1149

feat(shape_estimation): fix indent

b97b92f

Signed-off-by: Kaan Çolak <kaancolak95@gmail.com>

Merge branch 'main' into feat/add-deep-learning-based-shape-estimator

16a98d9

kaancolak enabled auto-merge (squash) August 1, 2024 10:39

Shin-kyoto reviewed Aug 1, 2024

View reviewed changes

YoshiRi added the tag:run-build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci) label Aug 2, 2024

kaancolak merged commit 04a00ef into autowarefoundation:main Aug 2, 2024
37 of 39 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(shape_estimation): add ml shape estimation #7860

feat(shape_estimation): add ml shape estimation #7860

kaancolak commented Jul 5, 2024 •

edited

Loading

github-actions bot commented Jul 5, 2024 •

edited

Loading

yukkysaito commented Jul 5, 2024

kaancolak commented Jul 5, 2024

kaancolak commented Jul 16, 2024 •

edited

Loading

kaancolak commented Jul 19, 2024

YoshiRi commented Jul 19, 2024

YoshiRi Jul 19, 2024

kaancolak Jul 27, 2024

kaancolak commented Jul 27, 2024

YoshiRi left a comment •

edited

Loading

YoshiRi Jul 29, 2024

kaancolak Jul 29, 2024

YoshiRi Jul 29, 2024

Shin-kyoto Aug 1, 2024

YoshiRi commented Aug 2, 2024

codecov bot commented Aug 2, 2024


		~TrtShapeEstimator();

		bool inference(const DetectedObjectsWithFeature & input, DetectedObjectsWithFeature & output);

	int remainer_count = static_cast<int>(input_pc_size) % point_size_of_cloud;
	int remainder_count = static_cast<int>(input_pc_size) % point_size_of_cloud;

feat(shape_estimation): add ml shape estimation #7860

feat(shape_estimation): add ml shape estimation #7860

Conversation

kaancolak commented Jul 5, 2024 • edited Loading

Description

Related links

How was this PR tested?

Notes for reviewers

Interface changes

Effects on system behavior

github-actions bot commented Jul 5, 2024 • edited Loading

yukkysaito commented Jul 5, 2024

kaancolak commented Jul 5, 2024

kaancolak commented Jul 16, 2024 • edited Loading

kaancolak commented Jul 19, 2024

YoshiRi commented Jul 19, 2024

Test with sample rosbag

Discussion

YoshiRi Jul 19, 2024

Choose a reason for hiding this comment

kaancolak Jul 27, 2024

Choose a reason for hiding this comment

kaancolak commented Jul 27, 2024

YoshiRi left a comment • edited Loading

Choose a reason for hiding this comment

YoshiRi Jul 29, 2024

Choose a reason for hiding this comment

kaancolak Jul 29, 2024

Choose a reason for hiding this comment

YoshiRi Jul 29, 2024

Choose a reason for hiding this comment

Shin-kyoto Aug 1, 2024

Choose a reason for hiding this comment

YoshiRi commented Aug 2, 2024

codecov bot commented Aug 2, 2024

Codecov Report

kaancolak commented Jul 5, 2024 •

edited

Loading

github-actions bot commented Jul 5, 2024 •

edited

Loading

kaancolak commented Jul 16, 2024 •

edited

Loading

YoshiRi left a comment •

edited

Loading