Is the texture map necessary?

[BitTuner-Jack]

Hi! Thank you for your excellent work. I have a few questions.

I noticed that in the official Python implementation, the model-based approach does not require a texture map (instead, it uses a simple method to generate one internally), and it achieved good results on my test data.

However, in your C++ implementation, a texture map is required. I tried using solid colors, UV unwrapped maps, and even aligned the implementation with the official Python version, but I found that different maps had a significant impact on the results. Moreover, none of them performed as well as the Python version directly. Could this be due to FP16 quantization?

I’m quite confused.

[zz990099]

Hi Jack! It seems I accidentally deleted my previous comment. :( Thanks for your interest in this project again!!!

The texture-free feature isn’t fully implemented yet, but we’ll prioritize adding it to our roadmap.

If possible, could you share the data with us? :)

[BitTuner-Jack]

Hi, zz! Thanks for your reply. I apologize for some statements that were not clearly expressed earlier. What I meant by "it uses a simple method to generate one internally" is that when my model does not have a texture map, it defaults to generating a gray-scale image and then replaces the texture map during rendering through interpolation. It worked.

I’m really looking forward to your texture-free feature, and once again, thanks for your excellent work.

As for the test data, due to certain protocol restrictions, I may not be able to share it publicly at this time—I sincerely apologize for that. Moving forward, I will test the texture-free feature as soon as it’s available and share the results with you. Thanks!

[zz990099]

Hi Jack!

Over the past few days, we've been exploring the texture-free pipeline in FoundationPose and noticed that our implementation was missing the use_light rendering component link. While we've now implemented this part, the results still don't align with FoundationPose's outputs.

Through our investigation, we found that the diffuse_intensity_map deviate significantly from expectations. We're currently uncertain whether this discrepancy stems from differences between the assimp library and trimesh, or if there might be other underlying factors.

Here are the diffuse_intensity_map of FoundationPose in Python and our implementation.

(left - FoundationPose, right - our implementation)

If you have bandwidth to look into this, we'd greatly appreciate your insights in identifying the root cause. Many thanks in advance for your consideration!

P.S. The relevant code will be pushed to a separate branch in the repository sometime next week for easier reference.

Best regards.

[zz990099]

We've made some progress on the input alignment issue:

1. We've excluded discrepancies between assimp and trimesh libraries
2. We conducted tests on the nvdiffrast::interpolate method by saving the input generated from the C++ code and having the Python code read and execute interpolate. However, we noticed that the outputs from the two implementations did not match.

So, now it is clear that the interpolate method performs differently between the Python project and the c++ implementation.

We would greatly appreciate any suggestions you may have!

[godfather1402]

Hi @zz990099, is it possible to change reading mesh file using open3d instead of assimp in the future. If texture free is supported, we can read .ply, .stl file or other format instead of only .obj file. open3d also use assimp internally

[zz990099]

Hi! @godfather1402 Thank you for your suggestion - it's valuable for the project. We plan to implement this feature in the future！ Perhaps a polymorphic approach could be adopted to handle different model loading method.

[BitTuner-Jack]

Hi Jack!

Over the past few days, we've been exploring the texture-free pipeline in FoundationPose and noticed that our implementation was missing the use_light rendering component link. While we've now implemented this part, the results still don't align with FoundationPose's outputs.

Through our investigation, we found that the diffuse_intensity_map deviate significantly from expectations. We're currently uncertain whether this discrepancy stems from differences between the assimp library and trimesh, or if there might be other underlying factors.

Here are the diffuse_intensity_map of FoundationPose in Python and our implementation. (left - FoundationPose, right - our implementation)

If you have bandwidth to look into this, we'd greatly appreciate your insights in identifying the root cause. Many thanks in advance for your consideration!

P.S. The relevant code will be pushed to a separate branch in the repository sometime next week for easier reference.

Best regards.

Hi zz! Sorry for the late reply. Thanks for your efforts in achieving texture-free rendering. From the images you provided, the rendering results from the Python implementation appear more natural, with more pronounced light and shadow variations, while the C++ implementation's rendering results look flatter, especially lacking in lighting details. Based on the official rendering code, I believe the following could be some potential issues and directions for troubleshooting:

1. Normalization of normals and light direction:
   The official code uses F.normalize to normalize vnormals_cam and light_dir_neg. If normalization is missing or implemented incorrectly, it may lead to errors in calculating diffuse intensity.
2. Clipping and range constraints:
   The official code uses .clip(0, 1) to constrain the diffuse intensity within the range [0, 1]. Please check if similar clipping operations are applied in the C++ code. Diffuse intensity values should be non-negative and are typically constrained to the range [0, 1].
3. Differences in interpolation methods:
   If the interpolation method used in the C++ code differs from nvdiffrast's dr.interpolate, it could likely result in differences in the rendering output. As far as I know, barycentric coordinate interpolation is commonly used in triangle mesh rendering.

[zz990099]

Update: We've resolved the nvdiffrast::interpolate method inconsistency, which stemmed from parameter misconfiguration during nvdiffrast migration in isaac_pose_estimation project. Details in issue_link.

The develop_texture_free_feature branch is now functional, though you might experience temporary precision degradation. We warmly welcome any optimization suggestions!

[BitTuner-Jack]

Great! I have tested the texture_free_feature, and the results are basically similar to the official implementation.
Additionally, please note that the provided OBJ model must include the v, vn, vt, and f attributes when in use. Otherwise, rendering failures may occur.
Thanks for your excellent work!

[zz990099]

Thank you for your insight! This is particularly helpful for developers like myself who are less familiar with 3D modeling workflows.