feat(modules_mapping): add skip_private_shared_objects parameter for hermetic gazelle file generation

[AFMiziara]

Summary

Adds skip_private_shared_objects parameter to modules_mapping rule to enable hermetic builds across platforms. When enabled, excludes private shared objects under .libs directories from modules mapping, which fixes inconsistent manifest generation between Linux and macOS for packages with platform-specific dependency libraries.

Problem

The gazelle_python_manifest generates different manifest files on different platforms due to private shared objects in .libs directories that appear in Linux wheels but not macOS wheels.

For example, with opencv-python-headless, numpy and probably many others:

Linux wheels contain:

opencv_python_headless.libs/libopenblas-r0-f650aae0.3.3.so
opencv_python_headless.libs/libssl-28bef1ac.so.1.1
numpy.libs/libgfortran-040039e1-0352e75f.so.5.0.0
numpy.libs/libscipy_openblas64_-56d6093b.so

macOS wheels omit these files because macOS uses system frameworks (like Accelerate) instead of bundled libraries like OpenBLAS.

These .libs files are private dependency libraries that are not directly imported in Python code - they exist solely for the underlying C/C++ dependencies. When the generator processes them, it creates module mappings that vary between platforms, breaking build hermiticity.

[gemini-code-assist]

Summary of Changes

Hello @AFMiziara, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant enhancement to the modules_mapping rule by adding an ignore_native_libs parameter. This feature allows the build system to disregard platform-specific native libraries during manifest generation, thereby resolving inconsistencies that arise when processing Python wheels across different operating systems. The primary impact is the enablement of truly hermetic builds, ensuring that the generated module mappings are identical regardless of the platform, which is crucial for packages with varying native dependencies like opencv-python-headless.

Highlights

• New Parameter ignore_native_libs: Introduced a new boolean parameter ignore_native_libs to the modules_mapping rule, defaulting to False for backward compatibility.
• Hermetic Builds Across Platforms: When ignore_native_libs is enabled, the modules_mapping generator will exclude platform-specific native libraries (.so, .dylib files) from the manifest, ensuring consistent manifest generation across different operating systems (e.g., Linux and macOS).
• Problem Solved: This change addresses the issue of non-hermetic builds caused by Python wheels containing platform-specific native libraries (like those found in opencv-python-headless), which previously led to inconsistent manifest files depending on the build platform.
• Documentation and Examples: The new parameter is documented in gazelle/docs/installation_and_usage.md and an example of its usage is added to examples/build_file_generation/BUILD.bazel.
• Unit Testing: New unit tests have been added to gazelle/modules_mapping/test_generator.py to verify the correct behavior of the ignore_native_libs functionality, ensuring native libraries are skipped when the flag is active.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a useful ignore_native_libs flag to improve build hermiticity for Python projects with platform-specific native libraries. The implementation is on the right track, but I've identified a key improvement to make it fully functional across platforms like macOS by handling .dylib files in addition to .so files. I've also suggested updates to the documentation and tests to reflect this change. Overall, this is a valuable addition.

[AFMiziara]

Another approach would be to support adding a list to ignore specific dependencies.

[aignas]

Right now users can add regex to exclude certain entries. I assume that this is how users currently work this around.

[AFMiziara]

@aignas do you think we should keep the use of this via regex? I was thinking one day this could actually be default behavior, because these .libs are safely skippable.