back2source: Refine ScanCode.io d2d pipeline for ELFs

[pombredanne]

Update pipelines steps for the binary to source analysis for native ELF using strings and symbols, and not only DWARFs.

The high level approach for this is:

• On the from side, collect and store source symbols using the "purl2sym" "collect_symbols" pipelines or it underlying functions
• On the to side, collect and store binary symbols using Lief or Pyelftools
• Build a step (or extend existing steps) to go into symbol matching, if the path match matching was not conclusive. Process the results the same way as path matching.

[AyanSinhaMahapatra]

From #1339:

There seem to be too many "requires-review" as a first glance in the summary report from a d2d project for ELF.

• https://github.com/aboutcode-org/back2source-data/blob/main/data/archive.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/debug/tree/a/atomix-debugsource-44.0-4.fc40.x86_64/d2d-summary.json#L36 has only one requires review

• https://github.com/aboutcode-org/back2source-data/blob/main/data/archive.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/debug/tree/a/atomix-debugsource-44.0-4.fc40.x86_64/d2d-details.json#L3535 has many correctly mapped paths that are marked as requires-review likely incorrectly

• Also we should not include directories and <artificial> and <built-in> (and this may something we could filter out in the elf-inspector ?)

  • like in https://github.com/aboutcode-org/back2source-data/blob/main/data/archive.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/debug/tree/a/atomix-debugsource-44.0-4.fc40.x86_64/d2d-details.json#L3398
  • or in https://github.com/aboutcode-org/back2source-data/blob/main/data/archive.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/debug/tree/a/atomix-debugsource-44.0-4.fc40.x86_64/d2d-details.json#L3493

[AyanSinhaMahapatra]

This is now completed.

To complete this feature we had the following PRs/items:

• C/C++/Rust/Go source symbols parsers were already present in https://github.com/aboutcode-org/source-inspector
• Use py-tree-sitter-swift v0.0.1 and support py3.13 source-inspector#36 Added tests for C/C++/Rust/Go symbols parsing and released source-inspector https://github.com/aboutcode-org/source-inspector/releases/tag/v0.6.1
• Get binary symbols from ELF elf-inspector#9 Get symbols from elf binaries using bids-analyzer
• Match binaries to source using elf symbols #1621 Add a new pipeline step to match symbols from source files to elf binaries

See the related pipe in SCIO documentation: https://scancodeio.readthedocs.io/en/latest/built-in-pipelines.html#scanpipe.pipelines.deploy_to_develop.DeployToDevelop.map_elf

This is available as a new optional step in the deploy_to_develop pipeline: Elf (you can select this through the UI or pass this as an option when using the API/CLI: deploy_to_develop:Elf )

Select the option Elf in the new project form like this: