CodebaseResource does not have all the features of commoncode Resource model

[AyanSinhaMahapatra]

Due to a need of maintaining a Codebase/Resource model to traverse and support all the scancode-toolkit functions we have two similar models with same functions to integrate scancode-toolkit in SCIO:

• the CodebaseResource model mirrors the commoncode.resource.Resource model
• the Project model has necessary functions with the same behavior from the commoncode.resource.Codebase model

But the support is currently limited to the widely used functions in scancode-toolkit only, and not all the functions from the commoncode.resource models, and this causes failures when a new function is used on the scancode-toolkit side, which might not be supported on the scancode.io side yet.

For example:
In aboutcode-org/scancode-toolkit#3983 we fixed some issues related to package resource assignment for jar manifests, and in the process we started using commoncode.resource.Resource.extracted_to from the suggestion at aboutcode-org/scancode-toolkit#3983 (comment), but this is not something that is supported in SCIO, so after this was released and used in SCIO, we had the following failure:

'CodebaseResource' object has no attribute 'extracted_from'

Traceback:
  File "/opt/scancodeio/aboutcode/pipeline/__init__.py", line 199, in execute
    step(self)
  File "/opt/scancodeio/scanpipe/pipelines/inspect_packages.py", line 61, in scan_for_application_packages
    scancode.scan_for_application_packages(
  File "/opt/scancodeio/scanpipe/pipes/scancode.py", line 436, in scan_for_application_packages
    assemble_packages(project=project)
  File "/opt/scancodeio/scanpipe/pipes/scancode.py", line 493, in assemble_packages
    for item in items:
                ^^^^^
  File "/opt/scancodeio/.venv/lib/python3.12/site-packages/packagedcode/maven.py", line 123, in assemble
    yield from JavaJarManifestHandlerMixin.assemble(package_data, resource, codebase, package_adder)
  File "/opt/scancodeio/.venv/lib/python3.12/site-packages/packagedcode/models.py", line 1185, in assemble
    cls.assign_package_to_resources(
  File "/opt/scancodeio/.venv/lib/python3.12/site-packages/packagedcode/maven.py", line 158, in assign_package_to_resources
    models.DatafileHandler.assign_package_to_resources(
  File "/opt/scancodeio/.venv/lib/python3.12/site-packages/packagedcode/models.py", line 1225, in assign_package_to_resources
    archive_resource = resource.extracted_from(codebase)

So we need to do a review of all the functions present in commoncode.resource models which could be used in scancode-toolkit, and implement all of them here, and add tests to make sure they are all supported.

[AyanSinhaMahapatra]

I will open a PR to support extracted_from and extracted_to functions just so we don't get these failures atleast, the rest can be done later.

[tdruez]

@AyanSinhaMahapatra Can this be closed following the merged of #1588 ?

[AyanSinhaMahapatra]

@tdruez Not really, my PR above only fixes the specific errors we were having (which is part of this issue but not all of it). We need to do a bit more to ensure this doesn't happen again in the future. From my comment above, the goal is to:

do a review of all the functions present in commoncode.resource models which could be used in scancode-toolkit, and implement all of them here, and add tests to make sure they are all supported

[abanoub-samy-farhan]

Hi @AyanSinhaMahapatra , I would like to contribute to this issue. Currently working on reviewing the CodebaseResource and it's mirror model commoncode.resource.Resource