Add argument to `make_codebase_resource` to make the created CodebaseResource path relative to the project work path rather than the project codebase path #378

JonoYang · 2021-12-08T02:48:32Z

I have a pipeline that performs something similar to a Merkle tree calculation on a codebase. In order for this to work, I need a root directory to roll my results up to. scanpipe currently does not have the concept of a root, so my current workaround is to use the project's codebase directory as the root directory.

I create a new pipeline that creates a CodebaseResource for the project codebase directory, but when I call scanpipe.pipes.scancode.make_codebase_resource, the created CodebaseResources do not have codebase as the prefix.

I've modified scanpipe.pipes.scancode.make_codebase_resource to have a new argument relative_to_work_path such that the path created for the CodebaseResource is related to the project work path, rather than the project codebase path. This means that the created CodebaseResource paths will be prefixed with codebase e.g. codebase/notice.NOTICE.

tdruez · 2021-12-09T07:44:29Z

@JonoYang I'm not confortable with the idea of storing the codebase/ prefix as a choice.
This lack of consistency in the way CodebaseResource path would be treated will likely be a pain in the future as features will now have to deal with both possibility (prefix or no-prefix).

In order for this to work, I need a root directory to roll my results up to

What about a special "root" CodebaseResource instance instead?
Maybe a simple . or ./ coule be good enough?

Note that we used to always store the codebase/ prefix in the database but this had a real estate cost in the UI and storage cost in the database without much benefits (at least at the time).
We could bring back the prefix is we do not find a better approach, but this would not be a choice and should be enforced for all CodebaseResource instances.

JonoYang · 2021-12-09T20:13:22Z

@tdruez Thanks for the review!

In order for this to work, I need a root directory to roll my results up to

What about a special "root" CodebaseResource instance instead?
Maybe a simple . or ./ coule be good enough?

This would be acceptable too. The thing I am interested in is the ability to use the ProjectCodebase class to iterate through the CodebaseResources using code that was originally written for use with scancode-toolkit's Codebase and VirtualCodebase classes. In particular, the code I have uses Codebase.walk() and Resource.walk(). If we are to create a special root CodebaseResource named . or ./, I would assume that all of the other CodebaseResources in our codebase would need to have . as a prefix since CodebaseResource.walk() in scancode.io uses the CodebaseResource path to determine its children.

For example:

./
./foo/bar.c
./README.txt

I'll modify my pipeline to create a CodebaseResource with the path ./ and then modify make_codebase_resource to allow me to create CodebaseResources with . as a path prefix.

tdruez · 2021-12-10T14:02:44Z

@JonoYang my point is that we do not want to store any prefix in the database.
I would suggest to refine the walk() functions logic instead to support a special case for a root.

JonoYang

@tdruez I've made some changes where we now create the "." CodebaseResource when running the scan_codebase pipeline. I've done this by modifying Project.walk_codebase_path() to return the project codebase/ directory and by modifying make_codebase_resource to handle the cases where it is making a CodebaseResource for the project codebase/ directory.

JonoYang · 2021-12-14T02:03:36Z

scanpipe/models.py

+        including the codebase/ directory itself.
        """
-        return self.codebase_path.rglob("*")
+        return itertools.chain(


@tdruez I am trying to get walk_codebase_path() to return the codebase/ directory in addition to the subdirectories and files of codebase/. Is there a better way to do this?

scanpipe/models.py

tdruez

I have not implemented a way to create "." when creating CodebaseResources for the docker pipeline. An immediate solution would be to check to see if a "." exists before creating the CodebaseResources for a layer (https://github.com/nexB/scancode.io/blob/main/scanpipe/pipes/docker.py#L134)

Maybe a better idea would be to create "." when the project is created?

No, let's not create resources at project creation time, resources should be created during pipelines execution.

tdruez · 2021-12-17T10:40:54Z

scanpipe/models.py

        if output_files:
            return output_files[-1]

    def walk_codebase_path(self):


We need a unit test for this method.

tdruez · 2021-12-17T10:58:51Z

scanpipe/models.py

        The current CodebaseResource is not included.
        """
-        return self.project.codebaseresources.filter(path__startswith=f"{self.path}/")
+        if self.path == ".":


Let's add a is_root model property for this code and replace here with "self.is_root"
This will make the code more readable and we'll be able to change the Root logic in the future by simply updating the property.
Also, the "." should be stored in a global variable, something like ROOT_SYMBOL, for the same reasons as above.

tdruez · 2021-12-17T11:01:59Z

scanpipe/models.py

        """
-        return self.project.codebaseresources.filter(path__startswith=f"{self.path}/")
+        if self.path == ".":
+            return self.project.codebaseresources.exclude(path=".")


This condition does not seem to be tested. Also, could you document in the dostring why the root need to be treated differently?

tdruez · 2021-12-17T11:02:36Z

scanpipe/models.py

        """
-        exactly_one_sub_directory = "[^/]+$"
-        children_regex = rf"^{self.path}/{exactly_one_sub_directory}"
+        if self.path == ".":


Same as above, it's not clear nor documented why we have to treat the root differently.

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* Remove create_discovered_packages2 and create_codebase_resources2 Signed-off-by: Jono Yang <jyang@nexb.com>

* Normalize package_uids before comparing results in tests * Update expected test results Signed-off-by: Jono Yang <jyang@nexb.com>

* Mark ProjectCodebase tests with expectedFailure * We will revisit ProjectCodebase and update it to fit our current models Signed-off-by: Jono Yang <jyang@nexb.com>

* We are using a scancode scan results for tests since asgiref-3.3.0_scan.json is not exactly the same format as scancode's json output Signed-off-by: Jono Yang <jyang@nexb.com>

* Update regen_test_data.py to generate asgiref-3.3.0_walk_test_fixtures.json Signed-off-by: Jono Yang <jyang@nexb.com>

* No need to explicity get license_clarity_score in make_results_summary() * Update expected test results Signed-off-by: Jono Yang <jyang@nexb.com>

* Add .vscode to .gitignore Signed-off-by: Jono Yang <jyang@nexb.com>

* Add argument to set the codebase path of a CodebaseResource created by make_codebase_resource relative to the project work path Signed-off-by: Jono Yang <jyang@nexb.com>

* `codebase` has been added as an unused argument to CodebaseResource.save() to provide compatibility with commoncode.resource.Resource Signed-off-by: Jono Yang <jyang@nexb.com>

* Have test to ensure that the codebase prefix is present when relative_to_work_path is True Signed-off-by: Jono Yang <jyang@nexb.com>

* Update expected test results * Revert previous changes to make_codebase_resource Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* Use different regex to determine the children of the "." CodebaseResource * Update test expectations Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* replace_root_path_and_name is a helper function that replaces the root of a Codebase with "." and strips the previous root prefix from the remaining Resources * Update tests Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

* Update docstrings * Update expected test results Signed-off-by: Jono Yang <jyang@nexb.com>

* Format code Signed-off-by: Jono Yang <jyang@nexb.com>

tdruez · 2022-06-14T16:49:26Z

@JonoYang now that we merged the toolkit upgrade, would it be a good time to rebase et complete this PR?

* Update expected test results Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang · 2022-06-14T19:10:01Z

@tdruez

I'm merging in the changes from main into this branch. I'm having trouble with replace_root_path_and_name (https://github.com/nexB/scancode.io/blob/180-projectcodebase-root-workaround/scanpipe/pipes/scancode.py#L504) that I made. My function renames the root resource path and name to ".", and removes the root path from the remaining resource paths. It's conflicting with the changes made to how Resource in a Codebase in scancode are related to one another. The walk method on commoncode.Resource uses the path of a Resource to find its children. If we walk from the root Resource of a Codebase, the walk fails because it cannot find the rest of the Resources in the codebase because no other Resource paths start with ".".

I think what I am trying to do in this function is best implemented somewhere in commoncode.Codebase, where I can pass some arguments to the Codebase to do this path transformations and modify the logic of walk to find the child of a root resource whose path is ".".

Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang · 2023-03-02T19:57:07Z

Closing this in favor of #624. Instead of creating a single root CodebaseResource, ProjectCodebase has been updated to walk, not from a single root, but from every CodebaseResource in the root of a Project. This effectively will get the same tree traversal done without large changes to the Project/CodebaseResource models.

JonoYang requested a review from tdruez December 8, 2021 02:48

JonoYang force-pushed the 180-projectcodebase-root-workaround branch 2 times, most recently from a4d4a8b to 4faafad Compare December 8, 2021 03:49

JonoYang commented Dec 14, 2021

View reviewed changes

JonoYang force-pushed the 180-projectcodebase-root-workaround branch 2 times, most recently from 8c690ce to cf8a8bf Compare December 15, 2021 20:52

tdruez requested changes Dec 17, 2021

View reviewed changes

tdruez and others added 6 commits May 2, 2022 11:31

Upgrade scancode-toolkit to latest beta release #411

14f36b7

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Add a test class to regen test data #411

e5b7696

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Upgrade container_inspector to latest 31.0.0 version #411

3519009

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Handle new scan format in scancode pipes #411

0110163

Signed-off-by: Jono Yang <jyang@nexb.com>

Handle package_uids for DiscoveredPackages #411

a029c26

* Remove create_discovered_packages2 and create_codebase_resources2 Signed-off-by: Jono Yang <jyang@nexb.com>

Update deprecated code #411

774c5b7

* Normalize package_uids before comparing results in tests * Update expected test results Signed-off-by: Jono Yang <jyang@nexb.com>

tdruez mentioned this pull request May 10, 2022

411 toolkit upgrade 2 #435

Merged

JonoYang added 13 commits May 10, 2022 12:11

Regenerate asgiref 3.3.0 test data #411

a6bffc5

* Mark ProjectCodebase tests with expectedFailure * We will revisit ProjectCodebase and update it to fit our current models Signed-off-by: Jono Yang <jyang@nexb.com>

Add asgiref-3.3.0_scancode_scan.json #411

4c354f1

* We are using a scancode scan results for tests since asgiref-3.3.0_scan.json is not exactly the same format as scancode's json output Signed-off-by: Jono Yang <jyang@nexb.com>

Add asgiref-3.3.0_walk_test_fixtures.json #411

4ed789f

* Update regen_test_data.py to generate asgiref-3.3.0_walk_test_fixtures.json Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

80d6095

Update make_results_summary() #411

bcceb76

* No need to explicity get license_clarity_score in make_results_summary() * Update expected test results Signed-off-by: Jono Yang <jyang@nexb.com>

Exclude system_environment from diff #411

b25c2c8

* Add .vscode to .gitignore Signed-off-by: Jono Yang <jyang@nexb.com>

Modify make_codebase_resource #180

98127ea

* Add argument to set the codebase path of a CodebaseResource created by make_codebase_resource relative to the project work path Signed-off-by: Jono Yang <jyang@nexb.com>

Add argument to CodebaseResource.save #180

d78c895

* `codebase` has been added as an unused argument to CodebaseResource.save() to provide compatibility with commoncode.resource.Resource Signed-off-by: Jono Yang <jyang@nexb.com>

Add test for make_codebase_resource #180

bbe588b

* Have test to ensure that the codebase prefix is present when relative_to_work_path is True Signed-off-by: Jono Yang <jyang@nexb.com>

Create "." directory for project codebase root #180

bb8c832

* Update expected test results * Revert previous changes to make_codebase_resource Signed-off-by: Jono Yang <jyang@nexb.com>

Modify CodebaseResource walk to handle "." #180

6dc403c

Signed-off-by: Jono Yang <jyang@nexb.com>

Update expected test results #180

7bca1a2

Signed-off-by: Jono Yang <jyang@nexb.com>

Update regex in CodebaseResource.children() #180

6ececfa

* Use different regex to determine the children of the "." CodebaseResource * Update test expectations Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang added 5 commits May 10, 2022 16:04

Handle root creation in scancode pipes #180

11bc9b2

Signed-off-by: Jono Yang <jyang@nexb.com>

Create replace_root_path_and_name #180

5d84d8e

* replace_root_path_and_name is a helper function that replaces the root of a Codebase with "." and strips the previous root prefix from the remaining Resources * Update tests Signed-off-by: Jono Yang <jyang@nexb.com>

Update docstring for CodebaseResource #180

d4cfac2

Signed-off-by: Jono Yang <jyang@nexb.com>

Use less paths for test expectations #180

06acd33

Signed-off-by: Jono Yang <jyang@nexb.com>

Add is_root property to CodebaseResource #180

a9410f8

* Update docstrings * Update expected test results Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang force-pushed the 180-projectcodebase-root-workaround branch from 48e489c to a9410f8 Compare May 11, 2022 02:24

Add test for Project.walk_codebase_path #180

24902bf

* Format code Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang force-pushed the 180-projectcodebase-root-workaround branch from 95ad5af to 24902bf Compare May 11, 2022 02:48

JonoYang added 2 commits June 14, 2022 10:52

Merge branch 'main' into 180-projectcodebase-root-workaround

b3c06cd

Do not save package_uids in extra_data #180

6831a55

* Update expected test results Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang added 3 commits June 17, 2022 11:06

Merge branch 'main' into 180-projectcodebase-root-workaround

adb7e3f

Merge branch 'main' into 180-projectcodebase-root-workaround

d73563e

Update expected test results #180

d3b985c

Signed-off-by: Jono Yang <jyang@nexb.com>

tdruez mentioned this pull request Mar 2, 2023

Update ProjectCodebase #624

Merged

JonoYang closed this Mar 2, 2023

JonoYang deleted the 180-projectcodebase-root-workaround branch March 3, 2023 17:38

Uh oh!

Add argument to make_codebase_resource to make the created CodebaseResource path relative to the project work path rather than the project codebase path #378

Add argument to make_codebase_resource to make the created CodebaseResource path relative to the project work path rather than the project codebase path #378

Uh oh!

Conversation

JonoYang commented Dec 8, 2021

Uh oh!

tdruez commented Dec 9, 2021

Uh oh!

JonoYang commented Dec 9, 2021

Uh oh!

tdruez commented Dec 10, 2021

Uh oh!

JonoYang left a comment

Choose a reason for hiding this comment

Uh oh!

JonoYang Dec 14, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tdruez left a comment

Choose a reason for hiding this comment

Uh oh!

tdruez Dec 17, 2021

Choose a reason for hiding this comment

Uh oh!

tdruez Dec 17, 2021

Choose a reason for hiding this comment

Uh oh!

tdruez Dec 17, 2021

Choose a reason for hiding this comment

Uh oh!

tdruez Dec 17, 2021

Choose a reason for hiding this comment

Uh oh!

tdruez commented Jun 14, 2022

Uh oh!

JonoYang commented Jun 14, 2022

Uh oh!

JonoYang commented Mar 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add argument to `make_codebase_resource` to make the created CodebaseResource path relative to the project work path rather than the project codebase path #378

Add argument to `make_codebase_resource` to make the created CodebaseResource path relative to the project work path rather than the project codebase path #378