Skip to content

Add Package Instances #2691 #2825

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 32 commits into from
Mar 5, 2022
Merged

Add Package Instances #2691 #2825

merged 32 commits into from
Mar 5, 2022

Conversation

@AyanSinhaMahapatra
Copy link
Member

@AyanSinhaMahapatra AyanSinhaMahapatra commented Jan 31, 2022 •
edited by pombredanne
Loading

This PR adds the PackageInstance class and functions to group package
manifests and package data as top level package instances.

Existing package data are ported to this new approach.

Reference: #2098
Reference: #2691
Reference: #2692
Reference: #2693
Reference: #2843
Reference: #2652
Signed-off-by: Ayan Sinha Mahapatra ayansmahapatra@gmail.com

Tasks

  • Reviewed contribution guidelines
  • PR is descriptively titled 📑 and links the original issue above 🔗
  • Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
    Run tests locally to check for errors.
  • Commits are in uniquely-named feature branch and has no merge conflicts 📁

@AyanSinhaMahapatra
Add initial package instance creation
c889023 
Adds the PackageInstance class and functions to group package manifests
and create a package instance out of them. Adds a sample example case for
python.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Add function to merge package data
fa010e4 
Adds functions to merge package data from multiple package
manifests into a package instance. Adds tests for a simple
python manifests case.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Adds PackageManifest classes for cran
9f9e3a6 
Creates PackageManifest classes for cran description manifests
and overrides the methods for detection and PackageManifest creation.

See #2748

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Adds PackageManifest class for maven
695ae3b 
Creates PackageManifest class for maven pom.xml  manifests
and overrides the methods for detection and PackageManifest creation.

See #2748

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@pombredanne pombredanne mentioned this pull request Feb 2, 2022
Closed
JonoYang
JonoYang reviewed Feb 2, 2022
View reviewed changes
Copy link
Member

@JonoYang JonoYang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AyanSinhaMahapatra Initial work on the PackageInstance model looks good. I just have some minor things I pointed out.

@AyanSinhaMahapatra
Modify package instance tests
0ef84fc 
Modifies package instance tests to remove package_uuid fields,
and adds tests for package instance/manifest type uniqueness.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Add proper package_uuid
88257a5 
Adds package_uuid as the pURL string for the package instance,
with it's UUID as a qualifier in the pURL.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra AyanSinhaMahapatra mentioned this pull request Feb 7, 2022
Merged
4 tasks
@pombredanne
Copy link
Member

pombredanne commented Feb 7, 2022

re: #2825 (comment)

I was having some problems with the package_uuid field as it had a UUID that was different for each test run. So I forgot to make this False, now this is resolved. I've also modified the check_json_scan to accept a new argument remove_package_uuid which removes the UUID qualifier from the package_uuid pURL string.

There are three ways to handle these cases:

  1. ignore it as you have done but you need to have at least one test case that would not ignore the UUID to ensure this is generated correctly.
  2. use mocks in tests so that you have a stable mocked UUID created and do not need to ignore it
  3. add something like an extra _test=True argument to the tested function(s) that is called only in tests and that creates a static UUID that is well known for testing.

The more generic solution is 2. ... use mock.

@AyanSinhaMahapatra
Update tests and expectations for python package instance
2ca018b 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Activate package instance creation for others
89b8302 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Update test expectations for top level packages
1301612 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Remove obsolete root_path
c4c6006 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Fix package instance creation bugs
35529b7 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Update test expectaions for package instances
3c86b7c 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Add for_packages file level attribute #2693
3f93f5e 
Adds `for_packages` attribute to files to have `package_uuid` fields for the
package instance this file belongs to.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Update test expectations with for_package #2693
beeef5a 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Modify "package_manifests" to "package_data" #2843
28a0121 
Modify and rename the resource-level attribute added for a package scan
to "package_data", replacing "package_manifests". Also replaces
"package_manifest_paths" in top-level "packages" to "package_data_paths".
Changes API function get_package_manifests to get_package_data.
Also renames other variables with similar naming.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Add changes in test data for "package_data" renaming #2843
83c6588 
Regenate tests after "package_manifests" -> "package_data" renaming
and add the corresponding chages to the test files.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Add top-level dependencies list #2843
00d4f71 
Adds DependencyInstance to packagedcode models.
Adds top-level dependencies list for DependencyInstance objects.
Adds DependencyInstance creation to process_codebase in packagedcode
plugin.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Modifies test expectations #2843
abc807e 
Regenerates test expectations with top-level `dependencies` list.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Modify remove_instance_uuid option to include dependencies
3324af8 
Modify `remove_package_uuid` flag to `remove_instance_uuid` to
include dependencies.
Add `ignore_instance_uuid` attribute and a new test that has UUIDs
in the test file.
Add XFAIL for failing test.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Merge branch 'develop' into 2098-top-level-packages
498080b
@AyanSinhaMahapatra
Fix test failures from merging develop
7497407 
Add node_modules as ignored paths for NPM package instance creation.
Regenerate test expectations after merging from develop.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Fix maven and nuget bugs
0f57bce 
Adds fix for nuget and maven bugs which were in the way of activating
their respective package instance creation.

Also fixes URL creation for lockfiles.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the 2098-top-level-packages branch from 2366754 to 32c160b Compare March 3, 2022 13:16
@AyanSinhaMahapatra
Fix package checksum reporting
9688dfc 
Package checksums were being wrongly ignored in package instance
data update from detected package data, this commit fixes that.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Fix dparse2 pypi detection bug
b6bb763 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the 2098-top-level-packages branch from 32c160b to b6bb763 Compare March 3, 2022 14:37
@AyanSinhaMahapatra
Update Package models names
c5a0059 
Update Package->PackageData, PackageInstance->Package and
PackageData->PackageDataFile for consistency. Also rename
"package_data_paths" to "package_data_files" and "is_package_data"
to "is_package_data_file".

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Remove redundant dependency info from packages
f2f998f 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Merge branch 'develop' into 2098-top-level-packages
facad19 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Update build_gradle.py to new model names
527e1dd 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Fix type/class ambiguity in package class lists
8abfe18 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Remove "root_path" package data attribute
c0475ea 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Update CHANGELOG with package detection changes
d4d54a2 
This updates the CHANGELOG with details about specific changes
in package detection, in the API and the data model.
Also bumps the scancode data format to 3.0.0 as there are
major data format changes.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Modify DependentPackage Atrributes
376abc6 
Rename "requirement" to "extracted_requirement" and add new attribute
"resolved_package".

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the 2098-top-level-packages branch from 7c95531 to 376abc6 Compare March 4, 2022 21:16
@AyanSinhaMahapatra
Copy link
Member Author

AyanSinhaMahapatra commented Mar 4, 2022

@pombredanne This is ready!

pombredanne
pombredanne approved these changes Mar 5, 2022
View reviewed changes
Copy link
Member

@pombredanne pombredanne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks! merging now.

@pombredanne pombredanne merged commit e080f83 into develop Mar 5, 2022
@pombredanne pombredanne deleted the 2098-top-level-packages branch March 5, 2022 05:50
This was referenced Mar 10, 2022
Closed
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JonoYang JonoYang JonoYang left review comments

@pombredanne pombredanne pombredanne approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@AyanSinhaMahapatra @pombredanne @JonoYang