Migrate Npm importer to aboutcode pipeline

[keshav-space]

• Reference Use the SCIO Pipeline Mechanism for Importers/Improvers #1509

[TG1999]

@keshav-space is there anything remaining here, can we go ahead and review / merge this ?

[keshav-space]

is there anything remaining here, can we go ahead and review / merge this ?

@TG1999 I will add a small data migration to change the value of created_by field on the old npm advisory to our new pipeline_id, and then ping you for review/merge.

[TG1999]

@keshav-space sure, let me know when it's ready for review

[keshav-space]

sure, let me know when it's ready for review

@TG1999 ready for your review :)

[TG1999]

ping @TG1999

[TG1999]

LGTM!

[keshav-space]

Log from a clean run.

❯ python3 manage.py import npm_importer
Importing data using npm_importer
INFO 2024-09-19 13:54:21.715 Pipeline [NpmImporterPipeline] starting
INFO 2024-09-19 13:54:21.715 Step [clone] starting
INFO 2024-09-19 13:54:21.715 Cloning `git+https://github.com/nodejs/security-wg`
INFO 2024-09-19 13:54:23.075 Step [clone] completed in 1 seconds
INFO 2024-09-19 13:54:23.076 Step [collect_and_store_advisories] starting
INFO 2024-09-19 13:54:23.157 Progress: 10% (47/468) ETA: 1 seconds
INFO 2024-09-19 13:54:23.251 Progress: 20% (94/468) ETA: 1 seconds
INFO 2024-09-19 13:54:23.327 Progress: 30% (141/468) ETA: 1 seconds
INFO 2024-09-19 13:54:23.380 Successfully collected 176 advisories
INFO 2024-09-19 13:54:23.380 Step [collect_and_store_advisories] completed in 0 seconds
INFO 2024-09-19 13:54:23.380 Step [import_new_advisories] starting
INFO 2024-09-19 13:54:23.381 Importing 176 new advisories
INFO 2024-09-19 13:54:23.567 Progress: 10% (18/176) ETA: 2 seconds
INFO 2024-09-19 13:54:23.738 Progress: 20% (36/176) ETA: 1 seconds
INFO 2024-09-19 13:54:23.925 Progress: 30% (53/176) ETA: 1 seconds
INFO 2024-09-19 13:54:24.182 Progress: 40% (71/176) ETA: 1 seconds
Failed to get exact purls for: AffectedPackage(package=PackageURL(type='npm', namespace=None, name='fury-adapter-swagger', version=None, qualifiers={}, subpath=None), affected_version_range=NpmVersionRange(constraints=(VersionConstraint(comparator='>=', version=SemverVersion(string='0.2.0')), VersionConstraint(comparator='>=', version=SemverVersion(string='0.8.0-pre')), VersionConstraint(comparator='<', version=SemverVersion(string='0.8.0')), VersionConstraint(comparator='>=', version=SemverVersion(string='0.8.0')), VersionConstraint(comparator='<', version=SemverVersion(string='0.8.1')), VersionConstraint(comparator='<=', version=SemverVersion(string='0.9.6')))), fixed_version=None) with error: InvalidConstraintsError("Invalid constraints sequence: ('>= 0.2.0','>= 0.8.0-pre') in [VersionConstraint(comparator='>=', version=SemverVersion(string='0.2.0')), VersionConstraint(comparator='>=', version=SemverVersion(string='0.8.0-pre')), VersionConstraint(comparator='<', version=SemverVersion(string='0.8.0')), VersionConstraint(comparator='>=', version=SemverVersion(string='0.8.0')), VersionConstraint(comparator='<', version=SemverVersion(string='0.8.1')), VersionConstraint(comparator='<=', version=SemverVersion(string='0.9.6'))]")
INFO 2024-09-19 13:54:24.419 Progress: 50% (88/176) ETA: 1 seconds
INFO 2024-09-19 13:54:24.616 Progress: 60% (106/176) ETA: 1 seconds
Failed to get exact purls for: AffectedPackage(package=PackageURL(type='npm', namespace=None, name='node.extend', version=None, qualifiers={}, subpath=None), affected_version_range=NpmVersionRange(constraints=(VersionConstraint(comparator='<', version=SemverVersion(string='1.1.7')), VersionConstraint(comparator='<', version=SemverVersion(string='2.0.1')))), fixed_version=None) with error: InvalidConstraintsError("Invalid constraints sequence: ('< 1.1.7','< 2.0.1') in [VersionConstraint(comparator='<', version=SemverVersion(string='1.1.7')), VersionConstraint(comparator='<', version=SemverVersion(string='2.0.1'))]")
INFO 2024-09-19 13:54:24.798 Progress: 70% (124/176) ETA: 1 seconds
INFO 2024-09-19 13:54:24.975 Progress: 80% (141/176)
INFO 2024-09-19 13:54:25.182 Progress: 90% (159/176)
INFO 2024-09-19 13:54:25.337 Progress: 100% (176/176)
Failed to get exact purls for: AffectedPackage(package=PackageURL(type='npm', namespace=None, name='electron', version=None, qualifiers={}, subpath=None), affected_version_range=NpmVersionRange(constraints=(VersionConstraint(comparator='<', version=SemverVersion(string='2.0.18')), VersionConstraint(comparator='<', version=SemverVersion(string='3.0.16')), VersionConstraint(comparator='<', version=SemverVersion(string='3.1.6')), VersionConstraint(comparator='<', version=SemverVersion(string='4.0.8')), VersionConstraint(comparator='<', version=SemverVersion(string='5.0.0-beta.5')))), fixed_version=None) with error: InvalidConstraintsError("Invalid constraints sequence: ('< 2.0.18','< 3.0.16') in [VersionConstraint(comparator='<', version=SemverVersion(string='2.0.18')), VersionConstraint(comparator='<', version=SemverVersion(string='3.0.16')), VersionConstraint(comparator='<', version=SemverVersion(string='3.1.6')), VersionConstraint(comparator='<', version=SemverVersion(string='4.0.8')), VersionConstraint(comparator='<', version=SemverVersion(string='5.0.0-beta.5'))]")
INFO 2024-09-19 13:54:25.363 Successfully imported 176 new advisories
INFO 2024-09-19 13:54:25.364 Step [import_new_advisories] completed in 2 seconds
INFO 2024-09-19 13:54:25.364 Step [clean_downloads] starting
INFO 2024-09-19 13:54:25.364 Removing cloned repository
INFO 2024-09-19 13:54:25.371 Step [clean_downloads] completed in 0 seconds
INFO 2024-09-19 13:54:25.371 Pipeline completed in 4 seconds