Repository navigation
feat: catalog CPAN distributions installed by perl clients - #5131
Conversation
Schema Change DetectionNew Schemas
|
fda4746 to
1827ce2
Compare
1827ce2 to
0f07da8
Compare
4119cf5 to
bd06995
Compare
Adds `perl` as a language and `cpan` as a package type, with two catalogers behind them.
- `perl-cpan-installed-cataloger` reads `.meta/*/install.json` (cpanm, cpm, carton) and `auto/**/.packlist` (anything installed through `ExtUtils::MakeMaker` or `Module::Build`, including CPAN.pm). Both globs are unanchored, so a local-lib or carton application tree is found the same as a system install.
- `perl-cpan-meta-cataloger` reads an unpacked release's own `META.json` or `META.yml`, gated on a sibling `MANIFEST` so source checkouts are ignored.
Packages are keyed on the **distribution**, not the module, because that is what CPAN, MetaCPAN and the advisory data all use. `LWP.pm` belongs to `libwww-perl` and reporting it as `LWP` would make every advisory for it unreachable. The distribution name comes from `install.json`'s `dist` field, and from the `auto/` path for packlists.
purls are `pkg:cpan/<distribution>@<version>`, with the PAUSE author added as an `author` qualifier when the evidence carries it. Metadata comes in two types: `cpan-distribution` for installed evidence and `cpan-unpacked-release` for a release sitting on disk. The tiers differ in more than a name, so a consumer should not have to string-match a cataloger name to tell them apart: an unpacked release has no PAUSE path and therefore no author and no file list, and "installed and loadable by the interpreter" is a materially stronger claim than "a source tree exists here".
A packlist's version comes from `perllocal.pod`. EUMM writes it and the packlist from the same variables in the same install target, so the `auto/` path segments and the perllocal `Module` name are the same key, and the recorded `VERSION` is what was evaluated at build time rather than what can be read back statically. Scraping `$VERSION` out of the main `.pm` is the fallback, since `NO_PERLLOCAL` suppresses the file and Module::Build never writes one. Three rules pick the stanza: dashed module name, longest `installed into` libdir that prefixes the packlist path, and last match wins because the file is append-only. Without the libdir rule a second perl on the image silently supplies the version.
Where scraping is the fallback, it reads more than a plain `our $VERSION = '...'`. A version declared in the package statement (`package Foo::Bar v1.0.0;`) counts, which matters because a distribution can declare it that way and carry no `$VERSION` at all: `CPAN::02Packages::Search` is one, and without this it reports no version and matches nothing. Fully qualified `$Foo::Bar::VERSION` counts too, which is what older Dist::Zilla emitted and what real `JSON::PP` 2.27300 still carries. `qv('1.2.3')` is handled. A version computed at runtime is not, and those are reported without one rather than guessed at.
Packlist entries are parsed the way `ExtUtils::Packlist` writes them: a line can carry space-separated metadata after the path (`/path/to/File.pm type=file`), which is what `installperl` produces, so the suffix is stripped rather than the line being cut at its first space.
`META.yml` is read alongside `META.json`, because about 43% of current CPAN releases ship no `META.json` and they skew old, which is where the advisories are. Where both sit in one directory the `META.json` wins. A `META.yml` a strict parser rejects skips that directory rather than failing the scan, which is routine rather than defensive for pre-spec releases.
A packlist is literally a file list, so its paths are surfaced through `pkg.FileOwner`. They are reported as recorded, unfiltered: a path the packlist claims and the filesystem lacks means the file was removed or overwritten out from under the installer, which is worth seeing rather than hiding.
Build leftovers under `~/.cpanm/work` and `~/.cpan/build` are skipped. They genuinely are unpacked release tarballs, `MANIFEST` included, so the `MANIFEST` gate admits them and a path exclusion is the only signal available. Without it every distribution on an image that did not clean up is reported twice. The cost is that a tarball deliberately kept under `~/.cpan/build` stops being reported.
Two behaviors that look wrong but are not:
- a distribution can be reported twice at different versions. `libwww-perl` 5.836 bundles `HTTP-Date`, `HTTP-Message` and `LWP-MediaTypes`, and installing it over the modern standalone releases overwrites their `.pm` files. Both versions are genuinely present, so the merge refuses to pair disagreeing versions rather than hiding one.
- a distribution with no readable version is reported without one rather than dropped, so it stays visible.
The coverage boundary is stated in full in the `package perl` doc comment. In short: modules installed from distro packages are out of scope, since packagers strip CPAN metadata with `NO_PACKLIST` and deb, rpm and apk already report them. Core and dual-life distributions bundled with the interpreter have no coverage, because nothing on disk carries their versions; the interpreter itself is reported separately. Vendored trees, `App::FatPacker` output and PAR archives are invisible, because what they carry is module identity with no offline map to a distribution. A packlist-derived name is the installer's `NAME` and may not be the distribution name, which is left to the vulnerability data to resolve.
One correction worth calling out, since it is easy to arrive at twice: a sibling `MANIFEST` is the only thing separating an unpacked release from a source checkout. An earlier version of the guard also rejected any tree containing a `dist.ini`, on the theory that it marked a Dist::Zilla source tree. dzil ships `dist.ini` *inside* the tarballs it builds and lists it in the generated `MANIFEST`, so that exclusion was silently skipping real releases, `URI` among them.
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The CPAN fixture added to `image-pkg-coverage` (`URI-5.35`) is shared with the CLI suite through a symlink, so the squashed package count goes from 42 to 43. Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
bd06995 to
750daf9
Compare
|
I think there's still a small version handling bug: CPAN distributions are reported under their distvname (and sometimes duplicated)
cpanm's This isn't rare. 3,573 distributions in PAUSE's current FROM perl:5.40-slim
# TRIAL release: dist=Try-Tiny-0.26-TRIAL, version=0.26. Dropped from the live mirrors, so fetch it
# from backpan; a PAUSE path through a mirror is what makes cpanm write install.json.
RUN cpanm --notest --mirror https://backpan.perl.org --mirror-only E/ET/ETHER/Try-Tiny-0.26-TRIAL.tar.gz
# same number written two ways: dist=Carp-Assert-More-2.9.0, version=2.009000
RUN cpanm --notest P/PE/PETDANCE/Carp-Assert-More-2.9.0.tar.gz
# different numbers: dist=Class-Rebirth-1.003, version=1.000
RUN cpanm --notest A/AH/AHERNIT/Class-Rebirth-1.003.tar.gz$ docker build -t cpan-distvname-repro .
$ go run ./cmd/syft docker:cpan-distvname-repro --select-catalogers perl -q
NAME VERSION TYPE
App-cpanminus 1.7049 cpan # from the base image
Carp-Assert-More 2.9.0 cpan # correct (.packlist + perllocal.pod)
Carp-Assert-More-2.9.0 2.009000 cpan # duplicate of the line above, with the distvname as its name (install.json)
Class-Rebirth 1.003 cpan # correct (.packlist + perllocal.pod)
Class-Rebirth-1.003 1.000 cpan # duplicate of the line above, with the distvname as its name (install.json)
Exporter-Tiny 1.006003 cpan # dependency of List-MoreUtils
IO-Socket-SSL 2.099 cpan # from the base image
List-MoreUtils 0.430 cpan # dependency of Class-Rebirth
Net-SSLeay 1.96 cpan # from the base image
Try-Tiny-0.26-TRIAL 0.26 cpan # should be Try-Tiny; no duplicate because both records said 0.26 and mergedThe ordinary releases are duplicated because the packlist record gets the release version, the Suggested fix: take both name and version from |
…dist cpanm writes the main module's `$VERSION` into `version` and the release distvname into `dist`, and the two often disagree (`Carp-Assert-More-2.9.0` records `2.009000`). Trimming `version` off `dist` failed on those, so the whole distvname became the package name, and the module version never paired with the packlist record for the same install, which reported the distribution twice. - name and version are now split out of `dist` at the last `-` followed by a version, as CPAN::DistnameInfo does - a `-TRIAL` suffix is dropped, so `Try-Tiny-0.26-TRIAL` reports as `Try-Tiny` `0.26` - the PAUSE path fallback uses the same split Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
* main: (91 commits) feat: catalog CPAN distributions installed by perl clients (#5131) fix(binary): correct traefik version detection on big-endian arches (#5281) Fix hang when scanning a Java resource adapter (.rar) file (#5348) chore(deps): update SPDX license list (#5315) fix: match opensource.org license URLs across SPDX URL forms (#5361) Speed up unit tests in CI (#5360) Fix missing file hashes when running on windows system (#5341) Stop reading the mount table for every single-file source (#5257) fix: conan expat CPE (#5259) fix(spdx): write a name for the root package when the source has none (#5349) fix(python): link uv.lock dependents to the locked version they name (#5351) fix: suppress misleading upstream CPE for PHP extension binaries (#5102) fix: report scanner errors and raise the line cap in metadata parsers (#5353) Harden cataloger parsers against truncated input (do not panic) (#5355) fix(cpe): bound CPE candidate generation for pathological package metadata (#5356) test(golang): make the UPX fixture exercise loader padding (#5354) fix(golang): account for UPX's loader padding when unpacking Go binaries (#5347) test(snap): bound the kernel changelog bomb fixture by ratio, not a fixed 1MB (#5352) test(debian): bound the bomb fixture by ratio, not a fixed 1MB (#5350) fix(ai): handle malformed GGUF headers without panicking (#5345) ...
Adds a cataloger for CPAN distributions installed by perl clients, along with a new
perllanguage andcpanpackage type.Two catalogers:
perl-cpan-installed-cataloger: reads.meta/*/install.json(cpanm, cpm, carton) andauto/**/.packlist(ExtUtils::MakeMaker, Module::Build, CPAN.pm). Covers system installs, local-lib, and carton trees.perl-cpan-meta-cataloger: readsMETA.json/META.ymlfrom unpacked releases. Only directories with a siblingMANIFESTare considered, which excludes source checkouts.Output:
libwww-perl), not the module (LWP), since that is what CPAN and advisory data usepkg:cpan/<distribution>@<version>, with anauthorqualifier when the PAUSE author is knowncpan-distribution(installed) andcpan-unpacked-release(release on disk)16.1.11Version resolution for packlists:
perllocal.podfirst, matching on module name and the longestinstalled intolibdir that prefixes the packlist path (so a second perl on the image doesn't supply the version)$VERSIONfrom the main.pm, includingpackage Foo v1.0.0;,$Foo::Bar::VERSION, andqv()formsWorth knowing:
libwww-perl5.836 bundles and overwritesHTTP-Messageand friends, so both versions really are on disk~/.cpanm/workand~/.cpan/buildare skipped, otherwise every distribution on an uncleaned image would be reported twiceMETA.ymlis supported since ~43% of CPAN releases ship withoutMETA.json. UnparseableMETA.ymlfiles are skipped, not fatalNot covered:
App::FatPackeroutput, and PAR archivesCloses #1906 (only for cpan, not ppm)