Skip to content

feat: catalog CPAN distributions installed by perl clients - #5131

Merged
wagoodman merged 3 commits into
mainfrom
add-perl-capabilities/cpan-cataloging
Sep 30, 2026
Merged

wagoodman merged 3 commits into
mainfrom
add-perl-capabilities/cpan-cataloging

Conversation

@wagoodman

@wagoodman wagoodman commented Jul 31, 2026 •

Copy link
Copy Markdown
Contributor

Adds a cataloger for CPAN distributions installed by perl clients, along with a new perl language and cpan package type.

Two catalogers:

  • perl-cpan-installed-cataloger: reads .meta/*/install.json (cpanm, cpm, carton) and auto/**/.packlist (ExtUtils::MakeMaker, Module::Build, CPAN.pm). Covers system installs, local-lib, and carton trees.
  • perl-cpan-meta-cataloger: reads META.json / META.yml from unpacked releases. Only directories with a sibling MANIFEST are considered, which excludes source checkouts.

Output:

  • packages are keyed on the distribution (e.g. libwww-perl), not the module (LWP), since that is what CPAN and advisory data use
  • purl is pkg:cpan/<distribution>@<version>, with an author qualifier when the PAUSE author is known
  • new metadata types: cpan-distribution (installed) and cpan-unpacked-release (release on disk)
  • packlist file paths are reported as owned files, unfiltered
  • JSON schema bumped to 16.1.11

Version resolution for packlists:

  • read from perllocal.pod first, matching on module name and the longest installed into libdir that prefixes the packlist path (so a second perl on the image doesn't supply the version)
  • falls back to scraping $VERSION from the main .pm, including package Foo v1.0.0;, $Foo::Bar::VERSION, and qv() forms
  • runtime-computed versions are not evaluated; those packages are reported with no version

Worth knowing:

  • a distribution can show up twice at different versions. e.g. libwww-perl 5.836 bundles and overwrites HTTP-Message and friends, so both versions really are on disk
  • build leftovers under ~/.cpanm/work and ~/.cpan/build are skipped, otherwise every distribution on an uncleaned image would be reported twice
  • META.yml is supported since ~43% of CPAN releases ship without META.json. Unparseable META.yml files are skipped, not fatal

Not covered:

  • modules installed via distro packages (deb/rpm/apk already report these)
  • core and dual-life modules bundled with the interpreter
  • vendored trees, App::FatPacker output, and PAR archives

Closes #1906 (only for cpan, not ppm)

@oss-housekeeper

Copy link
Copy Markdown

Schema Change Detection

New Schemas

  • schema/json/schema-16.1.11.json

@wagoodman wagoodman changed the title add perl capabilities/cpan cataloging feat: catalog CPAN distributions installed by perl clients Jul 31, 2026
@wagoodman
wagoodman force-pushed the add-perl-capabilities/cpan-cataloging branch 2 times, most recently from fda4746 to 1827ce2 Compare August 4, 2026 15:31
@wagoodman
wagoodman force-pushed the add-perl-capabilities/cpan-cataloging branch from 1827ce2 to 0f07da8 Compare September 29, 2026 15:00
@wagoodman
wagoodman marked this pull request as ready for review September 29, 2026 15:08
@wagoodman
wagoodman force-pushed the add-perl-capabilities/cpan-cataloging branch 3 times, most recently from 4119cf5 to bd06995 Compare September 30, 2026 14:33
Adds `perl` as a language and `cpan` as a package type, with two catalogers behind them.

- `perl-cpan-installed-cataloger` reads `.meta/*/install.json` (cpanm, cpm, carton) and `auto/**/.packlist` (anything installed through `ExtUtils::MakeMaker` or `Module::Build`, including CPAN.pm). Both globs are unanchored, so a local-lib or carton application tree is found the same as a system install.
- `perl-cpan-meta-cataloger` reads an unpacked release's own `META.json` or `META.yml`, gated on a sibling `MANIFEST` so source checkouts are ignored.

Packages are keyed on the **distribution**, not the module, because that is what CPAN, MetaCPAN and the advisory data all use. `LWP.pm` belongs to `libwww-perl` and reporting it as `LWP` would make every advisory for it unreachable. The distribution name comes from `install.json`'s `dist` field, and from the `auto/` path for packlists.

purls are `pkg:cpan/<distribution>@<version>`, with the PAUSE author added as an `author` qualifier when the evidence carries it. Metadata comes in two types: `cpan-distribution` for installed evidence and `cpan-unpacked-release` for a release sitting on disk. The tiers differ in more than a name, so a consumer should not have to string-match a cataloger name to tell them apart: an unpacked release has no PAUSE path and therefore no author and no file list, and "installed and loadable by the interpreter" is a materially stronger claim than "a source tree exists here".

A packlist's version comes from `perllocal.pod`. EUMM writes it and the packlist from the same variables in the same install target, so the `auto/` path segments and the perllocal `Module` name are the same key, and the recorded `VERSION` is what was evaluated at build time rather than what can be read back statically. Scraping `$VERSION` out of the main `.pm` is the fallback, since `NO_PERLLOCAL` suppresses the file and Module::Build never writes one. Three rules pick the stanza: dashed module name, longest `installed into` libdir that prefixes the packlist path, and last match wins because the file is append-only. Without the libdir rule a second perl on the image silently supplies the version.

Where scraping is the fallback, it reads more than a plain `our $VERSION = '...'`. A version declared in the package statement (`package Foo::Bar v1.0.0;`) counts, which matters because a distribution can declare it that way and carry no `$VERSION` at all: `CPAN::02Packages::Search` is one, and without this it reports no version and matches nothing. Fully qualified `$Foo::Bar::VERSION` counts too, which is what older Dist::Zilla emitted and what real `JSON::PP` 2.27300 still carries. `qv('1.2.3')` is handled. A version computed at runtime is not, and those are reported without one rather than guessed at.

Packlist entries are parsed the way `ExtUtils::Packlist` writes them: a line can carry space-separated metadata after the path (`/path/to/File.pm type=file`), which is what `installperl` produces, so the suffix is stripped rather than the line being cut at its first space.

`META.yml` is read alongside `META.json`, because about 43% of current CPAN releases ship no `META.json` and they skew old, which is where the advisories are. Where both sit in one directory the `META.json` wins. A `META.yml` a strict parser rejects skips that directory rather than failing the scan, which is routine rather than defensive for pre-spec releases.

A packlist is literally a file list, so its paths are surfaced through `pkg.FileOwner`. They are reported as recorded, unfiltered: a path the packlist claims and the filesystem lacks means the file was removed or overwritten out from under the installer, which is worth seeing rather than hiding.

Build leftovers under `~/.cpanm/work` and `~/.cpan/build` are skipped. They genuinely are unpacked release tarballs, `MANIFEST` included, so the `MANIFEST` gate admits them and a path exclusion is the only signal available. Without it every distribution on an image that did not clean up is reported twice. The cost is that a tarball deliberately kept under `~/.cpan/build` stops being reported.

Two behaviors that look wrong but are not:

- a distribution can be reported twice at different versions. `libwww-perl` 5.836 bundles `HTTP-Date`, `HTTP-Message` and `LWP-MediaTypes`, and installing it over the modern standalone releases overwrites their `.pm` files. Both versions are genuinely present, so the merge refuses to pair disagreeing versions rather than hiding one.
- a distribution with no readable version is reported without one rather than dropped, so it stays visible.

The coverage boundary is stated in full in the `package perl` doc comment. In short: modules installed from distro packages are out of scope, since packagers strip CPAN metadata with `NO_PACKLIST` and deb, rpm and apk already report them. Core and dual-life distributions bundled with the interpreter have no coverage, because nothing on disk carries their versions; the interpreter itself is reported separately. Vendored trees, `App::FatPacker` output and PAR archives are invisible, because what they carry is module identity with no offline map to a distribution. A packlist-derived name is the installer's `NAME` and may not be the distribution name, which is left to the vulnerability data to resolve.

One correction worth calling out, since it is easy to arrive at twice: a sibling `MANIFEST` is the only thing separating an unpacked release from a source checkout. An earlier version of the guard also rejected any tree containing a `dist.ini`, on the theory that it marked a Dist::Zilla source tree. dzil ships `dist.ini` *inside* the tarballs it builds and lists it in the generated `MANIFEST`, so that exclusion was silently skipping real releases, `URI` among them.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
The CPAN fixture added to `image-pkg-coverage` (`URI-5.35`) is shared with the CLI suite through a symlink, so the squashed package count goes from 42 to 43.

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
@wagoodman
wagoodman force-pushed the add-perl-capabilities/cpan-cataloging branch from bd06995 to 750daf9 Compare September 30, 2026 15:01
@willmurphyscode

Copy link
Copy Markdown
Contributor

I think there's still a small version handling bug:

CPAN distributions are reported under their distvname (and sometimes duplicated)

distributionName (parse_install_json.go:66-80) takes the package name from install.json by trimming "-" + version off dist. But cpanm doesn't write the same version into both fields:

  • dist is the release's distvname.
  • version is the main module's $VERSION.

cpanm's save_meta writes version => $provides->{$module_name}{version} || $dist->{version} (source: Menlo/CLI/Compat.pm, which cpanm 1.7 is built from). When the two strings differ, the trim fails and the whole distvname becomes the package name. The comment on that fallback says no installer writes such a record, but cpanm does.

This isn't rare. 3,573 distributions in PAUSE's current 02packages index have a main-module version string that differs from the release version, mostly in format only (2.9.0 vs 2.009000). That still breaks an exact-string trim. TRIAL releases always hit it.

FROM perl:5.40-slim

# TRIAL release: dist=Try-Tiny-0.26-TRIAL, version=0.26. Dropped from the live mirrors, so fetch it
# from backpan; a PAUSE path through a mirror is what makes cpanm write install.json.
RUN cpanm --notest --mirror https://backpan.perl.org --mirror-only E/ET/ETHER/Try-Tiny-0.26-TRIAL.tar.gz

# same number written two ways: dist=Carp-Assert-More-2.9.0, version=2.009000
RUN cpanm --notest P/PE/PETDANCE/Carp-Assert-More-2.9.0.tar.gz

# different numbers: dist=Class-Rebirth-1.003, version=1.000
RUN cpanm --notest A/AH/AHERNIT/Class-Rebirth-1.003.tar.gz
$ docker build -t cpan-distvname-repro .
$ go run ./cmd/syft docker:cpan-distvname-repro --select-catalogers perl -q
NAME                    VERSION   TYPE
App-cpanminus           1.7049    cpan    # from the base image
Carp-Assert-More        2.9.0     cpan    # correct (.packlist + perllocal.pod)
Carp-Assert-More-2.9.0  2.009000  cpan    # duplicate of the line above, with the distvname as its name (install.json)
Class-Rebirth           1.003     cpan    # correct (.packlist + perllocal.pod)
Class-Rebirth-1.003     1.000     cpan    # duplicate of the line above, with the distvname as its name (install.json)
Exporter-Tiny           1.006003  cpan    # dependency of List-MoreUtils
IO-Socket-SSL           2.099     cpan    # from the base image
List-MoreUtils          0.430     cpan    # dependency of Class-Rebirth
Net-SSLeay              1.96      cpan    # from the base image
Try-Tiny-0.26-TRIAL     0.26      cpan    # should be Try-Tiny; no duplicate because both records said 0.26 and merged

The ordinary releases are duplicated because the packlist record gets the release version, the install.json record gets the module version, and the merge won't pair records whose versions disagree. MetaCPAN names that release distribution: Try-Tiny, version: 0.26, maturity: developer (API): -TRIAL is a maturity flag, not part of the name.

Suggested fix: take both name and version from dist. Split at the last - followed by v?\d and drop -TRIAL, as CPAN::DistnameInfo does. Existing cases keep working (libwww-perl-6.83 → libwww-perl, CPAN-02Packages-Search-v1.0.0 → CPAN-02Packages-Search), and the install.json and packlist records then agree and merge into one package.

…dist

cpanm writes the main module's `$VERSION` into `version` and the release distvname into `dist`, and the two often disagree (`Carp-Assert-More-2.9.0` records `2.009000`). Trimming `version` off `dist` failed on those, so the whole distvname became the package name, and the module version never paired with the packlist record for the same install, which reported the distribution twice.

- name and version are now split out of `dist` at the last `-` followed by a version, as CPAN::DistnameInfo does
- a `-TRIAL` suffix is dropped, so `Try-Tiny-0.26-TRIAL` reports as `Try-Tiny` `0.26`
- the PAUSE path fallback uses the same split

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
@wagoodman
wagoodman merged commit 8d5567c into main Sep 30, 2026
25 of 26 checks passed
@wagoodman
wagoodman deleted the add-perl-capabilities/cpan-cataloging branch September 30, 2026 18:15
@wagoodman wagoodman added enhancement New feature or request and removed enhancement New feature or request labels Sep 30, 2026
spiffcs added a commit that referenced this pull request Sep 30, 2026
* main: (91 commits)
  feat: catalog CPAN distributions installed by perl clients (#5131)
  fix(binary): correct traefik version detection on big-endian arches (#5281)
  Fix hang when scanning a Java resource adapter (.rar) file (#5348)
  chore(deps): update SPDX license list (#5315)
  fix: match opensource.org license URLs across SPDX URL forms (#5361)
  Speed up unit tests in CI (#5360)
  Fix missing file hashes when running on windows system (#5341)
  Stop reading the mount table for every single-file source (#5257)
  fix: conan expat CPE (#5259)
  fix(spdx): write a name for the root package when the source has none (#5349)
  fix(python): link uv.lock dependents to the locked version they name (#5351)
  fix: suppress misleading upstream CPE for PHP extension binaries (#5102)
  fix: report scanner errors and raise the line cap in metadata parsers (#5353)
  Harden cataloger parsers against truncated input (do not panic) (#5355)
  fix(cpe): bound CPE candidate generation for pathological package metadata (#5356)
  test(golang): make the UPX fixture exercise loader padding (#5354)
  fix(golang): account for UPX's loader padding when unpacking Go binaries (#5347)
  test(snap): bound the kernel changelog bomb fixture by ratio, not a fixed 1MB (#5352)
  test(debian): bound the bomb fixture by ratio, not a fixed 1MB (#5350)
  fix(ai): handle malformed GGUF headers without panicking (#5345)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add package cataloger for perl ecosystem

3 participants