Archived license files not included in (all) reports when using the `ARTIFACT` source code origin

[sasa-boros-cp]

Describe the bug

The license files contained in the source artifacts, found by the use of a custom pattern, are not included in the reported notice even though they are found by the archiver during the scan. Not just the plain text, but also PDF and HTML notices.

Steps to reproduce the behavior:

1. Provide custom configuration of licenseFilePatterns, for e.g. include **/LICENSE*
2. Run a scan on the repository that contains dependencies with source artifacts that include LICENSE files, in META-INF for e.g. - default scan already has archiver enabled
3. Run a report on the scan result
4. Check notice file - it doesn't include the file license info

Expected behavior

File licenses found by the archiver should be included in the final notice.

Console/log output

During scanning:

DEBUG org.ossreviewtoolkit.model.utils.FileArchiver - Adding 'META-INF/LICENSE' to archive.

I see that the LICENSE was found and included.

During reporting:

INFO  org.ossreviewtoolkit.model.utils.FileArchiver - Unarchived data for ArtifactProvenance(sourceArtifact=RemoteArtifact(url=https://repo.maven.apache.org/maven2/org/apache/tika/tika-core/3.2.0/tika-core-3.2.0-sources.jar, hash=Hash(value=328e6b6cbc9be6d5b31e38a34a3a90f9981b280a, algorithm=SHA-1))) to '/tmp/ort-LicenseInfoResolver-archive15339033700059155317' in 4.312064ms.

I can see the unarchiving happening, but no files getting picked up.

Environment

• ORT version: ort-minimal:61.0.0
• Java version: 21
• OS: linux/amd64 (Docker)

config.yml:

ort:
  forceOverwrite: true
  licenseFilePatterns:
    licenseFilenames: [  "copying*",
                         "COPYING*",
                         "copyright",
                         "licence*",
                         "license*",
                         "*.licence",
                         "*.license",
                         "**/LICENCE*",
                         "**/LICENSE*",
                         "unlicence",
                         "unlicense"]
    patentFilenames: [ 'patents' ]
    otherLicenseFilenames: [ 'readme*' ]
  analyzer:
    skipExcluded: true
    enabled_package_managers: [ GradleInspector ]
  downloader:
    sourceCodeOrigins: [ARTIFACT, VCS]
    skipExcluded: true
  scanner:
    archive:
      enabled: true
      fileStorage:
        localFileStorage:
          directory: ${user.home}/.ort/scanner/archive
          compression: false
    # We don't want excluded stuff to be scanned, e.g. dev or test dependencies
    skipExcluded: true
    storages:
      local:
        backend:
          localFileStorage:
            directory: ${user.home}/.ort/scanner/results
            compression: false
      
    storageReaders: [local]
    storageWriters: [local]
  reporter:
    config:
      SpdxDocument:
        options:
          output.file.formats: json
          file.information.enabled: true
      CycloneDx:
        options:
          output.file.formats: json

Additional context

I am running in Docker, with a common volume for each ORT workflow step. I can see that the archiver archives the files in the right directory (during scan), as well as that unarchiving is happening during the report step. In the template, the package.licenseFiles is empty, and therefore no license is included in the final notice.

The Docker commands I am using:

Scan:

docker run \
  --rm \
  --name ort-scan-backend \
  --platform linux/amd64 \
  -v "$PWD"/:/workingdir \
  -v "$repo_abs_path":/repo \
  -v "$PWD"/.gitconfig:/home/ort/.gitconfig \
  -v ~/.netrc:/home/ort/.netrc:ro \
  -v ~/.gradle/gradle.properties:/home/ort/.gradle/gradle.properties:ro \
  -v "$ORT_DOCKER_VOLUME":/home/ort/.ort \
  -e ORT_CONFIG_DIR=/workingdir/config-backend \
  "$ORT_DOCKER_IMAGE" --debug scan \
  -i /repo/ort/results/backend/analyzer-result.yml \
  -o /repo/ort/results/backend \
  | tee scan-backend.log

Report:

docker run \
  --rm \
  --name ort-report-backend \
  --platform linux/amd64 \
  -v "$PWD"/:/workingdir \
  -v "$repo_abs_path":/repo \
  -v "$PWD"/.gitconfig:/home/ort/.gitconfig \
  -v ~/.netrc:/home/ort/.netrc:ro \
  -v ~/.gradle/gradle.properties:/home/ort/.gradle/gradle.properties:ro \
  -v "$ORT_DOCKER_VOLUME":/home/ort/.ort \
  -e ORT_CONFIG_DIR=/workingdir/config-backend \
  "$ORT_DOCKER_IMAGE" --debug report \
  -f PlainTextTemplate,StaticHtml,WebApp,PdfTemplate,HtmlTemplate,CycloneDx,SpdxDocument \
  -O PlainTextTemplate=template.path=/workingdir/config-general/notice.txt.ftl,/workingdir/config-general/notice.json.ftl \
  --license-classifications-file /workingdir/config-general/license-classifications.yml \
  -i /repo/ort/results/backend/evaluation-result.yml \
  -o /repo/ort/results/backend/reports/ \
  | tee report-backend.log

[sschuberth]

Whether verbatim / archived license fields are added to a report, or whether "normalized" license texts are used, depends on the reporter implementation. Currently, only the NOTICE_DEFAULT.ftl and disclosure_document.ftl reporter templates make use of archived license files:

ort/plugins/reporters/freemarker/src/main/resources/templates/plain-text/NOTICE_DEFAULT.ftl

Lines 74 to 75 in 4a7d58a

	[#-- List the content of archived license files and associated copyrights. --]
	[#list package.licenseFiles.files as licenseFile]

ort/plugins/reporters/asciidoc/src/main/resources/templates/asciidoc/disclosure_document.ftl

Lines 93 to 94 in 490a641

	[#-- List the content of archived license files and associated copyrights. --]
	[#list package.licenseFiles.files as licenseFile]

This has been a source of confusion for user in the past, and I agree it should be changed. But I'd like to get #10479 merged before that.

[sasa-boros-cp]

Thank you for your fast response!

We also tried using the NOTICE_DEFAULT.ftl and the license files are not included there as well, i.e., [#list package.licenseFiles.files as licenseFile] produces no results. We also use the same statement in our custom templates.

When I switch the order of source code origins: sourceCodeOrigins: [VCS, ARTIFACT], the licenses are correctly included in the report because they are found by the default patterns in the source code. When I include a custom pattern **/LICENSE*, those are never included even though they are found by the archiver, and I can see the found files in the associated archives.

I looked at the code, and my suspicion was

ort/plugins/commands/reporter/src/main/kotlin/ReportCommand.kt

Line 257 in 5d7b086

licenseFilePatterns = LicenseFilePatterns.getInstance()

i.e., that the default patterns are used and not custom ones provided by the config.yml licenseFilePatterns.

[sschuberth]

When I switch the order of source code origins: sourceCodeOrigins: [VCS, ARTIFACT], the licenses are correctly included in the report because they are found by the default patterns in the source code.

Hmm, so have you manually confirmed the license files that are matched for the VCS origin not to be present for the ARTIFACT origin?

I looked at the code, and my suspicion was

I can see how that code reads misleading, but it is actually executed after

ort/cli/src/main/kotlin/OrtMain.kt

Line 154 in 298a964

LicenseFilePatterns.configure(ortConfig.licenseFilePatterns)

which configures what LicenseFilePatterns.getInstance() returns later.

[sasa-boros-cp]

Hmm, so have you manually confirmed the license files that are matched for the VCS origin not to be present for the ARTIFACT origin?

Yes, I have confirmed that multiple times. None of the licenses get included when running with ARTIFACT origin. We have tested on a lot of different dependencies and their associated source artifacts.

After scanning, I find the archived files in scanner/archive/ folder and after unzipping the archive.zip of dependencies, I can clearly see the licenses in, for e.g., META-INF/LICENSE, but the reporter doesn't pick them up when compiling the reports.

I can see how that code reads misleading, but it is actually executed after

Ok, thanks for looking into it, didn't go that deeply into the code.

[sasa-boros-cp]

@sschuberth I used time to debug the issue and find the solution. The issue is a combination of two things:

1. The scanner's FileArchiver never gets the patterns that include the directories, just file names

Example:

    licenseFilenames: [  "copying*",
                         "COPYING*",
                         "copyright",
                         "licence*",
                         "license*",
                         "*.licence",
                         "*.license",
                         "unlicence",
                         "unlicense"]

The fix is to prepend them with /**/ in the FileArchiverConfiguration so that the scanner's archiver also archives files not in the root, line 92:

val patterns = LicenseFilePatterns.getInstance().allLicenseFilenames.map { "/**/$it" }

Now, the licenses that are not in the root are also found during the scan and are stored in archive.zip.

2. The LicenseInfoResolver only traverses the root directory of the archive to find licenses based on patterns

In the method createLicenseFileInfo, currently only the root of the archive (when unarchived) is traversed to match the patterns. All paths should be included instead, like the following:

val allFiles = archiveDir.walk().filter { it.isFile }.map {
                it.relativeTo(archiveDir).invariantSeparatorsPath
            }.toSet()

            // Collect all unique directories that contain files
            val allDirectories = allFiles.map { filePath ->
                val parentPath = filePath.substringBeforeLast('/', "")
                parentPath
            }.toSet()

            // Search for license files in all directories, not just the root
            val allLicenseFiles = pathLicenseMatcher.getApplicableLicenseFilesForDirectories(
                relativeFilePaths = allFiles,
                directories = allDirectories.toList()
            )

            // Collect license files from all directories
            allLicenseFiles.values.flatten().forEach { relativePath ->
                // Register files in `archiveDir` for deletion. Because files are deleted in reverse order than
                // registered, this will leave `archiveDir` empty to get properly deleted by the registration above.
                val file = archiveDir.resolve(relativePath).apply { deleteOnExit() }

                licenseFiles += ResolvedLicenseFile(
                    provenance = provenance,
                    licenseInfo.filter(provenance, relativePath),
                    relativePath,
                    file
                )
            }

With this fix, the license files are found during the reporter step and are therefore included in the final notice.

---

I can make a PR that solves the issue, but wanted to discuss the solution here first.