Merge branch 'develop' into add-license-detection

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Author: AyanSinhaMahapatra

.github/workflows/about-files-ci.yml

@@ -2,6 +2,9 @@ name: CI About Files
 
 on: [push, pull_request]
 
+permissions:
+  contents: read # to fetch code (actions/checkout)
+
 jobs:
   build:
     runs-on: ubuntu-20.04

.github/workflows/docs-ci.yml

@@ -2,6 +2,9 @@ name: CI Documentation
 
 on: [push, pull_request]
 
+permissions:
+  contents: read # to fetch code (actions/checkout)
+
 jobs:
   build:
     runs-on: ubuntu-20.04

.github/workflows/scancode-release.yml

@@ -17,9 +17,13 @@ on:
     tags:
       - "v*.*.*"
 
+permissions: {}
 jobs:
 
   build_scancode_for_pypi:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Build PyPI archives
     runs-on: ubuntu-20.04
 
@@ -67,6 +71,9 @@ jobs:
 
 
   build_scancode_for_release_linux:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Build Release for linux
     runs-on: ubuntu-20.04
     needs:
@@ -106,6 +113,9 @@ jobs:
 
 
   build_scancode_for_release_macos:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Build Release for mac
     runs-on: ubuntu-20.04
     needs:
@@ -145,6 +155,9 @@ jobs:
 
 
   build_scancode_for_release_windows:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Build Release for windows
     runs-on: ubuntu-20.04
     needs:
@@ -183,6 +196,9 @@ jobs:
 
 
   build_scancode_for_release_source:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Build source
     runs-on: ubuntu-20.04
     needs:
@@ -221,6 +237,9 @@ jobs:
 
 
   smoke_test_install_and_run_pypi_dists_posix:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Test POSIX PyPI wheels
     needs:
       - build_scancode_for_pypi
@@ -267,6 +286,9 @@ jobs:
 
 
   smoke_test_install_and_run_pypi_dists_windows:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Test Windows PyPI wheels
     needs:
       - build_scancode_for_pypi
@@ -312,6 +334,9 @@ jobs:
 
 
   smoke_test_install_and_run_app_archives_on_linux:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Test app on ${{ matrix.os }}
     needs:
       - build_scancode_for_release_linux
@@ -350,6 +375,9 @@ jobs:
 
 
   smoke_test_install_and_run_app_archives_on_macos:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Test app on ${{ matrix.os }}
     needs:
       - build_scancode_for_release_macos
@@ -388,6 +416,9 @@ jobs:
 
 
   smoke_test_install_and_run_app_archives_on_windows:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Test app on ${{ matrix.os }}
     needs:
       - build_scancode_for_release_windows
@@ -422,6 +453,9 @@ jobs:
           for %%F in (dist/*.zip) do python etc/release/scancode_release_tests.py dist/%%F
 
   publish_to_gh_release:
+    permissions:
+      contents: write # to create GitHub release (softprops/action-gh-release)
+
     name: Publish to GH Release
     needs:
       - smoke_test_install_and_run_app_archives_on_linux

.gitignore

@@ -11,6 +11,7 @@
 /src/*.egg-info
 *.egg-info
 !tests/packagedcode/data/pypi/source-package/pip-22.0.4/src/pip.egg-info
+!tests/packagedcode/data/pypi/unpacked_sdist/prefer-egg-info-pkg-info/celery/celery.egg-info
 /dist
 /build
 /bin

CHANGELOG.rst

@@ -14,6 +14,30 @@ Package detection:
   - OpenWRT packages.
   - Yocto/BitBake .bb recipes.
 
+- Update ``GemfileLockParser`` to track the gem which the Gemfile.lock is for,
+  which we assign to the new ``GemfileLockParser.primary_gem`` field. Update
+  ``GemfileLockHandler.parse()`` to handle the case where there is a primary gem
+  detected from a gemfile.lock. If there is a primary gem, a single ``Package``
+  is created and the detected gem data within the gemfile.lock are assigned as
+  dependencies. If there is no primary gem, then all of the dependencies are
+  collected into Package with no name and yielded.
+
+  https://github.com/nexB/scancode-toolkit/issues/3072
+
+- Fix issue where dependencies were not reported when scanning an extracted
+  Python project by modifying ``BaseExtractedPythonLayout.assemble()`` to favor
+  using package data from a PKG-INFO file from an egg-info directory. Package
+  data from a PKG-INFO file from an egg-info directory contains the dependency
+  information collected from the requirements.txt file along side PKG-INFO.
+
+  https://github.com/nexB/scancode-toolkit/issues/3083
+
+- Fix issue where we were returning incorrect purl package ``type`` for cocoapods.
+  ``pods`` was being returned  as a purl type for cocoapods, it should be
+  ``cocoapods`` instead.
+  https://github.com/package-url/purl-spec/blob/master/PURL-TYPES.rst#cocoapods
+
+  https://github.com/nexB/scancode-toolkit/issues/3081
 
 License detection:
 ~~~~~~~~~~~~~~~~~~~
@@ -30,7 +54,20 @@ License detection:
   detected license only once. This data can contain the reference license text
   as an option.
 
+v31.2.1 - 2022-10-05
+----------------------------------
+
+This is a minor release to fix a typo in a date.
+
+
+v31.2.0 - 2022-10-05
+----------------------------------
+
+This is a minor release with small bug fixes and minor feature updates.
 
+- Update SPDX license list to 3.18
+- Improve how we discard license matches that are "gibberish"
+- And new and improve existing license and license detection rules
 
 
 v31.1.1 - 2022-09-02

ROADMAP.rst

@@ -18,19 +18,27 @@ even if it is correct and it is technically correct.
 The goal of this improvement is to:
 
 - combine multiple related license matches in a single license detection
+
 - in a license detection, expose a primary license expression in addition
   to the complete, full license expression.
+
 - make the logic of selection of the primary license visible, at the minimum
   with a log of combination and primary license selection operations
 
 This is for SCTK first.
 
+Status: This has been completed in SCTK and also included in SCIO. We use
+an updated --summary option and a new license clarity score for this.
+Some work is still in progress as part of 3.) "detections"
+
 
 2. Package files.
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Reporting the set of package files for each package instance is important because
-it allows for natural grouping of these in one unit. 
+it allows for natural grouping of these in one unit.
+
+This has been completed in SCTK and also included in SCIO.
 
 
 3. Go to two-level reporting of detections to provide more effective detections

docs/source/explanations/overview.rst

@@ -27,18 +27,18 @@ ScanCode-Toolkit performs the scan on a codebase in the following steps :
 
 Scan results are provided in various formats:
 
-- a JSON file simple or pretty-printed,
-- SPDX tag value or XML, RDF formats,
-- CSV,
-- a simple unformatted HTML file that can be opened in browser or as a spreadsheet.
+* a JSON file simple or pretty-printed,
+* SPDX tag value or XML, RDF formats,
+* CSV,
+* a simple unformatted HTML file that can be opened in browser or as a spreadsheet.
 
 For each scanned file, the result contains:
 
-- its location in the codebase,
-- the detected licenses and copyright statements,
-- the start and end line numbers identifying where the license or copyright was found in the
+* its location in the codebase,
+* the detected licenses and copyright statements,
+* the start and end line numbers identifying where the license or copyright was found in the
   scanned file, and
-- reference information for the detected license.
+* reference information for the detected license.
 
 For archive extraction, ScanCode uses a combination of Python modules, 7zip and libarchive/bsdtar
 to detect archive types and extract these recursively.

docs/source/misc/faq.rst

@@ -68,7 +68,7 @@ Can I install ScanCode in a Unicode path?
 -----------------------------------------
 
 Yes but but it is best to void this for now. See
-https://github.com/nexB/scancode- toolkit/issues/867
+https://github.com/nexB/scancode-toolkit/issues/867
 
 There is a bug in virtualenv https://github.com/pypa/virtualenv/issues/457 that
 is now fixed but has not been extensively tested for ScanCode.