Skip to content

Commit

Permalink
CodeQL package management GA (#50221)
Browse files Browse the repository at this point in the history
Co-authored-by: James Fletcher <42464962+jf205@users.noreply.github.com>
  • Loading branch information
subatoi and jf205 authored Apr 26, 2024
1 parent c3e2221 commit ac10304
Show file tree
Hide file tree
Showing 14 changed files with 16 additions and 83 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -313,9 +313,7 @@ In this example, the default queries will be run for Java, as well as the querie

### Using query packs

{% data reusables.code-scanning.beta-codeql-packs-cli %}

To add one or more {% data variables.product.prodname_codeql %} query packs (beta), add a `with: packs:` entry within the `uses: {% data reusables.actions.action-codeql-action-init %}` section of the workflow. Within `packs` you specify one or more packages to use and, optionally, which version to download. Where you don't specify a version, the latest version is downloaded. If you want to use packages that are not publicly available, you need to set the `GITHUB_TOKEN` environment variable to a secret that has access to the packages. For more information, see "[AUTOTITLE](/actions/security-guides/automatic-token-authentication)" and "[AUTOTITLE](/actions/security-guides/encrypted-secrets)."
To add one or more {% data variables.product.prodname_codeql %} query packs, add a `with: packs:` entry within the `uses: {% data reusables.actions.action-codeql-action-init %}` section of the workflow. Within `packs` you specify one or more packages to use and, optionally, which version to download. Where you don't specify a version, the latest version is downloaded. If you want to use packages that are not publicly available, you need to set the `GITHUB_TOKEN` environment variable to a secret that has access to the packages. For more information, see "[AUTOTITLE](/actions/security-guides/automatic-token-authentication)" and "[AUTOTITLE](/actions/security-guides/encrypted-secrets)."

{% note %}

Expand Down Expand Up @@ -456,8 +454,6 @@ The settings in the configuration file are written in YAML format.

### Specifying {% data variables.product.prodname_codeql %} query packs

{% data reusables.code-scanning.beta-codeql-packs-cli %}

You specify {% data variables.product.prodname_codeql %} query packs in an array. Note that the format is different from the format used by the workflow file.

{% raw %}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,17 +74,10 @@ The queries are open source, so you can view and contribute to the queries in th

If you are scanning your code with advanced setup or an external CI system, you can run additional queries as part of your analysis.

{%- ifversion codeql-packs %}
These queries must belong to a published {% data variables.product.prodname_codeql %} query pack (beta) or a {% data variables.product.prodname_codeql %} pack in a repository. {% data variables.product.prodname_codeql %} packs (beta) provide the following benefits over traditional {% data variables.product.prodname_ql %} packs:
These queries must belong to a published {% data variables.product.prodname_codeql %} query pack or a {% data variables.product.prodname_codeql %} pack in a repository.

- When a {% data variables.product.prodname_codeql %} query pack (beta) is published to the {% data variables.product.company_short %} {% data variables.product.prodname_container_registry %}, all the transitive dependencies required by the queries and a compilation cache are included in the package. This improves performance and ensures that running the queries in the pack gives identical results every time until you upgrade to a new version of the pack or the CLI.
- {% data variables.product.prodname_ql %} packs do not include transitive dependencies, so queries in the pack can depend only on the standard libraries (that is, the libraries referenced by an `import LANGUAGE` statement in your query), or libraries in the same {% data variables.product.prodname_ql %} pack as the query.
- {% data variables.product.prodname_codeql %} query packs (beta) can be downloaded from multiple GitHub container registries. For more information, see "[AUTOTITLE](/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning#downloading-codeql-packs-from-github-enterprise-server)."
- When a {% data variables.product.prodname_codeql %} query pack is published to the {% data variables.product.company_short %} {% data variables.product.prodname_container_registry %}, all the transitive dependencies required by the queries and a compilation cache are included in the package. This improves performance and ensures that running the queries in the pack gives identical results every time until you upgrade to a new version of the pack or the CLI.

For more information, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/customizing-analysis-with-codeql-packs)."

{% data reusables.code-scanning.beta-codeql-packs-cli %}
- {% data variables.product.prodname_codeql %} query packs can be downloaded from multiple {% data variables.product.prodname_dotcom %} container registries. For more information, see "[AUTOTITLE](/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning#downloading-codeql-packs-from-github-enterprise-server)."

{%- else %}
The queries you want to run must belong to a {% data variables.product.prodname_ql %} pack in a repository. Queries must only depend on the standard libraries (that is, the libraries referenced by an `import LANGUAGE` statement in your query), or libraries in the same {% data variables.product.prodname_ql %} pack as the query.
{% endif %}
For more information, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/customizing-analysis-with-codeql-packs)."
Original file line number Diff line number Diff line change
Expand Up @@ -132,16 +132,7 @@ The following examples show how to run `database analyze` using {% data variable

### Running a {% data variables.product.prodname_codeql %} query pack

{% note %}

**Note**

The {% data variables.product.prodname_codeql %} package management functionality, including {% data variables.product.prodname_codeql %} packs, is currently available as a beta release and is subject to change. During the beta release, {% data variables.product.prodname_codeql %} packs are available only using {% data variables.product.prodname_registry %} - the {% data variables.product.prodname_dotcom %} {% data variables.product.prodname_container_registry %}. To use this beta functionality, install the latest version of the {% data variables.product.prodname_codeql_cli %} bundle from: https://github.com/github/codeql-action/releases.

{% endnote %}

To run an existing {% data variables.product.prodname_codeql %} query pack from the {% data variables.product.prodname_dotcom %} {% data variables.product.prodname_container_registry %}, you can specify one or more
pack names:
To run an existing {% data variables.product.prodname_codeql %} query pack from the {% data variables.product.prodname_dotcom %} {% data variables.product.prodname_container_registry %}, you can specify one or more pack names:

```shell
codeql database analyze <database> microsoft/coding-standards@1.0.0 github/security-queries --format=sarifv2.1.0 --output=query-results.sarif --download
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@ redirect_from:
- /code-security/codeql-cli/codeql-cli-reference/about-codeql-packs
---

{% data reusables.codeql-cli.beta-note-package-management %}

## About {% data variables.product.prodname_codeql %} packs

{% data reusables.code-scanning.codeql-cli-version-ghes %}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,6 @@ tools](https://developer.apple.com/downloads/index.action) and [Rosetta 2](https

You should always use the {% data variables.product.prodname_codeql %} bundle as this ensures compatibility and also gives much better performance than a separate download of the {% data variables.product.prodname_codeql_cli %} and checkout of the {% data variables.product.prodname_codeql %} queries. If you will only be running the CLI on one specific platform, download the appropriate `codeql-bundle-PLATFORM.tar.gz` file. Alternatively, you can download `codeql-bundle.tar.gz`, which contains the CLI for all supported platforms.

{% data reusables.code-scanning.beta-codeql-packs-cli %}

#### Download information for macOS "Catalina" (or newer) users

{% data reusables.codeql-cli.download-info-macos-catalina-or-newer %}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,6 @@ redirect_from:
- /code-security/codeql-cli/using-the-codeql-cli/creating-and-working-with-codeql-packs
---

{% data reusables.codeql-cli.beta-note-package-management %}

## About {% data variables.product.prodname_codeql %} packs and the {% data variables.product.prodname_codeql_cli %}

{% data reusables.code-scanning.codeql-cli-version-ghes %}
Expand Down Expand Up @@ -105,18 +103,6 @@ Once you've created a model pack, you can publish it in the same way as other {%

{% endif %}

## Modifying an existing legacy QL pack to create a {% data variables.product.prodname_codeql %} query pack

If you already have a `qlpack.yml` file, you can edit it manually to convert it into a {% data variables.product.prodname_codeql %} pack.

1. Edit the `name` property so that it matches the format `<scope>/<name>`, where `<scope>` is the name of the {% data variables.product.prodname_dotcom %} organization or user account that you will publish to.

1. In the `qlpack.yml` file, include a `version` property with a semver identifier, as well as an optional `dependencies` block.

1. Migrate the list of dependencies in `libraryPathDependencies` to the `dependencies` block. Specify the version range for each dependency. If the range is unimportant, or you are unsure of compatibility, you can specify `"\*"`, which indicates that any version is acceptable and will default to the latest version when you run `codeql pack install`.

For more information about the properties, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/customizing-analysis-with-codeql-packs)."

## Adding and installing dependencies on a {% data variables.product.prodname_codeql %} pack

{% note %}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,6 @@ redirect_from:
- /code-security/codeql-cli/using-the-codeql-cli/publishing-and-using-codeql-packs
---

{% data reusables.codeql-cli.beta-note-package-management %}

## Configuring the `qlpack.yml` file before publishing

{% data reusables.code-scanning.codeql-cli-version-ghes %}
Expand Down Expand Up @@ -499,7 +497,7 @@ Some extra notes on the following properties:

- `library`: Indicates that this is a library pack with no executable queries. It is only meant to be used as a dependency for other packs.

- `dbscheme` and `upgrades`: These properties are internal to the {% data variables.product.prodname_codeql_cli %} and should only be defined in the core QL pack for a language.
- `dbscheme` and `upgrades`: These properties are internal to the {% data variables.product.prodname_codeql_cli %} and should only be defined in the core {% data variables.product.prodname_codeql %} query pack for a language.

### Core query pack

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,6 @@ For more information about these metadata properties, see "[Metadata for {% data

## Packaging custom QL queries

{% data reusables.codeql-cli.beta-note-package-management %}

<!-- using reusable re-created by Felicity -->

When you write your own queries with the intention to share them with others, you should save them in a custom {% data variables.product.prodname_codeql %} pack. You can publish the pack as a {% data variables.product.prodname_codeql %} pack to {% data variables.product.prodname_registry %} - the {% data variables.product.prodname_dotcom %} {% data variables.product.prodname_container_registry %}. For more information, see "[AUTOTITLE](/code-security/codeql-cli/getting-started-with-the-codeql-cli/customizing-analysis-with-codeql-packs)."

<!-- tweaked to "For more information, see X" to match our style guide -->
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@ intro: 'You can view, write, and edit {% data variables.product.prodname_codeql
allowTitleToDifferFromFilename: true
---

{% data reusables.codeql-cli.beta-note-package-management %}

## Benefits of using the {% data variables.product.prodname_codeql %} extension for {% data variables.product.prodname_vscode %} to work with packs

With the {% data variables.product.prodname_codeql %} for {% data variables.product.prodname_vscode %} extension, you can:
Expand Down
9 changes: 0 additions & 9 deletions data/reusables/code-scanning/beta-codeql-packs-cli.md

This file was deleted.

14 changes: 7 additions & 7 deletions data/reusables/code-scanning/example-configuration-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,14 @@ name: "My {% data variables.product.prodname_codeql %} config"
disable-default-queries: true

queries:
- name: Use an in-repository {% data variables.product.prodname_ql %} pack (run queries in the my-queries directory)
- name: Use an in-repository {% data variables.product.prodname_codeql %} pack (run queries in the my-queries directory)
uses: ./my-queries
- name: Use an external JavaScript {% data variables.product.prodname_ql %} pack (run queries from an external repo)
uses: octo-org/javascript-qlpack@main
- name: Use an external query (run a single query from an external {% data variables.product.prodname_ql %} pack)
uses: octo-org/python-qlpack/show_ifs.ql@main
- name: Use an external JavaScript {% data variables.product.prodname_codeql %} pack (run queries from an external repo)
uses: octo-org/javascript-codeql-pack@main
- name: Use an external query (run a single query from an external {% data variables.product.prodname_codeql %} pack)
uses: octo-org/python-codeql-pack/show_ifs.ql@main
- name: Use a query suite file (run queries from a query suite in this repo)
uses: ./codeql-qlpacks/complex-python-qlpack/rootAndBar.qls
uses: ./codeql-packs/complex-python-codeql-pack/rootAndBar.qls

paths:
- src
Expand All @@ -37,7 +37,7 @@ The following configuration file only runs queries that generate alerts of sever

``` yaml
queries:
- name: Use an in-repository QL pack (run queries in the my-queries directory)
- name: Use an in-repository CodeQL query pack (run queries in the my-queries directory)
uses: ./my-queries
packs:
- codeql/java-queries
Expand Down
10 changes: 2 additions & 8 deletions data/reusables/code-scanning/run-additional-queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,13 @@ You can also specify the queries you want to exclude from analysis, or include i
{% endtip %}
{% endif %}

{% ifversion codeql-packs %}
You can run extra queries if they are part of a {% data variables.product.prodname_codeql %} pack (beta) published to the {% data variables.product.company_short %} {% data variables.product.prodname_container_registry %} or a {% data variables.product.prodname_ql %} pack stored in a repository. For more information, see "[AUTOTITLE](/code-security/code-scanning/introduction-to-code-scanning/about-code-scanning-with-codeql#about-codeql-queries)."
You can run extra queries if they are part of a {% data variables.product.prodname_codeql %} pack published to the {% data variables.product.company_short %} {% data variables.product.prodname_container_registry %} or a {% data variables.product.prodname_codeql %} pack stored in a repository. For more information, see "[AUTOTITLE](/code-security/code-scanning/introduction-to-code-scanning/about-code-scanning-with-codeql#about-codeql-queries)."

The options available to specify the additional queries you want to run are:

- `packs` to install one or more {% data variables.product.prodname_codeql %} query packs (beta) and run the default query suite or queries for those packs.
- `packs` to install one or more {% data variables.product.prodname_codeql %} query packs and run the default query suite or queries for those packs.
- `queries` to specify a single _.ql_ file, a directory containing multiple _.ql_ files, a _.qls_ query suite definition file, or any combination. For more information about query suite definitions, see "[Creating {% data variables.product.prodname_codeql %} query suites](https://codeql.github.com/docs/codeql-cli/creating-codeql-query-suites/)."

You can use both `packs` and `queries` in the same workflow.
{% else %}
Any additional queries you want to run must belong to a {% data variables.product.prodname_ql %} pack in a repository. For more information, see "[AUTOTITLE](/code-security/code-scanning/introduction-to-code-scanning/about-code-scanning-with-codeql#about-codeql-queries)."

You can specify a single _.ql_ file, a directory containing multiple _.ql_ files, a _.qls_ query suite definition file, or any combination. For more information about query suite definitions, see "[Creating {% data variables.product.prodname_codeql %} query suites](https://codeql.github.com/docs/codeql-cli/creating-codeql-query-suites/)."
{% endif %}

{% ifversion fpt or ghec %}We don't recommend referencing query suites directly from the `github/codeql` repository, for example, `github/codeql/cpp/ql/src@main`. Such queries would have to be recompiled, and may not be compatible with the version of {% data variables.product.prodname_codeql %} currently active on {% data variables.product.prodname_actions %}, which could lead to errors during analysis.{% endif %}
5 changes: 0 additions & 5 deletions data/reusables/codeql-cli/beta-note-package-management.md

This file was deleted.

1 change: 0 additions & 1 deletion data/variables/product.yml
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,6 @@ prodname_pages: 'GitHub Pages'

# CodeQL
prodname_codeql: 'CodeQL'
prodname_ql: 'QL'
prodname_codeql_cli: 'CodeQL CLI'
# The CodeQL CLI version recommended for each GHES version.
# CodeQL usually bumps its minor version for each minor version of GHES.
Expand Down

0 comments on commit ac10304

Please sign in to comment.