Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Respect proxy settings when streaming the download and extraction of the CodeQL bundle #2624

Merged
merged 17 commits into from
Dec 11, 2024
Merged

Respect proxy settings when streaming the download and extraction of the CodeQL bundle #2624

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
0dc76a9
feat: use http agent that respects environment proxy settings when st…
NlightNFotis Dec 4, 2024
8f2cb3a
build: add undici to node_modules folder - CI fails without it
NlightNFotis Dec 4, 2024
78be2f1
ci: add configuration tto run proxy with zstd package to test streami…
NlightNFotis Dec 3, 2024
9c48c8b
ci: add capability to handle container initialisation steps to sync.py
NlightNFotis Dec 5, 2024
b4bc093
ci: remove patch file and cleanup justfile
NlightNFotis Dec 5, 2024
b706e37
chore: add changelog entry
NlightNFotis Dec 5, 2024
182c5e7
chore: reword description in CHANGELOG.md
NlightNFotis Dec 6, 2024
a89fbc8
docs: add readme for justfile/pr-checks update
NlightNFotis Dec 9, 2024
c901aee
ci: remove dns for container running proxy steps (experimental)
NlightNFotis Dec 9, 2024
3951a82
Merge branch 'main' into NlightNFotis/detect_use_proxy_when_streaming
NlightNFotis Dec 9, 2024
51e71f8
docs: add documentation for the reason why we need to have a setup st…
NlightNFotis Dec 9, 2024
4c20d4f
feat: cleanup destination directory if we fail during streaming of th…
NlightNFotis Dec 9, 2024
c6454d5
Merge branch 'main' into NlightNFotis/detect_use_proxy_when_streaming
henrymercer Dec 10, 2024
9323695
Update PR checks readme instructions
aeisenberg Dec 9, 2024
88bcf64
feat: change wording of error reporting when a failure occurs during …
NlightNFotis Dec 11, 2024
9e8cd42
fix: move changelog entry to appropriate place
NlightNFotis Dec 11, 2024
1e5b591
Merge branch 'main' into NlightNFotis/detect_use_proxy_when_streaming
NlightNFotis Dec 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/__test-proxy.yml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ Note that the only difference between `v2` and `v3` of the CodeQL Action is the
## [UNRELEASED]

- We are rolling out a change in December 2024 that will extract the CodeQL bundle directly to the toolcache to improve performance. [#2631](https://github.com/github/codeql-action/pull/2631)
- We have added support for respecting the proxy settings present in environment variables in a runner when streaming the download and extraction of the CodeQL bundle. [#2624](https://github.com/github/codeql-action/pull/2624)
- Fixed an issue where streaming the download and extraction of the CodeQL bundle did not respect proxy settings. [#2624](https://github.com/github/codeql-action/pull/2624)
- Update default CodeQL bundle version to 2.20.0. [#2636](https://github.com/github/codeql-action/pull/2636)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this line added due to a bad merge?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I will have a look.

This was reported as a conflict in the GitHub UI, but when I did a git merge locally, it had no problem automatically resolving the conflict. Perhaps it did a weird thing because of my .gitconfig.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot find precisely what went wrong here, as the merge log is past my terminal character limit. I remember it saying that it resolved it automatically using the ort method, but it apparently went wrong there.

This is now fixed in 9e8cd42.

Going forward I will be more careful validating these merges went right.


## 3.27.6 - 03 Dec 2024

Expand Down
5 changes: 4 additions & 1 deletion lib/tools-download.js
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion lib/tools-download.js.map
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion pr-checks/checks/test-proxy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,10 @@ versions: ["linked", "nightly-latest"]
operatingSystems: ["ubuntu"]
container:
image: ubuntu:22.04
options: --dns 8.8.8.8
container-init-steps:
# These steps are required to initialise the `gh` cli in a container that doesn't
# come pre-installed with it. The reason for that is that this is later
# needed by the `prepare-test` workflow to find the latest release of CodeQL.
name: Set up GitHub CLI
NlightNFotis marked this conversation as resolved.
Show resolved Hide resolved
run: |
apt update
Expand Down
23 changes: 23 additions & 0 deletions pr-checks/readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# PR Checks

This folder contains the code supporting the workflows run when a PR is created.

## Update
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can simplify this. I can give some suggestions, but not quite right now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See here for an alternative readme: https://github.com/github/codeql-action/compare/NlightNFotis/detect_use_proxy_when_streaming...aeisenberg/justfile-readme?quick_pull=1#diff-b90d48e45a6813fad7195d74075f05bdf05919291a02116fe8a6b67691b1266a

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this! I agree it looks cleaner.

Will cherry-pick this in.


If you need to make a change to any of the PR checks, you need to perform the following
steps:

1. Make the change - the code for the PR checks is under the `pr-checks/checks/` folder.
2. Run the `sync.py` file to produce (and sync) the final workflow files under `.github/`

The second part requires some associated steps (create a virtual environment, download
the dependencies for the Python script, etc), so we have automated this with the `justfile`
included in this folder.

### 1-step update

1. Install https://github.com/casey/just by whichever way you prefer.
2. Run `$ just update-pr-checks` in your terminal.

If you don't wish to install `just`, you can also manually perform the steps
outlined in the `justfile` under the `update-pr-checks` action.
6 changes: 5 additions & 1 deletion src/tools-download.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,11 @@ export async function downloadAndExtract(
core.warning(
`Failed to download and extract CodeQL bundle using streaming. Falling back to downloading the bundle before extracting.`,
);
core.warning(getErrorMessage(e));
core.warning(`Error: ${getErrorMessage(e)}`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't prefix a warning with Error.

Suggested change
core.warning(`Error: ${getErrorMessage(e)}`);
core.warning(`Warning: ${getErrorMessage(e)}`);

But, do we even need the prefix at all?

Maybe reverting this is better:

Suggested change
core.warning(`Error: ${getErrorMessage(e)}`);
core.warning(getErrorMessage(e));

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for catching this.

Seeing it reported now, I agree it looks a bit silly.

My rationale for changing this was that this was rendering an error that was generated in the process of unpacking as a simple warning, with no other context whatsoever, which could be confusing to a user when interpreting the runner log (example).

I'm still motivated to decorate it somehow, as reverting may not satisfy this concern, but I will think about a cleaner decoration that doesn't look weird/silly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, this is what I came up with: 88bcf64

I've reworded the warnings to flow a bit more naturally for me, and to allow me to embed the error message in an indicative but non-invasive way.

Is this workable for you @aeisenberg ? Or do you have a strong preference for the previous state of affairs?


// If we failed during processing, we want to clean up the destination directory
// before we try again.
await cleanUpGlob(dest, "CodeQL bundle", logger);
}

const toolsDownloadStart = performance.now();
Expand Down
Loading