-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Autotune memory #15564
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Autotune memory #15564
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,84 @@ | ||
name: Make more memory available | ||
description: Stops services and adds swap | ||
|
||
inputs: | ||
max: | ||
description: Maximum memory to suggest using | ||
default: 50000 | ||
|
||
outputs: | ||
available: | ||
description: Available memory | ||
value: ${{ steps.memory.outputs.available }} | ||
suggested: | ||
description: Memory suggested for use (constrained by max and available) | ||
value: ${{ steps.memory.outputs.suggested }} | ||
|
||
runs: | ||
using: composite | ||
steps: | ||
- name: Stop services | ||
if: env.RUNNER_OS == 'Linux' | ||
shell: bash | ||
run: | | ||
sudo systemctl disable php8.1-fpm mono-xsp4 walinuxagent multipathd walinuxagent chrony cron getty@tty1 networkd-dispatcher rsyslog serial-getty@ttyS0 snapd multipathd.socket snapd.socket | ||
sudo systemctl stop php8.1-fpm mono-xsp4 walinuxagent multipathd walinuxagent chrony cron getty@tty1 networkd-dispatcher rsyslog serial-getty@ttyS0 snapd | ||
sudo killall mono | ||
|
||
- name: enable swap | ||
if: env.RUNNER_OS == 'Linux' | ||
shell: bash | ||
run: | | ||
sudo fallocate -l 10G /mnt/big-swapfile | ||
sudo chmod 600 /mnt/big-swapfile | ||
sudo mkswap /mnt/big-swapfile | ||
sudo swapon /mnt/big-swapfile | ||
|
||
- name: Report Available Memory (Linux) | ||
id: memory-linux | ||
if: runner.os == 'Linux' | ||
shell: bash | ||
run: | | ||
free | | ||
perl -ne ' | ||
next unless /Mem:.*\s(\d+)$/; | ||
my $m = int($1 / 102400)*100; | ||
print "available=$m\n"; | ||
' >> "$GITHUB_OUTPUT" | ||
|
||
- name: Report Available Memory (macOS) | ||
id: memory-mac | ||
if: runner.os == 'macOS' | ||
shell: bash | ||
run: | | ||
sysctl -a | | ||
perl -e ' | ||
my $m; | ||
while (<>) { | ||
next unless /.*memsize.*:\s*(\d+)/; | ||
my $m0 = int($1 >> 20); | ||
$m = $m0 unless ($m && $m > $m0); | ||
} | ||
print "available=$m\n"; | ||
' >> "$GITHUB_OUTPUT" | ||
|
||
- name: Report Available Memory (Windows) | ||
id: memory-windows | ||
if: runner.os == 'Windows' | ||
shell: bash | ||
run: | | ||
echo "available=10240" >> "$GITHUB_OUTPUT" | ||
Comment on lines
+65
to
+70
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As noted, there doesn't appear to be a consumer for this at this time, but having at least a stub implementation seemed better than nothing. |
||
|
||
- name: Report Memory | ||
id: memory | ||
shell: bash | ||
env: | ||
max: ${{ inputs.max }} | ||
available: ${{ steps.memory-linux.outputs.available || steps.memory-mac.outputs.available || steps.memory-windows.outputs.available }} | ||
run: | | ||
perl -e ' | ||
my $available = $ENV{available}; | ||
my $max = $ENV{max} || $available; | ||
my $s = ($m < $max ? $available : $max); | ||
print "suggested=$s\n"."available=$available\n"; | ||
' >> "$GITHUB_OUTPUT" |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,7 +5,10 @@ on: | |
paths: | ||
- "csharp/**" | ||
- "shared/**" | ||
- .github/workflows/csharp-qltest.yml | ||
- .github/actions/fetch-codeql/action.yml | ||
- .github/actions/cache-query-compilation/action.yml | ||
Comment on lines
+8
to
+10
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. these two are used but weren't checked |
||
- .github/actions/reasonable-memory/action.yml | ||
- codeql-workspace.yml | ||
branches: | ||
- main | ||
|
@@ -16,6 +19,8 @@ on: | |
- "shared/**" | ||
- .github/workflows/csharp-qltest.yml | ||
- .github/actions/fetch-codeql/action.yml | ||
- .github/actions/cache-query-compilation/action.yml | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. only one was missing here... |
||
- .github/actions/reasonable-memory/action.yml | ||
- codeql-workspace.yml | ||
branches: | ||
- main | ||
|
@@ -53,6 +58,9 @@ jobs: | |
slice: ["1/2", "2/2"] | ||
steps: | ||
- uses: actions/checkout@v4 | ||
- name: Allocate memory | ||
id: memory | ||
uses: ./.github/actions/reasonable-memory | ||
- uses: ./csharp/actions/create-extractor-pack | ||
- name: Cache compilation cache | ||
id: query-cache | ||
|
@@ -61,9 +69,10 @@ jobs: | |
key: csharp-qltest-${{ matrix.slice }} | ||
- name: Run QL tests | ||
run: | | ||
codeql test run --threads=0 --ram 50000 --slice ${{ matrix.slice }} --search-path extractor-pack --check-databases --check-undefined-labels --check-repeated-labels --check-redefined-labels --consistency-queries ql/consistency-queries ql/test --compilation-cache "${{ steps.query-cache.outputs.cache-dir }}" | ||
codeql test run --threads=0 --ram $memory --slice ${{ matrix.slice }} --search-path extractor-pack --check-databases --check-undefined-labels --check-repeated-labels --check-redefined-labels --consistency-queries ql/consistency-queries ql/test --compilation-cache "${{ steps.query-cache.outputs.cache-dir }}" | ||
env: | ||
GITHUB_TOKEN: ${{ github.token }} | ||
memory: ${{ steps.memory.outputs.suggested }} | ||
unit-tests: | ||
strategy: | ||
matrix: | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,8 +5,11 @@ on: | |
paths: | ||
- "ruby/**" | ||
- "shared/**" | ||
- .github/workflows/ruby-build.yml | ||
- .github/workflows/ruby-qltest.yml | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. bad copy paste? |
||
- .github/actions/fetch-codeql/action.yml | ||
- .github/actions/cache-query-compilation/action.yml | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. used but not checked |
||
- .github/actions/reasonable-memory/action.yml | ||
- csharp/actions/create-extractor-pack/action.yml | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this is amusing -- it probably should be moved to .github/actions -- the idea that ruby is using a csharp action is pretty confusing. |
||
- codeql-workspace.yml | ||
branches: | ||
- main | ||
|
@@ -17,10 +20,14 @@ on: | |
- "shared/**" | ||
- .github/workflows/ruby-qltest.yml | ||
- .github/actions/fetch-codeql/action.yml | ||
- .github/actions/cache-query-compilation/action.yml | ||
- .github/actions/reasonable-memory/action.yml | ||
- csharp/actions/create-extractor-pack/action.yml | ||
- codeql-workspace.yml | ||
branches: | ||
- main | ||
- "rc/*" | ||
workflow_dispatch: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's really handy to be able to test these workflows (especially given that they have branch constraints that prevent normal testing) |
||
|
||
env: | ||
CARGO_TERM_COLOR: always | ||
|
@@ -55,6 +62,9 @@ jobs: | |
fail-fast: false | ||
steps: | ||
- uses: actions/checkout@v4 | ||
- name: Allocate memory | ||
id: memory | ||
uses: ./.github/actions/reasonable-memory | ||
- uses: ./.github/actions/fetch-codeql | ||
- uses: ./ruby/actions/create-extractor-pack | ||
- name: Cache compilation cache | ||
|
@@ -64,6 +74,7 @@ jobs: | |
key: ruby-qltest | ||
- name: Run QL tests | ||
run: | | ||
codeql test run --threads=0 --ram 50000 --search-path "${{ github.workspace }}/ruby/extractor-pack" --check-databases --check-undefined-labels --check-unused-labels --check-repeated-labels --check-redefined-labels --check-use-before-definition --consistency-queries ql/consistency-queries ql/test --compilation-cache "${{ steps.query-cache.outputs.cache-dir }}" | ||
codeql test run --threads=0 --ram $memory --search-path "${{ github.workspace }}/ruby/extractor-pack" --check-databases --check-undefined-labels --check-unused-labels --check-repeated-labels --check-redefined-labels --check-use-before-definition --consistency-queries ql/consistency-queries ql/test --compilation-cache "${{ steps.query-cache.outputs.cache-dir }}" | ||
env: | ||
GITHUB_TOKEN: ${{ github.token }} | ||
memory: ${{ steps.memory.outputs.suggested }} |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -32,6 +32,7 @@ on: | |
- main | ||
- rc/* | ||
- codeql-cli-* | ||
workflow_dispatch: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's really handy to be able to test these workflows (especially given that they have branch constraints that prevent normal testing) |
||
|
||
jobs: | ||
# not using a matrix as you cannot depend on a specific job in a matrix, and we want to start linux checks | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This could be dropped