[compiler-v2][decompiler] Pulling fat-loop algorithm out of the prover #14488

wrwg · 2024-08-31T15:59:03Z

Description

The prover actually has a full-fledged loop analysis for stackless bytecode for loop invariant instrumentation and unrolling. This loop analysis is a major part of the logic for decompilation. This PR pulls out this functionality in a new general purpose module, fat_loop. Its mostly a refactoring of existing code.

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

Existing prover tests

trunk-io · 2024-08-31T15:59:06Z

⏱️ 1h 12m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
rust-targeted-unit-tests	17m	🟩
rust-move-unit-coverage	15m	🟩
cli-e2e-tests / run-cli-tests	10m	🟥 🟥
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-lints	3m	🟩
rust-build-cached-packages	2m	🟩
rust-cargo-deny	2m	🟩
general-lints	2m	🟩
check-dynamic-deps	41s	🟩
semgrep/ci	26s	🟩
file_change_determinator	12s	🟩
file_change_determinator	9s	🟩
permission-check	3s	🟩
permission-check	3s	🟩
permission-check	3s	🟩
permission-check	2s	🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

wrwg · 2024-08-31T15:59:19Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @wrwg and the rest of your teammates on Graphite

wrwg

@gedigi FYI regards ongoing discussion

@meng-xu-cs nice piece of code!

codecov · 2024-08-31T16:10:36Z

Codecov Report

Attention: Patch coverage is 86.60131% with 41 lines in your changes missing coverage. Please review.

Project coverage is 59.8%. Comparing base (e87224f) to head (52dcfa8).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
...ird_party/move/move-model/bytecode/src/fat_loop.rs	87.5%	35 Missing ⚠️
...move-prover/bytecode-pipeline/src/loop_analysis.rs	75.0%	6 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (e87224f) and HEAD (52dcfa8). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (e87224f) HEAD (52dcfa8)

2 1

Additional details and impacted files

@@             Coverage Diff             @@
##             main   #14488       +/-   ##
===========================================
- Coverage    71.9%    59.8%    -12.1%     
===========================================
  Files        2395      852     -1543     
  Lines      482134   207466   -274668     
===========================================
- Hits       346822   124219   -222603     
+ Misses     135312    83247    -52065

Flag	Coverage Δ
	`59.8% <86.6%> (-12.1%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

brmataptos

Why do you call this a "fat loop"?

brmataptos

Oh, I see "fat loop" was in the old code. I suppose it is intended to mean a loop with possibly multiple back-edges to the header. Looks good.

The prover actually has a full-fledged loop analysis for stackless bytecode for loop invariant instrumentation and unrolling. This loop analysis is a major part of the logic for decompilation. This PR pulls out this functionality in a new general purpose module, `fat_loop`. Its mostly a refactoring of existing code.

github-actions · 2024-09-23T21:08:48Z

✅ Forge suite `realistic_env_max_load` success on `52dcfa8e2403783867e429c650a94b44f0867268`

two traffics test: inner traffic : committed: 14244.75 txn/s, latency: 2789.40 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3300 ms), latency samples: 5416500
two traffics test : committed: 100.00 txn/s, latency: 1715.88 ms, (p50: 1500 ms, p70: 1600, p90: 1700 ms, p99: 10800 ms), latency samples: 1780
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.256, avg: 0.230", "QsPosToProposal: max: 1.117, avg: 1.084", "ConsensusProposalToOrdered: max: 0.328, avg: 0.294", "ConsensusOrderedToCommit: max: 0.424, avg: 0.406", "ConsensusProposalToCommit: max: 0.715, avg: 0.701"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.91s no progress at version 2197947 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.56s no progress at version 2197945 (avg 8.56s) [limit 15].
Test Ok

github-actions · 2024-09-23T21:10:54Z

✅ Forge suite `framework_upgrade` success on `25a081116546670e62ca927ba90478de78557056` ==> `52dcfa8e2403783867e429c650a94b44f0867268`

Compatibility test results for 25a081116546670e62ca927ba90478de78557056 ==> 52dcfa8e2403783867e429c650a94b44f0867268 (PR)
Upgrade the nodes to version: 52dcfa8e2403783867e429c650a94b44f0867268
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1254.12 txn/s, submitted: 1257.25 txn/s, failed submission: 3.13 txn/s, expired: 3.13 txn/s, latency: 2468.47 ms, (p50: 2400 ms, p70: 2700, p90: 3800 ms, p99: 5400 ms), latency samples: 112160
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1084.31 txn/s, submitted: 1086.94 txn/s, failed submission: 2.63 txn/s, expired: 2.63 txn/s, latency: 2689.47 ms, (p50: 2400 ms, p70: 3000, p90: 4200 ms, p99: 6900 ms), latency samples: 98940
5. check swarm health
Compatibility test for 25a081116546670e62ca927ba90478de78557056 ==> 52dcfa8e2403783867e429c650a94b44f0867268 passed
Upgrade the remaining nodes to version: 52dcfa8e2403783867e429c650a94b44f0867268
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1146.83 txn/s, submitted: 1148.85 txn/s, failed submission: 2.02 txn/s, expired: 2.02 txn/s, latency: 2635.79 ms, (p50: 2500 ms, p70: 2900, p90: 3900 ms, p99: 5400 ms), latency samples: 102120
Test Ok

github-actions · 2024-09-23T21:15:41Z

✅ Forge suite `compat` success on `25a081116546670e62ca927ba90478de78557056` ==> `52dcfa8e2403783867e429c650a94b44f0867268`

Compatibility test results for 25a081116546670e62ca927ba90478de78557056 ==> 52dcfa8e2403783867e429c650a94b44f0867268 (PR)
1. Check liveness of validators at old version: 25a081116546670e62ca927ba90478de78557056
compatibility::simple-validator-upgrade::liveness-check : committed: 13249.20 txn/s, latency: 2461.41 ms, (p50: 1800 ms, p70: 2000, p90: 4000 ms, p99: 14000 ms), latency samples: 444440
2. Upgrading first Validator to new version: 52dcfa8e2403783867e429c650a94b44f0867268
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6683.21 txn/s, latency: 4027.93 ms, (p50: 4500 ms, p70: 4600, p90: 5000 ms, p99: 5100 ms), latency samples: 138320
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7168.17 txn/s, latency: 4466.14 ms, (p50: 4500 ms, p70: 4700, p90: 6700 ms, p99: 6900 ms), latency samples: 242860
3. Upgrading rest of first batch to new version: 52dcfa8e2403783867e429c650a94b44f0867268
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7793.51 txn/s, latency: 3564.43 ms, (p50: 3800 ms, p70: 4100, p90: 4200 ms, p99: 4400 ms), latency samples: 144360
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7845.06 txn/s, latency: 4050.96 ms, (p50: 4200 ms, p70: 4300, p90: 5800 ms, p99: 6100 ms), latency samples: 259120
4. upgrading second batch to new version: 52dcfa8e2403783867e429c650a94b44f0867268
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 10248.51 txn/s, latency: 2574.34 ms, (p50: 2300 ms, p70: 3100, p90: 4000 ms, p99: 4800 ms), latency samples: 189840
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 8033.65 txn/s, latency: 3565.37 ms, (p50: 2900 ms, p70: 3600, p90: 5400 ms, p99: 12400 ms), latency samples: 325300
5. check swarm health
Compatibility test for 25a081116546670e62ca927ba90478de78557056 ==> 52dcfa8e2403783867e429c650a94b44f0867268 passed
Test Ok

wrwg requested review from brmataptos, fEst1ck, rahxephon89 and vineethk August 31, 2024 16:00

wrwg commented Aug 31, 2024

View reviewed changes

vineethk approved these changes Aug 31, 2024

View reviewed changes

rahxephon89 approved these changes Aug 31, 2024

View reviewed changes

wrwg requested a review from meng-xu-cs September 1, 2024 04:52

wrwg mentioned this pull request Sep 2, 2024

[compiler-v2] Ast generator from stackless bytecode, Source Gen from Ast #14494

Merged

15 tasks

brmataptos reviewed Sep 9, 2024

View reviewed changes

brmataptos approved these changes Sep 9, 2024

View reviewed changes

wrwg force-pushed the wrwg/fat-loop branch from 2dc4e05 to 8d449bb Compare September 23, 2024 06:48

wrwg enabled auto-merge (squash) September 23, 2024 06:49

This comment has been minimized.

Sign in to view

wrwg force-pushed the wrwg/fat-loop branch from 8d449bb to 7a3da0a Compare September 23, 2024 18:15

This comment has been minimized.

Sign in to view

wrwg added the CICD:run-framework-upgrade-test label Sep 23, 2024

Merge branch 'main' into wrwg/fat-loop

52dcfa8

This comment has been minimized.

Sign in to view

wrwg merged commit 51f896c into main Sep 23, 2024
50 of 51 checks passed

wrwg deleted the wrwg/fat-loop branch September 23, 2024 21:15

wrwg mentioned this pull request Oct 4, 2024

[compiler-v2] Add loop labels to the language #14868

Merged

16 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[compiler-v2][decompiler] Pulling fat-loop algorithm out of the prover #14488

[compiler-v2][decompiler] Pulling fat-loop algorithm out of the prover #14488

wrwg commented Aug 31, 2024

trunk-io bot commented Aug 31, 2024 •

edited

Loading

wrwg commented Aug 31, 2024 •

edited

Loading

wrwg left a comment

codecov bot commented Aug 31, 2024 •

edited

Loading

brmataptos left a comment

brmataptos left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Sep 23, 2024

github-actions bot commented Sep 23, 2024

github-actions bot commented Sep 23, 2024

[compiler-v2][decompiler] Pulling fat-loop algorithm out of the prover #14488

[compiler-v2][decompiler] Pulling fat-loop algorithm out of the prover #14488

Conversation

wrwg commented Aug 31, 2024

Description

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

trunk-io bot commented Aug 31, 2024 • edited Loading

wrwg commented Aug 31, 2024 • edited Loading

wrwg left a comment

Choose a reason for hiding this comment

codecov bot commented Aug 31, 2024 • edited Loading

Codecov Report

brmataptos left a comment

Choose a reason for hiding this comment

brmataptos left a comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Sep 23, 2024

✅ Forge suite realistic_env_max_load success on 52dcfa8e2403783867e429c650a94b44f0867268

github-actions bot commented Sep 23, 2024

✅ Forge suite framework_upgrade success on 25a081116546670e62ca927ba90478de78557056 ==> 52dcfa8e2403783867e429c650a94b44f0867268

github-actions bot commented Sep 23, 2024

✅ Forge suite compat success on 25a081116546670e62ca927ba90478de78557056 ==> 52dcfa8e2403783867e429c650a94b44f0867268

trunk-io bot commented Aug 31, 2024 •

edited

Loading

wrwg commented Aug 31, 2024 •

edited

Loading

codecov bot commented Aug 31, 2024 •

edited

Loading

✅ Forge suite `realistic_env_max_load` success on `52dcfa8e2403783867e429c650a94b44f0867268`

✅ Forge suite `framework_upgrade` success on `25a081116546670e62ca927ba90478de78557056` ==> `52dcfa8e2403783867e429c650a94b44f0867268`

✅ Forge suite `compat` success on `25a081116546670e62ca927ba90478de78557056` ==> `52dcfa8e2403783867e429c650a94b44f0867268`