fuzz results differ on local vs github CI

[xenide]

Component

Forge

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

forge 0.2.0 (7bba788 2023-05-10T00:11:46.049317000Z)

What command(s) is the bug in?

forge snapshot

Operating System

macOS (Apple Silicon)

Describe the bug

The fuzz run gas usages differ on my machine vs in github CI. This has been going on for 1-2 months already

Things I tried

• comparing the forge config on my machine vs in github CI
  • discovered that the default remappings for ds-test was defaulted to "ds-test/=lib/forge-std/lib/ds-test/src/" while it was "ds-test/=lib/solmate/lib/ds-test/src/" on the github CI
  • tried changing to both, but the fuzz runs gas outputs still differ slightly, and therefore CI tests failing

This is the original foundry.toml

[profile.default]
solc = "0.8.19"
#via_ir = true
bytecode_hash = "none"
optimizer_runs = 1_000_000
libs = ['lib']
remappings = [
    "@openzeppelin/=lib/openzeppelin-contracts/contracts/",
]
match_path = "test/unit/*.sol"
verbosity = 3
fs_permissions = [
    { access = "write", path = "./script/optimized-deployer-meta" },
    { access = "write", path = "./script/unoptimized-deployer-meta" },
]
ignored_error_codes = []

[profile.integration]
match_path = "test/integration/*.sol"

[profile.differential]
fs_permissions = [{ access = "read", path = "./reference/balancer-v2-monorepo" }]
match_path     = "test/differential/*.sol"

[profile.differential.fuzz]
runs = 10_000

[fmt]
bracket_spacing   = true
wrap_comments     = false
number_underscore = "thousands"
int_types         = "long"

[profile.script]
optimizer_runs = 1_000_000

[mds1]

Are you setting a fuzz seed? Most likely this is because each fuzz run uses different random values which affects gas costs

[xenide]

did not set a fuzz seed in foundry.toml as you can see. But when I run forge config it doesn't have anything related to the seed as well

[mds1]

Right, so if a fuzz seed is not present in the config (which is the default, and which is why you don't see it when running forge config), then it means "use a random seed each time forge test is invoked

[mds1]

did not set a fuzz seed in foundry.toml as you can see.

You may have been setting a fuzz seed with the FOUNDRY_FUZZ_SEED env var which overrides what's in the config (which is why I asked)

[xenide]

Alright so I just tried setting a fuzz seed and the github CI gas usage still differs from my local machine's.

Check out my branch and the failing CI gas check.

Also made sure that FOUNDRY_FUZZ_SEED is not set in both my local env and the ci's env

[xenide]

the only other difference that the github CI runs with the ci profile. But the config displayed by forge config does not show any difference compared to the default profile.

Any thoughts? @mds1

[mds1]

A few possibilities:

• The file you are checking against may be out of date and need to be rengenerated with the new fuzz seed
• Your CI has FOUNDRY_PROFILE: ci, but the config file has no ci profile
• The way you currently specify the seed only applies to the default profile, so if you use a different profile it may not have the seed set. Check out the foundry book for more info here
• Try seed = 1 e.g. a number instead of a hex string. I don't recall offhand which format is required, but I've always used numbers, so maybe that matters.