Fix: Auto-increment seed across batch_run iterations #2841

EwoutH · 2025-10-07T18:18:03Z

Problem

When using batch_run() with a single seed value and multiple iterations, all iterations use the same seed, producing identical results instead of independent replications.

parameters = {'seed': 42}
batch_run(MyModel, parameters, iterations=10)
# All 10 iterations use seed=42 → identical results

See #2835.

Solution

Modify _model_run_func to automatically increment the seed for each iteration: 42, 43, 44, etc.

if 'seed' in kwargs and kwargs['seed'] is not None and iteration > 0:
    seed_value = kwargs['seed']
    if isinstance(seed_value, (int, float)) and not isinstance(seed_value, bool):
        kwargs = kwargs.copy()
        kwargs['seed'] = int(seed_value) + iteration

Behavior changes

seed=42, iterations=3: currently all use 42, now uses 42, 43, 44
seed=[42, 43, 44], iterations=1: unchanged
No seed specified: unchanged (random)

Code that passes a single seed with multiple iterations will get different results. The current behavior seems like a bug (why run multiple identical iterations?), but this technically breaks existing code.

Review

I'm in doubt about this. What if users change have other random elements in their model? Do we do good obscuring this?

Secondly, is this a bugfix or a breaking change? Should we treat it as a fix and merge, or wait for a major version?

Might close #2835. @dylan-munson curious what you think.

github-actions · 2025-10-07T18:26:55Z

Performance benchmarks:

Model	Size	Init time [95% CI]	Run time [95% CI]
BoltzmannWealth	small	🔵 +0.5% [-0.2%, +1.2%]	🔵 -0.1% [-0.2%, +0.1%]
BoltzmannWealth	large	🔵 -0.3% [-1.4%, +0.8%]	🔵 +0.1% [-3.2%, +3.4%]
Schelling	small	🔵 -0.9% [-1.2%, -0.6%]	🔵 -2.1% [-2.5%, -1.6%]
Schelling	large	🔵 -1.5% [-2.2%, -0.6%]	🔵 -2.6% [-6.8%, +1.4%]
WolfSheep	small	🔵 -0.9% [-1.3%, -0.6%]	🔵 +1.0% [+0.7%, +1.2%]
WolfSheep	large	🔵 +2.1% [+1.1%, +3.0%]	🔴 +5.4% [+4.1%, +6.7%]
BoidFlockers	small	🔵 -1.5% [-2.3%, -0.5%]	🔵 -0.7% [-1.0%, -0.4%]
BoidFlockers	large	🔵 -1.3% [-2.2%, -0.4%]	🔵 -0.9% [-1.6%, -0.1%]

quaquel · 2025-10-07T20:20:08Z

I agree that this needs to be fixed. However, using subsequent integers with Mersenne Twister, Python's default RNG, is a bad idea.

From wikipedia: "A consequence of poor diffusion is that two instances of the generator, started with initial states that are almost the same, will usually output nearly the same sequence for many iterations". Also, using a seed with many zeros (like 42) is actually bad as well. One option is to just use time.time() every single time and return this seed value for reproducibility.

As an aside, numpy's rng is much better and I believe we should move all mesa code over to using this while deprecating the use of python's stdlib random library.

tpike3 · 2025-10-08T10:33:02Z

Considering how important this is, maybe we should just go all in and do the switch to numpy and its rng and then have seed options like system time and hierarchical seeding?

Does it have to be a breaking change? Could we keep the old behavior and just add a warning?

quaquel · 2025-10-08T14:03:34Z

Does it have to be a breaking change? Could we keep the old behavior and just add a warning?

Moving the internals over should be possible as a non-breaking change.

quaquel · 2025-10-28T16:21:43Z

mesa/batchrunner.py

+        seed_value = kwargs["seed"]
+        if isinstance(seed_value, (int, float)) and not isinstance(seed_value, bool):
+            kwargs = kwargs.copy()
+            kwargs["seed"] = int(seed_value) + iteration


Suggested change

kwargs["seed"] = int(seed_value) + iteration

kwargs["seed"] = seed_value + time.time()

This is all that is needed to ensure a much better spread of seeding values and thus better randomness.

EwoutH · 2025-10-28T18:03:02Z

As an aside, numpy's rng is much better and I believe we should move all mesa code over to using this while deprecating the use of python's stdlib random library.

Considering how important this is, maybe we should just go all in and do the switch to numpy and its rng and then have seed options like system time and hierarchical seeding?

Considering this, do we want to move forward with this PR?

return this seed value for reproducibility.

Where/how should we do this (without breaking API)?

When using batch_run() with a single seed value and multiple iterations, all iterations were using the same seed, producing identical results instead of independent replications. This defeats the purpose of running multiple iterations. This commit modifies _model_run_func to automatically increment the seed for each iteration (seed, seed+1, seed+2, ...) when a numeric seed is provided. This ensures: - Each iteration produces different random outcomes - Results remain reproducible (same base seed → same sequence) - Backward compatibility with seed arrays (no modification if seed is already an iterable passed via parameters) - Unchanged behavior when no seed is specified (each iteration gets random seed from OS) The fix only applies when: 1. A 'seed' parameter exists in kwargs 2. The seed value is not None 3. The iteration number is > 0 4. The seed is a single numeric value (int/float, not bool)

for more information, see https://pre-commit.ci

quaquel · 2025-10-28T20:53:17Z

Considering this, do we want to move forward with this PR?

Shifting to numpy rng requires changes in e.g., CellCollection and AgentSet, it's independent from this PR.

Where/how should we do this (without breaking API)?

The return is List[Dict[str, Any]], so in prinicple you could just insert a seed kwarg into the dict.

Alternatively, you can keep stuff as is and just document the behavior.

Or, perhaps even better: raise a ValueError if seed and iterations don't match. So, if you do iterations=10 and seed=[5,] you raise a ValueError because the number of iterations and the number of seeds don't match.

In my view, we might even consider deprecating iterations in favor of only seed, where seed is either a single SeedLike or a list of SeedLike.

for more information, see https://pre-commit.ci

quaquel · 2025-11-12T09:40:43Z

@EwoutH, I have updated this PR as discussed yesterday. I added a new kewword argument rng and deprecated iterations. I have also updated the tests accordingly.

While reviewing the code, I noticed the current use of run_id and iteration in the return value from batch_run. run_id is just the sum total of runs, while iteration gives the iteration-id of particular experiment. I am inclined to modify this (probably in a separate PR). In my mind, it makes sense to have 3 pieces of information: an identifier for the experiment (i.e., the exact parameter settings), an identifier for the iteration/replication (this might also just be the seed value instead of an integer starting from 0), and possibly an identifier for which run in total it is. Currently, batch_run only gives you the second and third, so grouping by experiment can be a bit tricky, while this is critical when calculating, e.g., the average across replications.

EwoutH

Thanks!

Since this is now a new, breaking feature, we should do it the proper way.

Could you add a section in the migration guide and link to it in the deprecationwarning? See #2872 for a recent example.

We also have to be careful about the order of our arguments. Some people use, positional arguments (stupid, I know), which we’re changing once we remove “iterations”.

Also we should properly explain this in the tutorial (can be a separate PR).

quaquel · 2025-11-12T13:40:04Z

It's technically not breaking because iterations will continue to work. It just issues a deprecation warning.
Yes, I'll take a look at adding it to the migration guide.
If you use arguments instead of keyword arguments, you deserve what you get :). Also, iterations is still there and still functions, but with a warning. So there is no problem.
Yes, I was planning to update the docs.

for more information, see https://pre-commit.ci

EwoutH · 2025-11-16T18:42:24Z

Good stuff.

I was thinking (not for this PR), wouldn’t it be useful if the results wasn’t just a dataframe, but an object?

quaquel · 2025-11-16T21:30:57Z

I was thinking (not for this PR), wouldn’t it be useful if the results wasn’t just a dataframe, but an object?

I am not sure. With the workbench, I have never seen the need for anything other than a dataframe with the experiments and a dictionary with the outcome names as keys and a numpy array as values. It might be different, however, if you want to store agent-level data over time. Still, for those use cases, you don't want to keep everything in memory.

quaquel · 2025-11-17T09:24:04Z

@EwoutH, this is ready for review. I added the docs, migration guide, and fixed the tests.

EwoutH

Looks good and complete, thanks a lot. I have a few minor comments/suggestions.

docs/migration_guide.md

EwoutH · 2025-11-17T10:33:28Z

tests/test_batch_run.py

+            "RunId": 0,
+            "iteration": 0,
+            "Step": 1000,
+            "reported_model_param": 42,
+            "AgentID": 1,
+            "agent_id": 1,
+            "agent_local": 250.0,
+            "seed": 42,


There seems to be a lot of duplicate information in this output format. Maybe we should clean this up (at some point)

I agree, but it's beyond the scope of this PR.

mesa/batchrunner.py

EwoutH · 2025-11-17T10:46:55Z

mesa/batchrunner.py

+    # establish to use seed or rng as name for parameter
+    model_parameters = inspect.signature(Model).parameters
+    rng_kwarg_name = "rng"
+    if "seed" in model_parameters:
+        rng_kwarg_name = "seed"


This behavior should be concisely explained in the batch_run docstring and tutorial

why?

Using both seed and rng on a model already gives an error. This just ensures that the seed is set to the keyword argument specified by the user in their model class, whether it is seed or rng.

Fair, that's true. Give me a moment to get up to speed with it myself.

Co-authored-by: Ewout ter Hoeven <15776622+EwoutH@users.noreply.github.com>

EwoutH · 2025-11-17T13:47:42Z

Thanks, I think I can resolve the last open comments and merge.

Can you make sure the PR description represents the final state of this PR?

EwoutH added bug Release notes label breaking Release notes label labels Oct 7, 2025

EwoutH requested a review from quaquel October 7, 2025 19:08

quaquel reviewed Oct 28, 2025

View reviewed changes

EwoutH and others added 2 commits October 28, 2025 19:03

[pre-commit.ci] auto fixes from pre-commit.com hooks

c03c6fa

for more information, see https://pre-commit.ci

EwoutH force-pushed the batch_seed branch from 85df43b to c03c6fa Compare October 28, 2025 18:03

quaquel mentioned this pull request Nov 10, 2025

Deprecating stdlib random #2884

Open

4 tasks

quaquel and others added 3 commits November 12, 2025 10:21

Merge remote-tracking branch 'upstream/main' into batch_seed

37a0839

add rng as kwarg and deprecate iterations

7f456af

[pre-commit.ci] auto fixes from pre-commit.com hooks

10136f9

for more information, see https://pre-commit.ci

quaquel added 2 commits November 12, 2025 10:41

fix for typo in value error message

904e796

Update batchrunner.py

7972638

EwoutH added the deprecation When a new deprecation is introduced label Nov 12, 2025

EwoutH commented Nov 12, 2025

View reviewed changes

quaquel and others added 7 commits November 16, 2025 16:28

Update migration_guide.md

529f3ac

Update migration_guide.md

7b6eaef

Update migration_guide.md

e53e16b

Update 9_batch_run.ipynb

4113a11

add support for both seed and rng

94c44e3

Update migration_guide.md

0778ff4

[pre-commit.ci] auto fixes from pre-commit.com hooks

5db0058

for more information, see https://pre-commit.ci

quaquel added 2 commits November 17, 2025 10:09

Update test_batch_run.py

27a777d

Update test_batch_run.py

0a4ea99

EwoutH commented Nov 17, 2025

View reviewed changes

quaquel and others added 4 commits November 17, 2025 12:37

Update docs/migration_guide.md

4726920

Co-authored-by: Ewout ter Hoeven <15776622+EwoutH@users.noreply.github.com>

Update mesa/batchrunner.py

7ba67df

Co-authored-by: Ewout ter Hoeven <15776622+EwoutH@users.noreply.github.com>

Update mesa/batchrunner.py

088425a

Co-authored-by: Ewout ter Hoeven <15776622+EwoutH@users.noreply.github.com>

Update batchrunner.py

435399c

	kwargs["seed"] = int(seed_value) + iteration
	kwargs["seed"] = seed_value + time.time()

Uh oh!

Fix: Auto-increment seed across batch_run iterations #2841

Are you sure you want to change the base?

Fix: Auto-increment seed across batch_run iterations #2841

Uh oh!

Conversation

EwoutH commented Oct 7, 2025

Problem

Solution

Behavior changes

Review

Uh oh!

github-actions bot commented Oct 7, 2025

Uh oh!

quaquel commented Oct 7, 2025

Uh oh!

tpike3 commented Oct 8, 2025

Uh oh!

quaquel commented Oct 8, 2025

Uh oh!

quaquel Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

EwoutH commented Oct 28, 2025

Uh oh!

quaquel commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

quaquel commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EwoutH left a comment

Choose a reason for hiding this comment

Uh oh!

quaquel commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EwoutH commented Nov 16, 2025

Uh oh!

quaquel commented Nov 16, 2025

Uh oh!

quaquel commented Nov 17, 2025

Uh oh!

EwoutH left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

EwoutH Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

quaquel Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

EwoutH Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

quaquel Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

EwoutH Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

EwoutH commented Nov 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

quaquel commented Oct 28, 2025 •

edited

Loading

quaquel commented Nov 12, 2025 •

edited

Loading

quaquel commented Nov 12, 2025 •

edited

Loading

quaquel Nov 17, 2025 •

edited

Loading