Mismatch in ForestFire implementation

[Corvince]

So I spent quite some time figuring out how to improve the performance of mesas ForestFire implementation. I couldn't believe such a large discrepancy for such a simple model. Most of the time is spent on attribute access and function calls. That is not a nice place to oprimize.

Then I looked more closely at the Agents.jl code and I think Agents.jl is almost cheating ;) Its a clever implementation, but it is very different from the other frameworks. Agents.jl uses dummy agents and a agent dummy. That is instead of using actual Agents on an actual Grid, it uses a matrix which holds the tree states. So instead of looking up tree attributes and calling a step function for each tree the logic is implemented only on this matrix.

Replicating the same approach in Python, I see a ~10x performance increase for Mesa.

Now I don't know which side should be changed - make agents.jl implementation more standard and explicit, but slower, or allow other frameworks to use the same tricks to make them faster. For Mesa, I could send in a PR

[Datseris]

Agents.jl isn't cheating, have a look at the declaration file:

https://github.com/JuliaDynamics/ABM_Framework_Comparisons/blob/main/ForestFire/DECLARATION.md#technical-implementation

the only reason to include forest fire is exactly to test what optimizations frameworks can do in cellular automata. If you have a faster version for Mesa go ahead and PR it, that would be totally fair.

Agents.jl has infrastructure that allows for such optimizations in cellular automata, such as the nearby_positions function that loops over nearby positions in an optimized radius-agnostic way.

[Corvince]

Ah yes, sorry, I have missed the readme. Indeed, it explicitly states that agents are not needed. I still find it strange to compare an optimized model with a non optimized in the first place, but I am happy to provide an optimized version for Mesa via PR

[Datseris]

sure, but we can't always know what's a Mesa optimized model, as we are Agents.jl users and not Mesa users, so contributions like yours are needed.

[Tortar]

I think that indeed with a numpy array it could be possibly also done in Mesa, but it is not clear to me how much it would be faster for the overhead in the conversion, but if you have a better implementation please do a pr @Corvince !

[Corvince]

I think that indeed with a numpy array it could be possibly also done in Mesa, but it is not clear to me how much it would be faster for the overhead in the conversion, but if you have a better implementation please do a pr @Corvince !

Yes with numpy one could build a more equivalent example, which might even be slightly faster, but I think a list of lists should also be fine. Submitted a PR

[Tortar]

I think that anyway ForestFire should be reimplemented in a way such that the fire starts randomly on the grid, we already discussed the implementation and a sort of optimization similar to yours in #13 but didn't decide to implement, of course I will merge your new way, but I think that it is a bit ill-suited the ForestFire model right now since the dynamics is a bit too simple

[Tortar]

But actually rethinking about it, the way it is now it's probably good enough

[Datseris]

personally I am not particularly happy with PRs #65 and #64 : I think we are losing track here. Focusing too much on the tree and losing sight of the forest (pun intended). When we originally designed this repository, we had in mind the Julia benchmarks . They compare programming patterns when implemented in each language. They do not compare "what's the fastest way to solve problem X with given language", and they make this clear in the benchmark declarations.

Yet, that's exactly what we have started doing here: "what's the fastest way to evolve an ABM with given rules in the used framework". We have shifted focus from comparing usage patterns in ABM to micro-optimizations. I would argue that neither #64 nor #65 are a "typical" programming pattern in ABMs, and in fact, neither uses any public API besides the function that iterates over nearby positions. In PR #64 the only line that actually uses Mesa is line 45: for nx, ny in self.grid.get_neighborhood(... and similarly for the Agents.jl implementation. We already compare this function in Schelling model, so what comparison do we gain from the Forest Fire model anymore? The rest of the code is how quickly Julia or Python iterates over a tracked list of 2-tuples.

@Tortar raised a good point: the forest fire is too trivial to highlight any meaningful usage of an ABM framework. Perhaps we should consider discarding it. Or, we should consider adding very strict technical implementation guidelines in the declaration that ensure that more of the API of each framework is used so that we compare ABM frameworks and not the core of the programming languages.

@Corvince feel free to counter-argue or prove me wrong about the above points of course.

[Datseris]

p.s. I am in favor of discarding it. We can add a graph based model if we want to have four examples.

[Corvince]

@Corvince feel free to counter-argue or prove me wrong about the above points of course.

Quite the opposite! I am very delighted to hear your opinion. I think you are on point with everything you wrote. Before opening this issue I thought about how to make a point about abandoning the forest fire example. After I realized that the Mesa example couldn't be sped up in a way that would reflect typical Mesa code, I was kind of annoyed to have this example here, because as you said it actually only benchmarks a single public API endpoint - the neighborhood 'search'. And yet it makes mesa look extremely slow compared to Agents.jl - 500x slower. When it only shows that neighborhood search is much faster in Agents.jl and more complex models like Wolf-Sheep are an order of magnitude faster.

Only after that I realized that Agents.jl wasn't actually using agents, didn't put any content on the actual grid and isn't using a scheduler. So I very much agree with your point that both models don't use regular patterns (after my PR) - I just think that was already true before #65. But now I think I should have made that point more explicit in the first place.

In any way I support removing ForestFire as being too simple. It shows that Julia is a fast programming language - but not much about framework code.

[Datseris]

ok i'll merge both #64 and #65 (mainly because we should keep @Corvince in the commit history) and then Tortar you can make a PR that removes the forest fire and adjust the comparison table accordingly.

[Tortar]

Then we all agree, I will remove the model soon, and I also agree on replacing it with a graph-based model instead. But which one? Do you have any proposal? It would be surely better if we can test a model which is already implemented in all frameworks.

[Corvince]

Actually I think most of the examples suffer from similar issues - the number of actual API calls is quite low in all of the examples.

In all honesty I don't see much benefit in these performance benchmarks as they are now. For small and simple models the difference is irrelevant - even the slowest implementation will be plenty fast. And for large and complex models most of their computational time will be spent outside of the framework code. I think it doesn't add much more than saying Julia is fast(er).

I think there are very few models where the performance difference between the frameworks tested here really matters. Again, for small models or classical ABMs it wont matter. And if someone is trying to simulate millions of agents for thousands of time steps I am pretty sure thats above Agents.jls limits with respect to the typical patterns used.

[Corvince]

Just to be clear about my opinion about Agents.jl. If someone were to ask me what modeling framework I would recommend for ABMs and they have no prior programming knowledge, I would say Agents.jl - but not because of its performance advantage.