Structure abm examples for agents.jl

[Tortar]

This file isn't working anymore with the latest version of Agents.jl since some example were removed from the models folder. My idea is to reproduce the same format as in the other frameworks, having a folder for each model containing the implementation. For the two models (Schelling and Flocking) which still are inside the models folder in the main repo I will just import them and create a benchmark.jl.

So this is still a work in progress. I'm waiting for some approval to go on with these changes :-)

[Datseris]

yes, each folder should have a model, and definitely the models should be written explicitly from scratch here. they should only use Agetns.jl API, not any predefined models.

[Tortar]

all examples are now present, I also benchmarked the models against the Mesa ones, if I recall well these are the results:

- flocking -> 100x - schelling -> 200x - predator_prey -> 50x - forestfire -> 300x

so good!! but the forestfire model seems a bit different from the Mesa one in terms of design

[Tortar]

I have to say that I'm for changing forestfire in a different and slower version, since I would not consider totally fair the current implementation for the reason that we could do the same in the Schelling model, but this would not be natural as it is not natural this implementation of forestfire, moreover this is not directly comparable (while the other three are almost one-to-one) with the one in Mesa, I would propose the older version + the new datastructures giving us

using Agents

@agent Tree GridAgent{2} begin
    status::Symbol  #:green, :burning, :burnt
end

function forest_fire(; density = 0.7, griddims = (100, 100))
    space = GridSpaceSingle(griddims; periodic = false, metric = :manhattan)
    forest = UnremovableABM(Tree, space, scheduler = Schedulers.fastest)
    for pos in positions(forest)
        if rand(forest.rng) < density
            state = pos[1] == 1 ? :burning : :green
            add_agent!(pos, forest, state)
        end
    end
    return forest, tree_step!, dummystep
end

function tree_step!(tree, forest)
    if tree.status == :burning
        for neighbor in nearby_agents(tree, forest)
            if neighbor.status == :green
                neighbor.status = :burning
            end
        end
        tree.status = :burnt
    end
end

but this is 4x slower than the current one.

Also the current implementation is better because it exploits the high density of the forest, with forests with a low density of trees (more uninteresting case though) the implementation above becomes better than the current one, while the current one has always the same time of execution because it's dominated by for I in findall(isequal(2), forest.trees)

[Tortar]

ops, actually the benchmarks used different initialization, my fault, do't consider those speed-up as representative

[Tortar]

ok, now everything should be fine, but there is still the problem about forestfire to discuss

results of the benchmark

Benchmarking Julia
Agents.jl WolfSheep (ms): 7.9513799999999994 ---> x49.4
Agents.jl Flocking (ms): 16.77779 ---> x90.5
Agents.jl Schelling (ms): 0.363712 ---> x211.9
Agents.jl ForestFire (ms): 2.058977 (here I used the slower version) ---> x71.3
Benchmarking Mesa
Mesa WolfSheep (ms): 393.10966200002895
Mesa Flocking (ms): 1518.7672120000002
Mesa Schelling (ms): 77.08737300004032
Mesa ForestFire (ms): 146.85305700004392

[Datseris]

What's the problem about forest fire?

[Tortar]

That the implementation of the step is so different from the one in Mesa and Netlogo so it's not directly comparable in my opinion, and it is also less natural, it's an extra optimization which wouldn't come to mind to a regular user implementing it in Agents.jl, it exploit the fact that for I in findall(isequal(2), forest.trees) is pretty fast given that trees is a matrix, but this is just something we could try to do in all other frameworks and probably it will give some speed-up (in Mesa at least and in Mason if we had the implementation), the other frameworks instead use the normal way to code the model, I think that we should try to create a comparison where the API is used in the most natural way, and indeed this is the case for the other 3 models, so I don't think we should do the opposite with one of them, also consider that the same optimization could be done on Schelling, but as I said the model will be less comparable. So in summary, Netlogo and Mesa uses very similar code, I think we should do the same

[Tortar]

mmmh actually looking at how it is done here on Mason on their benchmark: https://github.com/isislab-unisa/ABM_Comparison/blob/main/MASON/forestfire/ForestFire.java, they are actually doing it with the same matrix approach.

[Tortar]

But I'm actually thinking that we lack a proper scheduler to handle a situation like this one, where most of the agents aren't actually activated at each step, in this representation of forestfire model we activate 1% of the agents (the burning ones) at each step on average

[Tortar]

I actually made the Mesa forestfire x8 faster using this information, this is the code for the new model in Mesa:

from mesa import Model
from mesa import Agent
from mesa.space import SingleGrid
from mesa.time import BaseScheduler

class TreeCell(Agent):
    """
    A tree cell.
    Attributes:
        x, y: Grid coordinates
        condition: Can be "Fine", "On Fire", or "Burned Out"
        unique_id: (x,y) tuple.
    unique_id isn't strictly necessary here, but it's good
    practice to give one to each agent anyway.
    """

    def __init__(self, pos, model):
        """
        Create a new tree.
        Args:
            pos: The tree's coordinates on the grid.
            model: standard model reference for agent.
        """
        super().__init__(pos, model)
        self.pos = pos
        self.condition = "Fine"

    def step(self):
        """
        If the tree is on fire, spread it to fine trees nearby.
        """
        for neighbor in self.model.grid.iter_neighbors(self.pos, moore=False):
            if neighbor.condition == "Fine":
                neighbor.condition = "On Fire"
                self.model.schedule.add(neighbor)
        self.condition = "Burned Out"
        self.model.schedule.remove(self)

class ForestFire(Model):
    """
    Simple Forest Fire model.
    """

    def __init__(self, height=100, width=100, density=0.7):
        """
        Create a new forest fire model.
        Args:
            height, width: The size of the grid to model
            density: What fraction of grid cells have a tree in them.
        """
        # Set up model objects
        self.schedule = BaseScheduler(self)
        self.grid = SingleGrid(height, width, torus=False)

        # Place a tree in each cell with Prob = density
        for cont, x, y in self.grid.coord_iter():
            if self.random.random() < density:
                pos = (x, y)
                # Create a tree
                new_tree = TreeCell(pos, self)
                # Set all trees in the first column on fire.
                if x == 0:
                    new_tree.condition = "On Fire"
                    self.schedule.add(new_tree)
                self.grid.place_agent(new_tree, pos)

    def step(self):
        """
        Advance the model by one step.
        """
        self.schedule.step()

now it takes 19 ms while before it took 147 ms. If we could do the same in Agents.jl we would give a similar speed-up with this more natural version. Indeed, with a 1000x1000 grid this new version in Mesa it's just 10x slower than the fastest version in agents.jl. So for now we should create a custom scheduler so that we can do the same in Agents.jl.

[Datseris]

That the implementation of the step is so different from the one in Mesa and Netlogo so it's not directly comparable in my opinion, and it is also less natural, it's an extra optimization which wouldn't come to mind to a regular user implementing it in Agents.jl, it exploit the fact that for I in findall(isequal(2), forest.trees) is pretty fast given that trees is a matrix, but this is just something we could try to do in all other frameworks and probably it will give some speed-up (in Mesa at least and in Mason if we had the implementation), the other frameworks instead use the normal way to code the model, I think that we should try to create a comparison where the API is used in the most natural way, and indeed this is the case for the other 3 models, so I don't think we should do the opposite with one of them, also consider that the same optimization could be done on Schelling, but as I said the model will be less comparable. So in summary, Netlogo and Mesa uses very similar code, I think we should do the same

I am not so sure I agree. The only reason to include Forest Fire is to allow a cellular automaton, which by definition is a type of system that could be solved with the matrix approach. If other ABM softwares don't allow for such optimizations, that's their fault. If you don't go the matrix approach for forest fire, there is no reason to have it: the schelling model has everything the forest fire would use, if forest fire was done with an agent based approach. Hence, since Agents.jl has the infastructure to handle matrices intuitively, due to the existence of nearby_positions, then the Forest Fire model should use a matrix approach.

There is one more reason: Julia is faster than python in any array based operation. And of course, given that we made the consious choice of writing Agents.jl in Julia, then of course we want to highlight our lightning fast matrix-based models.

[Datseris]

But I'm actually thinking that we lack a proper scheduler to handle a situation like this one, where most of the agents aren't actually activated at each step, in this representation of forestfire model we activate 1% of the agents (the burning ones) at each step on average

You should open a feature request for this: a scheduler that activates agents based on condition.

[Datseris]

I actually made the Mesa forestfire x8 faster using this information, this is the code for the new model in Mesa:

I don't think this would have such a large impact on AGents.jl; if an agent isn't burning, the agent_step! function returns immediatelly, right?

In any case, we need to first agree on what we want to measure. As I said, if you want to measure forest fire explicitly as using individual agents, then remove forest fire. Schelling does this measurement.

However, I think a cellular automaton should be in this comparison, and each software may solve the automaton as they can the fastest, which is fair. Cellular automata can always be done as matrices, no matter the complexity.

[Tortar]

it has a big impact even if the step returns immediately, this is 5x faster than the best one with the matrix approach:

using Agents, Random

@agent Tree GridAgent{2} begin
    status::Symbol  #:green, :burning, :burnt
end

function forest_fire(; density = 0.7, griddims = (1000, 1000))
    space = GridSpaceSingle(griddims; periodic = false, metric = :manhattan)
    rng = Random.MersenneTwister()
    forest = ABM(Tree, space; rng, scheduler = Burning(Set{Int}()))
    for pos in positions(forest)
        if rand(forest.rng) < density
            state = pos[1] == 1 ? :burning : :green
            agent = Tree(nextid(forest), pos, state)
            add_agent_pos!(agent, forest)
            if agent.status == :burning
                push!(forest.scheduler.trees, agent.id)
            end
        end
    end
    
    #println("Before step ", Agents.schedule(forest))
    return forest, tree_step!, dummystep
end

function tree_step!(tree, forest)
    for neighbor in nearby_agents(tree, forest)
        if neighbor.status == :green
            neighbor.status = :burning
            push!(forest.scheduler.trees, neighbor.id)
        end
    end
    tree.status = :burnt
    delete!(forest.scheduler.trees, tree.id)
    #println("removed ", tree.id)
end

struct Burning
    trees::Set{Int}
end

function (scheduler::Burning)(model::ABM)
    return collect(scheduler.trees)
end

0.10 ms instead of 0.46

I don't know why but UnremovableABM doesn't work with this version, so I used the normal ABM, remove the comments to see the difference, so I think there should be some other bug in UnremovableABM.

I think that this is faster since the simple loop on the agents is actually the greatest part of the computations involved

[Tortar]

yes, indeed I agree that Forestfire doesn't offer much of novelty in respect to Schelling, the thing is that for this version of forestfire the matrix approach is also not the best one as the code above demonstrate. However it's maybe interesting since it requires a more flexible scheduler

[Tortar]

but I agree that probably a different ABM where cellular automata approach is best could be more interesting

[Tortar]

I'm also actually thinking that the custom scheduler I'm using here could be useful for some ABMs (like this one) where some propagation effect on a subset is the main part of the model, but this has to be handled by the user for the most part as I did here, maybe it could be documented somewhere or even implemented as something like UserhandlingScheduler :D

[Tortar]

this is almost 10x ! 0.054 vs 0.46

using Agents, Random

@agent Tree GridAgent{2} begin
    status::Symbol  #:green, :burning, :burnt
end

function forest_fire(; density = 0.7, griddims = (1000, 1000))
    space = GridSpaceSingle(griddims; periodic = false, metric = :manhattan)
    rng = Random.MersenneTwister()
    forest = ABM(Tree, space; rng, scheduler = Burning(Vector{Int}(), Vector{Int}()))
    for pos in positions(forest)
        if rand(forest.rng) < density
            state = pos[1] == 1 ? :burning : :green
            agent = Tree(nextid(forest), pos, state)
            add_agent_pos!(agent, forest)
            if agent.status == :burning
                push!(forest.scheduler.this_step, agent.id)
            end
        end
    end
    
    #println("Before step ", Agents.schedule(forest))
    return forest, tree_step!, dummystep
end

function tree_step!(tree, forest)
    for neighbor in nearby_agents(tree, forest)
        if neighbor.status == :green
            neighbor.status = :burning
            push!(forest.scheduler.this_step, neighbor.id)
        end
    end
    tree.status = :burnt
    #println("removed ", tree.id)
end

struct Burning
    this_step::Vector{Int}
    next_step::Vector{Int}
end

function (scheduler::Burning)(model::ABM)
    empty!(scheduler.next_step)
    append!(scheduler.next_step, scheduler.this_step)
    empty!(scheduler.this_step)
    return scheduler.next_step
end

Just for fun I ran this vs. matrix approach for a 5000x5000 forestfire and this took 4 seconds while the matrix approach took 50, I think these approaches can be generally useful in some cases where propagation of something is the main objective to model

[Datseris]

I think the forest fire is ill defined as its setup because of the reasons you said that most of the time nothing happens everywhere. Perhaps we should consider re-structuring it so that it is the "normal" version where grass regrows and can instantaneously combust and then the fire propagates randomly.

[Datseris]

BTW, we should structure this repo as follows: each folder is a model. In the folder there is a DECLARATION.md file that declares the model as specifically as possible. Then there is an implementation file for each software with the name of the software.

[Tortar]

yeah, I think that would be the best way to structure the benchmark without making errors, I also agree that ForestFire is a bit ill defined and should be at least rebuilt in someway, but some of the idea discussed here could be useful anyway.

Will merge this PR as it is now, so that we move forward in some other PR since the conversation here is becoming pretty long

[Tortar]

Hi @Datseris, I have modified a bit the repo, let me know what you thing of the changes when you have time, surely the README.md (soon DECLARATION.md if you think it's better) is quite basic right now. What I'm thinking is that we could try to run different configuration maybe automatically, setting the parameters from the outside considering all files as normal text files to be completed with parameters. One would define the parameters in the runall.sh and the script will modify the files before running the benchmarks

[Tortar]

I have also created this little procedure to automatically set everything for the benchmark (actually I realized now that I didn't consider dependencies and Agents itself xD -> done), but let me know what do you think in general in specifing all of this inside the main readme:

# fetch update software list
sudo apt-get update

# clone the repository and give permissions
sudo git clone https://github.com/JuliaDynamics/ABM_Framework_Comparisons.git
sudo chmod a+rwx ABM_Framework_Comparisons
sudo chmod -R 777 ABM_Framework_Comparisons

# install java
sudo apt install default-jre-headless
sudo apt install default-jdk-headless

# install julia
sudo wget https://julialang-s3.julialang.org/bin/linux/x64/1.9/julia-1.9.0-linux-x86_64.tar.gz
sudo tar zxvf julia-1.9.0-linux-x86_64.tar.gz
printf "\nexport PATH=\"\$PATH:"$(pwd)"/julia-1.9.0/bin\"" >> ~/.bashrc
exec bash

# install agents
julia --project=ABM_Framework_Comparisons -e 'using Pkg; Pkg.instantiate()'

# install mesa
sudo apt install python3-pip
pip install mesa==1.2.1

# install netlogo
sudo wget http://ccl.northwestern.edu/netlogo/6.3.0/NetLogo-6.3.0-64.tgz
sudo tar -xzf NetLogo-6.3.0-64.tgz

# move netlogo inside repository
sudo mv "NetLogo 6.3.0" netlogo
sudo mv netlogo ABM_Framework_Comparisons

# install parallel tool
sudo apt install parallel

# move to repo folder
cd ABM_Framework_Comparisons

[Tortar]

also, if the thing is sound, let me know if everything works after running these commands

I tested it on an already partially set up Ubuntu, hopefully it should work even from scratch

edit: tested carefully on Ubuntu, not it works very well