Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Small change to derive_predictors_and_scores for speed + normalization #119

Open
wants to merge 138 commits into
base: staging-collab-2
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 135 commits
Commits
Show all changes
138 commits
Select commit Hold shift + click to select a range
8365331
add_velocity function
PalkaPuri Aug 7, 2024
e2ee500
velocity predictor WIP
PalkaPuri Aug 8, 2024
6a19ae4
wip
PalkaPuri Aug 8, 2024
b5dde8a
format/lint
PalkaPuri Aug 8, 2024
a923457
specify forager column as int
PalkaPuri Aug 12, 2024
8cc4111
add warning
PalkaPuri Aug 12, 2024
892da93
format/lint
PalkaPuri Aug 12, 2024
f675d23
rename f,t for clarity
PalkaPuri Aug 13, 2024
1ee7caa
rerun notebook
PalkaPuri Aug 13, 2024
fcd34b2
add docstrings
PalkaPuri Aug 13, 2024
dc950c2
format/lint
PalkaPuri Aug 13, 2024
205235f
use inbuilt function for gaussian pdf
PalkaPuri Aug 13, 2024
b476a17
merge changes from pp-collab2-compute-velocity
PalkaPuri Aug 13, 2024
045e7d0
updated visualization
PalkaPuri Aug 13, 2024
05b3d90
add handling of nan values in visualization
PalkaPuri Aug 15, 2024
5774858
add _generate_pairwise_copying predictor
PalkaPuri Aug 15, 2024
41a7787
format/lint
PalkaPuri Aug 15, 2024
ab6ee7e
type hints
PalkaPuri Aug 15, 2024
47679f3
add pairwise predictor and animation function
PalkaPuri Aug 21, 2024
fac3f10
change velocity to backward looking
PalkaPuri Aug 21, 2024
ddf4572
Merge branch 'pp-collab2-compute-velocity' of https://github.com/Basi…
PalkaPuri Aug 21, 2024
4b242d8
docstrings and type hints
PalkaPuri Aug 21, 2024
82223c0
generate function and typehints
PalkaPuri Aug 21, 2024
11f5dce
Merge branch 'staging-collab-2' of https://github.com/BasisResearch/c…
PalkaPuri Aug 21, 2024
0dec71b
fixed naming, nan handling
PalkaPuri Aug 21, 2024
ea399b2
refactoring
PalkaPuri Aug 22, 2024
9a2ec97
vicsek wip
PalkaPuri Aug 23, 2024
033e565
update normalization of velocity predictors
PalkaPuri Aug 26, 2024
9c6854b
distance_to_next_move WIP
PalkaPuri Aug 26, 2024
8bfa2d0
Merge branch 'pp-collab2-vicsek-predictor' of https://github.com/Basi…
PalkaPuri Aug 26, 2024
e61915f
added alternates for distance_to_next_pos
PalkaPuri Aug 26, 2024
e79917e
lint/format
PalkaPuri Aug 26, 2024
ab7df53
updated nb example
PalkaPuri Aug 26, 2024
21e0361
update scores wip
PalkaPuri Aug 27, 2024
fca3235
merge rafal's changes
PalkaPuri Aug 27, 2024
d9cadcf
absolute paths for imports in modules
PalkaPuri Aug 27, 2024
3999139
update module to allow chnaging n
PalkaPuri Aug 27, 2024
540b11b
renaming
PalkaPuri Aug 27, 2024
a7b0495
format
PalkaPuri Aug 27, 2024
d0018bd
trace code in place
rfl-urbaniak Aug 27, 2024
319aa97
rename n
PalkaPuri Aug 28, 2024
cf3d543
merge staging-collab-2
PalkaPuri Aug 28, 2024
e47d161
Merge branch 'staging-collab-2' of https://github.com/BasisResearch/c…
PalkaPuri Aug 29, 2024
92a1957
add derive_predictors
PalkaPuri Aug 29, 2024
fd2a3f3
test inference wip
PalkaPuri Aug 29, 2024
d573538
Merge branch 'staging-collab-2' of https://github.com/BasisResearch/c…
PalkaPuri Aug 29, 2024
e2d3615
derive WIP
PalkaPuri Aug 29, 2024
f897604
test inference pipeline
PalkaPuri Aug 29, 2024
5230e77
format/lint
PalkaPuri Aug 29, 2024
e8dba19
format again
PalkaPuri Aug 29, 2024
a0f024c
one more format
PalkaPuri Aug 29, 2024
e07b3ff
lint
PalkaPuri Aug 29, 2024
5faaa13
format
PalkaPuri Aug 29, 2024
b01c8cf
adding doc to local_windows WIP
PalkaPuri Aug 29, 2024
e7bfeaf
small typos in docstrings
rfl-urbaniak Aug 30, 2024
4476c6b
added predictor time to logging
rfl-urbaniak Aug 30, 2024
bcbd8d6
added time to score logging
rfl-urbaniak Aug 30, 2024
72d28f4
time to logging, a few typos
rfl-urbaniak Aug 30, 2024
8d6a2ce
format lint
rfl-urbaniak Aug 30, 2024
5c70d1f
refactored proximity
rfl-urbaniak Aug 30, 2024
eb406a7
removed old proximity code
rfl-urbaniak Aug 30, 2024
bfbf187
format lint
rfl-urbaniak Aug 30, 2024
065518c
refactored trace
rfl-urbaniak Aug 30, 2024
20b89c1
Merge branch 'staging-collab-2' of https://github.com/BasisResearch/c…
rfl-urbaniak Aug 30, 2024
bdc5bff
added generate food to init
rfl-urbaniak Aug 30, 2024
0de9af7
format, lint
rfl-urbaniak Aug 30, 2024
1f0e506
update parameter names in docstring
PalkaPuri Sep 3, 2024
a7c55cc
explanation of score_kwargs
PalkaPuri Sep 3, 2024
b14029d
update next_step_score notebook
PalkaPuri Sep 3, 2024
2b8126f
update warning and add docstring to dataObject
PalkaPuri Sep 3, 2024
f00ce02
removing reusing velocity warning
PalkaPuri Sep 3, 2024
4d11e85
removed individual drop warning, add to derive_predictors_and_scores
PalkaPuri Sep 3, 2024
66688b8
lint/format
PalkaPuri Sep 3, 2024
c3f875e
fix bug in test
PalkaPuri Sep 3, 2024
d7fe531
format
PalkaPuri Sep 3, 2024
21083e8
Merge branch 'pp-collab2-derivepredictors' into pp-collab2-housekeeping
PalkaPuri Sep 3, 2024
2b8ecb2
local_windows update constraint implementation, add docstring
PalkaPuri Sep 3, 2024
5df34c7
rename variables
PalkaPuri Sep 3, 2024
62f0cb2
filter_by_distance update constraint implementation
PalkaPuri Sep 3, 2024
728a182
remove fps argument from subsampling func
PalkaPuri Sep 3, 2024
c7bc9bf
velocity- rename predictorID, update constraint implementation, docst…
PalkaPuri Sep 3, 2024
b583275
ensure all test notebooks run
PalkaPuri Sep 3, 2024
a621300
format/lint WIP
PalkaPuri Sep 3, 2024
e33c5f8
Merge branch 'ru-new-trace' into pp-collab2-housekeeping
PalkaPuri Sep 3, 2024
1b28cc3
ensure rafals nbs run
PalkaPuri Sep 3, 2024
c4ddd34
Merge branch 'staging-collab-2' of https://github.com/BasisResearch/c…
PalkaPuri Sep 4, 2024
2d9829e
type hints for kwargs in constraint func
PalkaPuri Sep 4, 2024
2ea9f12
add collab2 notebooks to automatic testing
PalkaPuri Sep 4, 2024
953558e
small fixes to local_windows.py docstrings
rfl-urbaniak Sep 4, 2024
f75ec48
interpunction in filtering.py
rfl-urbaniak Sep 4, 2024
d9249c1
proximity notebook now works
rfl-urbaniak Sep 5, 2024
575db78
wip
rfl-urbaniak Sep 5, 2024
2bd0546
fixed trace and derivation notebooks
rfl-urbaniak Sep 5, 2024
5913050
resolved type hinting problems
rfl-urbaniak Sep 5, 2024
f3f7e4f
fix some bugs
PalkaPuri Sep 5, 2024
bba492b
Merge branch 'staging-collab-2' of https://github.com/BasisResearch/c…
PalkaPuri Sep 5, 2024
dd1816c
notebook issue fix
PalkaPuri Sep 5, 2024
b4edb96
fix type hint in proximity
PalkaPuri Sep 5, 2024
ca693a8
Merge branch 'pp-collab2-housekeeping' of https://github.com/BasisRes…
rfl-urbaniak Sep 5, 2024
b88937e
grid_constraint_params updated to dict
PalkaPuri Sep 9, 2024
35f9131
interaction_constraint_params updated to dict
PalkaPuri Sep 9, 2024
1bcf668
make sure all notebooks pass + format/lint
PalkaPuri Sep 9, 2024
5572734
lint
rfl-urbaniak Sep 9, 2024
59e6215
lint
rfl-urbaniak Sep 9, 2024
f9b92da
lint
rfl-urbaniak Sep 9, 2024
bdd9965
lint
rfl-urbaniak Sep 9, 2024
1e1a548
lint
rfl-urbaniak Sep 9, 2024
5467b05
lint
rfl-urbaniak Sep 9, 2024
cc63147
lint
rfl-urbaniak Sep 9, 2024
c2f4c82
suspended old velocity test
rfl-urbaniak Sep 9, 2024
235fb07
changed DF merge to concat
PalkaPuri Sep 11, 2024
954d9e5
rhf old update
rfl-urbaniak Sep 17, 2024
032b3f4
Merge branch 'ru-random-hungry-2' of https://github.com/BasisResearch…
rfl-urbaniak Sep 17, 2024
944b20a
re-ran hungry and random foragers
rfl-urbaniak Sep 17, 2024
5780fef
following foragers replicated
rfl-urbaniak Sep 17, 2024
4ee8c47
format, lint
rfl-urbaniak Sep 18, 2024
1d31c6b
change to sublinear
rfl-urbaniak Sep 18, 2024
f0293fb
fixing velocity_predicts WIP
rfl-urbaniak Sep 18, 2024
e3d75c6
updated trace_predictor
rfl-urbaniak Sep 18, 2024
c1c0b8b
suspend animations in velocity predictors
rfl-urbaniak Sep 18, 2024
5205da4
added access predictor
rfl-urbaniak Sep 19, 2024
ec21ab4
updated random with access
rfl-urbaniak Sep 19, 2024
233c693
updateg hungry foragers
rfl-urbaniak Sep 19, 2024
bab23a2
updated followers with access
rfl-urbaniak Sep 19, 2024
b349262
format, lint
rfl-urbaniak Sep 19, 2024
9b6e058
Merge branch 'staging-collab-2' of https://github.com/BasisResearch/c…
rfl-urbaniak Sep 19, 2024
52b339b
fixed lint
rfl-urbaniak Sep 19, 2024
22104de
fixed imports in random foragers
rfl-urbaniak Sep 19, 2024
f20a472
fixing init
rfl-urbaniak Sep 19, 2024
605ed17
format, lint
rfl-urbaniak Sep 19, 2024
f3a6aa1
restored init fix
rfl-urbaniak Sep 19, 2024
8f61e4e
bump pyro to 1.9.1
rfl-urbaniak Sep 20, 2024
e0dfbaf
revert (chirho), remove deterministic from rendering
rfl-urbaniak Sep 20, 2024
1b204de
updated warning and added eCDF normalization
PalkaPuri Sep 25, 2024
52d0f14
Some polishing of the random/hungry/followers notebooks and fixing st…
dimkab Oct 2, 2024
2150069
format lint
rfl-urbaniak Oct 3, 2024
58f526c
Merge branch 'ru-random-hungry-2' of https://github.com/BasisResearch…
rfl-urbaniak Oct 3, 2024
63e3a87
Merge branch 'staging-collab-2' into pp-collab2-upgrade-derive
PalkaPuri Oct 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -155,3 +155,11 @@ data/communicators/communicators_data/*
data/central_park_birds_2022/*
data/foraging/central_park_birds_cleaned_2022/*
.vscode/launch.json
docs/foraging/random-hungry-followers/random_hungry_followers.ipynb
docs/foraging/random-hungry-followers/random_hungry_followers.ipynb
data/foraging/locust/ds/length_s0_e10_i500_15EQ20191202.pkl
data/foraging/locust/ds/priors_sam30_15EQ20191202_s0_e10.pkl
docs/foraging/communicators/communicators_derivedDF_0.pkl
docs/foraging/communicators/communicators_derivedDF_6.pkl
docs/foraging/communicators/communicators0.pkl
docs/foraging/communicators/communicators6.pkl
6 changes: 3 additions & 3 deletions collab/foraging/locust/ds_locust_functions.py
Original file line number Diff line number Diff line change
Expand Up @@ -218,9 +218,9 @@ def forward(self, X: State[torch.Tensor]):

def bayesian_locust(base_model=LocustDynamics) -> Dynamics[torch.Tensor]:
with pyro.plate("attr", size=5):
attraction = pyro.sample("attraction", dist.Uniform(0.00001, 0.1))
attraction = pyro.sample("attraction", dist.Uniform(0.00001, 0.1)) # type: ignore
with pyro.plate("wond", size=4):
wander = pyro.sample("wander", dist.Uniform(0.00001, 0.3))
wander = pyro.sample("wander", dist.Uniform(0.00001, 0.3)) # type: ignore
locust_model = base_model(attraction, wander)
return locust_model

Expand All @@ -237,7 +237,7 @@ def locust_noisy_model(X: State[torch.Tensor]) -> None:
with pyro.plate("data", len(X["edge_l"])):
pyro.sample(
"counts_obs",
dist.Multinomial(total_count, probs=probs), # .to_event(event_dim)
dist.Multinomial(total_count, probs=probs), # type: ignore
)


Expand Down
6 changes: 3 additions & 3 deletions collab2/foraging/locust/ds_locust_functions.py
Original file line number Diff line number Diff line change
Expand Up @@ -218,9 +218,9 @@ def forward(self, X: State[torch.Tensor]):

def bayesian_locust(base_model=LocustDynamics) -> Dynamics[torch.Tensor]:
with pyro.plate("attr", size=5):
attraction = pyro.sample("attraction", dist.Uniform(0.00001, 0.1))
attraction = pyro.sample("attraction", dist.Uniform(0.00001, 0.1)) # type: ignore
with pyro.plate("wond", size=4):
wander = pyro.sample("wander", dist.Uniform(0.00001, 0.3))
wander = pyro.sample("wander", dist.Uniform(0.00001, 0.3)) # type: ignore
locust_model = base_model(attraction, wander)
return locust_model

Expand All @@ -237,7 +237,7 @@ def locust_noisy_model(X: State[torch.Tensor]) -> None:
with pyro.plate("data", len(X["edge_l"])):
pyro.sample(
"counts_obs",
dist.Multinomial(total_count, probs=probs), # .to_event(event_dim)
dist.Multinomial(total_count, probs=probs), # type: ignore
)


Expand Down
13 changes: 7 additions & 6 deletions collab2/foraging/random_hungry_followers/follower_birds.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import numpy as np
import pandas as pd

from collab.foraging.toolkit import (
from collab2.foraging.random_hungry_followers.rhf_helpers import (
construct_visibility,
generate_proximity_score,
update_rewards,
Expand All @@ -19,6 +19,7 @@ def add_follower_foragers(
getting_worse=1.5,
optimal=4,
proximity_decay=1,
initial_positions=None,
):
"""
A function to add follower foragers to a simulation.
Expand All @@ -35,9 +36,9 @@ def add_follower_foragers(
# TODO Check if different forager types mix well
how_many_foragers_already = len(old_foragers)

new_foragers = sim.generate_random_foragers(num_follower_foragers, size=1)[
"random_foragers"
]
new_foragers = sim.generate_random_foragers(
num_follower_foragers, size=1, initial_positions=initial_positions
)["random_foragers"]

for new_forager in new_foragers:
new_forager["forager"] = new_forager["forager"] + how_many_foragers_already
Expand Down Expand Up @@ -81,8 +82,8 @@ def add_follower_foragers(
new_row = {
"x": new_x,
"y": new_y,
"time": t + 2,
"forager": b + 1,
"time": t + 1,
"forager": b,
"type": "follower",
}

Expand Down
15 changes: 7 additions & 8 deletions collab2/foraging/random_hungry_followers/hungry_birds.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,12 @@
import numpy as np
import pandas as pd

from collab.foraging.toolkit import (
from collab2.foraging.random_hungry_followers.rhf_helpers import (
construct_visibility,
rewards_to_trace,
update_rewards,
)

logging.basicConfig(level=logging.INFO, format="%(asctime)s: %(message)s")


def add_hungry_foragers(
sim,
Expand Down Expand Up @@ -53,9 +51,10 @@ def add_hungry_foragers(
end=t + 1,
)["visibility"]

sim.rewards = update_rewards(
sim, sim.rewards, new_foragers, start=t, end=t + 1
)["rewards"]
if t > 0: # DB: no need to update rewards for the first frame
sim.rewards = update_rewards(
sim, sim.rewards, new_foragers, start=t, end=t + 1
)["rewards"]

sim.traces = rewards_to_trace(
sim.rewards,
Expand All @@ -79,8 +78,8 @@ def add_hungry_foragers(
new_row = {
"x": new_x,
"y": new_y,
"time": t + 2,
"forager": b + 1,
"time": t + 1,
"forager": b,
"type": "hungry",
}

Expand Down
40 changes: 24 additions & 16 deletions collab2/foraging/random_hungry_followers/random_foragers.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,10 @@
import numpy as np
import pandas as pd

from collab.foraging.toolkit import generate_grid, update_rewards
from collab2.foraging.random_hungry_followers.rhf_helpers import (
generate_grid,
update_rewards,
)


class Foragers:
Expand Down Expand Up @@ -45,7 +48,7 @@ def __init__(

include_random_foragers (bool): Whether to include random foragers in
the final output (if you only use the sim as
a starting point for another foragint strategy,
a starting point for another foraging strategy,
you might want to set this to False).

Other attributes:
Expand Down Expand Up @@ -122,9 +125,16 @@ def __call__(self):
self.rewards = rew["rewards"]
self.rewardsDF = rew["rewardsDF"]

def generate_random_foragers(self, num_foragers, size=None):
def generate_random_foragers(self, num_foragers, size=None, initial_positions=None):
if size is None:
size = self.num_frames

# set to (grid_size/2,grid_size/2) for each forager if not provided
if initial_positions is None:
initial_positions = np.array(
[[self.grid_size / 2, self.grid_size / 2]] * num_foragers
)

random_foragers = []

size_warning_flag = False
Expand All @@ -137,14 +147,12 @@ def generate_random_foragers(self, num_foragers, size=None):
p=self.probabilities,
replace=True,
)
) + (
self.grid_size / 2
) # make centered
) + (initial_positions[forager, 0])

if any(forager_x < 0) or any(forager_x > self.grid_size):
if any(forager_x < 0) or any(forager_x >= self.grid_size):
size_warning_flag = True
forager_x[forager_x < 0] = 0
forager_x[forager_x > self.grid_size] = self.grid_size
forager_x[forager_x >= self.grid_size] = self.grid_size - 1

forager_y = np.cumsum(
np.random.choice(
Expand All @@ -153,11 +161,11 @@ def generate_random_foragers(self, num_foragers, size=None):
p=self.probabilities,
replace=True,
)
) + (self.grid_size / 2)
) + (initial_positions[forager, 1])

if any(forager_y < 0) or any(forager_y > self.grid_size):
if any(forager_y < 0) or any(forager_y >= self.grid_size):
forager_y[forager_y < 0] = 0
forager_y[forager_y > self.grid_size] = self.grid_size
forager_y[forager_y >= self.grid_size] = self.grid_size - 1

if size_warning_flag:
warnings.warn(
Expand All @@ -170,8 +178,8 @@ def generate_random_foragers(self, num_foragers, size=None):
{
"x": forager_x,
"y": forager_y,
"time": range(1, size + 1),
"forager": forager + 1,
"time": range(0, size),
"forager": forager,
"type": "random",
}
)
Expand All @@ -187,11 +195,11 @@ def generate_random_foragers(self, num_foragers, size=None):
def generate_random_rewards(self, size=None):
if size is None:
size = self.num_rewards
rewardsX = np.random.choice(range(1, self.grid_size + 1), size=size)
rewardsY = np.random.choice(range(1, self.grid_size + 1), size=size)
rewardsX = np.random.choice(range(0, self.grid_size), size=size)
rewardsY = np.random.choice(range(0, self.grid_size), size=size)

rewards = []
for t in range(1, self.num_frames + 1):
for t in range(0, self.num_frames):
rewards.append(pd.DataFrame({"x": rewardsX, "y": rewardsY, "time": t}))

return {"rewards": rewards, "rewardsDF": pd.concat(rewards)}
Expand Down
Loading
Loading