Optimization suggests too expensive transfers which are not possible with current money

[wattsra]

after running airsenal_run_pipeline. The suggested optimisation shown below is not possible given the budget. (I am 0.1m short)

Do I need to clean out the database and start again everytime I refresh it?

[jack89roberts]

You may need to run airsenal_update_db first to get the latest player prices @wattsra , but I'll look into whether there's a bug (we have had problems with this before but I thought we'd fixed it).

[wattsra]

I thought it might have been that as well. but I deleted my database and then started from scratch. First I ran:

airsenal_setup_initial_db
then
airsenal_check_data
then
airsenal_update_db
then
airsenal_run_prediction --weeks_ahead 3

all ran through and then gave me the same result.

only things to note are:

Adding transaction: gameweek: 5 removing player 403 for 69
Adding transaction: gameweek: 5 adding player 284 for 66

and

SUMMARY
-------
Seasons: ['2021', '1920', '1819', '1718']

season_num_teams: OK!
season_num_new_teams: OK!
season_num_fixtures: OK!
fixture_player_teams: OK!
fixture_num_players: FAIL! 115 errors.
fixture_num_goals: FAIL! 1 errors.
fixture_num_assists: OK!
fixture_num_conceded: OK!

OVERALL: Passed 6 out of 8 tests with 116 errors.

[jack89roberts]

Could you let me know your team ID so I can try to recreate it myself? It's all working as expected as far as I can tell for our team ID.

[wattsra]

Sure thanks for your help. My team_id is 4598470

[wattsra]

Is it normal to use these seasons
Seasons: ['2021', '1920', '1819', '1718']

[jack89roberts]

Yes, we currently fit all the models with data from the current season + the previous 3 seasons, so that's normal.

I think it's behaving correctly for me with your ID, or at least it correctly says you are £0.1m short of affording those transfers.

Did you install AIrsenal quite recently or was it a while ago? If it was a while ago you could try updating by doing this from the parent AIrsenal directory:

git pull
pip install --upgrade --force-reinstall .

(and then re-run everything)

Not sure what else to try next but will think about it tomorrow.

[jack89roberts]

Hmm I get the same recommended transfers as you, so this must be a bug in the optimisation code somewhere.

[jack89roberts]

Actually no, I'm wrong - I get the same recommended transfers in, but it suggests transferring out Alonso and Van de Beek, which you can afford with £0.3m spare (according to AIrsenal at least).

I'd try reinstalling as described above, if that doesn't work I'm out of ideas.

[wattsra]

Yes, we currently fit all the models with data from the current season + the previous 3 seasons, so that's normal.

I stupidly thought this was a bug and was trying to use information from the year 1920 not for the 2019 to 2020 season!!

So I did as you suggested and did the full reinstall which worked - though I am now dubious on the algorithm as it has left me with two players that are injured on the team... but that's just my luck I guess! 😆

[jack89roberts]

Yeah, I think it's choosing to remove your more expensive players that have been benched/haven't played much recently, rather than your cheaper injured players. Improving our estimate of how likely someone is to play is definitely something we want to improve, though.

[wattsra]

Thanks, yeah - its a pretty interesting tool in any case.

I don't know if it is related, but whenever I try and run_optimization for more than 3 weeks. I get a range of sqlalchemy errors.
including:

sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) database disk image is malformed
[SQL: SELECT player_attributes.id AS player_attributes_id, player_attributes.player_id AS player_attributes_player_id, player_attributes.season AS player_attributes_season, player_attributes.gameweek AS player_attributes_gameweek, player_attributes.price AS player_attributes_price, player_attributes.team AS player_attributes_team, player_attributes.position AS player_attributes_position, player_attributes.transfers_balance AS player_attributes_transfers_balance, player_attributes.selected AS player_attributes_selected, player_attributes.transfers_in AS player_attributes_transfers_in, player_attributes.transfers_out AS player_attributes_transfers_out
FROM player_attributes
WHERE ? = player_attributes.player_id]

and

Traceback (most recent call last):%|                                                                                                                                                                                 | 0/100 [00:02<?, ?it/s]
  File "/home/robert/anaconda3/lib/python3.8/site-packages/sqlalchemy/engine/result.py", line 1284, in fetchall                                                                                                      | 0/100 [00:00<?, ?it/s]
    l = self.process_rows(self._fetchall_impl())                                                                                                                                                         | 1/43906 [00:00<2:17:39,  5.32it/s]
  File "/home/robert/anaconda3/lib/python3.8/site-packages/sqlalchemy/engine/result.py", line 1230, in _fetchall_impl
    return self.cursor.fetchall()
sqlite3.DatabaseError: database disk image is malformed

[jack89roberts]

Hmm, there are some nasty sqlalchemy/multiprocessing issues hanging around, e.g. see #81

We don't have fixes, but not using conda or running on a single thread has worked for some people. Or you could try running AIrsenal in a docker container (see the readme).