sqlite3.DatabaseError

[RajeshTailor1]

Hi - Firstly this is a great project keep up the good work!

I seem to be encountering one issue during the following command:

airsenal_run_prediction --weeks_ahead 3

I get the following error:

sqlite3.DatabaseError: database disk image is malformed.

I have included a screen shot if this helps. Also I am attempting to run this in RHEL (Red Hat Enterprise).

[jack89roberts]

Hi @RajeshTailor1 , this seems to be happening more often so we should definitely figure out the cause and fix it properly. Could you try installing and running AIrsenal with the system python rather than in a conda environment? This has fixed it for other Linux users (e.g. see here #81 (comment))

[RajeshTailor1]

Thanks for this info - but you mention to install airsenal to the system Python how is that done? Also once that is done are the commands to populate the database etc the same as in your readme?

[jack89roberts]

Sorry I should have explained. You can follow the installation instructions in the readme but ignore the conda parts basically. You already have the repo now so the following should work (run from the AIrsenal directory):

conda deactivate
pip install .

You should then be able to run all the commands as normal (but without doing conda activate airsenalenv before).

P.S. The above will work assuming you have python 3.5, 3.6 or 3.7, which you can check by doing python --version after conda deactivate above. If you don't get a version starting with 3.5, 3.6, 3.7 we might need to change things a bit.

[RajeshTailor1]

So looks like python --version didn't work however python3 --version returns Python 3.8.2

Would I need to install Python v3.7?

[RajeshTailor1]

Apologies I also don't have pip but pip3 seems to work fine.

[jack89roberts]

Yes you can replace the install above with pip3 in that case. I had a problem running it on 3.8 recently (#242 ) but you can give it a try.

[nbarlowATI]

I think one thing we can try here is have an option that doesn't use the multiprocessing...
I'll have a go at that and test it on an Ubuntu VM...

[nbarlowATI]

So branch bugfix/251-db-error-linux changes the default behaviour of airsenal_run_prediction so it runs single-threaded, and this seems to fix this issue when running on Ubuntu in the conda environment.

The previous multiprocessing behaviour can be accessed (for those on other systems) with the --num_thread <N> argument (where there's no real benefit to having N>4).

[RajeshTailor1]

This is an awesome development 👍🏽

What would be the optimal setup for the fastest processing time? 4cores?

[jack89roberts]

The points prediction is done with one thread per position (GK, DEF, MID, FWD) maximum, so you won't get much/any benefit from having more than 4 cores for that. The transfer optimisation will benefit from having as many cores as possible though - it should continue to get faster with up to 27 threads for the 3 week optimisation (one thread per possible transfer strategy)!

[nyutuml]

Seems the bugfix for linux is the issue. I am WSL user, sometime when i run "airsenal_run_prediction --weeks_ahead 3", the prediction only conducted for 1 position only, and it is random. Sometime, it doesn't give any results at all.

Sorry for my english

[nbarlowATI]

Hi @nyutuml I also saw this on an Ubuntu VM, caused by the sqlite3 error "database disk image is malformed".
Have you tried with the branch bugfix/251-db-error-linux though? This fixed it for me (basically just by running single-threaded).

[nyutuml]

@nbarlowATI I misunderstood things then. Instead of an issue, branch bugfix/251-db-error-linux is the solution, isnt it? So how do i apply this bugfix?

[nbarlowATI]

@nyutuml if you do (from the main AIrsenal/ directory)

git pull
git checkout bugfix/251-db-error-linux
pip install .

then retry, it should use the code on this branch, that should fix the issue for Ubuntu.

[nyutuml]

done that, here's the result

i also experienced issue #257
what should i do next?
Thanks

[nbarlowATI]

Hi @nyutuml , for #257 we will look to have a proper fix implemented in the next weeks, but in the meantime, if your team was created for gameweek 2, you could make the temporary change suggested by @samscnp in that issue (change the argument in get_players_for_gameweek from 1 to 2 in the fill_initial_team function in transaction_utils.py.

However, the above is still not even getting as far as the optimization - the prediction still isn't working :(
Are there errors above the screenshot your posting? Anything SQLite3 related?

[nyutuml]

yes, there was sqlite3.DatabaseError like what @RajeshTailor1 mentioned above.

I also have tried on Mac, but i got error due to bpl like @jack89roberts has mentioned in #81 . I have tried using conda install -c psi4 gcc-5, it returned with package libcxx conflict.

Thanks a lot for helping out, really appreciate it.

[nbarlowATI]

Hi @nyutuml could you copy/paste in the lines just above the sqlite3 error? I'm interested in whether there are any lines like the

Process Process-4:
Process Process-3:

in the original error message above - these would suggest that multiprocessing is still running, which it shouldn't be if that branch is working correctly...

(Also, just to check, you're not adding the --num_thread argument are you? If so, could you try without?)

[nyutuml]

Yes I was adding the --num_thread argument. Here's the error without the argument

There was also error Process Process-10 for every position (GK, DEF, MID, FW) prediction error

[RajeshTailor1]

It might be worth a try to run the same command but run it with the --num_thread 1 command so forcing it to use a single thread, this may resolve the DB errors.

[nbarlowATI]

@nyutuml I think for some reason it's not picking up the bugfix/251-db-error-linux branch - in the error message you show above it is definitely still running multithreaded, while in that branch if you don't add --num_thread it should run single-threaded.

(We can see if it's running single threaded - there shouldn't be any Process lines in the output, and after each Generating player history dataframe line it will go through all the "Filling history dataframe" lines.)

Unfortunately there is a bug in develop that even the num_threads is always 4 no matter what you set it to on the command line - this will be fixed when we merge the above branch into develop.

Were there any errors when you did the pip install . on the bugfix/251-db-error-linux branch ?

[nyutuml]

Hi @RajeshTailor1 , did add the --num_tread 1, the error still there.
@nbarlowATI no error thou

[nbarlowATI]

Hi @nyutuml this is very strange - I don't know how/why it's still doing multithreading when using that branch.
If you have a bit of time, could you try the following?

python
>>> from airsenal.framework.utils import *
>>> from airsenal.scripts.fill_predictedscore_table import *
>>> team_model = get_fitted_team_model(CURRENT_SEASON, session)
>>> player_model = get_player_model()
>>> predictions = calc_predicted_points_for_pos("FWD", [3,4,5], team_model, player_model, CURRENT_SEASON, "dummy", session)
>>> for preds in predictions.values():
...    for gw_pred in preds:
...             session.add(gw_pred)
...
>>> session.commit()

If that works without any trouble, it will have added all the predictions for FWDs to the database - you can then repeat with "GK", "MID", "DEF" in place of "FWD" in the call to calc_predicted_points_for_pos.

Basically this is just doing the steps in the fill_predictedscore_table script by hand, but explicitly doing the positions one-at-a-time means there is no chance that multi-threading problems can come up...

[RajeshTailor1]

So maybe try the following

git checkout bugfix/251-db-error-linux

followed by

git pull

followed by

pip install --upgrade --force-reinstall .

Then try removing the database

rm /tmp/data.md

Then try rebuilding the database with the normal command

Then try running the prediction.

[nyutuml]

@nbarlowATI @RajeshTailor1 Guys it worked! Have you just rolled the update?
Thanks a lot for helping me out

[RajeshTailor1]

@nyutuml - thats great to hear what did you do to get it working?

[nyutuml]

just reinstalled it. tried to pull branch bugfix/251-db-error-linux as you said above, the result was already up to date (previously, i couldn't pull the bugfix).
I ran the prediction, and voila!!

[nbarlowATI]

@nyutuml Excellent news! :)
I did push a small fix this afternoon about when predictions are added to the database but I don't think that was related to the problems you were seeing before.
Great that it's working now though!

I think when you run the optimization (airsenal_run_optimization), you might also need to use --num_thread 1 to avoid multithreading-related issues there...

[nyutuml]

lol, just encountered the issue when i ran the optimization. saw Process Process-(number) error again.
@nbarlowATI well noted. Thanks