Commandline arguments to fill / build certain tables

[mpope9]

Right now all tables will be populated. play_by_play tables takes a while, and maybe it isn't needed by everyone. A inclusion or exclusion list would be useful.

[avadhanij]

Hey can I pick this issue up? The load time for even a single season play_by_play table is bugging me. I am currently interested in some of the other tables.

[mpope9]

Go for it!

[avadhanij]

Planning to introduce a new option for this called - skip-tables. There are issues though, mainly because of the dependencies.

For now, outside of the base tables, both the player_game_log and play_by_play tables depend on the game table.

So to skip certain tables, these dependencies need to be factored in. E.g.:, if a user skips game by mistake, it should automatically imply skipping the ones dependent on it.

As the scope of this project increases, so do the dependencies. We could implement a simple dependency graph using a dictionary.

E.g.:

dependencies = {'player': [], 'event_message_type': [], 'play_by_play': ['player', 'game'] ....... }

Or we could just make game a base table for now, and address the dependencies later on.

What do you think?

[mpope9]

I missed this comment. So the way I set things up was relying on the player_game_log_requester to build out the game table, as the source of game id's, as there wasn't a great way to get them all. That is why I didn't consider it a base table, although it really is. HOWEVER, nothing is stopping us from just hitting the player_game_log endpoint, getting the game ids, then not filling the player_game_log endpoint. It was logic I was trying to avoid, but it seems there isn't an obvious way of going about this, so I am fine with it. Unless, we make sure that the game ids start at a certain ID.

There will always be a catch22 of complexity, so I don't mind adding more. I like the dependency graph idea, less foot-guns for users. We could poke at the ORM to generate this. However, I do want to move away from peewee and use something like pyodbc, but peewee is too convenient at the moment.

Anyways that was a bit of a ramble. Yes to game being a base table if we can sanely skip the player_game_log table, and yes to a dependency graph.

[avadhanij]

So, for now, the PR I opened, and you just merged, assumes game is a default table.

I like the idea of getting the dependencies from the ORM instead of hard cording it. I will play around this a bit more to see how to extract that info.

[avadhanij]

Up to you if you want to close this ticket and approach the other thing(dependency tracking) as a secondary enhancement.

[mpope9]

Missed this one a second time! Created a follow up issue for this. I love this feature, it has made developing this much more smooth, I don't need to mass comment and uncomment code to skip tables!