Query for set information

[kevinlul]

Collecting set information is the last piece for YAML Yugi to exceed parity with other solutions. Unlike the other data collected so far, which are contained in flat categories, sets are indexed on Yugipedia in hierarchical categories. This means that instead of a target category for sets directly containing an article about a set, categories may be nested. When querying the MediaWiki API, only the immediate members of a category are returned, including the names of child categories, but the members of those child categories are not returned. Therefore, new code is required in order to download entire category hierarchies and subscribe to updates on them. Category hierarchies are allowed to contain cycles, and while this is not expected of the categories for sets, our code should be correct even if cycles are encountered and not fall into an infinite loop.

Design

Either create or extend the current full download script to recursively download a targeted category, without falling into infinite loops. For example, after fetching https://yugipedia.com/api.php?action=query&redirects=true&generator=categorymembers&prop=revisions&rvprop=content&format=json&formatversion=2&gcmlimit=50&gcmtitle=Category:Yu-Gi-Oh!_Master_Duel_sets, the the ns=14 category items in the response should be stored in a ordered set for additional follow-up requests once the current category is completely downloaded.

To subscribe to incremental updates, the existing script can be used, but each time, it should be called with all the known descendant categories cached from the last full download, in addition to the top-level category itself. This is because the MediaWiki API only provides the immediate parent categories of an article, not all ancestor categories.

Subtasks

• Create or extend full download script to be recursive
• Test full download of https://yugipedia.com/wiki/Category:Yu-Gi-Oh!_Master_Duel_sets
• Set up incremental updates for Yu-Gi-Oh!_Master_Duel_sets
• Set up regular full downloads and incremental updates for other game formats

[kevinlul]

These are fairly scattered due to the varying types of product that exist. Examples:

• https://yugipedia.com/wiki/Category:Worldwide_English_Set_Card_Lists
• https://yugipedia.com/wiki/Category:TCG_Booster_Packs
• https://yugipedia.com/wiki/Category:Booster_Packs

[kevinlul]

Recursive on

• https://yugipedia.com/wiki/Category:TCG_sets
• https://yugipedia.com/wiki/Category:OCG_sets
• https://yugipedia.com/wiki/Category:Yu-Gi-Oh!_Rush_Duel_sets
• https://yugipedia.com/wiki/Category:Yu-Gi-Oh!_Master_Duel_sets
• https://yugipedia.com/wiki/Category:Yu-Gi-Oh!_Duel_Links_sets

Top level for exploration: https://yugipedia.com/wiki/Category:Sets

[kevinlul]

Recursive full download logic is okay. Capture the ns=14 category items in the response https://yugipedia.com/api.php?action=query&redirects=true&generator=categorymembers&prop=revisions&rvprop=content&format=json&formatversion=2&gcmlimit=50&gcmtitle=Category:Yu-Gi-Oh!_Master_Duel_sets and store them for additional follow-up requests. It's less clear whether just the top-level category is effective for recent changes, or if all target categories need to be queried.

[kevinlul]

Must list all immediate containing categories to be effective.

[kevinlul]

Notes:

Recursive full download needs to return a list of found categories (ns=14) after each page downloaded
In the main loop, this is appended to an OrderedSet
There's an additional outer loop iterating over the OrderedSet, thus fetching all categories,
without infinite recursion in the case of cycles, because the category will already be in the set and
have been iterated

Should I just add the second return value for the list of categories or switch to OOP?

[xyj-3]

Is OrderedSet a specific thing? Also what do you mean by "add the second return value for the list of categories or switch to OOP"?

Also do you want the new downloaded files to be flat in the top level category folder or nested?

[xyj-3]

How does gcmcontinue and grccontinue work, like when are you using it and what value do you give it

[kevinlul]

I'm describing the changes that need to happen to the main logic in the download function in https://github.com/DawnbrandBots/yaml-yugipedia/blob/master/src/utils.py

Currently the category is specified to the MediaWiki API by the gcmtitle URL parameter in main.py. However, this only retrieves direct members of the category, so the download logic needs to keep track of child category pages that were retrieved, to be downloaded by another request to the MediaWiki API. I mentioned an OrderedSet because that is one way to keep track of the categories already downloaded and newly discovered in order to avoid infinite looping.

gcmcontinue and grccontinue are pagination tokens in the response from MediaWiki APIs when a generator is used, when the results don't fit on a single page of results. In the download scripts, this is populated from the previous request so it downloads all pages, but can also be provided on the command-line to restart a previous set of downloads from the middle. The parameter varies by script. For main.py, the generator CategoryMembers is used, so the parameter is GCMcontinue. For incremental.py, the generator RecentChanges is used, so the parameter is GRCcontinue.

https://yugipedia.com/api.php
https://www.mediawiki.org/wiki/API:Query#query:generator
https://www.mediawiki.org/w/api.php?action=help&modules=query%2Bcategorymembers
https://www.mediawiki.org/w/api.php?action=help&modules=query%2Brecentchanges

[xyj-3]

I got it to work with recursion and a second return value but it doesn't look that nice right now so I'm considering restructuring it.

The biggest issue so far is actually getting an identifier for the top category for preventing loops. The generator=categorymembers doesn't return any info about the category itself.

I figure you can use pageid or title to track if there is looping. So in that case it looks like you either have to

• change your categories.txt to be a list of pageids
• Make a request for every category to get the pageid/title
• Do some string manipulation to transform between "TCG_Speed_Duel_Forbidden_%26_Limited_Lists" and "Category:TCG Speed Duel Forbidden & Limited Lists"

What do you think? Do you have any preferences because otherwise I'm probably picking making another request for every category.

[kevinlul]

Feel free to restructure. I already anticipated it would be necessary and there's actually very little code in this repository. The only interface that needs to be respected for full downloads is the command-line interface. Everything else is an implementation detail.