Parse Infoboxes in Wikimedia dumps?

[rexdouglass]

This package is excellent, I've been trying to parse the raw xml dump for infoboxes and this package does it very well.

Is there a way to point wptools at a local copy of a wikipedia dump, so I can parse every single info box locally without having to hit wikipedia's servers over and over?

[siznax]

Thanks for your question, @rexdouglass!

If you are running a local instance of MediaWiki, then you can use wiki=<URI> to access it with wptools like this, for example:

>>> page = wptools.page('A', wiki='http://localhost:8080')

If you have a local copy of a Wikimedia dump, then we may be able to work with it. We don't have a solution out-of-the-box yet. It's a splendid idea. Let's talk about how to accomplish it...

First of all, we do NOT attempt to parse wikitext or HTML. Instead, we parse the XML parse tree offered by the API parse module.

So, if your dump article contains the XML parse tree, then we can just point wptools.utils.get_infobox() at it. If not, we can either transform it with some other tool, or add a new method to handle it as-is. (As you probably know, creating a correct article parse tree outside of a MediaWiki instance is non-trivial—or impossible—so we really do need that XML parse tree.)

Alternatively, you could setup a local instance of Mediawiki, import an XML dump, and then point to the local wiki's API with the wiki argument as described above.

What does an article from your local dump look like?

[rexdouglass]

Excellent! There is a lot of specialized knowledge here, so I'm going describe my experience to date without any assumptions it's shared experience, and likewise I haven't worked much with MediaWiki or the live APIs.

My direct experience is with enwiki-latest-pages-articles.xml.bz2
Which is available https://dumps.wikimedia.org/enwiki/latest/
I believe the format and details for how it's created are described here https://en.wikipedia.org/wiki/Help:Export

Here are the first 1000 lines of the dump https://pastebin.com/hfDAadux

The example on the page is representative of xml from the dump, millions and millions blocks.

<mediawiki xml:lang="en">
   <page>
     <title>Page title</title>
     <!-- page namespace code -->
     <ns>0</ns>
     <id>2</id>
     <!-- If page is a redirection, element "redirect" contains title of the page redirect to -->
     <redirect title="Redirect page title" />
     <restrictions>edit=sysop:move=sysop</restrictions>
     <revision>
       <timestamp>2001-01-15T13:15:00Z</timestamp>
       <contributor>
         <username>Foobar</username>
         <id>65536</id>
       </contributor>
       <comment>I have just one thing to say!</comment>
       <text>A bunch of [[text]] here.</text>
       <minor />
     </revision>
     <revision>
       <timestamp>2001-01-15T13:10:27Z</timestamp>
       <contributor><ip>10.0.0.2</ip></contributor>
       <comment>new!</comment>
       <text>An earlier [[revision]].</text>
     </revision>
     <revision>
       <!-- deleted revision example -->
       <id>4557485</id>
       <parentid>1243372</parentid>
       <timestamp>2010-06-24T02:40:22Z</timestamp>
       <contributor deleted="deleted" />
       <model>wikitext</model>
       <format>text/x-wiki</format>
       <text deleted="deleted" />
       <sha1/>
     </revision>
   </page>
   
   <page>
     <title>Talk:Page title</title>
     <revision>
       <timestamp>2001-01-15T14:03:00Z</timestamp>
       <contributor><ip>10.0.0.2</ip></contributor>
       <comment>hey</comment>
       <text>WHYD YOU LOCK PAGE??!!! i was editing that jerk</text>
     </revision>
   </page>
 </mediawiki>

[siznax]

Thanks for the examples, @rexdouglass. Hehe, yes, specialized knowledge. I'll share some more and try to avoid being pedantic. The XML dump is tantalizing, and working with it is what inspired me to make this package. However, it has a problem with respect to extracting data. If you take a look at the content model from your examples, you'll see that it is "wikitext":

      <model>wikitext</model>

Because wikitext is unstructured, the difficulty in parsing it (outside of a MediaWiki instance) is significant. Infobox templates contain all manner of wikitext: formatting, links, HTML/entities, comments, various other metadata, as well as other templates!

It may seem like a reasonably sane (Infobox) world, but the devil are in the details. Let me explain. You may have some exciting success parsing select wikitext examples, but then you'll need to add more and more corner cases to accommodate more weirdness. You may even want to hit the API to parse some idiosyncratic wikitext coventions in use in many places. Take a look at Taxoboxes for instance (which warns: "This template employs intricate features of template syntax.") and some of our related issues #62, #66, #91, #109.

Eventually, you'll discover Wikidata... (the heavens ring out in harmony: AUHHH!) which solves two huge problems with getting data out of Wikipedia. It has a well-defined, standard structure, and that structure is designed to support all languages. A surprise for me was discovering that wikitext syntax can be different in different language instances of Wikipedia (e.g. "Ficha de taxón" [es] versus "Taxobox" [en]). Some of us are programming in English, but there are wikitexts in many other languages. Wikidata dispenses with the complexity of idiosyncratic/surprising wikitext and it works the same for all languages. All hail Wikidata!

This package supports Wikidata and I encourage you to use wptools.page.get_wikidata(). The problem with Wikidata is that it is often data-poor where the analogous Infobox is data-rich. This is because Wikidata is relatively young. Another goal of this package is to help that situation. Let's get the info out of Infoboxes and into Wikidata.

You can browse the wikitext of Infoboxes to get a better sense of what we're up against simply by adding ?action=raw&section=0 to a Wikipedia URL. This will let you inspect the wikitext of the so-called lead-section. Here's one from the example you posted:

https://en.wikipedia.org/wiki/Autism?action=raw&section=0

{{Infobox medical condition (new)
| name            = Autism
| image           = Autism-stacking-cans 2nd edit.jpg
| alt             = Boy stacking cans
| caption         = Repetitively stacking or lining up objects is associated with autism.
| field           = [[Psychiatry]]
| symptoms        = Trouble with [[Interpersonal relationship|social interaction]], impaired [[communication]], restricted interests, repetitive behavior<ref name=Land2008/>
| complications   =
| onset           = By age two or three<ref name=NIH2016>{{cite web |title= NIMH » Autism Spectrum Disorder |url= https://www.nimh.nih.gov/health/topics/autism-spectrum-disorders-asd/index.shtml |website= nimh.nih.gov |accessdate= 20 April 2017 |language=en |date= October 2016}}</ref><ref name=DSM5/>
| duration        = Long-term<ref name=NIH2016/>
| causes          = [[Heritability of autism|Genetic]] and environmental factors<ref name=Ch2012/>
| risks           = 
| diagnosis       = Based on behavior and developmental history<ref name=NIH2016/>
| differential    = [[Reactive attachment disorder]], [[intellectual disability]], [[schizophrenia]]<ref>{{cite book |first1= Jacqueline |last1= Corcoran |first2=Joseph |last2=Walsh |title= Clinical Assessment and Diagnosis in Social Work Practice |url= https://books.google.com/books?id=y28kokLoe78C&pg=PA72 |publisher= Oxford University Press, USA |date=9 February 2006 |isbn= 9780195168303 |via=Google Books |page=72 }}</ref>
| prevention      =
| treatment       = Early speech and [[Early intensive behavioral intervention|behavioral interventions]]<ref name=CCD2007/>
| medication      =
| prognosis       = Frequently poor<ref name=Ste106/>
| frequency       = 24.8 million (2015)<ref name=GBD2015Pre>{{cite journal|last1=GBD 2015 Disease and Injury Incidence and Prevalence|first1=Collaborators.|title=Global, regional, and national incidence, prevalence, and years lived with disability for 310 diseases and injuries, 1990-2015: a systematic analysis for the Global Burden of Disease Study 2015.|journal=Lancet|date=8 October 2016|volume=388|issue=10053|pages=1545–1602|pmid=27733282|doi=10.1016/S0140-6736(16)31678-6|pmc=5055577}}</ref>
| deaths          = 
}}

That's actually quite sane to me, and our tool handles it reasonably well by parsing the API parse tree:

>>> page = wptools.page('Autism')
>>> page.get_parse()
>>> page.data['infobox']
{'alt': 'Boy stacking cans',
 'caption': 'Repetitively stacking or lining up objects is associated with autism.',
 'causes': '[[Heritability of autism|Genetic]] and environmental factors',
 'diagnosis': 'Based on behavior and developmental history',
 'differential': '[[Reactive attachment disorder]], [[intellectual disability]], [[schizophrenia]]',
 'duration': 'Long-term',
 'field': '[[Psychiatry]]',
 'frequency': '24.8 million (2015)',
 'image': 'Autism-stacking-cans 2nd edit.jpg',
 'name': 'Autism',
 'onset': 'By age two or three',
 'prognosis': 'Frequently poor',
 'symptoms': 'Trouble with [[Interpersonal relationship|social interaction]], impaired [[communication]], restricted interests, repetitive behavior',
 'treatment': 'Early speech and [[Early intensive behavioral intervention|behavioral interventions]]'}

The trick is to get that parse tree. One way to do this locally is by running your own Mediawiki instance. It's not that difficult, but it will take hours to import the dump. Then, just point wptools at it as though it were out on the network. You can do this for as many dumps in as many languages as you like.

Another approach could be to extract the wikitext for all the Infoboxes from the dumps (another parsing problem), and then pass that text to the API to get the correct parse trees. However, then you are hitting the API again.

There may be another alternative: if you can find those parse trees sitting on some network endpoint somewhere. Then, you can envoke a simple module to bunzip2 them, stream parse them, and wptool them:

# bzip2, expat, and extract a parse tree

ptree = '<template><title>Speciesbox</title><part><name>name</name><equals>=</equals><value>Okapi</value></part></template>'

def infobox(ptree):
    import wptools
    return wptools.utils.get_infobox(ptree)  # {'name': 'Okapi'}

I think the challenge there is that executing a dump is already a (computationally) expensive process. Dumping the parse trees must be more expensive. How much more? That's a good question.

Maybe the parse trees are lurking somewhere in the dumps, and I just haven't found them yet. Can someone please send a page down into the stacks to fetch a copy of the parse trees? If they ever return, please post them here. Really, maybe they are there. Maybe DBPedia has them?

With regard to easing the load on Wikimedia servers, another possibility is to use the more robust and performant RESTBase API. However, it does not currently offer parse trees. If we could get them to offer a /page/parsetree/{title} endpoint, then that may help Wikimedia. One way to motivate progress like that would be to show a high load from wptools (fiendishly rubs hands together).

That's basically a little history of this package. I hope it helps. There's a good chance I may have some of this wrong. If anyone has any further thoughts or corrections, please let us know!

[rexdouglass]

Thank you very much for the extensive feedback and advice.

I tried two things today, one failed and one successful. The failed attempt was at getting a local copy of mediawiki running. It was a great idea, there are just lots of hurdles involved in importing the xml dump. I tripped in two places- (1) unicode characters not being configured somewhere either on the MSQL end or on the mwdumper side generated duplicate key conflicts when importing, and (2) rendering infoboxes require installing and configuring extensions correctly.

Because of your package I knew infoboxes must be parsable to a dictionary at some level and so went down the mwparserfromhell route. I'm now going through the xml document by and parsing anything with an infobox template into keys and values and dumping them to a csv file. It's still somewhat slow, but embarrassingly parallel and something I can run locally without hitting someone's servers. Thanks much for the feedback.

[siznax]

Brilliant! I'm glad you found a workable path. And thank you for sharing the details!

[narayana1043]

@rexdouglass can you please share your code to parse the infoboxes, i would like learn how you parsed form the infoxes

[siznax]

@narayana1043 I believe he used https://github.com/earwig/mwparserfromhell as noted above.