Parse wikimedia URLs as input param

[divanvb]

I'm unable to fetch and parse from the following wiki: https://dota2.gamepedia.com/Abaddon

wtfWikipedia.fetch('https://dota2.gamepedia.com/Abaddon', (err, doc) => {
    if (err) console.log(err);
    console.log(doc);
});`

From what I read through the other issues, mediawiki's should be supported or am I mistaken?

[niebert]

wtf_wikipedia was meant to fetch from Wikipedia, Wikiversity, .... see https://github.com/spencermountain/wtf_wikipedia/wiki - Fork library and modify cross-fetch call in directory /src/_fetch/.

[spencermountain]

hi @divanvb yeah, it looks like that wiki has an api
so it should work

it may just be some custom url path to it, or something

here's the url we want:
https://dota2.gamepedia.com/api.php?action=query&prop=revisions&rvprop=content&maxlag=5&rvslots=main&format=json&origin=*&redirects=true&titles=Abaddon

[spencermountain]

oh, this works:

wtf.fetch('Abaddon', 'dota2', {
  wikiUrl: 'https://dota2.gamepedia.com/api.php'
}).then(function(doc) {
  console.log(doc.json())
})

i like that idea of passing-in the url though

[divanvb]

Indeed, passing the url through gives a bit more flexibility towards specifying the exact wiki etc. I'll have a look at the code above and let you know how well it works with what I'm trying to achieve.

[divanvb]

Indeed, passing the url through gives a bit more flexibility towards specifying the exact wiki etc. I'll have a look at the code above and let you know how well it works with what I'm trying to achieve.

[divanvb]

I've been playing around with the above-mentioned solution, seems to be working quite well. Can you please provide me with a link that shows the options that can be passed through as the 3rd argument. I can't find anything in the documentation on what options are available. It just states [options] with no details.

[spencermountain]

yeah sorry, these were added in prs and need to be cleaned-up.

i think these are the supported params right now:

• userAgent
• wikiUrl
• follow_redirects

[niebert]

add extension of wtf_wikipedia to the Github Wiki

[divanvb]

This is breaking with the latest version for me. Unable to fetch anymore.

import wtf from 'wtf_wikipedia';

TypeError: Cannot read property 'fetch' of undefined
at AppController.getHello (/Users/divan.van.biljon/Workspace/Personal/Cover Me/scraper/dist/app.controller.js:20:33)
at /Users/divan.van.biljon/Workspace/Personal/Cover Me/scraper/node_modules/@nestjs/core/router/router-execution-context.js:37:29
at InterceptorsConsumer.intercept (/Users/divan.van.biljon/Workspace/Personal/Cover Me/scraper/node_modules/@nestjs/core/interceptors/interceptors-consumer.js:10:20)
at /Users/divan.van.biljon/Workspace/Personal/Cover Me/scraper/node_modules/@nestjs/core/router/router-execution-context.js:45:60
at /Users/divan.van.biljon/Workspace/Personal/Cover Me/scraper/node_modules/@nestjs/core/router/router-proxy.js:8:23
at Layer.handle [as handle_request] (/Users/divan.van.biljon/Workspace/Personal/Cover Me/scraper/node_modules/express/lib/router/layer.js:95:5)
at next (/Users/divan.van.biljon/Workspace/Personal/Cover Me/scraper/node_modules/express/lib/router/route.js:137:13)
at Route.dispatch (/Users/divan.van.biljon/Workspace/Personal/Cover Me/scraper/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (/Users/divan.van.biljon/Workspace/Personal/Cover Me/scraper/node_modules/express/lib/router/layer.js:95:5)
at /Users/divan.van.biljon/Workspace/Personal/Cover Me/scraper/node_modules/express/lib/router/index.js:281:22

[spencermountain]

hey @divanvb thanks, I'll need to know more information about your setup, it seems to work for me, and the tests.

import wtf from 'wtf_wikipedia'
wtf.fetch('Toronto').then(doc => {
  console.log(doc.categories())
})

node --experimental-modules index.mjs
cheers

[spencermountain]

this works now, in v8.0.0
check on the docs here
thanks