Skip to content

How to link fetch requests with their results in bundled calls #417

Open
@Vacilando

Description

Consider 5 web pages that need to be enhanced with 5 images from these Wikipedia articles:

wikiArticles = [
    'Richard P. Feynman',
    'Thor Heyerdahl',
    'Interstellar travel',
    'Religion',
    'Cosmology',
]

Then fetch all of these asynchronously:

var [page] = await Promise.all([
  //wtf.fetch(wikiArticle, wikiLang, { 'Api-User-Agent': wikiUserAgentMail, }).then((doc) => doc.summary()),
  wtf
    .fetch(wikiArticles, wikiLang, { 'Api-User-Agent': wikiUserAgentMail })
    .then(docList => {
      return docList.map((doc, i) => {
        return {
          requestArticle: wikiArticles[i],
          title: doc.title(),
          firstImage: doc.image(0).url(), 
        }
      })
    }),
])

As you see we use requestArticle to track which request fetched which image.

The problem is that the returned list of fetches is in a different order, e.g.

  • Interstellar travel
  • Religion
  • Richard Feynman
  • Thor Heyerdahl
  • Cosmology

Matching the original article name with doc.title() would not be a solution because of redirects (e.g. Richard P. Feynman correctly fetches the redirected page Richard Feynman).

Any ideas how to solve this in a safe way?

NB Due to this, and #414 and #416 we are more and more inclined to abandon the bundling and do the fetches sequentially. But that of course increases the risk of being blocked by Wikipedia's bots.

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions