Skip to content

Web scraping Discord bot that notifies if new item appears

Notifications You must be signed in to change notification settings

jjablonski-it/webscraper-bot

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

🚨 There's a new, more advanced version of this bot that you can checkout for free 🚨

Visit notifast.me to learn more

webscraper-bot

Discord web scraping bot used to scrape websites with dynamic content and send notifications when there is a new item.

Instructions

1.Adding bot to your server

This version development has been paused but you can try out the new version or self host it (e.g. using Render or Railway),

2.Adding new scraper job

Use /create-job command and fill in the form:

  • name - name of the job
  • url - url that the bot will scrape
  • selector - selector for an a tag with link to element (examples below)
  • interval - interval of the job in minutes (min 1 minute)
  • active? - default true, if the job should be active right away (you can always enable/disable job with [enable/disable]-job commands)
  • channel? - defualt channel where the command was run which channel should be messaged when the new item appear
  • clean? - default true, if the query params should be ignored (essential for some sites like Ebay)

image

After that you can do basic CRUD opeartions on jobs with commands like /list-jobs /update-job /delete-job...

3.Waiting...

After that you simply wait for the job run and it will send message when new items are found:

image

You can also run the job manually with /run-job. Note that the first run will get all the elements that will be on the webiste. image

How to get selector

The selector is a querySelectorAll string that the bot uses to get unique a tags linking to items you want to scrape. To verify if your selector is working you can run document.querySelectorAll(<your selector>) in browser console and check if the function returns items you want to scrape.

Examples of selectors

  • Ebay: .srp-river-results .s-item__image a
  • Otodom: [data-cy=listing-item-link]
  • Olx: [data-cy=l-card] > a

Feel free to contribute, there are a lot of things to improve :)

About

Web scraping Discord bot that notifies if new item appears

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published