web-scrape-indexer

Scrapes websites using Scrapy and makes it searchable in Elasticsearch

How to run?

cd indexer
./gradlew build
./gradlew bootRun

In a separate terminal

cd scrapper
poetry shell
cd simple-scrapper
scrapy crawl news-spider
deactivate

Play around the endpoints