Web Scraper

Scrapes a list of URLs and saves each page as Markdown with images downloaded locally.

Setup

npm install

Usage

1. Add URLs to urls.txt — one per line, # for comments:

# My sites
https://example.com/page-1
https://example.com/page-2

2. Run:

node scraper.js

Output lands in output/:

• output/<page-slug>.md — page content as Markdown
• output/assets/ — all downloaded images (referenced locally in the .md files)

---

How Bot Blocking is Handled

What's built in

Technique	Detail
Rotating User-Agents	Picks a random Chrome / Firefox / Safari UA each request
Browser-like headers	Sends Accept, Referer, Sec-Fetch-*, Cache-Control etc.
Random delay	Waits 1.5–4s between pages to avoid rate-limit triggers
Auto-retry	Retries up to 3× on 403 / 429 / 503 / 504, respects Retry-After header

Still getting blocked?

403 Forbidden / Cloudflare / bot detection

The site is doing JavaScript-based fingerprinting. Switch to a headless browser:

npm install playwright
npx playwright install chromium

Then replace the fetchWithRetry function with a Playwright page.goto() call. Playwright renders the page like a real browser and bypasses most JS-based challenges.

429 Too Many Requests

Increase the delay. Edit these two values in scraper.js:

// Line: return delay(1500 + Math.random() * 2500);
return delay(5000 + Math.random() * 5000);  // 5–10s

IP banned

Use a rotating proxy. Pass it via axios:

// In fetchWithRetry(), add to the axios config:
proxy: {
  host: "your.proxy.host",
  port: 8080,
  auth: { username: "user", password: "pass" }
}

Free proxy lists are unreliable — use a paid residential proxy service (e.g. Bright Data, Oxylabs) for consistent results.

SSL / certificate errors

// In fetchWithRetry(), add to the axios config:
httpsAgent: new https.Agent({ rejectUnauthorized: false })

Also add import https from "https"; at the top of scraper.js.

---

Output Format

Each .md file starts with:

# Page Title

> Source: https://original-url

... page content ...

Images are saved as:

output/assets/site-com-images-photo-jpg.jpg

And referenced in the Markdown as:

![alt text](assets/site-com-images-photo-jpg.jpg)

---

Troubleshooting

Problem	Fix
urls.txt not found	Create the file in the same directory as scraper.js
Images not downloading	The image URL may require auth or be lazy-loaded via JS
Markdown looks messy	The site has heavy JS rendering — use Playwright instead
All pages fail	Check your internet connection or try a single URL first