RSS Feed Crawler

A terminal-based RSS feed aggregator. Follow feeds, run a background scraper to collect posts, and browse the latest articles — all from the command line.

Prerequisites

• Node.js v18 or higher
• npm v9 or higher
• A running PostgreSQL instance

Configuration

The program reads its configuration from ~/.gatorconfig.json. Create that file before running any commands:

{
  "db_url": "postgres://user:password@localhost:5432/aggregator",
  "current_user_name": ""
}

• db_url — connection string for your PostgreSQL database. Required.
• current_user_name — the currently logged-in user. Leave it empty; the login and register commands manage this automatically.

Installation

git clone https://github.com/caverraaa/rss-feed-crawler
cd aggregator
npm install

Database setup

Run the migrations to create all tables:

npm run migrate

Running commands

All commands are run with:

npm start <command> [arguments]

Commands

User management

Register a new user account:

npm start register <username>

Log in as an existing user:

npm start login <username>

List all registered users (current user is marked with *):

npm start users

Managing feeds

Add a feed and automatically follow it:

npm start addfeed <name> <url>

Example:

npm start addfeed "Hacker News" https://news.ycombinator.com/rss
npm start addfeed "TechCrunch" https://techcrunch.com/feed/

List all feeds in the database:

npm start feeds

Follow an existing feed by URL:

npm start follow <url>

Unfollow a feed by URL:

npm start unfollow <url>

List feeds you are currently following:

npm start following

Aggregating posts

Run the scraper in a continuous loop. It fetches feeds round-robin, always picking the one that was fetched least recently first. The argument is the delay between fetches — supports ms, s, m, and h units.

npm start agg <interval>

Examples:

npm start agg 30s    # fetch a feed every 30 seconds
npm start agg 5m     # fetch a feed every 5 minutes

Leave this running in a separate terminal. Press Ctrl+C to stop it gracefully.

Browsing posts

Show the latest posts from all feeds you follow. The optional argument sets how many posts to return (defaults to 2):

npm start browse [limit]

Examples:

npm start browse       # shows 2 posts
npm start browse 20    # shows 20 posts

Typical workflow

# 1. Set up config, then register and log in
npm start register bob

# 2. Add some feeds
npm start addfeed "Hacker News" https://news.ycombinator.com/rss

# 3. Start the aggregator in one terminal
npm start agg 30s

# 4. Browse collected posts in another terminal
npm start browse 10

Database schema

users
+----+------------+------------+------+
| id | created_at | updated_at | name |
+----+------------+------------+------+

feeds
+----+------------+------------+------+-----+-----------------+---------+
| id | created_at | updated_at | name | url | last_fetched_at | user_id |
+----+------------+------------+------+-----+-----------------+---------+
                                                                    |
                                                             FK -> users.id

feed_follows
+----+------------+------------+---------+---------+
| id | created_at | updated_at | user_id | feed_id |
+----+------------+------------+---------+---------+
                                    |          |
                             FK -> users.id    FK -> feeds.id

posts
+----+------------+------------+-------+-----+-------------+--------------+---------+
| id | created_at | updated_at | title | url | description | published_at | feed_id |
+----+------------+------------+-------+-----+-------------+--------------+---------+
                                                                               |
                                                                        FK -> feeds.id

Relationships

users ──< feed_follows >── feeds
  |                           |
  |      (many-to-many)       |
  |   a user follows          |
  |   many feeds,             |
  |   a feed is followed      |
  |   by many users           |
  |                           |
  └──< feeds                  └──< posts
       (one-to-many)               (one-to-many)
       a user owns                 a feed contains
       many feeds                  many posts

• users -> feeds: one-to-many. A user can add many feeds; each feed has one owner.
• users -> feed_follows: one-to-many. A user can follow many feeds.
• feeds -> feed_follows: one-to-many. A feed can be followed by many users.
• users <-> feeds (via feed_follows): many-to-many. A user follows many feeds; a feed is followed by many users.
• feeds -> posts: one-to-many. A feed contains many posts; each post belongs to one feed.