A simple ASYNC RSS parser. To make retrieving RSS feeds ultra fast.
Note
This is module works and is used! You are invited to use it and make IT better! Extensive exception validation is left out on purpose. This to keep the code simple, clean, and mean!
pip install ultrafastrss
In your Python program call the async function with a (large) list of URLs:
process_rssfeeds(urls)
To validate and see the working:
Create a simple demo program, create a python file fastrss_demo.py like (just copy-paste):
from ultrafastrss.ultrafastrss import process_rssfeeds
import asyncio
urls = [
'https://nocomplexity.com/rss',
'https://www.eff.org/rss/updates.xml',
'https://www.freebsd.org/news/rss.xml',
'https://ubuntu.com/blog/feed',
'https://nlnet.nl/feed.atom',
'https://j3s.sh/feed.atom',
'https://blog.research.google/atom.xml'
]
# Run the async RSS feed processing, all results are as json stored in the variable results
results = asyncio.run(process_rssfeeds(urls))
print(results)After running this demo in a terminal by doing:
python ultrafastrss_demo.py
You should seen directly the result of all RSS information gathered!
You can do all kind of fun things with the returned json object which contains the crucial parts from the parsed rss or atom feeds!
I did some benchmarking test with feedparser, a module that is widely used within many Python programs for parsing rss feeds.
Results are minimal 10 times faster with 500 URLs.
ultrafastrss is distributed under the terms of the GPL-3.0-or-later license.
This ultrafastrss parser is designed to parse RSS or Atom return JSON data. The parsing is delegating to specific parsers functions (rss_parser, atom_parser or rdf_parser). Only the attributes title, link and date are retrieved.
The key design principle for this parser is to retrieve JSON data from a significant number of feeds asynchronously.
Feed content or summary is not parsed on purpose. This is also not a RSS reader. But by clicking the retrieved link, the retrieved item can be viewed in a browser.
The fields that are parsed and stored for every feed are:
"title": "Exphormer: Scaling transformers for graph-structured data",
"link": "http://blog.research.google/2024/01/exphormer-scaling-transformers-for.html",
"date": "2024-01-23",
"tags": [
"Deep Learning",
"Graphs"All kind but edge cases are left out on purpose to keep the needed code simple. Creating a generic module for RSS/ATOM parsing is not simple. So the design decision is to only incorporate logic that I need for the URLs I use.
Great you want to contribute!
Simple Guidelines:
- Questions, Feature Requests, Bug Reports please use on the
Github Issue Tracker.
Note
This is module works and is used frequently on thousands of RSS feeds! Extensive exception validation is left out on purpose for now in the code. Feature requests are possible, but will only be realised after a quote and paid invoice. But this code is so Simple that I encourage everyone to make a special parser and share the code. So we all benefit! RSS feed parsing is a minefield. The standards leaves room and many sites struggle to publish clean RSS feeds. Also some sites mesh up RSS feeds on purpose.
Besides a manual code review, this project is checked with the static application security testing (SAST) tool: Python Code Audit.
But note: You should never parse untrusted XML data from unknown sources. So recommended is to validate the RSS feeds you want retrieve for parsing before using this or any library before parsing XML! The used default Python libraries used in this module are considered safe, but parsing XML from malicious sites should never be done. All RSS readers have the same risk, so always practice security by design principles