OData Scraper is a Python application that analyzes the schema of an OData API and creates a matching SQLite schema. It then fetches all data from the OData API and stores it in the SQLite database.
-
Clone the repository:
git clone https://github.com/daniel-lerch/odata-scraper cd odata-scraper -
Run the scraper with
uv(this installs dependencies automatically on first run):uv run main.py -u https://api.example.org/OData.svc/ -o test.db -
You can look into the resulting
test.dbfile at sqliteviewer.app or with any other SQLite viewer
| Argument | Description |
|---|---|
-u, --url |
Base URL of the OData service. |
-o, --output |
Output SQLite database file path. |
--basic-auth |
Basic Auth credentials as user:password. Falls back to the ODATA_BASIC_AUTH environment variable. |
--auth-header |
Raw Authorization header value (e.g. Bearer <token>). Falls back to the ODATA_AUTH_HEADER environment variable. |
--max-page-size |
Asks the server to cap each page via a Prefer: odata.maxpagesize=<n> header on data requests. |
Why SQLite?
OData's data model — entity sets, keys, and navigation properties between them — maps very naturally onto tables, primary keys, and foreign keys in a relational database. Some OData concepts (like type inheritance) don't have a perfect one-to-one equivalent in SQL, but overall a relational database is the best fit for storing what the scraper fetches.
SQLite in particular is widely supported and stores the entire database in a single file, so the result is easy to copy around and open with countless existing tools, which makes it a convenient format for further processing the scraped data.
What happens if scraping fails partway through, e.g. due to an error or a timeout?
The scraper processes one table (entity set) at a time and commits its data to the database as soon as that table is done. If a table fails — for example because of an HTTP error or a network timeout — the scraper logs the error, skips that table, and moves on to the next one. Tables that were already fetched keep their data in test.db.
You can simply run the same command again afterwards. Already-stored rows are safely overwritten (not duplicated), so re-running effectively resumes the job by retrying whatever failed or was skipped before.
I keep getting timeouts — what can I do?
Try setting --max-page-size to a lower value. This asks the server to return smaller pages, which reduces the chance of a single request timing out.
Who this document is for
- Written for: Steven (software developer)