README.md

scrapiq-python

Official Python client for Scrapiq — turn any URL into clean JSON / text / markdown for LLM & RAG pipelines. No LLM in the loop, deterministic output, ~60–200ms per page, self-hostable.

pip install scrapiq-python   # not yet on PyPI — pip install git+https://github.com/NG-PR0JECT/scrapiq-python.git

Quickstart

from scrapiq import Scrapiq

with Scrapiq() as client:               # defaults to https://scrapiq.io
    md = client.markdown("https://example.com")
    # "<h1>Example Domain</h1> ..." — clean markdown, boilerplate removed

    data = client.json("https://example.com")          # -> {"text": "..."}
    # with a JSON schema the same call returns matched fields:
    article = client.json(
        "https://example.com/article",
        schema={"title": "string", "author": "string", "published": "string"},
    )

Plain text and full control:

txt = client.text("https://example.com")

# Full response incl. metadata (title, author, language, word_count…) and timing
resp = client.extract("https://example.com", format="markdown", include_metadata=True)
print(resp["metadata"]["title"], resp["extraction_time_ms"])

Async:

import asyncio
from scrapiq import AsyncScrapiq

async def main():
    async with AsyncScrapiq() as client:
        md = await client.markdown("https://example.com")
        print(md)

asyncio.run(main())

Self-hosted instance or API key:

client = Scrapiq(base_url="http://localhost:8001", api_key=os.environ["SCRAPIQ_API_KEY"])

What you get

format	field	contents
json	data	schema-matched fields, or data.text
text	content	clean plain text (boilerplate removed)
markdown	content	clean markdown with links
any	metadata	title, description, author, date, language…
any	extraction_time_ms	server-side extraction time

Behavior

• Retries: transparently retries 429/5xx (exponential backoff, honors Retry-After) and connection errors.
• Errors: non-2xx responses raise ScrapiqHTTPError with .status and .detail; transport errors raise ScrapiqError.
• Timeouts: 30s default, configurable via timeout=.

Ecosystem

• scrapiq — the API (MIT, self-hostable)
• scrapiq-cli — dependency-free CLI
• scrapiq-node — TypeScript/Node client
• scrapiq-langchain-loader — LangChain DocumentLoader
• scrapiq-mcp-server — MCP server for Claude/Cursor/agent tools

License

MIT — see LICENSE.