Scrapeless + Dify

Give Dify agents and workflows live web data by importing one OpenAPI file as a custom tool. Powered by the Scrapeless Scraper API.

No plugin to install, no code to deploy — scrapeless-openapi.yaml is a validated OpenAPI 3.0.3 schema that Dify turns into a callable tool.

Use case

A Dify agent that cannot fetch a page can only answer from its model and your uploaded documents. Adding this tool lets you build:

• Research agents that search, open results, and answer from live pages.
• Lead-gen workflows that pull local businesses for a category and city into a dataset.
• Product/price assistants that read a live Amazon listing on demand.
• Knowledge pipelines that fetch pages on a schedule and write them into a Dify knowledge base.

Requirements

• Dify — cloud or self-hosted
• A Scrapeless API key — create a free account

Setup

1. Import the schema. In Dify: Tools → Custom → Create Custom Tool. Paste the contents of scrapeless-openapi.yaml, or point Dify at its raw URL. One tool appears: scraperRequest.

2. Set authentication. Choose API Key, then set every field — the two Dify pre-fills are both wrong for this API:

Field	Value
Auth type	API Key
Header name	x-api-token — not the default Authorization
Value	your Scrapeless key
Header prefix	Custom — not the default Basic

Two of these default to the wrong thing, and both produce the same failure — 401 with {"code":14404,"message":"invalid access token"}:

• Header name defaults to Authorization. The schema declares x-api-token.
• Header prefix defaults to Basic, which makes Dify send x-api-token: Basic <your-key>. Scrapeless expects the bare key, so pick Custom. Bearer fails the same way.

3. Test it. Use Dify's built-in test panel with:

{ "actor": "scraper.google.search", "input": { "q": "web scraping api" } }

You should get a JSON response containing organic_results.

Dify parses input as a string parameter rather than an object, because it flattens nested request-body properties. Both forms work — the object above, and the same thing as a JSON string ("{\"q\": \"web scraping api\"}") — so paste whichever your node produces. In a Workflow, a preceding Code node returning a JSON string is the reliable way to build input dynamically.

4. Use it. Attach the tool to an Agent so the LLM calls it when it needs live data, or add it as a Tool node in a Workflow for a deterministic fetch step.

What the tool accepts

One operation, scraperRequest, with two fields — actor picks the scraper and input carries its parameters:

Goal	Body
Google SERP	{"actor":"scraper.google.search","input":{"q":"web scraping api"}}
Local businesses	{"actor":"scraper.google.search","input":{"q":"plumbers in Austin, TX","tbm":"lcl"}}
Next page of local results	add "start": 20 (20 per page; num is ignored)
Amazon product	{"actor":"scraper.amazon","input":{"action":"product","url":"https://www.amazon.com/dp/B09B8V1LZ3"}}
Amazon search	{"actor":"scraper.amazon","input":{"action":"keywords","keywords":"smart speaker"}}

The schema ships all four as named examples, so they appear in Dify's request builder.

Prompting an agent to use it

Agents call a tool reliably when the instruction says what data is needed, not just what to answer:

When a question depends on current web content, call scraperRequest with
actor "scraper.google.search" and input {"q": "<the search terms>"}, read the
organic_results, and answer from those. Never answer from memory when the
question is about current prices, rankings, or availability.

For a local-business workflow, pin the actor in a Tool node instead and let the LLM fill only the query — deterministic beats persuasive.

Response shapes

• scraper.google.search flattens the SERP at the top level: organic_results[], or local_results.places[] with tbm: lcl.
• scraper.amazon nests the product under result.

In a Dify workflow, a Code node after the Tool node is the natural place to flatten before writing to a knowledge base or database.

Quirks worth knowing, all observed live:

• Sporadic 400s that succeed on retry. Enable retry on the Tool/HTTP node.
• Padded strings — local-pack phone, type, and hours arrive with a leading space. Trim in a Code node.
• Empty local fields — place_id, gps_coordinates, thumbnail come back empty for local results.
• scraper.google.shopping is retired (15002), and tbm: "shop" returns no shopping results.

Troubleshooting

Symptom	Cause and fix
401 + {"code":14404}	One of the two auth defaults. Header name must be x-api-token, not Authorization, and header prefix must be Custom, not Basic.
input field expects a string	Correct — Dify flattens the nested body object. An object or a JSON string both work.
Import rejected	Paste the full YAML including the openapi: line. The file validates as OpenAPI 3.0.3 — if Dify complains, it is a truncated paste.
400 that passes on retry	Known API behavior. Enable retries on the node.
Empty local_results	The query lacks local intent. "plumbers in Austin, TX" works; "plumbing" often does not.
Agent never calls the tool	Instruction is too soft. Name the tool and the actor explicitly, as above.
15002 in the response	Retired actor. Use scraper.google.search or scraper.amazon.

Verification status

Verified end to end on a self-hosted Dify 1.16.1 (2026-08-12). Every step above was executed against a real instance, not inferred from the schema:

Step	Result
Import the schema	Accepted. Produces exactly one tool, scraperRequest, method POST, parameters actor and input.
API-key auth with x-api-token + prefix Custom	Live call returned 9 organic_results (~18 KB).
Same call with prefix Basic or Bearer	401 {"code":14404,"message":"invalid access token"} — the default is wrong, hence the warning in step 2.
Same call with header Authorization	401, identical failure.
input as an object vs a JSON string	Both accepted; 9 results either way.
Persist the provider	Created, and scraperRequest then lists as an attachable tool.

• The schema is validated — scrapeless-openapi.yaml passes openapi-spec-validator as OpenAPI 3.0.3, and every request example in it was executed against the live API on 2026-08-11.
• One caveat about self-hosting: Dify routes tool HTTP through its ssrf_proxy container. If that service is not running, tool calls fail with [Errno -3] Temporary failure in name resolution rather than a network error naming the proxy. Bring up the full compose stack, not just api.
• UI wording can differ between Dify versions; the field names above (api_key_header, api_key_header_prefix) are what Dify stores, so trust those if a label looks different.

Project structure

scrapeless-dify/
├── scrapeless-openapi.yaml   # validated OpenAPI 3.0.3 — import this
└── LICENSE

Related

• Product: Scraping API · Deep SERP API
• Integration page: Scrapeless with Dify
• Guide: Build smart business news monitoring with Dify

License

MIT — see LICENSE.