docs: Add guide for running crawler in web server #1174
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pijukatel
added
documentation
There was a problem hiding this comment.
Pull Request Overview
This PR adds a guide for running the crawler in a web server by including new FastAPI server and crawler code examples along with configuration updates.
- Updated pyproject.toml to include new file paths and disable specific error codes for the web server examples.
- Added a FastAPI server example (server.py) to illustrate how to run the crawler from a web endpoint.
- Introduced an asynchronous crawler implementation (crawler.py) with lifecycle management using an async context manager.
Reviewed Changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| pyproject.toml | Updated configuration to include new file mappings for docs examples and added mypy overrides. |
| docs/guides/code_examples/running_in_web_server/server.py | Introduces a FastAPI server with endpoints for running and interacting with a crawler. |
| docs/guides/code_examples/running_in_web_server/crawler.py | Adds an asynchronous crawler setup with a default request handler and lifecycle management. |
Files not reviewed (1)
- docs/guides/running_in_web_server.mdx: Language not supported
vdusek
requested changes
Apr 25, 2025
|
|
||
| # Set up a web server | ||
|
|
||
| There are many popular web server frameworks for Python, such as Flask, Django, Pyramid, ... In this guide, we will use the [FastAPI](https://fastapi.tiangolo.com/) to keep things simple. |
There was a problem hiding this comment.
links to the mentioned projects?
| - `/` - The index is just giving short description of the server with example link to the second endpoint. | ||
| - `/scrape` - This is the endpoint that receives a `url` parameter and returns the page title scraped from the URL | ||
|
|
||
| To run the example server, make sure that you have installed the [fastapi[standard]](https://fastapi.tiangolo.com/#installation) and you can use the command `fastapi dev server.py` from the directory where the example code is located. |
There was a problem hiding this comment.
could we have a separate triple-backticks (```) command here for executing the server?
|
|
||
| This will be our core server setup: | ||
|
|
||
| <CodeBlock className="language-python"> |
There was a problem hiding this comment.
since we have 2 files here, could we use filename arg for code block?
|
|
||
| We will create a standard <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink> and use the `keep_alive=true` option to keep the crawler running even if there are no requests currently in the <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink>. This way it will always be waiting for new requests to come in. | ||
|
|
||
| <CodeBlock className="language-python"> |
There was a problem hiding this comment.
since we have 2 files here, could we use filename arg for code block?
| @@ -244,8 +247,15 @@ module = [ | |||
| "cookiecutter.*", # Untyped and stubs not available | |||
| "inquirer.*", # Untyped and stubs not available | |||
| ] | |||
| disable_error_code = ["misc"] | |||
| import Crawler from '!!raw-loader!./code_examples/running_in_web_server/crawler.py'; | ||
| import Server from '!!raw-loader!./code_examples/running_in_web_server/server.py'; | ||
|
|
||
| # Running in web server |
There was a problem hiding this comment.
This should not be here, as titles are rendered based on the title field in the --- header.
Suggested change
| # Running in web server |
|
|
||
| We will build a simple HTTP server that receives a page URL and returns the page title in the response. | ||
|
|
||
| # Set up a web server |
There was a problem hiding this comment.
2nd level heading (1st only for page title)
|
|
||
| To run the example server, make sure that you have installed the [fastapi[standard]](https://fastapi.tiangolo.com/#installation) and you can use the command `fastapi dev server.py` from the directory where the example code is located. | ||
|
|
||
| # Create a crawler |
There was a problem hiding this comment.
2nd level heading (1st only for page title)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add guide for running crawler in web server
Issues