@speedyapply
Problem:
JobSpy currently lacks support for Wellfound, a major platform for startup jobs. Historically, scraping Wellfound has been difficult because they use strict DataDome anti-bot protection that blocks standard HTTP requests with 403 Forbidden errors.
Proposed Solution:
I have built a native WellFound module to solve this and will be submitting a PR.
Here is how the implementation works:
- Bypassing DataDome: Uses
curl_cffi with impersonate="chrome124" to automatically generate authentic TLS signatures and browser headers, successfully passing DataDome checks.
- Robust Extraction: Instead of fragile HTML scraping, it extracts the embedded Next.js
<script id="__NEXT_DATA__"> payload and parses the Apollo GraphQL cache (JobListingSearchResult) to get rich, structured data (exact compensation, remote policies, company info).
- Complex Filtering: Dynamically slugifies locations/roles and includes a custom tokenized search engine that filters jobs if keywords match the title, company, or description.
- Architecture Integration: Fully adheres to JobSpy's package structure, maps all data to the
JobPost schema, and supports proxy rotation.
Context:
I have successfully tested the module locally, scraping 200+ jobs sequentially. I will open a Pull Request shortly with the full code.
Repo: https://github.com/Vivek02Sharma/JobSpy.git
Branch: wellfound
@speedyapply
Problem:
JobSpy currently lacks support for Wellfound, a major platform for startup jobs. Historically, scraping Wellfound has been difficult because they use strict DataDome anti-bot protection that blocks standard HTTP requests with
403 Forbiddenerrors.Proposed Solution:
I have built a native
WellFoundmodule to solve this and will be submitting a PR.Here is how the implementation works:
curl_cffiwithimpersonate="chrome124"to automatically generate authentic TLS signatures and browser headers, successfully passing DataDome checks.<script id="__NEXT_DATA__">payload and parses the Apollo GraphQL cache (JobListingSearchResult) to get rich, structured data (exact compensation, remote policies, company info).JobPostschema, and supports proxy rotation.Context:
I have successfully tested the module locally, scraping 200+ jobs sequentially. I will open a Pull Request shortly with the full code.
Repo: https://github.com/Vivek02Sharma/JobSpy.git
Branch: wellfound