fix: bahn.de API 403 errors #131

alexanderroidl · 2025-09-30T21:59:02Z

This MR aims to fix the 403 errors which were causing HTTP responses with status 500 on our end (as described in #57).

The Bahn-API was responding 403 errors as it deemed our requests forbidden. This MR solves this consistently as I was able to verify over several days of testing.

Therefore instead of doing plain HTTP requests via the fetch API this MR aims to replace it by:

Using a headless browser which has extra stealth capabilities and sets a random user agent/viewport.
...which opens a page for journey information at https://www.bahn.de/buchung/start?vbid=<vbid>
...which always triggers HTTP calls on the client-side (our browser) to the Bahn-API at /web/api/angebote/recon and /web/api/angebote/verbindung/<vbid>. This is not done by us but expected behavior of the page itself, as it would happen for an actual user.
These calls are intercepted, their responses extracted and evaluated (as before, but without using fetch).

Side-effects of this approach:

We no longer need to handle any response cookies as our browser automatically saves and re-uses them.
Also we no longer need to reverse-engineer the Recon logic as the actual implementation is used by our browser via the page's client-side JS.

Preview:

 GET /discount?url=https%3A%2F%2Fwww.bahn.de%2Fbuchung%2Fstart%3Fvbid%3D230ffc6f-f4e3-40ef-b079-50f08a1acce9&bahnCard=50&hasDeutschlandTicket=true&passengerAge=29&travelClass=2 200 in 501ms
🌐 Browser was setup with user agent "Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0 Mobile/15E148 Safari/604.1", platform "iPhone" and viewport 402x660.
🌐 Browser is visiting "https://www.bahn.de/buchung/start?vbid=230ffc6f-f4e3-40ef-b079-50f08a1acce9".
🌐 Browser has intercepted Verbindung response.
🌐 Browser has intercepted Recon response.
🌐 Browser was closed.
From: Berlin Hbf (8098160) | To: Passau Hbf (8000298) | Date: 2025-10-01 | Time: 03:45 | Class: Second
 POST /api/parse-url 200 in 6385ms

🔄 API Counter reset to 0
(...)

alexanderroidl · 2025-09-30T22:01:05Z

next-env.d.ts

@@ -1,5 +1,6 @@
 /// <reference types="next" />
 /// <reference types="next/image-types/global" />
+/// <reference path="./.next/types/routes.d.ts" />


I'm not sure why Next.js automatically added this. 🤷

Somebody please double-check this for correctness.

alexanderroidl · 2025-09-30T22:02:27Z

next.config.js

-	}
+		ignoreBuildErrors: true, // temporarily, since some type errors still exists and are ambiguous
+	},
+	serverExternalPackages: ["puppeteer-extra", "puppeteer-extra-plugin-stealth"],


Those must be stated as external to avoid bundling.

alexanderroidl · 2025-09-30T22:04:32Z

utils/schemas.ts

 export type ValidatedVendoJourney = z.infer<typeof validatedVendoJourneySchema>;

-export const vbidSchema = z.object({
+export const VerbindungResponseSchema = z.object({


This schema has nothing to do with the Bahn's VBIDs.

It is about the HTTP response from their API at /web/api/angebote/verbindung/<vbid>.

The VBID is just the sole parameter of the route, but the schema should be named after the route itself acting as a descriptor.

alexanderroidl · 2025-09-30T22:05:17Z

utils/schemas.ts

-export type VbidSchema = z.infer<typeof vbidSchema>;
+export type VerbindungResponse = z.infer<typeof VerbindungResponseSchema>;
+
+const reconLegSchema = z.object({


These were over at parseHinfahrtRecon.ts which got deleted.

alexanderroidl · 2025-09-30T22:07:31Z

utils/parseHinfahrtRecon.ts

We no longer need to reverse-engineer the Recon logic as we use the actual implementation by the Bahn itself via our browser-based approach, which executes client-side JS as a normal user would.

FunctionDJ · 2025-09-30T22:44:01Z

I don't know how I feel with going back to a headless browser / puppeteer when it was a small milestone to get rid of it, making setup and run requirements much leaner. I doubt that a headless browser will circumvent scrape protection from cloud provider IPs.

I'm expecting the same type of proof of this actually fixing #57 as detailed here #98 (comment)

I'll only consider this PR if affected users confirm that this branch/fork solves their issue.

alexanderroidl · 2025-09-30T22:48:54Z

I don't know how I feel with going back to a headless browser / puppeteer when it was a small milestone to get rid of it, making setup and run requirements much leaner. I doubt that a headless browser will circumvent scrape protection from cloud provider IPs.

I'm expecting the same type of proof of this actually fixing #57 as detailed here #98 (comment)

I'll only consider this PR if affected users confirm that this branch/fork solves their issue.

Generally I agree such extras should be avoided wherever possible. In our case though, where we're avoiding bot detection, we most likely will need to make our program mimic an actual user to avoid getting caught. I've even received error pages from the website itself until I added random user agents and viewports. But let's see...

FunctionDJ · 2025-09-30T22:49:46Z

PS: Please ask users to leave their testimonial on this PR instead of #57 or #98.

FunctionDJ · 2025-09-30T22:54:12Z

I've even received error pages from the website itself until I added random user agents and viewports.

I understand, but we should try our best to avoid adding a headless browser (again). In the end, almost everything a browser does - from the perspective of backends - can be emulated without running one. And not having a headless browser makes the application more lightweight, easier to deploy, and faster.

chore: remove some comments and add spacing

hackmybeer · 2025-10-01T17:03:20Z

Works for me!

jsschmid · 2025-10-01T18:18:23Z

@alexanderroidl
I tried it by checking out your 131

after sudo docker image rm d40e5aa07805

by running the docker image. Are your changes incorporated or do I need to rebuild the image?

Anyway...still got the "Fehler: Server error (500). Diese Problem ist uns bekannt und wir arbeiten daran, es zu beheben. Ein Status über den Fehler finden Sie unter #57"

alexanderroidl · 2025-10-01T21:36:00Z

@alexanderroidl I tried it by checking out your 131

after sudo docker image rm d40e5aa07805

by running the docker image. Are your changes incorporated or do I need to rebuild the image?

Anyway...still got the "Fehler: Server error (500). Diese Problem ist uns bekannt und wir arbeiten daran, es zu beheben. Ein Status über den Fehler finden Sie unter #57"

Dang, I forgot to adjust the Docker setup. Right now it's failing because it's missing Chromium.

Can you try it without Docker until I take care of it in the next couple of days, so we can see whether it generally works?

CactiChameleon9 · 2025-10-09T08:30:37Z

This PR fixed it for me. I am running on a residential IP but I seemed to be blocked anyway... Although, I would prefer things to work without a whole chromium instance tbh

github-actions · 2025-11-13T02:53:33Z

This PR is stale because it has been open for 30 days with no activity. It will be closed in 14 days if there is no activity.

laura-3 · 2025-11-27T18:29:52Z

could this be merged soon?

FunctionDJ · 2025-12-09T18:59:20Z

could this be merged soon?

I won't merge it for the time being because only 2 users report that this works and 1 reports an error.

This PR massively changes the build and deployment requirements for Betterbahn and imo we need more data (i.e. more reports by affected users) to justify this change.

To give a concrete number, I'd say if we get 10 reports by different users and it fixes #57 for at least 5 of them, this could be a worthwhile change. Otherwise if the interest is too low or those affected don't want to test this PR, I won't be able to help because I'm not affected since I don't have cloud infrastructure to run Betterbahn on.

koehdaniel · 2025-12-21T19:07:35Z

Tried it locally on a residental IP and doesn‘t work.
I cloned alexanderroidl:fix/bahn-api-403-errors and ran the Docker-build.
Throws an 500 error and refers to #57

alexanderroidl commented Sep 30, 2025

View reviewed changes

alexanderroidl force-pushed the fix/bahn-api-403-errors branch from 6ea49a8 to e12662a Compare September 30, 2025 22:28

alexanderroidl marked this pull request as ready for review September 30, 2025 22:39

This was referenced Sep 30, 2025

Fix/browsify scraping #98

Closed

[FEAT] Dealing with Akamai scrape protection #57

Open

FunctionDJ added the proof pending Effect of PR can't be verified by maintainers and requires testimonial by affected users label Sep 30, 2025

alexanderroidl added 6 commits October 1, 2025 11:56

fix: bahn api 403 errors

9f2e432

chore: rebase lockfile

2aa3ddc

chore: remove some comments and add spacing

fix: linting error

f4c8bcc

fix: linting error

6eb65ed

fix: linting errors

390968d

chore: re-install dependencies after rebase

ee13b1d

alexanderroidl force-pushed the fix/bahn-api-403-errors branch from ecb1634 to ee13b1d Compare October 1, 2025 09:57

ezzcodeezzlife approved these changes Oct 13, 2025

View reviewed changes

github-actions bot added the Stale label Nov 13, 2025

github-actions bot removed the Stale label Nov 28, 2025

fix: bahn.de API 403 errors #131

Are you sure you want to change the base?

fix: bahn.de API 403 errors #131

Conversation

alexanderroidl commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexanderroidl Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexanderroidl Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

alexanderroidl Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexanderroidl Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

alexanderroidl Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

FunctionDJ commented Sep 30, 2025

Uh oh!

alexanderroidl commented Sep 30, 2025

Uh oh!

FunctionDJ commented Sep 30, 2025

Uh oh!

FunctionDJ commented Sep 30, 2025

Uh oh!

hackmybeer commented Oct 1, 2025

Uh oh!

jsschmid commented Oct 1, 2025

Uh oh!

alexanderroidl commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CactiChameleon9 commented Oct 9, 2025

Uh oh!

github-actions bot commented Nov 13, 2025

Uh oh!

laura-3 commented Nov 27, 2025

Uh oh!

FunctionDJ commented Dec 9, 2025

Uh oh!

koehdaniel commented Dec 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

alexanderroidl commented Sep 30, 2025 •

edited

Loading

alexanderroidl Sep 30, 2025 •

edited

Loading

alexanderroidl Sep 30, 2025 •

edited

Loading

alexanderroidl commented Oct 1, 2025 •

edited

Loading

koehdaniel commented Dec 21, 2025 •

edited

Loading