Fix/browsify scraping

[Anatol-Beck]

Disclamer

This PR builts on the fork of defekkt who added a cookie fix.

Issue Description

The application was encountering 403 Forbidden errors when attempting to access the Deutsche Bahn API endpoint https://www.bahn.de/web/api/angebote/recon. This was happening specifically in the search functionality when users enter a search link in the UI.

Root Cause

Deutsche Bahn has implemented anti-scraping protection on their APIs that detects and blocks requests that don't appear to come from a legitimate browser. Our implementation was missing proper browser-like headers and had issues with cookie handling.

Changes Made

1. Enhanced Browser Headers in API Requests:

• Added comprehensive browser-like headers to the parseHinfahrtReconWithAPI function including:
  • Modern User-Agent string
  • Appropriate Accept headers
  • Origin and Referer headers matching the Deutsche Bahn website
  • Security-related headers (Sec-Fetch-* series)

2. Improved Cookie Handling:

• Implemented better cookie formatting in getResolvedUrlBrowserless
• Used a Set to eliminate duplicate cookies
• Properly formatted cookies with correct separators
• Fixed spacing issues in cookie string concatenation

3. Enhanced Error Diagnostics:

• Added detailed logging for 403 responses in fetchAndValidateJson
• Included request headers in error logs for debugging
• Added response body inspection for additional error details
• Improved error messages to make troubleshooting easier

How It Works

These changes make our application's requests appear more like legitimate browser traffic, which helps bypass Deutsche Bahn's anti-scraping measures. By properly formatting cookies and including all the headers a real browser would send, we minimize the likelihood of being detected as an automated script.

[FunctionDJ]

These changes make our application's requests appear more like legitimate browser traffic, which helps bypass Deutsche Bahn's anti-scraping measures. By properly formatting cookies and including all the headers a real browser would send, we minimize the likelihood of being detected as an automated script.

Sorry for the nit-picking, but I'd really like some more concrete evidence and facts rather than "appear more like legitimate browser traffic" and "minimize the likelihood".

Please ask affected users in #57 to test your fork/branch and report if the fork fixes the issue for them as the issue depends on how/where the app is deployed and us maintainers don't have the resources to test on all the different VPS providers.

[Anatol-Beck]

These changes make our application's requests appear more like legitimate browser traffic, which helps bypass Deutsche Bahn's anti-scraping measures. By properly formatting cookies and including all the headers a real browser would send, we minimize the likelihood of being detected as an automated script.

Sorry for the nit-picking, but I'd really like some more concrete evidence and facts rather than "appear more like legitimate browser traffic" and "minimize the likelihood".

Please ask affected users in #57 to test your fork/branch and report if the fork fixes the issue for them as the issue depends on how/where the app is deployed and us maintainers don't have the resources to test on all the different VPS providers.

Agree, I have not tested it thoroughly yet, so more testing might be better.

[jsschmid]

@Anatol-Beck
TLDR; Doens't work for me unfortunately.

Tested using an oracle server, IP 141.144.239.XYZ

I pulled your fork
https://github.com/Anatol-Beck/betterbahn

Tried with

Verbindung am Do. 02.10.2025
• von Berlin Hbf, Abfahrt 09:09 Uhr Gl. 13 mit ICE 1050
• nach Osnabrück Hbf, Ankunft 12:41 Uhr Gl. 12 mit WFB 95776
Verbindung ansehen: https://www.bahn.de/buchung/start?vbid=4bbeeff4-2217-4a6d-85d4-2fddf9862bd7

and got the error message pointing to the issue

#57

...so it doesn't seem to change anything for me unfortunately. The changes you made do also apply to the docker, right?

[alexanderroidl]

@Anatol-Beck Your fix did not work for me either unfortunately, I'm still getting the 403 responses. Can you check and comment my MR below please?

@jsschmid Can you check if my MR #131 solves this problem on your end and comment there?

[CactiChameleon9]

Unfortunately this didn't work for me. Already commented on #131

[github-actions]

This PR is stale because it has been open for 30 days with no activity. It will be closed in 14 days if there is no activity.

[Anatol-Beck]

Thank you for all the testing, I understand the issue better now. I agree that this fix would most probably not work on servers and even for private machines it's not a general solution.

I would therefore, take the learning, try to come up with a better solution and close this PR.