[WIP] Update fetchHTML to support gl query parameter

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

Original prompt

Change the Play Store HTML fetch and the HTTP handlers to support the gl query parameter so consumers can request ratings for any country (not just US).

Requirements:

1. Update fetchHTML signature and behavior

   • Change fetchHTML(packageID string) to fetchHTML(packageID string, gl string).
   • Normalize gl (uppercase, trim). If empty, default to "US" to preserve backward compatibility.
   • Use a Redis cache key that includes gl (e.g., ${packageID}:gl=${gl}) so HTML is cached per-country.
   • Append &gl=<GL> to the Play Store URL used for fetching HTML: https://play.google.com/store/apps/details?id=<id>&gl=<GL>.
   • Update logs to include the gl value.

2. Propagate gl through all call sites

   • Find every call site that invokes fetchHTML(packageID). Update these call sites to accept and forward a gl string.
   • If the caller is an HTTP handler (e.g., the /json route) or can access request query params, read the gl query param from the request (r.URL.Query().Get("gl")) and pass it through.
   • If a caller cannot access a request, provide a sensible default ("US") or accept an additional parameter to forward gl.

3. Update HTTP /json handler

   • Read gl from the request query string (e.g., ?id=com.example.app&gl=IN).
   • Validate/normalize gl (uppercase, trim). Leave validation lenient — only normalization and defaulting to "US" if empty.
   • Pass gl into fetchHTML and downstream functions that require it.
   • Ensure error handling and cache behavior remain consistent.

4. Tests & examples

   • Update any unit tests or mocks that call fetchHTML to pass the new gl parameter (or update to use a helper default if appropriate).
   • Update README or examples to show passing gl in requests (optional but preferred): e.g., /json?id=com.spotify.music&gl=GB.

5. Backwards compatibility and migration

   • Keep behavior identical for existing clients that do not pass gl by defaulting to "US".
   • Keep existing Redis keys intact; new keys should be used for per-gl caching.

Implementation notes for the agent

• The agent should search the repository for all call sites of fetchHTML(packageID) and update them.
• Update function signatures transitively where required so compilation succeeds.
• Update the /json route handler to read the gl query parameter and pass it down.
• Ensure imports and gofmt formatting are correct.
• Update any tests referencing the old function signature.

Files likely to change (agent should search and modify as needed):

• functions.go (change fetchHTML signature and caching behavior)
• main.go (or whichever file contains the /json handler) — read gl param and pass it to fetchHTML
• Other files that call fetchHTML — update signatures and calls
• Tests under *_test.go that call fetchHTML

Deliverable

• Create a new branch and open a pull request with all changes, a clear PR title, and a summary of modifications and rationale. The PR should build and keep previous behavior when gl is not provided.

Do the work: search the repo for usages, make the code changes, run gofmt (or ensure formatting), update tests, and open a PR containing all edits. If you encounter ambiguous call sites where gl cannot be obtained, thread gl as an additional parameter and document the changes in the PR description.

This pull request was created as a result of the following prompt from Copilot chat.

Change the Play Store HTML fetch and the HTTP handlers to support the gl query parameter so consumers can request ratings for any country (not just US).

Requirements:

1. Update fetchHTML signature and behavior

   • Change fetchHTML(packageID string) to fetchHTML(packageID string, gl string).
   • Normalize gl (uppercase, trim). If empty, default to "US" to preserve backward compatibility.
   • Use a Redis cache key that includes gl (e.g., ${packageID}:gl=${gl}) so HTML is cached per-country.
   • Append &gl=<GL> to the Play Store URL used for fetching HTML: https://play.google.com/store/apps/details?id=<id>&gl=<GL>.
   • Update logs to include the gl value.

2. Propagate gl through all call sites

   • Find every call site that invokes fetchHTML(packageID). Update these call sites to accept and forward a gl string.
   • If the caller is an HTTP handler (e.g., the /json route) or can access request query params, read the gl query param from the request (r.URL.Query().Get("gl")) and pass it through.
   • If a caller cannot access a request, provide a sensible default ("US") or accept an additional parameter to forward gl.

3. Update HTTP /json handler

   • Read gl from the request query string (e.g., ?id=com.example.app&gl=IN).
   • Validate/normalize gl (uppercase, trim). Leave validation lenient — only normalization and defaulting to "US" if empty.
   • Pass gl into fetchHTML and downstream functions that require it.
   • Ensure error handling and cache behavior remain consistent.

4. Tests & examples

   • Update any unit tests or mocks that call fetchHTML to pass the new gl parameter (or update to use a helper default if appropriate).
   • Update README or examples to show passing gl in requests (optional but preferred): e.g., /json?id=com.spotify.music&gl=GB.

5. Backwards compatibility and migration

   • Keep behavior identical for existing clients that do not pass gl by defaulting to "US".
   • Keep existing Redis keys intact; new keys should be used for per-gl caching.

Implementation notes for the agent

• The agent should search the repository for all call sites of fetchHTML(packageID) and update them.
• Update function signatures transitively where required so compilation succeeds.
• Update the /json route handler to read the gl query parameter and pass it down.
• Ensure imports and gofmt formatting are correct.
• Update any tests referencing the old function signature.

Files likely to change (agent should search and modify as needed):

• functions.go (change fetchHTML signature and caching behavior)
• main.go (or whichever file contains the /json handler) — read gl param and pass it to fetchHTML
• Other files that call fetchHTML — update signatures and calls
• Tests under *_test.go that call fetchHTML

Deliverable

• Create a new branch and open a pull request with all changes, a clear PR title, and a summary of modifications and rationale. The PR should build and keep previous behavior when gl is not provided.

Do the work: search the repo for usages, make the code changes, run gofmt (or ensure formatting), update tests, and open a PR containing all edits. If you encounter ambiguous call sites where gl cannot be obtained, thread gl as an additional parameter and document the changes in the PR description.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[abhisheksoni27]

close it