feat: support GitHub mirrors and proxy URLs in repo import#103
Open
iarhaaan wants to merge 6 commits into
Open
feat: support GitHub mirrors and proxy URLs in repo import#103iarhaaan wants to merge 6 commits into
iarhaaan wants to merge 6 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR updates the GitHub repository import pipeline to accept GitHub mirror/proxy URL formats (including “double-protocol” proxy prefixes) and to construct download ZIP URLs in a way that works better with public proxies/mirrors that block codeload.github.com.
Changes:
- Extended
parseGitHubUrlto extract a proxy prefix and return{ hostname, proxyPrefix }alongside{ owner, repo, branch }. - Updated
fetchGitHubRepoZIP URL construction to use GitHub “archive” URLs for proxy/mirror scenarios instead of always usingcodeload.github.com. - Added integration tests covering parsing and ZIP URL generation for shorthand, standard GitHub, proxy-prefixed, and mirror-domain inputs.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
src/content/files/github-reader.js |
Relaxes parsing to support proxy/mirror URLs and changes ZIP download URL generation accordingly. |
tests/integration/github-reader.test.js |
Adds integration coverage for parsing and URL construction across proxy/mirror scenarios. |
Comments suppressed due to low confidence (1)
src/content/files/github-reader.js:167
- When a proxy prefix or mirror hostname is used, the constructed download URL is no longer
codeload.github.com, and the background fetcher intentionally will not attach a GitHub token (it only sends tokens tocodeload.github.com). This means users cannot import private repos via proxy/mirror URLs, and the resulting failures can be misleading. Consider failing fast with a clear error when a token is provided with a proxy/mirror URL.
const { owner, repo, branch, hostname, proxyPrefix } = parsed;
const token = String(
typeof options.token === "string" ? options.token : ""
).trim();
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
87
to
91
| try { | ||
| const url = new URL(trimmed); | ||
| if (url.hostname !== "github.com") return null; | ||
| const hostname = url.hostname; | ||
|
|
||
| const parts = url.pathname.split("/").filter(Boolean); |
| hostname: "hub.fastgit.xyz", | ||
| proxyPrefix: "https://hk.gh-proxy.org/", | ||
| }); | ||
| expect(parseGitHubUrl("https://example.com/nope")).toBeNull(); |
Comment on lines
74
to
+78
| // owner/repo shorthand | ||
| if (/^[a-zA-Z0-9_.-]+\/[a-zA-Z0-9_.-]+$/.test(trimmed)) { | ||
| const [owner, repo] = trimmed.split("/"); | ||
| return { owner, repo, branch: "main" }; | ||
| return { owner, repo, branch: "main", hostname: "github.com", proxyPrefix: "" }; | ||
| } |
Add todo/checklist card with markdown export
…aaan/better-deepseek into feat/support-github-proxies
Comment on lines
+64
to
+68
| function isPrivateOrLocal(hostname) { | ||
| const host = hostname.toLowerCase().trim(); | ||
| if (host === "localhost" || host === "[::1]") { | ||
| return true; | ||
| } |
Comment on lines
+259
to
+263
| if (lastFailure && lastFailure.status === 404 && token && (proxyPrefix || hostname !== "github.com")) { | ||
| throw new Error( | ||
| "Private repositories cannot be imported using proxy or mirror URLs. Please use a direct GitHub URL to use your access token safely." | ||
| ); | ||
| } |
Comment on lines
128
to
136
| try { | ||
| const url = new URL(trimmed); | ||
| if (url.hostname !== "github.com") return null; | ||
| if (url.protocol !== "http:" && url.protocol !== "https:") return null; | ||
| if (isPrivateOrLocal(url.hostname)) return null; | ||
|
|
||
| const hostname = url.hostname; | ||
|
|
||
| const parts = url.pathname.split("/").filter(Boolean); | ||
| if (parts.length < 2) return null; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR resolves an issue where the repository import feature strictly validates and rejects GitHub mirror domains and proxy URLs. In regions where ISPs heavily throttle direct GitHub connections, users rely on mirrors or proxy prefixes to fetch code at reasonable speeds.
Changes
parseGitHubUrlto extract double-protocol proxy prefixes (e.g.https://hk.gh-proxy.org/https://github.com/...).github.comhostname checking to support mirror domains (e.g.github.com.cnpmjs.org,hub.fastgit.xyz).fetchGitHubRepoto detect proxy/mirror hostnames and automatically map downloads to standard archive URLs (/archive/refs/heads/branch.zip) instead ofcodeload.github.com, preventing403 Forbiddenblocks common on public proxies.github-reader.test.jsto assert parsing and URL generation for shorthand, standard, proxy-prefixed, and domain-mirror URL schemes.All unit and integration tests are passing and the extension compiles cleanly.