Failed to clone repository

[mstimvol]

When Sourcebot syncs my repositories (around 480), a few of them (8) are failing with the following error:

[backend] | 2025-04-23T15:51:19.622Z error: [RepoManager] Failed to sync repository 447, attempt 2/3. Sleeping for 10s... Error: Error: Failed to clone repository

My conf looks like:

{
    "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/refs/heads/main/schemas/v3/index.json",
    "connections": {
        "some-connection1": {
            "type": "github",
            "repos": ["myorga/repo"],
            "revisions": {
                "branches": [
                    "main",
                    "stable_v1",
                    "stable_v2"
                ]
            }
        },
        "some-connection2": {
            "type": "github",
            "repos": ["Behat/Gherkin"]
        },
        "some-connection3": {
            "type": "gitlab",
            "token": {
                "env": "GITLAB_TOKEN"
            },
            "groups": [
                "plugins"
            ],
            "url": "https://my-self-hosted-instance.tld"
        }
    }
}

I first thought it's because of the API limit in Gitlab or because one or two projects are archived, but I figured out that it also fails to sync sometimes "some-connection1" and "some-connection2" and furthermore, the 8 repositories are always other repos - not the same. Any ideas why the indexing fails?

I'm using v3.0.4 of Sourcebot

[msukkari]

Hey Tim, sorry you're hitting this. Since the specific repos that are failing is changing I suspect this might be a networking issue when cloning the repos (I'm assuming the fail is happening during cloning not indexing from your error log). We've hit this before in certain environments where the networking is limited.

A workaround I would suggest you try is to drop the maxRepoIndexingJobConcurrency in the config: https://demo.sourcebot.dev/~/browse/github.com/sourcebot-dev/sourcebot@HEAD/-/blob/schemas/v3/index.json?highlightRange=46%3A1%2C50%3A19

Let me know if that helps!

[mstimvol]

Hi Michael, changing maxRepoIndexingJobConcurrency from 8 to 2 finally fixed the error. Thank you very much!

[securisec]

@msukkari having the same issue with gitea. Here is my quick test setup. The user admin has 4 repos in gitea.

docker-compose.yml

version: "3"

networks:
  gitea:
    external: false

services:
  gitea:
    image: docker.gitea.com/gitea:1.23.7
    environment:
      - USER_UID=1000
      - USER_GID=1000
    restart: no
    networks:
      - gitea
    volumes:
      - ./data:/data
      # - /etc/timezone:/etc/timezone:ro
      # - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      # - "222:22"

  sourcebot:
    image: ghcr.io/sourcebot-dev/sourcebot:latest
    volumes: 
      - ./sb:/data
    networks:
      - gitea
    ports:
      - 3001:3000
    environment:
      - CONFIG_PATH=/data/config.json
      - GITEA_TOKEN=...

config.json

{
  "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/refs/heads/main/schemas/v3/index.json",
  "settings": {
    "maxConnectionSyncJobConcurrency": 1,
    "maxRepoIndexingJobConcurrency": 2
  },
  "connections": {
    "connection-1": {
      "type": "gitea",
      "users": [
        "admin"
      ],
      "url": "http://gitea:3000",
      "token": {
        "env": "GITEA_TOKEN"
      }
    }
  }
}

The logs show

sourcebot-1  | [backend] | 2025-05-10T05:19:47.283Z error: [RepoManager] Failed to sync repository gitea:3000/admin/chepy (id: 2) after 3 attempts. Error: Error: Failed to clone repository
sourcebot-1  | [backend] | 2025-05-10T05:19:47.285Z error: [RepoManager] Failed to sync repository gitea:3000/admin/foo (id: 1) after 3 attempts. Error: Error: Failed to clone repository

In sourcebot, I do see that it touches the 4 repos (3 marked as private and all are failing). The repos are fine inside gitea itself.

To make sure it is not a network issue, I obtained a shell inside the sourcebot container, and can manually clone any of the repos via specifying the token as the password.

Any suggestions would be great! Maybe a feature request, more verbose error messages would be wonderful.

[msukkari]

@securisec I added some better logging to git operations in 3.1.4 https://github.com/sourcebot-dev/sourcebot/releases/tag/v3.1.4

Can you try with this new version? It should hopefully help us know whats going on

[securisec]

Thank you for adding the extended logging @msukkari. I think I see the reason why it fails to clone. The following are assumptions of course. I would need to debug it.

TLDR

It seems like the issue is due to a mismatch in clone_url from the api response, and the gitea host that is set in config.json.

In https://github.com/sourcebot-dev/sourcebot/blob/main/packages/backend/src/gitea.ts#L26, I can see that it will use the gitea host that i am specifying in the config.json, but when it uses the gitea api to fetch the repos, https://github.com/sourcebot-dev/sourcebot/blob/main/packages/backend/src/gitea.ts#L42-L59, it will will receive back the gitea api response as gitea sees it https://github.com/sourcebot-dev/sourcebot/blob/main/packages/backend/src/gitea.ts#L142, but it never updates the git urls https://github.com/sourcebot-dev/sourcebot/blob/main/packages/backend/src/repoCompileUtils.ts#L185.

So in this case, instead of seeing the repo url as the gitea host (ex. http://gitea:3000/admin/cliam) (although specified in the config (it also gets the repo names correctly because of this - we can validate that in the screenshot above because it shows the valid repo names)), it incorrectly sees the repo url as how gitea sees it and includes in the api response, (ex http://localhost:3000/admin/cliam).

The raw response from the gitea api is

...
		"ssh_url": "git@localhost:admin/cliam.git",
		"clone_url": "http://localhost:3000/admin/cliam.git",
		"original_url": "",
		"website": "",
...

which means https://github.com/sourcebot-dev/sourcebot/blob/main/packages/backend/src/repoCompileUtils.ts#L106 sees the wrong url and fails to clone which is why we see the error

...
sourcebot-1  | [backend] | fatal: repository 'http://localhost:3000/admin/cliam.git/' not found
...

The ideal fix would be in https://github.com/sourcebot-dev/sourcebot/blob/main/packages/backend/src/repoCompileUtils.ts#L106 I would imagine. Something like

const repo = { // what source bot sees after processing gitea api result.
  clone_url: 'http://localhost:3000/admin/cliam.git/'
}

const config = {
  host: 'http://gitea:3000'
}

const url = new URL(repo.clone_url);
console.log(url.toString())
// http://localhost:3000/admin/cliam.git/ (incorrect gitea clone url)

const configUrl = new URL(config.host)

url.host = configUrl.host
console.log(url.toString())
// http://gitea:3000/admin/cliam.git/ (correct gitea clone url)

[msukkari]

I think your explanation of the the issue makes sense. Your gitea instance isn't aware that it's running in a docker network with Sourcebot so the clone_url it gives back refers to localhost not gitea.

We could support this use case in Sourcebot by adding an clone_url override of some sort in the config. However, I think a quick fix in your case is to spin up your gitea and sourcebot instances seperately and open up the gitea instance port so that sourcebot can hit it using localhost

[securisec]

I just validated the fix by modifying https://github.com/sourcebot-dev/sourcebot/blob/main/packages/backend/src/repoCompileUtils.ts#L106 with

        const configUrl = new URL(hostUrl);
        const cloneUrl = new URL(repo.clone_url!);
        cloneUrl.host = configUrl.host

I believe updating the clone_url is the correct path. The thought is that it would not be an uncommon deployment practice to deploy in compose files, routed via edge routers or k8s clusters where the current implementation of using the gitea supplied clone_url will always break. Specially considering the gitea host url is user configurable, perhaps it would be reasonable to assume that the clone_url will always be hostUrl? Because the current assumption is that hostUrl is the host for clone_url which in the implementation details is not right.

Happy to make a PR if you folks think this fix is the right direction to go. But also agree that if the clone_url base is configurable, that is also an idea path that results in the same outcome.

[msukkari]

This fix should probably be fine if we contain it to gitea config compilation. I'd prefer to not change the way we fetch clone_url for other platforms unless we fully understand the side effects. Thanks for looking into this!

[securisec]

Happy to help. Yes, I believe this impacts gitea only for now (though possible selfhosted gitlab). For now, I will create a PR for gitea only.