Add logging / UI cues when a repository fails to clone

[jerrykan]

I seem to have hit a problem where Sourcebot only indexes repositories with public visibility and not internal or private visibility on our self-hosted GitLab instance.

I have create a dedicated Sourcebot account in GitLab (marked as an external user) and generated an "Impersonation Token" (essentially the same as a "Personal Access Token) for the account via the GitLab Admin interface.

The Sourcebot account has been added as a member to a few different repositories and groups (of which most repositories have their visibility set to either "Internal" or "Private").

The "Impersonation Token" has been set in the GITLAB_TOKEN environment variable when starting the container. I can see requests to the GitLab API coming from Sourcebot and can confirm that the responses for the /api/v4/projects?membership=true&order_by=id&per_page=100&sort=asc endpoint contain all the repositories that the Sourcebot account has access to are indeed being returned. However, it seems that Sourcebot is only indexing the repositories which have a visibility of public.

I'm not sure if this is indented behaviour, or a bug, but it would be good if there were a way ensure that Sourcebot indexes GitLab repositories with a visibility of internal or private.

[jerrykan]

Here is my config file:

{
  "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/index.json",
  "Configs": [
    {
      "GitLabURL": "https://<self-hosted host>",
      "Type": "gitlab"
    }
  ]
}

I should also note that explicitly setting "OnlyPublic": false made no difference.

[brendan-kellam]

Hey @jerrykan - thanks for raising a issue! This is definitely a bug. Could you validate if the repositories are being successfully cloned in the .sourcebot/repos directory? Also could you check if the .zoekt indexes are in the .sourcebot/index directory? You'll need to mount a volume with -v $(pwd):/data to inspect the contents of .sourcebot.

The issue is likely either (a) in the mirror step (handled by zoekt-mirror-gitlab), in which case the repos are likely missing in the .sourcebot/repos directory, or (b) in the indexing step (handled by zoekt-indexserver) in which case the indexes will be missing.

For good measure, do you mind also sending along any console logs outputted?

[brendan-kellam]

Also to sanity check, did you set the GITLAB_HOSTNAME environment variable to your instance's hostname? https://github.com/sourcebot-dev/sourcebot?tab=readme-ov-file#using-a-self-hosted-gitlab--github-instance

[jerrykan]

Hi @brendan-kellam

Hey @jerrykan - thanks for raising a issue! This is definitely a bug. Could you validate if the repositories are being successfully cloned in the .sourcebot/repos directory? Also could you check if the .zoekt indexes are in the .sourcebot/index directory? You'll need to mount a volume with -v $(pwd):/data to inspect the contents of .sourcebot.

The issue is likely either (a) in the mirror step (handled by zoekt-mirror-gitlab), in which case the repos are likely missing in the .sourcebot/repos directory, or (b) in the indexing step (handled by zoekt-indexserver) in which case the indexes will be missing.

The private repositories are not showing up in either the .sourcebot/repos or the .sourcebot/index directories. So it would seem to be a problem with zoekt-mirror-gitlab

---

For good measure, do you mind also sending along any console logs outputted?

[Info] Disabling telemetry since SOURCEBOT_TELEMETRY_DISABLED was set.
[Info] Using config file at: '/data/config.json'.
[Info] Private GitHub repositories will not be indexed since GITHUB_TOKEN was not set.
[Info] Configuring GitLab credentials with hostname '<redacted-gitlab-host>'.
2024-10-09 01:59:04,305 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2024-10-09 01:59:04,308 INFO supervisord started with pid 2
2024-10-09 01:59:05,313 INFO spawned: 'node-server' with pid 68
2024-10-09 01:59:05,325 INFO spawned: 'zoekt-indexserver' with pid 69
2024-10-09 01:59:05,341 INFO spawned: 'zoekt-webserver' with pid 70
2024/10/09 01:59:05 run [zoekt-mirror-gitlab -dest /data/.sourcebot/repos -url https://<redacted-gitlab-host>]
{"SeverityText":"WARN","Timestamp":1728439145635753523,"InstrumentationScope":"metricsRegistration.mountPointInfo.deviceNameDiscovery","Caller":"mountinfo@v0.0.0-20240201124957-b314c0befab1/info.go:48","Function":"github.com/sourcegraph/mountinfo.NewCollector","Body":"skipping metric registration","Resource":{"service.name":"zoekt-webserver","service.instance.id":"0.0.0.0"},"Attributes":{"mountName":"indexDir","mountFilePath":"/data/.sourcebot/index","reason":"failed to discover device name","error":"discovering device path: discoverSysfsDevicePath: failed to evaluate sysfs symlink \"/sys/dev/block/254:0\": lstat /sys/dev/block/254:0: no such file or directory"}}
2024/10/09 01:59:05 loading 1 shard(s): <redacted-gitlab-host>%2F<redacted-repo-group>%2F<redacted-repo>_v16.00000.zoekt
2024/10/09 01:59:05 run [zoekt-git-index -require_ctags -parallelism=1 -repo_cache /data/.sourcebot/repos -index /data/.sourcebot/index -incremental /data/.sourcebot/repos/<redacted-gitlab-host>/<redacted-repo-group>/<redacted-repo>.git]
  ▲ Next.js 14.2.10
2024-10-09 01:59:06,408 INFO success: node-server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-10-09 01:59:06,409 INFO success: zoekt-indexserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-10-09 01:59:06,409 INFO success: zoekt-webserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
  - Local:        http://localhost:3000
  - Network:      http://0.0.0.0:3000

 ✓ Starting...
 ✓ Ready in 344ms
2024/10/09 02:59:05 run [zoekt-git-index -require_ctags -parallelism=1 -repo_cache /data/.sourcebot/repos -index /data/.sourcebot/index -incremental /data/.sourcebot/repos/<redacted-gitlab-host>/<redacted-repo-group>/<redacted-repo>.git]
[zoektClient] POST /api/list 200 1094ms
[zoektClient] POST /api/list 200 16ms
2024/10/09 03:59:05 run [zoekt-git-index -require_ctags -parallelism=1 -repo_cache /data/.sourcebot/repos -index /data/.sourcebot/index -incremental /data/.sourcebot/repos/<redacted-gitlab-host>/<redacted-repo-group>/<redacted-repo>.git]
2024/10/09 04:59:06 run [zoekt-git-index -require_ctags -parallelism=1 -repo_cache /data/.sourcebot/repos -index /data/.sourcebot/index -incremental /data/.sourcebot/repos/<redacted-gitlab-host>/<redacted-repo-group>/<redacted-repo>.git]
2024/10/09 05:59:05 run [zoekt-git-index -require_ctags -parallelism=1 -repo_cache /data/.sourcebot/repos -index /data/.sourcebot/index -incremental /data/.sourcebot/repos/<redacted-gitlab-host>/<redacted-repo-group>/<redacted-repo>.git]

Note only 1 repository is being indexed ("visibility": "public") , while the GitLab user also has access to 4 other repositories ("visibility": "private")

---

Also to sanity check, did you set the GITLAB_HOSTNAME environment variable to your instance's hostname? https://github.com/sourcebot-dev/sourcebot?tab=readme-ov-file#using-a-self-hosted-gitlab--github-instance

Yes, the GITLAB_HOSTNAME environment variable is correctly set (which would seem to be confirmed by the fact that the public repositories are being indexed correctly.

[brendan-kellam]

What role does the external GitLab user have on the groups you gave it access to?

I was able to reproduce the issue you are describing when the role is set to Guest. When I change the role to Reporter or higher, I am able to clone both private and internal repositories it seems.

The docs seem to suggest that makes sense: https://docs.gitlab.com/17.4/ee/user/public_access.html

Private:

https://docs.gitlab.com/17.4/ee/user/public_access.html#private-projects-and-groups

For private projects, only members of the private project or group can:
Clone the project.
View the public access directory (/public).

Users with the Guest role cannot clone the project.

Internal:

https://docs.gitlab.com/17.4/ee/user/public_access.html#private-projects-and-groups

For internal projects, any authenticated user, including users with the Guest role, can:

Clone the project.
View the public access directory (/public).
Only internal members can view internal content.

External users cannot clone the project.

The docs don't outline what happens in the "External user & Reporter+ Role" scenario, but it seems like it allows the user to clone the project.

[jerrykan]

@brendan-kellam Thanks for the follow-up. I didn't realise that Guest users are not able to clone projects. I've change the membership of the GitLab user to Reporter and it seems to be indexing the private repositories now.

It might be useful to output some sort of error indicating if a repo can't be cloned to help debug problems (or is there a way to adjust the logging level to show these sorts of errors?)

[brendan-kellam]

Perfect glad it's working now. And yea definitely - improved logging is on the list as part of our current effort to revamp how the index server / mirror commands work. Changed title & tags to track that.