FR: Support indexing code in local working copies

[olivierdagenais]

I would like to configure Sourcebot to (recursively) index all the working copies in a given directory tree.

This way, I could search anywhere in repositories I have previously cloned and it didn't matter where they came from (support would not be required for, say, Bitbucket DC).

Thanks!

[brendan-kellam]

Heya - we have support for indexing local directories in V2 (see this part of the schema) - would that suffice for your use-case?

[vguleaev]

You do not plan to support local repos in V3?

[olivierdagenais]

Hello, @brendan-kellam, I finally had time to try it out and, after crafting the following configuration file:

{
    "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json",
    "connections": {
        "oss": {
            "type": "local",
            "path": "/home/native/oss",
            "watch": true
        }
    }
}

...Sourcebot 3.0.4 emitted this error in the log:

[web] | Error: Config file '/sourcebot/config.json' is invalid: data/connections/oss must NOT have additional properties, data/connections/oss must NOT have additional properties, data/connections/oss must NOT have additional properties, data/connections/oss must have required property 'url', data/connections/oss must match exactly one schema in oneOf
[web] |     at tN (/app/packages/web/.next/server/chunks/2900.js:7:107815)
[web] |     at async tx (/app/packages/web/.next/server/chunks/2900.js:7:109424)
[web] |     at async /app/packages/web/.next/server/chunks/2900.js:7:109577

Eventually, I noticed that the sample configuration file for self-hosting with a container was pointing to v3 while the link you introduced was V2... so now I see they are very different configuration file schemas and I finally understand the follow-up question about local repositories not being supported in V3.

So, my feature request still stands. We operate in an isolated network with no access to GitHub, nor any of the other source control systems/services that Sourcebot apparently supports in v3.

[brendan-kellam]

You do not plan to support local repos in V3?

Sorry @vguleaev I missed your reply. See my reply to @olivierdagenais below.

Hello, @brendan-kellam, I finally had time to try it out and, after crafting the following configuration file:

{
"$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json",
"connections": {
"oss": {
"type": "local",
"path": "/home/native/oss",
"watch": true
}
}
}
...Sourcebot 3.0.4 emitted this error in the log:

[web] | Error: Config file '/sourcebot/config.json' is invalid: data/connections/oss must NOT have additional properties, data/connections/oss must NOT have additional properties, data/connections/oss must NOT have additional properties, data/connections/oss must have required property 'url', data/connections/oss must match exactly one schema in oneOf
[web] |     at tN (/app/packages/web/.next/server/chunks/2900.js:7:107815)
[web] |     at async tx (/app/packages/web/.next/server/chunks/2900.js:7:109424)
[web] |     at async /app/packages/web/.next/server/chunks/2900.js:7:109577

Eventually, I noticed that the sample configuration file for self-hosting with a container was pointing to v3 while the link you introduced was V2... so now I see they are very different configuration file schemas and I finally understand the follow-up question about local repositories not being supported in V3.

So, my feature request still stands. We operate in an isolated network with no access to GitHub, nor any of the other source control systems/services that Sourcebot apparently supports in v3.

For context, we removed support for local indexing from V3 since it was slightly obtuse to support a repository that is non-git (i.e., any random directory) since many parts of the application assumed a repo uses git.. so we made the decision to remove support for this in V3.

I think a compromise that we can make is to add support for local git repos, i.e., any directory that contain a .git folder at the top-level. My hope is that this will cover most people's use cases for local indexing.

Would this cover your use-case @olivierdagenais? @vguleaev?

[olivierdagenais]

Yes, configuring multiple Git working copy directories is the direction I was headed in. Before I spent time generating the corresponding configuration, I figured I would point it at a directory tree containing a bunch of Git repositories to see if I could just get that working.

[olivierdagenais]

any directory that contain a .git folder at the top-level

Does that mean Git worktrees would not be supported? I was planning on creating a few of those for some of our repositories that have more than one active branch; they don't have a .git folder but rather a .git file that contains the path to the origin/reference working copy from with the worktree was created.

Granted, this could be a phase 2 feature.

[brendan-kellam]

Hmm good point - I don't think it should matter too much whether it is a folder or a file (besides worktrees, git submodules also come to mind where .git can be a file), just as long as we are able to run git commands against it.

However, I think if we proceed with the base assumption that we are dealing with a git directory, multi-branch indexing can be achieved without git worktrees - instead, we should be able to specify revisions in the schema similar to what we are doing with other connection types.

What I'm thinking: we re-introduce the git connection type that previously existed in v2, with additional support for file:// resources. For example:

{
    "$schema": "./schemas/v3/index.json",
    "connections": {
        // Remote path
        "connection-a": {
            "type": "git",

            // Must be a valid git remote URL. If it is, it will be
            // cloned into .sourcebot/repos
            "url": "https://github.com/sourcebot-dev/sourcebot.git"
        },

        // Local path
        "connection-b": {
            "type": "git",

            // This path must be a git working directory.
            "url": "file:///path/to/local/repo",

            // Example of multiple revisions
            "revisions": {
                "branches": [
                    "branch-a",
                    "branch-b"
                ]
            }
        }
    }
}

For use-cases such as yours (bunch of repositories in a single directory), we would probably benefit from supporting glob patterns for urls that have the file: protocol. For example:

{
    "$schema": "./schemas/v3/index.json",
    "connections": {
        "connection-c": {
            "type": "git",

            // Glob pattern that matches all git working
            // directories within `base_dir`. E.g.,:
            //
            // - /path/to/base_dir/foo
            // - /path/to/base_dir/bar
            // - /path/to/base_dir/baz
            //
            // (assuming foo, bar, and baz are valid git working directories)
            "url": "file:///path/to/base_dir/*"
        }
    }
}

[olivierdagenais]

Hmm good point - I don't think it should matter too much whether it is a folder or a file (besides worktrees, git submodules also come to mind where .git can be a file), just as long as we are able to run git commands against it.

Yes, I believe the git client is fine with both, however, after recently having tried a competitor (when I found Sourcebot wasn't working with local working copies), I was horrified to see it attempted to run git reset --hard origin/<default branch> as well as git fetch and other git commands that could have lost me work (thankfully, I was running via a container and had configured the volume as read-only), I would like to qualify my feature request as:

I would like to configure Sourcebot to index all one or more Git working copies (as they currently exist, with no mutations attempted) by configuring one or more connections to local directories.

I'm OK with a watch feature (which could also be re-introduced at a later phase), since the use-case would be to keep the Sourcebot app running while I work, such that I could search code I wrote 30 seconds ago as easily as I could search code I wrote 30 days ago.

I may also not need the revisions feature right away if I can configure some of the connections to worktrees, especially if it's not possible to provide that feature without needing to configure read/write access to my working copies.

[brendan-kellam]

Is the intention to run Sourcebot on your local machine or in a deployed environment? I think we will need write access for the purposes of fetching new revisions of the repository.

Supporting watch gets tricky since our current assumption is that the "source of truth" for the repository is on the remote, so reindexing implies first refetching from that remote. Things can be special-cased of-course to support this use-case, but tbh the core-scenario we are thinking about is when Sourcebot is running as a deployment rather than locally.

[olivierdagenais]

Is the intention to run Sourcebot on your local machine

Yes, that was my intention, mostly because not everybody has access to all Git repositories (we would first need more authentication options, such as LDAP, with some mechanism for synchronization of authorization with our hosting provider to make sure people can't see code they would otherwise have access to), plus I really want to search all the code across the repositories I have cloned, even/especially if I haven't pushed it yet.

Sorry if it sounds like I'm derailing your use-cases. I was contacted to ask what my experience was because I had starred the repo, probably after seeing an announcement on Hacker News. Maybe my current position (and its associated restrictions) means I'm not your target customer.

[brendan-kellam]

Hey @olivierdagenais - we just shipped support for indexing local copies in #307 . This doesn't support a watch mode at this time, but it's something we are thinking about. You can find the docs here.