Git Proxy 2.x architecture

[coopernetes]

Tracking issue to revisit Git Proxy's internal architecture and figuring out how best to design the application going forward. I'm including my initial sketch and using this issue to bounce around ideas, flesh out the right interfaces & API boundaries as well as discuss overall what we want to change in Git Proxy internals.

Issues that are blocked on the current architecture or would greatly benefit from a new one:

• SSH support #27
• Patch oriented workflow #61
• Sign outgoing contributions #62
• Provide facility to manage multiple forks of the same upstream repository #66
• Synchronous Webhooks #10
• Asynchronous management of a push request  #428
• Scan for leaked credentials in PRs and code diff using TruffleHog OSS 🐷  #414

Related issues:

• TypeScript and ESM support #276
• Open Policy Agent checks #99

[coopernetes]

Here's first stab at a new architecture. Some initial thoughts:

• the idea of a "chain" of actions that Git Proxy executes in some sequence is extremely valuable. I don't think we need to rework this that much as we currently have a series of Step/Action types that the application cycles through for every git push.
• @JamieSlome brought up some interesting points in Open Policy Agent checks #99 around policy management. What constitutes a "policy" vs a "filter" is not immediately clear to me but some separation between a decision to permit and the underlying git request seems like a useful abstraction. It would also allow us to implement an interface that is generic to other policy engines like OPA
• Each phase of the request lifecycle should be able to fire off events in an agnostic way. How we persist those events can be abstracted into a persistence interface and backed by a variety of sources (object storage, SQL, Mongo, etc).
• I included an idea of using git-proxy purely in a localized sense such as a library running on a development machine. This may be preferable to a hosted server so I've separated those out into their own components. At the very least, this allows a developer to directly use git-proxy without the new to run it as a server.

Much of this will likely result in breaking changes so I consider this a version 2.x of git-proxy so we're free to iterate on the new design.

EDIT: Added the dashboard UI, API and user interactions.

[maoo]

Really like this new architecture, and the drawing.

For the latter, I think it would help to draw where users connect to, specifically:

• a Developer (connecting directly to git-proxy-server, or the local block close to it)
• a Compliance manager, which should connect to git-proxy-ui, though I don't see it mentioned ; I also assume that the UI connects to an API layer that is exposed by the core package, which would be helpful to draw.

WDYT?

[JamieSlome]

I'd like to bring in #428 to 2.x as well. Likely sits in the filter chain in the architecture diagram above.

@coopernetes - thoughts?

[coopernetes]

I'll add the API and dashboard (UI) components in here. They should get decoupled and exist as separate packages with various interaction points with a few of the core components.

[JamieSlome]

🔔 Reminder to include for Monday working group agenda @coopernetes @maoo

[coopernetes]

Updated diagram above.

[maoo]

Here's a recap of async discussions with @coopernetes and @JamieSlome around project layout and structure.

• Codebase for 2.x restructuring is in https://github.com/coopernetes/git-proxy/tree/experiment-2.x
• At root level, the packages/ folder contains core application components and the plugins/ folder contains extensions which may be bundled with the app but are standalone functionality
• We plan to leverage Yarn 3+ workspaces to decompose the current monolithic package.json into plugins and packages

Anything to add?

[JamieSlome]

@coopernetes - looking fresh 🚀

One thing I'd like to reflect on is the home for plugins. Whilst it makes sense for us to bundle out-of-the-box plugins with Git Proxy, I wonder if the plugin functionality should itself be a separate repository, housed elsewhere?

It follows on from the discussion and closing comments in #438 as well conversations with GitLab around re-use of our plugins. I'd liken it to the modules ecosystem in Nuxt (massive props to @danielroe and team). I equate plugins for us to highly reusable modules that support a customisable Git Proxy, CI/CD or otherwise experience.

@coopernetes - I'd like to get your thoughts here...

[coopernetes]

@JamieSlome I'm agnostic on the approach. We can keep "in tree" features under the packages/ workspace and leave plugins to be outside the main repo. I think it's worth shipping at least one "example" or "hello world" extension but that can be hosted separately too. That can be provided out as a template repository as well.