Open Policy Agent checks

[coopernetes]

Hi all! 👋

Given the target use case of git proxy of performing security & compliance policy-based checks, it would be nice to reuse existing policies in Rego with Open Policy Agent running either as a sidecar or separate process along with the proxy.

There is also a WASM implementation but I don't have much experience with that myself so not sure if that fits this project.

Describe the solution you'd like

1. Provide a JSON body as input data into a rego policy from git-proxy.

{
  "author": "tom@example.com",
  "committer": "dick@example.com",
  "date": "1970-01-01T00:00:00",
  "commitSHA": "74d2a005a8619f9f748436269bbb0ae7d28fc15f",
  "url": "https://github.com/finos/test-allowed-repo",
  "branch": "feat/new-widget",
  ...
}

2. Launch the proxy with the following config.json using OPA in server mode or using embedded Rego & WASM.

"openPolicyAgent": {
  "enabled": true,
  "mode": "client",
  "url": "https://localhost:8181",
  "policies": [
    "authorized_committers",
    "contains_readme",
    "..."
  ]
  }

"openPolicyAgent": {
  "enabled": true,
  "mode": "embedded",
  "policies": [
    "resources/authorized_committers.rego",
    "resources/contains_readme.rego",
    "..."
  ]
}

3. Proxy sends sends an HTTP request to the OPA server or evaluates the policies in-place.

HTTP POST localhost:8081/v1/data/<policy_name>/allow
{
  "inputs": {
    "author": "tom@example.com",
    "committer": "dick@example.com",
    "date": "1970-01-01T00:00:00",
    "commitSHA": "74d2a005a8619f9f748436269bbb0ae7d28fc15f",
    "url": "https://github.com/finos/test-allowed-repo",
    "branch": "feat/new-widget"
  }
}

Response
{"result":false}

Describe alternatives you've considered
It may be worth while to write a simple interface around querying external systems and allowing the use of other policy engines such as Hashicorp Sentinel.

Additional context
Related issue(s):
#47

[JamieSlome]

Hi @coopernetes, thanks for raising the feature request 👏 I'm not familiar with Open Policy Agent so bear with me whilst I do some reading of the documentation.

A policy agent or engine is something we should take advantage of. There is a question as to whether this in itself should be configurable, as you rightly mentioned. As an example, as a developer deploying and using Git Proxy within an organization, I may want to use a policy framework or policy language I'm familiar with, nice 👍

I love your examples above; can you just explain what the benefits might be between these two for the developer?

Proxy sends sends an HTTP request to the OPA server or evaluates the policies in-place.

I'll drop further thoughts once I've read through the docs 📝

[coopernetes]

I love your examples above; can you just explain what the benefits might be between these two for the developer?

Proxy sends sends an HTTP request to the OPA server or evaluates the policies in-place.

Ah, I may not have been clear. These are really just two different deployment methods. OPA can run as a server with a generic HTTP API to make authorization decisions based on Rego policy language. Since git-proxy is currently implemented in Express, there is also a OPA WASM npm module to compile Rego in binary form which can be embedded as part of git-proxy itself.

When deploying git-proxy, you can either have WASM compiled policies as binaries which would ship with git-proxy itself using that npm package or you deploy OPA as a server running along side git-proxy either on the same host or separate infrastructure. git-proxy would then make an HTTP request to ask for a policy decision in combination with the input data. There are a few limitations with the WASM approach so the HTTP API is the most straight-forward to integrate with.

The rationale behind incorporating OPA is that it would cover a lot of potential use cases with the extensibility of the proxy. Between developers, security and compliance officers, you could then have a common language to define policy using a more application-driven approach to these sorts of custom checks. There's also an existing ecosystem to leverage.

OPA has the concept of input data. This is where most of the heavy-lifting for git-proxy would lay as it would need to provide a JSON structure as an input into the Rego policy and would based off of the commit data. I think a lot of this is already in place. I'm new to the project so this is just an idea at the moment. I guess this would be a custom Step? I haven't dug into the code base enough yet but I assume it would be trivial to use that as inputs into OPA.

I'll provide some examples here when I get the chance to demonstrate some use cases.

[coopernetes]

Here's a general design:

flowchart TB
    subgraph OPA
        direction LR
        C[OPA server] -.->|"{input: {email, commiter}}"| D[authorized_committers.rego]
        C -.->|"{input: {url}}"| E[authorized_repos.rego]
        C -.->|"{input: {...}}"| F[...]
    end
    A(committer) -->|push| B(git proxy)
    B <-->|http| C
    B -->|"push"| G[GitHub]

Loading

Some example policies written in Rego.

Authorized committer & repos

Unauthorized committer & repos

[JamieSlome]

Sorry for the delay in my response!

@coopernetes, thanks for the time you've put into the above 👍 Definitely provides a clearer image of the potential deployment approaches for Git Proxy.

It feels like the separation of the policy engine and the execution of Git Proxy is desirable. Ultimately, we want the experience of Git Proxy to be highly configurable. It may hold that a team wants to use Rego to enforce their requirements or perhaps an alternative.

Maybe we need to consider making Git Proxy generic enough to interface with "any" form of policy description, whether it be allowing a developer to implement Rego policies or attaching custom JavaScript logic.

Would love to get your thoughts on this level of abstraction. Just thinking in terms of the familiarity of developers with Rego. I am not that well exposed to it yet, not to say that we shouldn't pursue this as the default option or an option.

@maoo, would love to get your thoughts here too... ❤️

[maoo]

Thanks Jamie for the mention.

@coopernetes - very interesting proposal, this is something that would help git-proxy becoming as generic as originally designed. I think it's worth trying to work together and build a Proof of Concept (PoC); below I'll share some random thoughts, eager to hear your feedback.

1. I'd suggest to start a conversation around the high level architectural diagram shared by @coopernetes 2 comments ago. When we have a clear idea of current and desired diagrams, it's going to be easier to come up with a list of actions and assign tasks
2. I think that jumping on a quick meeting with the 3 of us (and anyone else that volunteers to put sweat equity into this) could drastically accelerate the development. WDYT? If you like the idea, please share your availability and timezone at help [at] finos [dot] org for next week; I'll take it from there
3. I give my +1 on the OPA sidecar implementation (VS rego parsing from git-proxy), as it leads to less overlap between git-proxy and OPA . We could even consider git-proxy as an OPA "plugin" that provides an input endpoint for the git protocol (and the exercise on step 1 could confirm - or not - this approach)
4. I agree with Jamie's point to still allow custom JS implementation - and potentially explore the possibility to integrate OPA with custom JS (this really depends on git-proxy use cases VS rego features and flexibility). We can consider this as one of the requirements when designing the PoC on step 1.

Off-topic - @coopernetes could well be one of the best GH usernames I've ever seen 😄

[JamieSlome]

@maoo @coopernetes - I'll schedule an open invite call and we can start hashing this out.

Generally, happy with reducing overlap where we can between Git Proxy and OPA, but really keen to keep ease of use, installation and deployment as simple as possible.

[grovesy]

I really like this - Some time back, in the early architecture, the central 'chain' loop that runs through actions, it was intended there would be a mechanism to post Synchronous and Asynchronous webhooks out

.. To do this we needed a standard to be defined, and I think Open Policy is a great candidate for the standard here

Synchronous meaning - Call this webservice, wait for a 'yes/no' response
Asynchronous meaning - call out to a webservice posting the Open OPA payload - at some point a callback will come back with a yes/no response .

The current 'manual approval' flow a asynchronous flow - i.e. we are waiting for a human to come in and hit an approve button to unblock the processing chain.... Potentially the manual review/approval flow could also conform to the OPA standard?

[grovesy]

Just to echo @JamieSlome, It's key that git-proxy works out of the box and has a really simple "out of box" deployment model (i.e. minimal services/deployments)

The flip side is to maximize compatibility with any open, adopted standard - so I do like the embedded idea of OPA controls

[coopernetes]

Sorry for the delay. I do see the need to balance simplicity in git-proxy via built-in features and checks vs embedding another policy framework or engine into the project.

This probably is a good candidate for an extension point in the form of a plugin. Happy to help where I can to prove this out in PoC and/or contribute an implementation. We have a small but growing Rego policy development practice that would be a natural fit as described here but appreciate it may not be suitable to be directly implemented in the proxy.

[JamieSlome]

@coopernetes - awesome! 🎉 Any help is massively appreciated.

I'll be scheduling a catch-up for Git Proxy soon, where we can all jump on a call and bash heads.

Just finishing up the tail end of some clean up work of the library, so we can actually start coding up features post our chat. I'll create an issue with the outstanding work as a precursor to our call.

[coopernetes]

Let's keep this issue to track the specific OPA enhancement. I'll comment on the original extensibility issue (#47) and add details and a design proposal for how we can support a plugin-style ecosystem to extend git-proxy from its core functionality.

[JamieSlome]

Perfect @coopernetes 🙌

[coopernetes]

The current 'manual approval' flow a asynchronous flow - i.e. we are waiting for a human to come in and hit an approve button to unblock the processing chain.... Potentially the manual review/approval flow could also conform to the OPA standard?

@grovesy this is definitely doable. OPA has a few mechanisms for handling external data. It depends on where we think the best place for the "state" of this sort of asynchronous flow should live.

Thinking more, this is another good candidate for increased modularity. By default, we want to avoid tight coupling between git-proxy, its own data sources, OPA & its potential sources and any other system needed to make authorization decisions. OPA and git-proxy can share a sink but that should be configurable too. OPA is very generic in this regard so it presents an opportunity for multiple use cases through it as a standard API. For now, I see it as an option alongside the current config-based approach if people want to further externalize a policy.