Argo-Diff

Application written in go that provides Pull Request comments for changes to Kubernetes manifests when those manifests are delivered via ArgoCD:

Overview

Argo-diff is designed to receive webhook notifications from Github for pull_request and issue_comment events. When events are received, it uses the ArgoCD CLI to identify the ArgoCD application(s) that are configured with the pull request's repository as a source and to generate a diff of those applications against the live state.

Argo-diff will comment on the associated pull request with markdown displaying the diffs for the applications with potential changes.

Argo-diff will not run in the following situations:

• Base branch of the pull request (branch in which the PR will be merged) is not the target revision for the Argo application. (eg: Your Argo application targets production, but your PR is to be merged into dev.)

Why argo-diff?

There are multiple ways to produce diff previews of proposed manifest changes to ArgoCD applications in Github Pull Requests, but they often require per-repository configuration. Because argo-diff is triggered by Github event notifications, and because argo-diff auto-discovers ArgoCD applications configured in the source code repository, no configuration is needed when repositories and/or ArgoCD applications are added and removed.

Argo-diff natively supports ArgoCD's manifest-generate-paths annotation. This annotation is useful for monorepos that contain the manifests for many ArgoCD applications. For applicatations with it set, ArgoCD will only attempt to produce diffs for the applications whose manifest-generate-paths match the pull request's files changed (which is fetched via the Github API).

Argo-diff supports multi-source ArgoCD applications, including helm applications whose Application specs are managed by ArgoCD. This means if, in source control, you have a pull request that updates the chart version in the Application spec of a helm application, argo-diff will produce a diff of affected application.

Argo-diff can also be quickly deployed via Github actions.

Screenshots

Below are screenshots of Github Pull Request comments generated by argo-diff.

Simple comment of healthy application

Long ling in diff truncated

Application out of sync

Helm application (argo-diff via Github Actions)

Deploying as a Webhook receiver

• Access to the Github API will need to be granted either via a Personal Access Token (good for small environments or a proof of concept) or a Github App.

  • If you using a Personal Access Token, it is recommended to create a separate Github account in your organization specifically for argo-diff. Argo diff will look for the PAT in the env var GITHUB_TOKEN or GITHUB_PERSONAL_ACCESS_TOKEN.
    • Permissions for the PAT should match the permissions specified below for Github Apps.
  • If you using a Github App, you'll need to create a new application in your organization under Developer Settings:
    • Name can be argo-diff.
    • Homepage URL can be the base URL of your argocd instance or the base URL of what will be your webhook URL.
    • Callback URL should be empty, as should Setup URL.
    • Webhooks should not be active.
    • The webhook URL can be the URL you will configure Github webhooks below.
    • Permissions:
      • Administration: Read-only
      • Commit statuses: Read and write
      • Metadata: Read-only
      • Pull requests: Read and write
    • Where can this GitHub App be installed? -> Only on this account
    • Upon creating the Github App, take note of the App ID and then generate a new Private Key, which will download a .pem file locally.
    • Install App to your organization. Settings for the installion will have the URL formatted like: /organizations/<ORG_NAME>/settings/installations/<INSTALLATION_ID>. Take note of the installation id on the URL.
    • Argo Diff will look for the API client configuration in the following environment variables: GITHUB_APP_ID, GITHUB_APP_INSTALLATION_ID, and GITHUB_APP_PRIVATE_KEY (the latter of which should be contents of the .pem file downloaded above)

• Create a user in your ArgoCD instance. This user should have read-only access to all applications

  • If you installed ArgoCD from the helm chart, you can add an argo-diff user via the chart's values:

    configs:
      cm:
        accounts.argodiff: apiKey
        accounts.argodiff.enabled: "true"

  • In policy.csv: g, argodiff, role:ci and p, role:ci, applications, get, *, allow NOTE: This may not be needed if users in your ArgoCD installation default to the role:readonly role
  • This user shouldn't need a password but does need an API token to be generated. This can be done by an admin user via the UI or the CLI. The generated API token should the value of ARGOCD_AUTH_TOKEN

• Generate a webhook secret that will be shared both by the argo-diff deployment and Github webhook config.

• Deploy to Kubernetes:

  • The recommended approach is to use the Helm chart included in charts/argo-diff/:

    $ helm install my-release oci://ghcr.io/vince-riv/chart/argo-diff

  • You can also use the example manifests in the docs/k8s/ directory to deploy argo-diff to the argocd namespace of your Kubernetes cluster. An Ingress or IngressRoute will need to be added to allow webhooks in from Github to the /webhook endpoint on the argo-diff Service.

• Configure organizational (or repository level) webhook notifications to argo-diff. The Payload URL should map the ingress configured in your cluster, and the secret should be the webhook secret previously generated. Individual event types to configure:

  • Issue comments (allows for a comment of argo diff to re-trigger argo-diff on the pull request)
  • Pull requests

  Note: Argo-diff only supports pull request events. Push events to branches are ignored.

• After the webhook is activated, the ping event should be received and verified by argo-diff and this will validate connectivity from Github to argo-diff

Github Actions

Support for Github Actions has been added. Running via Github Actions requires that your ArgoCD instance be available to Github actions runners: ArgoCD is publicly available (and you understand the risks associated with this) or you are using self-hosted runners within your environment.

The benefit of deploying via Github Actions is that neither a Github account nor a Github application need to be provisioned. Just create an ArgoCD user for argo-diff:

• Create a user in your ArgoCD instance. This user should have read-only access to all applications
  • If you installed ArgoCD from the helm chart, you can add an argo-diff user via the chart's values:

    configs:
      cm:
        accounts.argodiff: apiKey
        accounts.argodiff.enabled: "true"

  • In policy.csv: g, argodiff, role:ci and p, role:ci, applications, get, *, allow NOTE: This may not be needed if users in your ArgoCD installation default to the role:readonly role
  • This user shouldn't need a password but does need an API token to be generated. This can be done by an admin user via the UI or the CLI.
  • Save auth token to repository's Actions secrets - this must be passed to argocd_auth_token input

The downside is that Github workflows are typically managed on a repo-by-repo basis. Performance is a litle slower than the deployed version.

Here's an example workflow snippet assuming a self-hosted running and that an ArgoCD auth token has been added to the Github repository's secrets:

---

on:
  pull_request:
    branches: [main]

permissions:
  pull-requests: write

jobs:
  argo-diff:
    runs-on: [self-hosted]  # remove when running on Github's public runners and ArgoCD is publicly available
    name: Argo Diff
    steps:
      - name: ArgoCD Diff
        id: argo-diff
        uses: vince-riv/argo-diff@actions-v1
        with:
          argocd_auth_token: ${{ secrets.ARGOCD_AUTH_TOKEN }}
          argocd_server: argocd-server.argocd.svc.cluster.local:80  # argocd.domain.example when publicly available via https
          argocd_server_insecure: true                              # typically false when publicly available via https
          argocd_server_plaintext: true                             # typically false when publicly available via https
          argocd_ui_base_url: https://argocd.domain.example
          argocd_grpc_web: true
          github_token: ${{ secrets.GITHUB_TOKEN }}
          log_level: trace
          repo_default_ref: main

Configuration

When deployed as a web service, argo-diff currently accepts all configuration options via environment variables. When used via Github Actions, relevant configuration options are accepted via inputs. The table following table describes accepted environment variables and their respective Github actions inputs.

Important: Valid Github API credentials are required to run. So either the GITHUB_APP_* variables must be set, or GITHUB_TOKEN or GITHUB_PERSONAL_ACCESS_TOKEN must be set.

Environment Variable	Input Name	Required	Default	Description
APP_ENV	N/A	no		Set to 'dev' during local development.
ARGOCD_AUTH_TOKEN	argocd_auth_token	yes		Bearer token for ArgoCD (value passed to --auth-token)
ARGOCD_CLI_CMD_NAME	N/A	no	argocd	When set, specifies the argocd cli command name. (Can be used to override the version of argocd argo-diff but requires the user to make it available
ARGOCD_GRPC_WEB	argocd_grpc_web	no	false	Set --grpc-web flag for argocd cli (true/false)
ARGOCD_GRPC_WEB_ROOT_PATH	argocd_grpc_web_root_path	no		Value for --grpc-web-root-path for argocd cli
ARGOCD_SERVER_ADDR	argocd_server	yes		ArgoCD server name (value passed to --server)
ARGOCD_SERVER_INSECURE	argocd_server_insecure	no	false	Set --insecure flag for argocd cli (true/false)
ARGOCD_SERVER_PLAINTEXT	argocd_server_plaintext	no	false	Set --plaintext flag for argocd cli (true/false)
ARGOCD_UI_BASE_URL	argocd_ui_base_url	no		Base URL of ArgoCD UI (usually server name prefixed with https://)
ARGO_DIFF_COMMENT_PREAMBLE	comment_preamble	no		String/markdown prefixed to comments. Keep to 150 chars or less in length
ARGO_DIFF_CONTEXT_STR	context_str	no		Unique identifier of argo-diff instance. Use when deploying multiple instances (eg: one per cluster). Recommended to be a brief cluster nickname
COMMENT_LINE_MAX_CHARS	comment_line_max_chars	no	175	Any individual line in Pull Request comments by argo-diff longer than this are truncated.
GITHUB_APP_ID	N/A	no		Github Application Id (see deployment instructions)
GITHUB_APP_INSTALLATION_ID	N/A	no		Github Application Installation Id (see deployment instructions)
GITHUB_APP_PRIVATE_KEY	N/A	no		Github Application Private Key (see deployment instructions)
GITHUB_PERSONAL_ACCESS_TOKEN	N/A	no		Bearer token for github API calls; same as GITHUB_TOKEN
GITHUB_TOKEN	github_token	yes for GHA		Bearer token for github API calls (in Github Actions workflow, usually secrets.GITHUB_TOKEN); Required in Github Actions
GITHUB_WEBHOOK_SECRET	N/A	yes for deployed		Shared secret for Github webhook validation; Required when deployed
LOG_LEVEL	log_level	no	info	Log level of argo-diff
REPO_DEFAULT_REF	repo_default_ref	no		Default branch of repository (eg: "main"). NOTE: only needed in Github Actions when HEAD is specified as target revision in ArgoCD application source

NOTE: When running via Github Actions, argo-diff will use the following environment variables set by Github: GITHUB_ACTIONS, GITHUB_BASE_REF, GITHUB_EVENT_NAME, GITHUB_HEAD_REF, GITHUB_REF, and GITHUB_REPOSITORY. Do not set these yourself.

Running locally

Set environment variables used by argo-diff and then execute go run cmd/main.go.

For example, you can place these in a file called .env.sh:

GITHUB_PERSONAL_ACCESS_TOKEN='github_pat_XXXX'
ARGOCD_AUTH_TOKEN='ABCDEFGHIJKLMNOPQRSTUVWXYZ'
ARGOCD_SERVER_ADDR='argocd.your.domain:443'
ARGOCD_UI_BASE_URL='https://argocd.your.domain'
APP_ENV='dev'

Then source it and execute go run to start the server:

$ set -o allexport ; . .env.sh ; set +o allexport
$ go run cmd/main.go

Or execute to go run and process a manually crafted event:

$ set -o allexport ; . .env.sh ; set +o allexport
# go run cmd/main.go -f path/to/event_info.json

Craft a local file with event data

When running locally, the easiest way to trigger argo-diff is to create a json file that can be unmarshalled into an EventInfo struct (defined in internal/webhook/process.go):

{
    "ignore": false,
    "owner": "GITHUB_ORG_NAME",
    "repo": "REPOSITORY_NAME",
    "default_ref": "main",
    "commit_sha": "LONG_SHA_OF_COMMIT",
    "pr": 123,
    "change_ref": "BRANCH_NAME_OF_PR",
    "base_ref": "main"
}

Description of those fields:

• ignore: tells argo-diff to ignore the event
• owner: Github organization name
• repo: Github repository name
• default_ref: the default branch of the repository
• commit_sha: the sha that triggered the event - should be HEAD of the branch of the PR if you want it to comment
• pr: pull request number (duh)
• change_ref: the source branch of the PR (the feature branch)
• base_ref: the branch to which the PR is getting merged

This json file can be passed to argo-diff via the -f argument or posted to the /dev http enpoint.

Testing Http Endpoints Locally

To simulate Github webhook requests, you can copy webhook request headers and payloads to temp/curl-headers.txt and temp/curl-payload.json and use the post-local.sh script to ship them to the local server.

There is also the /dev endpoint that gets enabled when APP_ENV=dev - this endpoint is handled by devHandler() in main.go. This bypasses webhook processing is triggers argo-diff for specific event.

When in APP_ENV is dev, argo-diff will comment but not set status checks.

Development Notes

This was originally developed by @vrivellino as a way to learn Go. Its functionality replicates that of an internal tool written by smart people at a previous job.