BigBang Template

This is a mirror of a government repo hosted on Repo1 by DoD Platform One. Please direct all code changes, issues and comments to https://repo1.dso.mil/platform-one/big-bang/customers/template**

[[TOC]]

This folder contains a template that you can replicate in your own Git repo to get started with Big Bang configuration. If you are new to Big Bang it is recommended you start with the Big Bang Quickstart before attempting customization.

The main benefits of this template include:

Isolation of the Big Bang product and your custom configuration
- Allows you to easily consume upstream Big Bang changes since you never change the product
- Big Bang product tags are explicitly referenced in your configuration, giving you control over upgrades
GitOps for your deployments configurations
- Single source of truth for the configurations deployed
- Historical tracking of changes made
- Allows tighter control of what is deployed to production (via merge requests)
- Enables use of CI/CD pipelines to test prior to deployment
- Avoids problem of helm upgrade using values.yaml that are not controlled
- Allows you to limit access to production Kubernetes cluster since all changes are made via Git
Shared configurations across deployments
- Common settings across deployment environments within an impact level can be configured in one place.
- Secrets (e.g. image pull credentials) can be shared across deployments.
  
  NOTE: SOPS supports multiple keys for encrypting the same secret so that each environment can use a different SOPS key but share a secret.

Prerequisites

To deploy Big Bang, the following items are required:

Kubernetes cluster ready for Big Bang
- [Optional] Code to setup a production-ready infrastructure on AWS with an RKE2 Kubernetes cluster is provided in the terraform directory.
A git repo for infrastructure and configuration as code
Kubectl
GPG (Mac users need to read this important note)
SOPS
Git
Iron Bank Personal Access Token - Under your User Profile, copy the CLI secret.
Repo1 Personal Access Token - You will need read_repository permissions.
Helm

In addition, the following items are recommended to assist with troubleshooting:

K9S

Setup

This template supports two distinct workflows for staying up-to-date with Big Bang:

package-strategy represents an environment that prioritizes the fastest path to receiving package updates as they are released from Big Bang development.
- This update strategy keeps updates to the environment and Big Bang deployment small (i.e. a single package update at a time)
- Allows for critical security patches to reach the environment as quickly as possible
- This can be coupled with renovate (See the renovate section below) to automate the delivery of updates for an environment for each package as updates released.
- As a result of updating packages - Updates to Big Bang patch and/or minor releases may not end up modifying the environment.
umbrella-strategy represents an environment that receives updates to all Big Bang packages exclusively from new releases of Big Bang.
- This update strategy allows Big Bang development to test the integration of all packages in an end-to-end testing workflow to ensure backwards compatibility and/or any required integration has occurred.
- It can deliver updates to many applications simultaneously with built-in dependency management for which updates occur first.
- It ensures that any integration that needs to be managed at the Big Bang umbrella layer has occurred before package deployment.

Each strategy consists of a Kubernetes manifest containing Flux resources (bigbang.yaml), a Kustomization file (kustomization.yaml), values to pass to Big Bang (configmap.yaml), secrets (secrets.enc.yaml), and additional files used to deploy resources.

We recommend that you adopt a multi environment workflow based off of one of these strategies. If following this path each environment can share a base folder to allow reusability of values between environments.

Git Repository

Create Git Repository

We need to work off our own Git repo for storing configuration. So, you should fork this repo into a private Git repo owned by yourself or your project. Then, clone your repo locally.

git clone https://<your domain>/<your repo>.git
cd <your repo>

# Create branch for your changes
git checkout -b template-demo

# Copy one of the strategy directories into named-environment directories
cp -r package-strategy/ dev
# update path - switch to gsed on MacOS
sed -i 's/package-strategy/dev/g' dev/bigbang.yaml

cp -r umbrella-strategy/ prod
# update path - switch to gsed on MacOS
sed -i 's/umbrella-strategy/prod/g' prod/bigbang.yaml

It is recommended that you create your own branch so that you can pull the original repository's main branch as a mirror to keep it in sync.

Git Repository Best Practices

If you plan to deploy clusters to multiple impact levels, then you should follow these best practices:
- Have at least one unique git repository for each impact level where a cluster will be deployed to.
- Match the IL where the git repository will be hosted to the impact level where the corresponding cluster will be hosted.
  - Example: The git repository corresponding to an IL4 Cluster Deployment should be hosted on an IL4 approved hosting environment. The Infrastructure as code for an IL4 Cluster Deployment shouldn't be hosted on an IL2 git repository.
  - The reason for this is that:
    - Infrastructure as code secrets corresponding to a cluster deployed to IL4 (or IL5), would be considered CUI (Controlled Unclassified Information), and thus should be isolated from IL2.
    - Even though both IL2 and IL4 secrets would be encrypted, it's still best practice to keep them in separate repositories, because when server side git hooks are not in place, then it's possible for human error to result in a non-encrypted secrets being committed to a git repository.
    - Any encrypted infrastructure as code secrets added to /base, would get shared by all clusters, if multiple impact levels existed in the same repo, then secrets added to /base could accidentally end up being shared across impact levels.
The intent of the /base folder is to allow configuration and in some cases secrets to be shared between clusters.
- Examples of how to properly use /base to share a secret between clusters in the same impact level.
  - If you have several developer sandbox, development, and test clusters operating at IL2, you may wish to have some level of shared configuration between them.
  - If you have a production and acceptance environment for an IL4 cluster, you may wish for them to share an image pull secret between them.
It's worth mentioning that while the above best practices are applicable to the majority of scenarios, Authorizing Officials(AOs) can always choose to make exceptions and accept risks if necessary to meet the needs of unique mission requirements, and the presence of a Cross Domain Solution could also result in nuances.

Create GPG Encryption Key

To make sure your pull secrets are not compromised when uploaded to Git, you must generate your own encryption key:

# Figure out what version of gpg your running
gpg --version | head -n 1
# Centos 7 and older AmazonLinux tend to run gpg 2.0.x
# Centos 8, Ubuntu 20, and Debian 11 tend run gpg 2.2.x
# Centos 9 and Mac tend to run gpg 2.3.x

# Generate a GPG master key
# Note:
# By default, gpg 2.3.x generated keys, will generate errors when
# consumed by clients running gpg 2.0.x - 2.2.x
# gpg 2.3.x users must use the following multiline keygen command for
# compatibility with sops and older clients using gpg 2.0.x - 2.2.x
# gpg 2.2.x users should also user the following key gen multiline command
# gpg 2.0.x users substitute --full-generate-key for --gen-key

# Use the following command to generate a compatible key.
# Compatibility issues may arise with GPG versions older than 2.4.3 unless explicitly specified above.

gpg --batch --full-generate-key --rfc4880 --digest-algo sha512 --cert-digest-algo sha512 <<EOF
    %no-protection
    # %no-protection: means the private key won't be password protected
    # (no password is a fluxcd requirement, it might also be true for argo & sops)
    Key-Type: RSA
    Key-Length: 4096
    Subkey-Type: RSA
    Subkey-Length: 4096
    Expire-Date: 0
    Name-Real: bigbang-dev-environment
    Name-Comment: bigbang-dev-environment
EOF

# The following command will store the GPG Key's Fingerprint in the $fp variable
# (The following command has been verified to work consistently between multiple versions of gpg: 2.0.x, 2.2.x, 2.3.x)
export fp=$(gpg --list-keys --fingerprint | grep "bigbang-dev-environment" -B 1 | grep -v "bigbang-dev-environment" | tr -d ' ' | tr -d 'Keyfingerprint=')
echo $fp

# The above command will make a key that doesn't expire
# You can optionally run the following if you need the key to expire after 1 year.
gpg --quick-set-expire ${fp} 1y

# cd to the location of the .sops.yaml, then run the following to set the encryption key
# sed: stream editor is like a cli version of find and replace
# This ensures your secrets are only decryptable by your key

## On linux
sed -i "s/pgp: FALSE_KEY_HERE/pgp: ${fp}/" .sops.yaml

## On MacOS
sed -i "" "s/pgp: FALSE_KEY_HERE/pgp: ${fp}/" .sops.yaml

# Save encrypted secrets into Git
# Configuration changes must be stored in Git to take affect
git add .sops.yaml
git commit -m "chore: update default encryption key"
git push --set-upstream origin template-demo

Sops Tip and Notes

File Editing Tip:
- Linux users: if you install Visual Studio Code or Sublime Text, the install process tends to add the binary to the $PATH, so linux users can easily edit files using vi file_to_edit, subl file_to_edit, or code file_to_edit
- Mac users: you can do the following to add the binary to your systems PATH variable to get the same functionality (zsh users adjust appropriately)
```
sudo ln -s "/Applications/Sublime Text.app/Contents/SharedSupport/bin/subl" /usr/local/bin/subl
echo 'export PATH="$PATH:/Applications/Visual Studio Code.app/Contents/Resources/app/bin"' >> ~/.bashrc
tail ~/.bashrc
source ~/.bashrc
```

Sops Tip: You can edit a sops encrypted file with your preferred text editor.

# Note the following commands require .sops.yaml to have a valid gpg fingerprint configured
sops ~/file_to_edit #Will edit the file using the default editor, usually vi
EDITOR="subl --wait" sops ~/file_to_edit # Will edit the file using Sublime Text
EDITOR="code --wait" sops ~/file_to_edit # Will edit the file using Visual Studio Code
# You can also permanently add the EDITOR variable to your profile if desired:
echo 'export EDITOR="subl --wait"' >> ~/.bashrc

Sops Note: Sops isn't guaranteed to preserve the yaml formatting of a file it encrypts
Sops Note: sops is an abstraction layer that allows several secret backends to be used.
- AGE and GPG (also known as PGP) are generic cloud agnostic options that have no dependencies. It's important to note that while the sops repo recommends AGE over PGP, For DoD use cases it's preferable to use GPG, because GPG offers NIST approved crypto algorithms. (AGE is just as secure as GPG; however, AGE's crypto algorithms, X25519 and ChaCha20-Poly1305 are not NIST approved algorithms (X25519 is used to generate asymmetric key pairs and ChaCha20-Poly1305 is a symmetric encryption algorithm. The example above has GPG leverage RSA 4096 and AES 256 which are both NIST approved)
- GPG backed sops is shown as an example above, because it's generic and cloud agnostic.
- If possible Cloud Service Provider based Key Management Service based solutions like AWS KMS, GCP KMS, and Azure Key Vault should be preferred over GPG, for the following reasons:
  - CSP KMS solutions are FIPS 140-2 approved
  - KMS solutions leverage identity based decryption, since the service never exposes the key and does encryption / decryption operations on behalf of users, the decryption key can't leak.
  - Identity based decryption means you can protect access to decryption rights using RBAC, and you can revoke access to the ability to decrypt data.
  - CSP KMS solutions have easy to satisfy dependencies
- Hashicorp Vault is also a good option; however, it's worth pointing out that:
  - Using Vault means introducing Vault as a dependency, and standing up a production grade instance of Vault requires a good amount of work.
  - If a FIPS 140-2 compliant instance of Vault is needed, then Vault would need to be backed by an HSM, and that requires a Vault Enterprise License.
- If you want to leverage a KMS based solution, you'll need to update .sops.yaml based on the following links:

Add TLS Certificates

The base/configmap.yaml is setup to use the domain dev.bigbang.mil by default. (Which results in sites that look like this: https://*.dev.bigbang.mil) A demo TLS wildcard certificate is provided in base/bigbang-dev-cert.yaml to use.

Important Security Note

Since the private key of the demo cert is in a public repo all traffic sent to a demo site should be treated as compromised and can be decrypted by MITM packet sniffers, so it's necessary to do one of the following:

Use the demo cert for demos over private IP space / trusted networks. (VPNs and tools like shuttle can make this easier, (shuttle can allow Linux and Mac users to treat an ssh bastion like a VPN), Note: Certain firewall and endpoint protection software can block the functionality of sshuttle.)
If using Cloud Infrastructure with Public IPs to test, ensure no Cloud IAM rights are attached to the instances. (Otherwise it'd be possible for a MITM attacker to listen in on traffic to a service like ArgoCD, and sniff ArgoCD admin creds, which could have kubectl admin creds, and if the kubernetes cluster has Cloud IAM rights, that could result in lateral movement deeper into a cloud account/potential compromise of resources beyond a demo cluster.)
Another alternative is to not use the demo HTTPS cert and provide your own.

cd base

# Encrypt the existing certificate
sops -e bigbang-dev-cert.yaml > secrets.enc.yaml

# Save encrypted TLS certificate into Git
git add secrets.enc.yaml
git commit -m "chore: add dev.bigbang.mil tls certificates"
git push

Add Pull Credentials

You will need pull credentials for Iron Bank to retrieve images for Big Bang.

Secrets can be specific to an environment if they are located in that environment's folder (e.g. prod, dev). Or, they can be shared between environments if located in the base directory.

# Edit the same secret holding your TLS certificates to add the pull credentials
sops secrets.enc.yaml

Add the following contents to the newly created sops secret. Put your Iron Bank user/PAT where it states replace-with-your-iron-bank-user and replace-with-your-iron-bank-personal-access-token.

The name of the secret must be common-bb if the secret is in the base folder or environment-bb if the secret is in the dev or prod folder. The environment-bb values take precedence over the common-bb values.

apiVersion: v1
kind: Secret
metadata:
  name: common-bb
stringData:
  values.yaml: |-
    registryCredentials:
    - registry: registry1.dso.mil
      username: replace-with-your-iron-bank-user
      password: replace-with-your-iron-bank-personal-access-token
    istio:
      gateways:
# ...
# Clarifying Note: The 'Add TLS Certificates', section above had you
# create this file from a pre-existing file with an embedded demo HTTPS
# cert. So when you go to edit this file you should see a pre-existing
# section with istio.gateways.public.tls.key, etc.
# This note is to clarify that you should manually edit the pre-existing
# file to append the registryCredentials section as shown.

When you save the file, it will automatically re-encrypt your secret using SOPS.

# Save pull credentials into Git
git add secrets.enc.yaml
git commit -m "chore: added iron bank pull credentials"
git push

Your private key to decrypt these secrets is stored in your GPG key ring. You must NEVER export this key and commit it to your Git repository since this would compromise your secrets.

Configure for GitOps

We need to reference your git repository so that Big Bang will use the configuration. Add your repository into the GitRepository resource in dev/bigbang.yaml:

cd ../dev/

Replace your forked Git repo where it states replace-with-your-git-repo. Replace replace-with-your-branch with your branch name (e.g. template-demo as created above).

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
   name: environment-repo
   namespace: bigbang
spec:
   interval: 1m
   url: https://replace-with-your-git-repo.git
   ref:
     branch: replace-with-your-branch
   secretRef:
     name: private-git

The Kustomization resource contains the path in your repo to the kustomization.yaml to start with. If your folder changes, makes sure to update spec.path with the new path.

Now, save and commit your change:

git add bigbang.yaml
git commit -m "chore: updated git repo"
git push

Configure All GitRepositories with Credentials and CA Certificate

If you are in an airgap or high side environment and will need to be re-hosting BigBang packages and are not able to use SSH to have Fluxv2 pull the repos, you can still use HTTPS with a "self-signed" DoD certificate via configuring the following:

Create a sops encrypted secret template within prod/repo-ca-credentials.yaml.enc :

apiVersion: v1
kind: Secret
metadata:
  name: repo-ca-credentials
  namespace: bigbang
stringData:
  username: USERNAME
  password: PASS
  caCert: |
    -------PEM ENCODED CERT--------

Encrypt the secret with sops: sops -e -i prod/repo-ca-credentials.yaml.enc .

Ensure the above file is added to prod/kustomization.yaml under the resources section so it gets created in the bigbang namespace:

...
resources:
  - repo-ca-credentials.yaml.enc

Add the following patches to base/kustomization.yaml to add the following secretRef configuration for every GitRepository you will have enabled, eg Jaeger:

patchesStrategicMerge:
...
- |-
  apiVersion: helm.toolkit.fluxcd.io/v2beta1
  kind: HelmRelease
  metadata:
    name: bigbang
    namespace: bigbang
  spec:
    postRenderers:
      - kustomize:
          patchesStrategicMerge:
          - apiVersion: source.toolkit.fluxcd.io/v1
            kind: GitRepository
            metadata:
              name: jaeger
              namespace: bigbang
            spec:
              secretRef:
                name: https-ca-credentials

This will deploy the BigBang HelmRelease and patch all above GitRepositories with the https-ca-credentials secret.

# Save pull credentials into Git
git add prod/repo-ca-credentials.yaml.enc prod/kustomization.yaml base/kustomization.yaml
git commit -m "chore: added gitrepo creds secret and overlays"
git push

Configure deployment to use OCI Signed BigBang and Verify Chart using Cosign

The BigBang Chart is available to be deployed from either Git (GitRepository) or OCI Registry (HelmRepository). The BigbBang Helm Chart is currently signed using Cosign. If using the HelmRepository then we have the ability to validated the chart using the Public Key. To do this, we have to perform the following steps:

Enable additional templates to be installed in the /base/ using base/kustomization.yaml.

Update base/kustomization.yaml to enable/add helmrepo.yaml and cosignsecret.yaml under the bases: section:

# When updating the version of BigBang, make sure to update
#   both the bases reference and the GitRepository reference
bases:
- https://repo1.dso.mil/platform-one/big-bang/bigbang.git//base?ref=2.22.0
- helmrepo.yaml
- cosignsecret.yaml

Enable override to the HelmRelease to use the HelmRepository by updating the base/kustomization.yaml:

patchesStrategicMerge:
#Enable this section if you wish to use the OCI HelmRepo with Cosign Verfiy instead of the GitRepository.
- |-
  apiVersion: helm.toolkit.fluxcd.io/v2beta2
  kind: HelmRelease
  metadata:
    name: bigbang
  spec:
    interval: 1m
    chart:
      spec:
        chart: bigbang
        version: 2.22.0
        sourceRef:
          kind: HelmRepository
          name: registry1
        verify:
          provider: cosign
          secretRef:
            name: bigbang-cosign-pub

During the deploy step, create a Docker secret in the bigbang namespace allowing Flux to pull the OCI chart. This command is defined below in Step 1 of the Deploy section.
Once pointed to the HelmRepository and it successfully pulling, each package within BigBang can then be pointed to their helmRepo sourceType value in BigBang eg:

istio:
  sourceType: "helmRepo"

Deploy

Big Bang follows a GitOps approach to deployment. All configuration changes will be pulled and reconciled with what is stored in the Git repository. The only exception to this is the initial manifests (e.g. bigbang.yaml) which points to the Git repository and path to start from.

Deploy SOPS private key for Big Bang to decrypt secrets

# The private key is not stored in Git (and should NEVER be stored there).  We deploy it manually by exporting the key into a secret.
kubectl create namespace bigbang
gpg --export-secret-key --armor ${fp} | kubectl create secret generic sops-gpg -n bigbang --from-file=bigbangkey.asc=/dev/stdin

Create imagePullSecrets for Flux

# Image pull secrets for Iron Bank are required to install flux.  After that, it uses the pull credentials we installed above
kubectl create namespace flux-system

# Adding a space before this command keeps our PAT out of our history
kubectl create secret docker-registry private-registry --docker-server=registry1.dso.mil --docker-username=<Your IronBank Username> --docker-password=<Your IronBank Personal Access Token> -n flux-system

Create Git credentials for Flux

# Flux needs the Git credentials to access your Git repository holding your environment
# Adding a space before this command keeps our PAT out of our history
 kubectl create secret generic private-git --from-literal=username=<Your Repo1 Username> --from-literal=password=<Your Repo1 Personal Access Token> -n bigbang

Optionally add Secret for OCI HelmRepository Deployment method If you are using HelmRepository to validate the OCI Signed BigBang Chart instead of the GitRepository, create the private-registry secret to pull from the Helm Chart from the OCI repository defined in /base/helmrepo.yaml.

# The private key is not stored in Git (and should NEVER be stored there).  We deploy it manually by exporting the key into a secret.
kubectl create secret docker-registry private-registry -n bigbang --docker-server=registry1.dso.mil --docker-username=<Your IronBank Username>--docker-password=<Your IronBank Personal Access Token>

Deploy Flux to handle syncing

# Flux is used to sync Git with the the cluster configuration
# If you are using a different version of Big Bang, make sure to update the `?ref=2.22.0` to the correct tag or branch.
kubectl apply -k https://repo1.dso.mil/platform-one/big-bang/bigbang.git//base/flux?ref=2.22.0

# Wait for flux to complete
kubectl get deploy -o name -n flux-system | xargs -n1 -t kubectl rollout status -n flux-system

Deploy Big Bang

kubectl apply -f bigbang.yaml

# Verify 'bigbang' namespace is created
kubectl get namespaces

# Verify Pull from Git was successful
kubectl get gitrepositories -A

# Verify Kustomization was successful
# NOTE: The Kustomization resource may fail at first with an error about the istio-system namespace.  This is normal since the Helm Release for istio will create that namespace and it has not run yet.  This should resolve itself within a few minutes
kubectl get -n bigbang kustomizations

# Verify secrets and configmaps are deployed
# At a minimum, you will have the following:
#  secrets: sops-gpg, private-git, common-bb, and environment-bb
#  configmaps: common, environment
kubectl get -n bigbang secrets,configmaps

# Watch deployment
watch kubectl get hr,po -A

# Test deployment by opening a browser to "kiali.dev.bigbang.mil" to get to the Kiali application deployed by Istio.
# Note that the owner of "dev.bigbang.mil" has setup the domain to point to 127.0.0.1 for this type of testing.
# If you are deployed on a remote host you will need to point "kiali.dev.bigbang.mil" to your cluster master node via your /etc/hosts file

If you cannot get to the main page of Kiali, it may be due to an expired certificate. Check the expiration of the certificate in base/configmap.yaml.

For troubleshooting deployment problems, refer to the Big Bang documentation.

You now have successfully deployed Big Bang. Your next step is to customize the configuration.

Customize

Enable a package

In dev/configmap.yaml, enable Twistlock
```
twistlock:
  enabled: true
```

Push changes to Git

git add configmap.yaml
git commit -m "feat: enable twistlock"
git push

Big Bang will automatically pick up your change and make the necessary changes.

# Watch deployment for twistlock to be deployed
watch kubectl get hr,po -A

# Test deployment by opening a browser to "twistlock.dev.bigbang.mil" to get to the Twistlock application

Update the Big Bang Version

To minimize the risk of an unexpected deployment of a BigBang release, the BigBang release version is explicitly stored in the kustomization.yaml files and can be updated for a planned upgrades. The default release is stored in base/kustomization.yaml, but can be overridden in a specific environment like dev/kustomization.yaml.

Reference for the Big Bang kustomize base:

bases:
- https://repo1.dso.mil/platform-one/big-bang/bigbang.git/base/?ref=2.22.0

To update dev/kustomization.yaml, you would create a mergePatch like the following:

patchesStrategicMerge:
- |-
  apiVersion: source.toolkit.fluxcd.io/v1
  kind: GitRepository
  metadata:
    name: bigbang
  spec:
    interval: 1m
    ref:
      $patch: replace
      tag: "2.22.0"

This does not update the kustomize base, but it is unusual for that to change.

Then, commit your change:

   git add kustomization.yaml
   git commit -m "feat(dev): update bigbang to 2.22.0"
   git push

It may take Big Bang up to 10 minutes to recognize your changes and start to deploy them. This is based on the interval set for polling. You can force Big Bang to recheck by running the sync.sh script.

It is recommended that you track Big Bang releases using the version. However, you can use branch in place of tag if needed. The kustomize base uses Go-Getter's syntax for the reference. The helm release (GitRepository) resource uses the GitRepository CRD's syntax.

When you are done testing, you can update the reference in base (and delete this setting in dev) to update Big Bang in all environments.

Do not forget to also update the base/kustomization.yaml's base: reference to point to the new release.

Update the domain

Big Bang deploys applications to *.dev.bigbang.mil by default. You can override the dev.bigbang.mil domain to your domain by updating base/configmap.yaml and adding the following:

domain: insert-your-domain-here

In addition, you will need to update the TLS certificates by updating base/secrets.enc.yaml.

# Open and edit the encrypted file
sops base/secrets.enc.yaml

After saving the secrets.enc.yaml file, it will be automatically re-encrypted.

# Push changes to Git
git add base/configmap.yaml base/secrets.enc.yaml
git commit -m "chore: updated domain and tls certificates"
git push

If you have different certificates for dev and prod, you can also put the values in dev/secrets.enc.yaml or prod/secrets.enc.yaml respectively. The name of the secret must be common-bb if the secret is in the base folder or environment-bb if the secret is in the dev or prod folder. The environment-bb values take precedence over the common-bb values. Make sure to add the file to kustomization.yaml as a resource if it is not already.

Use NodePorts for Istio Ingress Gateway

Istio Ingress Gateway is deployed with the LoadBalancer service type by default. You can configure Istio to use nodePorts instead by updating dev/configmap.yaml:

istio:
  ingressGateways:
    public-ingressgateway:
      type: "NodePort"
      nodePortBase: 30000

Node ports are assigned starting from nodePortBase. The nodePortBase specifies the start of a range of 4 unused node ports. Node port will be assigned as follows: Port 15021 (Status) = nodePortBase, Port 80 = nodePortBase+1, Port 443 = nodePortBase+2, Port 15443 (SNI) = nodePortBase+3. Node port base should be in the range from 30000 to 32764. Alternatively, the kubernetesResourceSpec can be used to configure all port parameters. If nodePortBase isn't specified ports will be assigned randomly.

Additional Big Bang values

For additional configuration options, refer to the Big Bang and Big Bang Package documentation. Big Bang values can be passed down in the configmap.yaml or secrets.enc.yaml. See the Kubernetes documentation on configmaps and secrets for differences between the two. Secrets should always be SOPS encrypted before committing to Git.

NOTE: The dev template includes several overrides in the configmap.yaml to minimize resource usage and increase polling time in a development environment. They are provided for convenience and are NOT required.

Additional resources

Using Kustomize, you can add additional resources to the deployment if needed. Read the Kustomization documentation for further details.

Key Rotation

If you need to rotate your GPG encryption keys for any reason, you will also need to re-encrypt any encrypted secrets.

Update .sops.yaml configuration file .sops.yaml holds all of the key fingerprints used for SOPS. Update pgp's value to the new key's fingerprint. You can list your locally stored fingerprints using gpg -k.
```
creation_rules:
- encrypted_regex: '^(data|stringData)$'
  pgp: INSERT_NEW_KEY_FINGERPRINT_HERE
```
Re-encrypt secrets.enc.yaml with your new SOPS keys. This will decrypt the file with the old key and re-encrypt with the new key.
```
sops updatekeys base/secrets.enc.yaml -y
# Repeat this for all other encrypted files (e.g. dev/secrets.enc.yaml)
```

Deploy new SOPS private key for Big Bang to decrypt secrets

# The private key is not stored in Git (and should NEVER be stored there).  We deploy it manually by exporting the key into a secret.
kubectl delete secret sops-gpg -n bigbang
gpg --export-secret-key --armor INSERT_NEW_KEY_FINGERPRINT_HERE | kubectl create secret generic sops-gpg -n bigbang --from-file=bigbangkey.asc=/dev/stdin

Commit changes

   git add .sops.yaml **/secrets.enc.yaml
   git commit -m "chore: rekey secrets"
   git push

Multiple Keys

You can encrypt files with SOPS using more than one key to allow different keys to decrypt the same file. The encrypted file contains copies of the data encrypted with each key and all of the public keys needed to re-encrypt the file if changes are made.

Only one of the private keys is required to decrypt the file

Add the second key's fingerprint to .sops.yaml:

creation_rules:
- encrypted_regex: '^(data|stringData)$'
  pgp: ORIGINAL_KEY
      ,INSERT_SECOND_KEY_HERE

Re-encrypt all encrypted files with your new SOPS keys. This will decrypt the file with the original key and re-encrypt with both of the keys.
```
sops updatekeys base/secrets.enc.yaml -y
# Repeat this for all other encrypted files (e.g. dev/secrets.enc.yaml)
```

Commit changes

   git add .sops.yaml **/secrets.enc.yaml
   git commit -m "chore: added second key to secrets"
   git push

Different keys for different environments

In our template, we have a dev and a prod environment with a shared base. Let's say we wanted the following:

Shared Iron Bank pull credential
Different database passwords for dev and prod
Different SOPS keys for dev and prod

Setup .sops.yaml for multiple folders:

creation_rules:
# Base is shared, so add fingerprints of both keys
- path_regex: base/.*
  encrypted_regex: '^(data|stringData)$'
  pgp: INSERT_DEV_KEY_FINGERPRINT_HERE
    ,INSERT_PROD_KEY_FINGERPRINT_HERE
- path_regex: dev/.*
  encrypted_regex: '^(data|stringData)$'
  pgp: INSERT_DEV_KEY_FINGERPRINT_HERE
- path_regex: prod/.*
  encrypted_regex: '^(data|stringData)$'
  pgp: INSERT_PROD_KEY_FINGERPRINT_HERE

Re-encrypt all encrypted files with your SOPS keys. This will decrypt the file with the original private key and re-encrypt with the new keys according to your path_regex.

If you do not have secrets.enc.yaml in dev or prod, you can can copy the one in base to test out these commands.

sops updatekeys base/secrets.enc.yaml -y
sops updatekeys dev/secrets.enc.yaml -y
sops updatekeys prod/secrets.enc.yaml -y

There is an excellent tutorial on multiple key SOPS here.

Commit changes

   git add .sops.yaml **/secrets.enc.yaml
   git commit -m "chore: split dev and prod keys"
   git push

Modifying an encrypted file

Updating values in an encrypted file can be achieved by simply opening the file with sops:

sops base/secrets.enc.yaml

When you save the file, sops automatically re-encrypts it for all of the keys specified in .sops.yaml.

Multi-environment Workflow

In this template, we provide the package-strategy and umbrella-strategy deployment strategies. These can be used in any combination you see fit to manage multiple environments. Optimally updates are introduced to your environments through quality gates - where you would test updates in a dev environment before promoting to a prod environment.

In this instance - you can copy the strategy directory you select for each environment and name them accordingly. We will use dev and prod in this example.

This accomplishes:

Test changes in the dev environment before deploying into prod
Keep dev as close as possible to prod by sharing values
Maintain dev and prod specific settings for resources, external connections, and secrets

To start, we may have the following in each folder:

base
- Iron Bank pull credentials
- Big Bang release reference
- Application settings
dev
- Dev domain name
- Dev TLS certificates
- Minimized resource values (e.g. memory, cpu)
- Dev external connections and credentials
prod
- Prod domain name
- Prod TLS certificates
- Prod external connections and credentials

Big Bang dev value changes can be made by simply modifying dev/configmap.yaml. base and dev create two separate configmaps, named common and environment respectively, with the environment values taking precedence over common values in Big Bang.

The same concept applies to dev secret changes, with two separate secrets named common-bb and environment-bb used for values to Big Bang, with the environment-bb values taking precedence over the common-bb values in Big Bang.

If a new resource must be deployed, for example a TLS cert, you must add a resources: section to the kustomization.yaml to refer to the new file. See the base directory for an example.

Renovate Bot

As documented in Big Bang, optimally the repository that maintains your GitOps state (this template) is being monitored by a tool such as Renovate to provide alerting and automation around updates for Big Bang and packages.

The Renovate configuration in this repository provides an example that will target Repo1 git repositories for both individual packages as well the Big Bang umbrella chart and provide automated merge requests for updates. If following the package-strategy for consuming updates, Renovate will open pull/merge-requests on a per-package basis.

The Renovate Deployment documentation supports deploying Renovate on a Big Bang cluster that can monitor both this template as well as dependencies in other repositories as a generic capability.

Once an environment directory has been created from a given strategy directory, the strategy directories can be omitted from renovate update by adding an ignorePaths option to the renovate.json:

"ignorePaths": ["package-strategy/**", "umbrella-strategy"],

Name		Name	Last commit message	Last commit date
Latest commit History 219 Commits
base		base
package-strategy		package-strategy
terraform		terraform
umbrella-strategy		umbrella-strategy
.gitignore		.gitignore
.sops.yaml		.sops.yaml
CHANGELOG.md		CHANGELOG.md
CODEOWNERS		CODEOWNERS
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
renovate.json		renovate.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

BigBang Template

Prerequisites

Setup

Git Repository

Create Git Repository

Git Repository Best Practices

Create GPG Encryption Key

Sops Tip and Notes

Add TLS Certificates

Important Security Note

Add Pull Credentials

Configure for GitOps

Configure All GitRepositories with Credentials and CA Certificate

Configure deployment to use OCI Signed BigBang and Verify Chart using Cosign

Deploy

Customize

Enable a package

Update the Big Bang Version

Update the domain

Use NodePorts for Istio Ingress Gateway

Additional Big Bang values

Additional resources

Key Rotation

Multiple Keys

Different keys for different environments

Modifying an encrypted file

Multi-environment Workflow

Renovate Bot

About

Releases

Packages

Contributors 14

Languages

License

DoD-Platform-One/template

Folders and files

Latest commit

History

Repository files navigation

BigBang Template

Prerequisites

Setup

Git Repository

Create Git Repository

Git Repository Best Practices

Create GPG Encryption Key

Sops Tip and Notes

Add TLS Certificates

Important Security Note

Add Pull Credentials

Configure for GitOps

Configure All GitRepositories with Credentials and CA Certificate

Configure deployment to use OCI Signed BigBang and Verify Chart using Cosign

Deploy

Customize

Enable a package

Update the Big Bang Version

Update the domain

Use NodePorts for Istio Ingress Gateway

Additional Big Bang values

Additional resources

Key Rotation

Multiple Keys

Different keys for different environments

Modifying an encrypted file

Multi-environment Workflow

Renovate Bot

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Contributors 14

Languages

Packages