Skip to content

filipeforattini/ff-iac-github-actions

Repository files navigation

Github Actions Fast Pipelines

semantic-release

This is a personal work in progress. Keep in mind your suggestions are welcome! :)

These workflows are highly opinionated kubectl-apply or helm-upgrade pipelines.

tldr;

  1. Config tour repository with these secrets.
  2. Create a directory .github/workflow in your repository and add this file.
  3. Create a another directory manifests and add this file.
  4. Commit and open your repository actions page! :)

Introduction

Features

  • Versioning with Semantic-Release
  • Linter:
    • Hadolint for Dockerfiles
    • ESLint for Javascript
    • PyLint for Python
  • Static analysis:
    • GitLeaks for repository
    • Trivy for repository and image
    • Open Source Static Analysis Runner

Repository Patterns

This pipeline assumes you have just 4 types of repositories:

Name Short Description Result
Web Application app Front-end application with internet-facing ingress language-based pod
Mobile Application mob Mobile application apk
Service svc Microservice that may - or may not - have ingress nginx-based pod
Infrastructure as Code iac Code that generates cloud infrastructure -

Those repositories must obey a name pattern.

{ecosystem}-{type}-{name/client/integration}

Examples:

  • ff-svc-clients: microservice that manages clients' data
  • ff-app-budget: application that organizes the company finances
  • ff-mob-auth: 2FA mobile application
  • ff-iac-aws: infra as code to manage the aws environment

Checkout the test repositories:

Type Solution Repository Pipeline Deploy
app filipeforattini/ff-app-react
app filipeforattini/ff-app-vue
svc filipeforattini/ff-svc-express
svc filipeforattini/ff-svc-fastapi
svc filipeforattini/ff-svc-flask
svc filipeforattini/ff-svc-moleculer
svc filipeforattini/ff-svc-nestjs
svc filipeforattini/ff-svc-nextjs

Environments

Name Short Description
Development dev Env for you and your team to test and explore
Staging stg Stable env for code shipping
Sandbox sbx Production-like env for external developers
Production prd Where the magic happens
Disaster Recovery dry Production copy

Usage

Flow

Loading
flowchart
  start[Start]
  analysis[Analysis]
  node-test[Node Tests]
  python-test[Python Tests]
  static-analysis[Static Analysis]
  
  start --- analysis

  analysis --- |event/push| static-analysis
  analysis --- |event/pull_request| static-analysis
  analysis --- |event/workflow_dispatch| trigger-manual

  static-analysis --- |lang/javascript| node-test
  static-analysis --- |lang/python| python-test
  static-analysis --- |lang/go| go-test
  
  subgraph node:
    node-test --- node-release
    node-release --- node-trigger
  end

  subgraph python:
    python-test --- python-release
    python-release --- python-trigger
  end

  subgraph go:
    go-test --- go-release
    go-release --- go-trigger
  end

  node-trigger --- |deployment/dev| finish
  python-trigger --- |deployment/dev| finish
  go-trigger --- |deployment/dev| finish

  trigger-manual --- |env/xxx| finish

  analysis --- |event/deployment| build
  build --- |env/dev| env-dev
  build --- |env/stg| env-stg
  build --- |env/prd| env-prd

  subgraph dev:
    env-dev --- |app.dev.domain.io| DEV
    env-dev --- |app-commit.dev.domain.io| DEV
  end

  subgraph stg:
    env-stg --- |app.stg.domain.io| STG
  end
  
  subgraph prd:
    env-prd --- |app.prd.domain.io| PRD
  end

Repository Structure

├─ .github
│  └─ workflows
│  │  └─ pipeline.yml
│  └─ dependabot.yml
├─ manifests
│  ├─ configs
│  │  └─ dev.env
│  ├─ dependencies
│  │  └─ dev.yml
│  ├─ secrets
│  │  └─ dev.gpg
│  ├─ k8s.yml
│  └─ helm.yml
├─ build
│  // distibuition version of our code
└─ src
   // our code goes here

Repository Secrets

Name Description
GPG_PASSPHRASE
KUBE_CONFIG Your ~/.kube/config file as base64.
PIPELINE_DEPLOY_SECRET A GitHub token, see the permissions below.
REGISTRY_USERNAME Registry username.
REGISTRY_PASSWORD Registry password.

Pipeline

Add this pipeline to your repository creating a file pipeline.yml in your .github/workflows directory.

For SVC

name: pipeline

on: 
  push:
  deployment:
  release:
    types: [created]
  pull_request:
    types: [opened, reopened]

  workflow_dispatch:
    inputs:
      environment:
        description: 'Environment'
        required: true
        type: choice
        default: 'dev'
        options:
        - dev
        - stg
        - prd

jobs:

  SVC:
    uses: filipeforattini/ff-iac-github-actions/.github/workflows/svc.yml@main
    secrets: inherit
    with:
      mainBranch: main
      containerRegistry: ghcr.io
      environmentsAsNamespaces: true

Parameters

Name Default Description
containerRegistry ghcr.io Container registry host that you will use
environmentsASnamespaces false

Deploy with kubetl apply

Create a file k8s.yml in your manifests directory.

#@data/values
---
port: 1234

env:
  - name: TZ
    value: America/Sao_Paulo

ingress:
  enable: true
  type: traefik

  tls:
    enable: true
    domain: your.domain

Requirements

Configure your k8s cluster and get your ~/.kube/config.

Daily work

Commits & Versioning

git commit -m "action(scope): subject"

Where the actions:

  • feat: new feature for the user, not a new feature for the build script
  • fix: bug fix for the user, not a fix for a build script
  • docs: documentation changes
  • style: formatting, lack of semicolons, etc; no changes to the production code
  • refactor: refactoring the production code, for example. renaming a variable
  • test: adding missing tests, refactoring tests; no changes to the production code
  • chore:updating grunted tasks, etc; no changes to the production code

Adds BREAKING CHANGE in the commit message and it will generate a new major version.

Secrets

gpg -v \
  --symmetric \
  --cipher-algo AES256 \
  --output ./manifests/secrets/dev.gpg \
  ./manifests/secrets/dev.env

Thanks to:

Example ecosystem

This ecosystem generates few data per second as samples for our apis.

Architecture

Full independent

In this implementation, each service has its own resources.

Loading
flowchart
  https---ingress

  subgraph k8s
    ingress---|ff-svc-nestjs.dev.forattini.app|nestjs
    ingress---|ff-svc-nextjs.dev.forattini.app|nextjs
    ingress---|ff-svc-fastapi.dev.forattini.app|fastapi
    ingress---|ff-svc-moleculer.dev.forattini.app|moleculer

    subgraph ff-svc-nextjs
      nextjs---rabbitmq-nextjs[rabbit]
    end
    
    subgraph ff-svc-moleculer
      moleculer---postgres-moleculer[postgres]
      moleculer---mysql-moleculer[mysql]
      moleculer---redis-moleculer[redis]
      moleculer---rabbitmq-moleculer[rabbit]
      moleculer---etcd-moleculer[etcd]
      moleculer---nats-moleculer[nats]

      moleculer---rabbitmq-nextjs
    end

    subgraph ff-svc-fastapi
      fastapi---postgres-fastapi[postgres]
      fastapi---rabbitmq-fastapi[rabbit]
      
      fastapi---rabbitmq-nextjs
    end

    subgraph ff-svc-nestjs
      nestjs---postgres-nestjs[postgres]
      nestjs---rabbitmq-nestjs[rabbitmq]

      nestjs---rabbitmq-nextjs
    end
  end

Shared resources

In this implementation, all services connects to a shared resource.

Loading
flowchart
  https---ingress

  subgraph k8s
    ingress---|ff-svc-nestjs.dev.forattini.app|nestjs
    ingress---|ff-svc-nextjs.dev.forattini.app|nextjs
    ingress---|ff-svc-fastapi.dev.forattini.app|fastapi
    ingress---|ff-svc-moleculer.dev.forattini.app|moleculer

    subgraph ff-svc-moleculer
      moleculer---postgres-moleculer[postgres]
      moleculer---mysql-moleculer[mysql]
      moleculer---redis-moleculer[redis]
      moleculer---rabbitmq-moleculer[rabbitmq]
      moleculer---etcd-moleculer[etcd]
      moleculer---nats-moleculer[nats]
    end

    subgraph ff-svc-nextjs
      nextjs---rabbitmq-moleculer
      nextjs---postgres-moleculer
      nextjs---mysql-moleculer
      nextjs---redis-moleculer
    end

    subgraph ff-svc-fastapi
      fastapi---postgres-moleculer
      fastapi---rabbitmq-moleculer
    end

    subgraph ff-svc-nestjs
      nestjs---postgres-moleculer
      nestjs---rabbitmq-moleculer
    end
  end