Kubernetes Response Engine powered by OpenFaaS

Although Falco can be used to detect any intrusion attempts and sends alerts to channels according to the given rules afterwards, it does not have any remediation system. This is why we need something called Kubernetes Response Engine. It simply aims to catch alerts and take actions on it. These actions can be designed as fine-grained serverless functions.

Think of a scenario you want to take action to your alerts that being notified by Falco, but the important thing that we should notice here is we are going to do different things for the different levels of alerts. For example, we are going to send the notification for notice level alert but for the warning level alert we are going to delete the pod, the overall architecture will be like this:

                +-----------+
                |   Falco   +
                +-----^-----+
                      |
              +-------v-------+
              >   OpenFaaS    +
              +-------v-------+
+-----------+         |          +-----------+
| notify-fn <---------+----------> delete-fn |
+-----v-----+ notice     warning +-----+-----+
      |                                |
      | send alert          delete pod |
      |                                |
+-----v-----+                    +-----v-----+       
|   Slack   |                    | Pwned Pod |
+-----------+                    +-----------+

🎁 Table of Contents

• 🧰 Prerequisites
• 🎯 Target Audience
• 🎁 What is ...?
  • CIVO
  • K3S
  • Falco
  • OpenFaaS
• 👨‍💻 Hands-on Demonstration

Prerequisites

• civo cli v0.7.6
• Helm v3.5.1
• faas-cli
• kubectl v1.20.2

We are going to do this demo on macOS Catalina 1.15.7, you can find the prerequisites on brew.

Target Audience

If you want to:

• Set up a K3S cluster on CIVO
• Set up the Falco
• Create custom serverless functions using OpenFaaS
• Subscribe Falco events from serverless functions

What is ...?

CIVO

Built for speed and simplicity, with K3s under the hood

Join the #Kube100 beta: Apply to join today (Get free credit to test-drive the world’s first K3s-powered, managed Kubernetes service)

• Simple, fast and powerful

Spin up Kubernetes in under 2 minutes, without the bloat, using the lightweight K3s distribution

• Management CLI

Manage your clusters with the custom-built CLI and streamline your deployment with simple REST API.

• Application marketplace

Launch clusters with preinstalled applications, or install on the fly using Civo's Kubernetes marketplace.

• Bandwidth Pooling

Use all your bandwidth allowance across all your Civo services. Only pay for what you use.

• Marketplace

Install applications quickly and easily to your clusters during or after set up with Civo marketplace. Also you can contribute an application to marketplace

• Custom Firewalls

Configure custom firewalls from your Civo dashboard using OpenStack’s security groups.

K3S

The certified Kubernetes distribution built for IoT & Edge computing

• Perfect for Edge

K3s is a highly available, certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances. Simplified & Secure

• Simplified & Secure

K3s is packaged as a single <40MB binary that reduces the dependencies and steps needed to install, run and auto-update a production Kubernetes cluster.

• Optimized for ARM

Both ARM64 and ARMv7 are supported with binaries and multiarch images available for both. K3s works great from something as small as a Raspberry Pi to an AWS a1.4xlarge 32GiB server.

Falco

Cloud-Native runtime security, de facto Kubernetes threat detection engine

• Strengthen container security

The flexible rules engine allows you to describe any type of host or container behavior or activity.

• Reduce risk via immediate alerts

You can immediately respond to policy violation alerts and integrate Falco within your response workflows.

• Leverage most current detection rules

Falco out-of-the box rules alert on malicious activity and CVE exploits.

OpenFaaS

OpenFaaS® makes it simple to deploy both functions and existing code to Kubernetes

• Anywhere

Avoid lock-in through the use of Docker. Run on any public or private cloud.

• Any code

Build both microservices & functions in any language. Legacy code and binaries.

• Any scale

Auto-scale for demand or to zero when idle.

Hands-on Demonstration

Create CIVO Playground

• Download the civo cli from here
• Copy and save the API key from your Civo account - Security Dashboard

$ civo apikey save my-awesome-key $KEY
Saved the API Key $KEY as my-awesome-key

$ civo apikey list
+----------------+---------+
| Name           | Default |
+----------------+---------+
| my-awesome-key | $KEY    |
+----------------+---------+

$ civo apikey current my-awesome-key
Set the default API Key to be my-awesome-key

• Create a cluster

$ civo kubernetes create playground --size=g3.k3s.medium --nodes=3 --region NYC1 --wait
Created Kubernetes cluster playground

• Show the playground on Kubernetes Dashboard

$ civo kubernetes show playground
          ID : 79435efe-2dac-403d-bfd2-f6644988830a
        Name : playground
       Nodes : 3
        Size : g3.k3s.medium
      Status : ACTIVE
     Version : 1.20.0-k3s2
API Endpoint : https://212.2.243.151:6443
   Master IP : 212.2.243.151
DNS A record : 79435efe-2dac-403d-bfd2-f6644988830a.k8s.civo.com
Nodes:
+-------------+---------------+--------+---------------+-----------+------+----------+
| Name        | IP            | Status | Size          | Cpu Cores | Ram  | SSD disk |
+-------------+---------------+--------+---------------+-----------+------+----------+
| master-7c8a | 212.2.243.151 | ACTIVE | g3.k3s.medium |         2 | 4096 |       25 |
| node-04ed   |               | ACTIVE | g3.k3s.medium |         2 | 4096 |       25 |
| node-5258   |               | ACTIVE | g3.k3s.medium |         2 | 4096 |       25 |
+-------------+---------------+--------+---------------+-----------+------+----------+

• Configure the playground

$ civo kubernetes config playground --save --local-path ./kubeconfig
Access your cluster with:
KUBECONFIG=./kubeconfig kubectl get node

• Ensure all is OK

$ export KUBECONFIG=./kubeconfig
$ kubectl get node
NAME                                  STATUS   ROLES                  AGE   VERSION
k3s-playground-66b18d51-node-04ed     Ready    <none>                 40h   v1.20.2+k3s1

You can find more details about civo cli here.

• Set up OpenFaaS

You can install OpenFaaS from Civo marketplace -

or via helm

$ helm repo add openfaas https://openfaas.github.io/faas-netes/
$ kubectl create namespace openfaas-fn
$ helm upgrade openfaas --install openfaas/openfaas \
    --namespace openfaas --create-namespace \
    -f openfaas-override.yaml
Release "openfaas" does not exist. Installing it now.
NAME: openfaas
LAST DEPLOYED: Thu Mar  4 14:28:32 2021
NAMESPACE: openfaas
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
To verify that openfaas has started, run:

  kubectl -n openfaas get deployments -l "release=openfaas, app=openfaas"

• Set up Falco

$ helm repo add falcosecurity https://falcosecurity.github.io/charts
$ helm upgrade falco --install falcosecurity/falco \
--namespace falco --create-namespace \
-f override.yaml
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: ./kubeconfig
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: ./kubeconfig
Release "falco" does not exist. Installing it now.
NAME: falco
LAST DEPLOYED: Thu Mar  4 14:34:54 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.


No further action should be required.


Tip:
You can easily forward Falco events to Slack, Kafka, AWS Lambda and more with falcosidekick.
Full list of outputs: https://github.com/falcosecurity/charts/falcosidekick.
You can enable its deployment with `--set falcosidekick.enabled=true` or in your values.yaml.
See: https://github.com/falcosecurity/charts/blob/master/falcosidekick/values.yaml for configuration values.

After everyting is ok, configure the faas-cli in order to deploy functions.

export OPENFAAS_URL="http://79435efe-2dac-403d-bfd2-f6644988830a.k8s.civo.com:31112"

Let's see what functions present.

$ faas-cli list
Function                        Invocations     Replicas

You should see nothing.

NOTE: Don't forget to change Docker ID in the function's description files.

Now Deploy the functions.

$ faas-cli up -f dispatch-fn.yml
$ faas-cli up -f notifier-fn.yml
$ faas-cli up -f delete-pod-fn.yml
...

Once it's completed, you should see the pods that's running on openfaas-fn namespace, and you should also see functions by typing faas-cli list command.

$ faas-cli list
Function                        Invocations     Replicas
dispatch-fn                     37              1
notifier-fn                     23              1
delete-pod-fn                   0               1

$ kubectl get pods --namespace=openfaas-fn
Found existing alias for "kubectl get pods". You should use: "kgp"
NAME                             READY   STATUS    RESTARTS   AGE
dispatch-fn-b5b68df77-r7crl      1/1     Running   0          7m23s
notifier-fn-5fd846c887-hgpf4     1/1     Running   0          4m48s
delete-pod-fn-55cc89965c-fs55q   1/1     Running   0          37s

If everything works well, you should see the Slack messages in the #falco-alert channel. Here is the screenshot above that shows this.

Learn more about how you can set up a Slack workspace from this link.

The next thing that you need to do this, you should create an alert at Warning level, in order to that, you need to create a Pod based on the alpine image, and you should connect to it, and access some important folders like /etc/shadow. Once this is done, you should notice that the Pod gets killed by the delete-pod-fn function.

Let's test it with creating a Pod.

$  kubectl run alpine -n default --image=alpine --restart='Never' -- sh -c "sleep 600"
Found existing alias for "kubectl". You should use: "k"
pod/alpine created

Connect to it.

$ kubectl exec -ti alpine -- sh
Found existing alias for "kubectl exec -ti". You should use: "keti"
/ #

Once you connect it, open a second terminal and watch the state of the Pod.

$ watch kubectl get pods
Every 2.0s: kubectl get pods
NAME     READY   STATUS    RESTARTS   AGE
alpine   1/1     Running   0          106s

Then, in the first terminal, run the cat /etc/shadow, then look at the second terminal, you should notice that Pod's status changed from Running to Terminating, and your connection to the Pod is lost.