Skip to content

Commit

Permalink
BUILD/MAJOR: cn: update client native to v3
Browse files Browse the repository at this point in the history
  • Loading branch information
oktalz committed May 12, 2022
1 parent 21a2a57 commit fdf9db7
Show file tree
Hide file tree
Showing 71 changed files with 498 additions and 244 deletions.
36 changes: 18 additions & 18 deletions crs/README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# Custom Resources

## Purpose
This document aims to provide basic understanding of the workflow of writing a new custom resource for HAProxy Ingress Controller.
Custom resources should be derived, whenever possible, from the [HAProxy CN (Client Native)](https://github.com/haproxytech/client-native) by reusing [HAProxy Models](https://github.com/haproxytech/client-native#haproxy-models).
This guide describes how to create a CR (Custom Resource) to configure the global HAProxy section.
This document aims to provide basic understanding of the workflow of writing a new custom resource for HAProxy Ingress Controller.
Custom resources should be derived, whenever possible, from the [HAProxy CN (Client Native)](https://github.com/haproxytech/client-native) by reusing [HAProxy Models](https://github.com/haproxytech/client-native#haproxy-models).
This guide describes how to create a CR (Custom Resource) to configure the global HAProxy section.

## Prerequisites
We suppose that an HAProxy CN Model, corresponding to the kubernetes custom resource, is already available. This is the case for the HAProxy global section, that we are dealing with in this guide, where we are going to reuse the following [global model](https://github.com/haproxytech/client-native/blob/master/models/global.go).
We suppose that an HAProxy CN Model, corresponding to the kubernetes custom resource, is already available. This is the case for the HAProxy global section, that we are dealing with in this guide, where we are going to reuse the following [global model](https://github.com/haproxytech/client-native/blob/master/models/global.go).
If it is not the case, this needs to be done in [HAProxy Native client](https://github.com/haproxytech/client-native) by providing [swagger specification](https://github.com/haproxytech/client-native/blob/master/specification/build/haproxy_spec.yaml) of the Model and generate it via [make models](https://github.com/haproxytech/client-native/blob/master/Makefile).

## Directory Layout
Expand Down Expand Up @@ -35,9 +35,9 @@ crs

## Custom Resource Creation
### CRD
Let's start by creating the Custom Resource Definition describing the Global CR and defining the kind `Global` in the API Group `core.haproxy.org`
To follow the above directory layout convention, the Global CRD should be in *crs/definitions/core/global.yaml*
The following is the beginning of the CRD, full content can be found in the corresponding yaml file:
Let's start by creating the Custom Resource Definition describing the Global CR and defining the kind `Global` in the API Group `core.haproxy.org`
To follow the above directory layout convention, the Global CRD should be in *crs/definitions/core/global.yaml*
The following is the beginning of the CRD, full content can be found in the corresponding yaml file:
```
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
Expand All @@ -59,10 +59,10 @@ spec:
properties:
spec:
type: object
required:
required:
- config
properties:
config:
properties:
config:
title: Global
description: HAProxy global configuration
type: object
Expand All @@ -82,23 +82,23 @@ spec:
type: string
enum: [enabled, disabled]
```
The CRD is created via `apiextensions.k8s.io/v1` where the definition of a [structural schema](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation) is mandatory and used for OpenAPI v3.0 validation.
The CRD is created via `apiextensions.k8s.io/v1` where the definition of a [structural schema](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation) is mandatory and used for OpenAPI v3.0 validation.
The Custom Resource Definition is constructed by providing:
- API Group Name: `core.haproxy.org`, this is the group where you can find custom resources related to HAProxy configuration.
- Kind Name of the Resource: `Global` (and `globals` for plural name)
- Resource Scope: in this case it is `Namespaced`
- Group Version: `v1alpha1`
- openAPIV3Schema: For schema validation you just copy the [global schema](https://github.com/haproxytech/client-native/blob/master/specification/models/configuration.yaml#L2) from Client Native repository and insert it in the `config` field of the CRD.
- openAPIV3Schema: For schema validation you just copy the [global schema](https://github.com/haproxytech/client-native/blob/master/specification/models/configuration.yaml#L2) from Client Native repository and insert it in the `config` field of the CRD.
**NB:** a helper script is available under `crs/get-crd-schema.sh` in order to automatically retreive the schema from the CN repository. The script takes the name of Client Native model as parameter (ex: `get-crd-schema.sh global`) and returns to stdout the schema after applying the necessary changes to make it a [valid schema](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation) for kubernetes.

### GoLang Type
In this guide the API group is `core.haproxy.org` (this group is used for any resource dealing with haproxy configuration) and the API version is `alpha1v1`.
In this guide the API group is `core.haproxy.org` (this group is used for any resource dealing with haproxy configuration) and the API version is `alpha1v1`.
So to follow the */crs/api/<group>/<version>/* convention, the GoLang Type describing the `Global` CR should be in *crs/api/core/alpha1v1/global.go* with the following content:
```
package v1alpha1
import (
"github.com/haproxytech/client-native/v2/models"
"github.com/haproxytech/client-native/v3/models"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
Expand Down Expand Up @@ -140,18 +140,18 @@ type GlobalList struct {
- The GoLang types (Global for a specific resource and GlobalList for a list of resources) need to embed `TypeMeta` and `ObjectMeta` as they are mandatory for the Kubernetes type system and any resource served by Kubernetes API.
- The Spec reuses the [global configuration](https://github.com/haproxytech/client-native/blob/master/models/global.go) from HAProxy CN Models via the CR `config` field.
- Additional fields can added later to Spec if need be, but any input related to the HAProxy Global section config is provided via the `config` field.
- The deepcopy-gen tags are used to automatically generate deepCopy methods, required for any Kubernetes object, via the [code-generator](https://github.com/kubernetes/code-generator) tool. However Code Generators cannot generate the deepCopy method for types in external packages (in this case "github.com/haproxytech/client-native/v2/models") so this needs to be done manually by relying on the `MarshalBinary` and `UnmarshalBinary` of HAProxy Models.
- The deepcopy-gen tags are used to automatically generate deepCopy methods, required for any Kubernetes object, via the [code-generator](https://github.com/kubernetes/code-generator) tool. However Code Generators cannot generate the deepCopy method for types in external packages (in this case "github.com/haproxytech/client-native/v3/models") so this needs to be done manually by relying on the `MarshalBinary` and `UnmarshalBinary` of HAProxy Models.

### Code Generation
Custom Resources require more code and methods in order to be served by Kubernetes API but fortunately most of it can be generated automatically by Kubernetes code generators.
Custom Resources require more code and methods in order to be served by Kubernetes API but fortunately most of it can be generated automatically by Kubernetes code generators.
The tool we are using is [k8s.io/code-generator](https://github.com/kubernetes/code-generator) which comes with the following code generators for CRs:
- deepcopy-gen: generates the new Type's DeepCopy methods.
- register-gen: generates methods to register the new Type in [Kubernetes scheme](https://github.com/kubernetes/apimachinery/blob/ef51ab160544f9d05b68e132a4af0b0fab459954/pkg/runtime/scheme.go#L47) and make it known to the Kubernetes type system.
- client-gen: generates the client set that will give us access to the new CR
- informer-gen: generates informer methods in order to watch and react to the CR changes.
- lister-gen: generates lister methods which provide a read-only caching layer for GET and LIST requests.

Before generating code, some global tags need to be set in the package of the versioned API group which should be in *crs/api/<group>/<version>/* directory.
Before generating code, some global tags need to be set in the package of the versioned API group which should be in *crs/api/<group>/<version>/* directory.
Usually this goes into the package's doc.go, so in this case it would be in *crs/api/core/alpha1v1/doc.go* with the following content:
```
// Package v1alpha1 contains the core v1alpha1 API group
Expand All @@ -166,7 +166,7 @@ Now you can generate all necessary code for all CRs under the */crs/api* directo


## Custom Resource Management
A custom resource manager already [exists](../controller/crmanager.go) as a single entry point for any custom resource defined inside HAProxy Ingress Controller.
A custom resource manager already [exists](../controller/crmanager.go) as a single entry point for any custom resource defined inside HAProxy Ingress Controller.
Its role is to be the unique delegate for all CR related task via the following CR interface:
```
type CR interface {
Expand Down
2 changes: 1 addition & 1 deletion crs/api/core/v1alpha1/backend.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
package v1alpha1

import (
"github.com/haproxytech/client-native/v2/models"
"github.com/haproxytech/client-native/v3/models"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

Expand Down
2 changes: 1 addition & 1 deletion crs/api/core/v1alpha1/defaults.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
package v1alpha1

import (
"github.com/haproxytech/client-native/v2/models"
"github.com/haproxytech/client-native/v3/models"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

Expand Down
2 changes: 1 addition & 1 deletion crs/api/core/v1alpha1/global.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
package v1alpha1

import (
"github.com/haproxytech/client-native/v2/models"
"github.com/haproxytech/client-native/v3/models"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

Expand Down
36 changes: 18 additions & 18 deletions crs/get-crd-schema.sh
Original file line number Diff line number Diff line change
@@ -1,40 +1,40 @@
#!/usr/bin/env bash
set -o errexit
set -o pipefail
#!/usr/bin/env bash

set -o errexit
set -o pipefail

command -v yq >/dev/null 2>&1 || { echo >&2 "yq not installed. Aborting."; exit 1; }
command -v jq >/dev/null 2>&1 || { echo >&2 "jq not installed. Aborting."; exit 1; }
CN_COMMIT=$(go list -m github.com/haproxytech/client-native/v2 | sed 's/^.*-//')

CN_COMMIT=$(go list -m github.com/haproxytech/client-native/v3 | sed 's/^.*-//')

if [ -z "$1" ]; then echo >&2 "No model name supplied. Aborting."; exit 1; fi
if [ -z "$CN_COMMIT" ]; then echo >&2 "Unable to get git commit for CN module. Aborting."; exit 1; fi

curl -sk https://raw.githubusercontent.com/haproxytech/client-native/$CN_COMMIT/specification/models/configuration.yaml |
yq |
jq --arg MODEL $1 '.|
jq --arg MODEL $1 '.|
reduce paths as $p(.;
if $p[0] == $MODEL and $p[-1] == "$ref" then
if $p[0] == $MODEL and $p[-1] == "$ref" then
setpath($p[0:-1];getpath(getpath($p) | split("/")[-1]| split(" ")))
else
.
else
.
end
) |
.[$MODEL] |
) |
.[$MODEL] |
walk(
if type == "object" then with_entries(
if type == "object" then with_entries(
if .key == "x-nullable" then
if .value == false then
empty
else
.key = "nullable"
end
elif (.key | contains("x-")) then
empty
else
.
end
elif (.key | contains("x-")) then
empty
else
.
end
) else . end
)' |
yq -y
7 changes: 4 additions & 3 deletions deploy/tests/integration/customresources/suite_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import (
"path/filepath"
"testing"

"github.com/haproxytech/client-native/v2/models"
"github.com/haproxytech/client-native/v3/models"
corev1alpha1 "github.com/haproxytech/kubernetes-ingress/crs/api/core/v1alpha1"
c "github.com/haproxytech/kubernetes-ingress/pkg/controller"
"github.com/haproxytech/kubernetes-ingress/pkg/haproxy/env"
Expand Down Expand Up @@ -84,12 +84,13 @@ option dontlog-normal
frontend stats
mode http
bind *:1024
bind 0:0:0:0:1024
http-request set-var(txn.base) base
http-request use-service prometheus-exporter if { path /metrics }
stats enable
stats uri /
stats refresh 10s`
stats refresh 10s
`

func (suite *CustomResourceSuite) GlobalCRFixture() (eventChan chan k8s.SyncDataEvent, s store.K8s, globalCREvt k8s.SyncDataEvent) {
var osArgs utils.OSArgs
Expand Down
7 changes: 4 additions & 3 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ go 1.17
require (
github.com/go-test/deep v1.0.7
github.com/google/renameio v1.0.1
github.com/haproxytech/client-native/v2 v2.5.2-0.20211021104411-038f5cf8ec14
github.com/haproxytech/config-parser/v4 v4.0.0-rc2.0.20211021093817-f9021b6ca61c
github.com/haproxytech/client-native/v3 v3.1.1-0.20220510114028-57f50c20a1bf
github.com/haproxytech/config-parser/v4 v4.0.0-rc2.0.20220428133329-7d0ec01198d4
github.com/jessevdk/go-flags v1.4.0
github.com/pires/go-proxyproto v0.6.1
github.com/prometheus/client_golang v1.12.1
Expand Down Expand Up @@ -43,10 +43,11 @@ require (
github.com/google/gofuzz v1.1.0 // indirect
github.com/google/uuid v1.1.2 // indirect
github.com/googleapis/gnostic v0.5.5 // indirect
github.com/haproxytech/go-logger v1.0.0 // indirect
github.com/haproxytech/go-logger v1.0.1-0.20211022075555-178f1cdf4d84 // indirect
github.com/imdario/mergo v0.3.5 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/kr/pretty v0.3.0 // indirect
github.com/mailru/easyjson v0.7.1 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
github.com/mitchellh/mapstructure v1.2.2 // indirect
Expand Down
Loading

0 comments on commit fdf9db7

Please sign in to comment.