Skip to content

Commit

Permalink
chore: integrate kb-agent with controller (#7821)
Browse files Browse the repository at this point in the history 
Co-authored-by: Ursasi <zshprint@163.com>
  • Loading branch information
@leon-inf @Ursasi
leon-inf and Ursasi authored Aug 14, 2024
1 parent b61cbd4 commit 44edc0d
Show file tree
Hide file tree
Showing 67 changed files with 2,576 additions and 2,487 deletions.
6 changes: 5 additions & 1 deletion Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ help: ## Display this help.
@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m<target>\033[0m\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf " \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)

.PHONY: all
all: manager dataprotection lorry reloader ## Make all cmd binaries.
all: manager dataprotection kbagent ## Make all cmd binaries.

##@ Development

Expand Down Expand Up @@ -277,6 +277,10 @@ manager: cue-fmt generate manager-go-generate test-go-generate build-checks ## B
dataprotection: generate test-go-generate build-checks ## Build dataprotection binary.
$(GO) build -ldflags=${LD_FLAGS} -o bin/dataprotection ./cmd/dataprotection/main.go

.PHONY: kbagent
kbagent: generate test-go-generate build-checks
$(GO) build -ldflags=${LD_FLAGS} -o bin/kbagent ./cmd/kbagent/main.go

CERT_ROOT_CA ?= $(WEBHOOK_CERT_DIR)/rootCA.key
.PHONY: webhook-cert
webhook-cert: $(CERT_ROOT_CA) ## Create root CA certificates for admission webhooks testing.
Expand Down
45 changes: 10 additions & 35 deletions apis/apps/v1alpha1/componentdefinition_types.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1113,12 +1113,9 @@ type ComponentLifecycleActions struct {
// It ensures replicas are correctly labeled with their respective roles.
// Without this, services that rely on roleSelectors might improperly direct traffic to wrong replicas.
//
// The container executing this action has access to following environment variables:
// The container executing this action has access to following variables:
//
// - KB_POD_FQDN: The FQDN of the Pod whose role is being assessed.
// - KB_SERVICE_PORT: The port used by the database service.
// - KB_SERVICE_USER: The username with the necessary permissions to interact with the database service.
// - KB_SERVICE_PASSWORD: The corresponding password for KB_SERVICE_USER to authenticate with the database service.
//
// Expected output of this action:
// - On Success: The determined role of the replica, which must align with one of the roles specified
Expand Down Expand Up @@ -1156,15 +1153,10 @@ type ComponentLifecycleActions struct {
// implementation, or automatically by the database kernel or a sidecar utility like Patroni that implements
// a consensus algorithm.
//
// The container executing this action has access to following environment variables:
// The container executing this action has access to following variables:
//
// - KB_SERVICE_PORT: The port used by the database service.
// - KB_SERVICE_USER: The username with the necessary permissions to interact with the database service.
// - KB_SERVICE_PASSWORD: The corresponding password for KB_SERVICE_USER to authenticate with the database service.
// - KB_PRIMARY_POD_FQDN: The FQDN of the primary Pod within the replication group.
// - KB_MEMBER_ADDRESSES: A comma-separated list of Pod addresses for all replicas in the group.
// - KB_NEW_MEMBER_POD_NAME: The pod name of the replica being added to the group.
// - KB_NEW_MEMBER_POD_IP: The IP address of the replica being added to the group.
// - KB_JOIN_MEMBER_POD_FQDN: The pod FQDN of the replica being added to the group.
// - KB_JOIN_MEMBER_POD_NAME: The pod name of the replica being added to the group.
//
// Expected action output:
// - On Failure: An error message detailing the reason for any failure encountered
Expand All @@ -1177,11 +1169,8 @@ type ComponentLifecycleActions struct {
// - bash
// - -c
// - |
// ADDRESS=$(KB_MEMBER_ADDRESSES%%,*)
// HOST=$(echo $ADDRESS | cut -d ':' -f 1)
// PORT=$(echo $ADDRESS | cut -d ':' -f 2)
// CLIENT="mysql -u $KB_SERVICE_USER -p$KB_SERVICE_PASSWORD -P $PORT -h $HOST -e"
// $CLIENT "ALTER SYSTEM ADD SERVER '$KB_NEW_MEMBER_POD_IP:$KB_SERVICE_PORT' ZONE 'zone1'"
// CLIENT="mysql -u $SERVICE_USER -p$SERVICE_PASSWORD -P $SERVICE_PORT -h $SERVICE_HOST -e"
// $CLIENT "ALTER SYSTEM ADD SERVER '$KB_POD_FQDN:$SERVICE_PORT' ZONE 'zone1'"
// ```
//
// Note: This field is immutable once it has been set.
Expand All @@ -1198,15 +1187,10 @@ type ComponentLifecycleActions struct {
// The process typically includes updating configurations and informing other group members about the removal.
// Data migration is generally not part of this action and should be handled separately if needed.
//
// The container executing this action has access to following environment variables:
// The container executing this action has access to following variables:
//
// - KB_SERVICE_PORT: The port used by the database service.
// - KB_SERVICE_USER: The username with the necessary permissions to interact with the database service.
// - KB_SERVICE_PASSWORD: The corresponding password for KB_SERVICE_USER to authenticate with the database service.
// - KB_PRIMARY_POD_FQDN: The FQDN of the primary Pod within the replication group.
// - KB_MEMBER_ADDRESSES: A comma-separated list of Pod addresses for all replicas in the group.
// - KB_LEAVE_MEMBER_POD_FQDN: The pod name of the replica being removed from the group.
// - KB_LEAVE_MEMBER_POD_NAME: The pod name of the replica being removed from the group.
// - KB_LEAVE_MEMBER_POD_IP: The IP address of the replica being removed from the group.
//
// Expected action output:
// - On Failure: An error message, if applicable, indicating why the action failed.
Expand All @@ -1218,11 +1202,8 @@ type ComponentLifecycleActions struct {
// - bash
// - -c
// - |
// ADDRESS=$(KB_MEMBER_ADDRESSES%%,*)
// HOST=$(echo $ADDRESS | cut -d ':' -f 1)
// PORT=$(echo $ADDRESS | cut -d ':' -f 2)
// CLIENT="mysql -u $KB_SERVICE_USER -p$KB_SERVICE_PASSWORD -P $PORT -h $HOST -e"
// $CLIENT "ALTER SYSTEM DELETE SERVER '$KB_LEAVE_MEMBER_POD_IP:$KB_SERVICE_PORT' ZONE 'zone1'"
// CLIENT="mysql -u $SERVICE_USER -p$SERVICE_PASSWORD -P $SERVICE_PORT -h $SERVICE_HOST -e"
// $CLIENT "ALTER SYSTEM DELETE SERVER '$KB_POD_FQDN:$SERVICE_PORT' ZONE 'zone1'"
// ```
//
// Note: This field is immutable once it has been set.
Expand All @@ -1238,9 +1219,6 @@ type ComponentLifecycleActions struct {
// The container executing this action has access to following environment variables:
//
// - KB_POD_FQDN: The FQDN of the replica pod whose role is being checked.
// - KB_SERVICE_PORT: The port used by the database service.
// - KB_SERVICE_USER: The username with the necessary permissions to interact with the database service.
// - KB_SERVICE_PASSWORD: The corresponding password for KB_SERVICE_USER to authenticate with the database service.
//
// Expected action output:
// - On Failure: An error message, if applicable, indicating why the action failed.
Expand All @@ -1260,9 +1238,6 @@ type ComponentLifecycleActions struct {
// The container executing this action has access to following environment variables:
//
// - KB_POD_FQDN: The FQDN of the replica pod whose role is being checked.
// - KB_SERVICE_PORT: The port used by the database service.
// - KB_SERVICE_USER: The username with the necessary permissions to interact with the database service.
// - KB_SERVICE_PASSWORD: The corresponding password for KB_SERVICE_USER to authenticate with the database service.
//
// Expected action output:
// - On Failure: An error message, if applicable, indicating why the action failed.
Expand Down
40 changes: 24 additions & 16 deletions cmd/kb_agent/main.go → cmd/kbagent/main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ package main

import (
"flag"
"fmt"
"os"
"os/signal"
"strings"
Expand All @@ -34,14 +35,27 @@ import (
ctrl "sigs.k8s.io/controller-runtime"
kzap "sigs.k8s.io/controller-runtime/pkg/log/zap"

"github.com/apecloud/kubeblocks/pkg/kb_agent/cronjobs"
"github.com/apecloud/kubeblocks/pkg/kb_agent/handlers"
"github.com/apecloud/kubeblocks/pkg/kb_agent/httpserver"
kbagent "github.com/apecloud/kubeblocks/pkg/kbagent"
"github.com/apecloud/kubeblocks/pkg/kbagent/server"
viper "github.com/apecloud/kubeblocks/pkg/viperx"
)

const (
defaultPort = 3501
defaultMaxConcurrency = 8
)

var serverConfig server.Config

func init() {
viper.AutomaticEnv()

pflag.StringVar(&serverConfig.Address, "address", "0.0.0.0", "The HTTP Server listen address for kb-agent service.")
pflag.StringVar(&serverConfig.UnixDomainSocket, "unix-socket", "", "The path of the Unix Domain Socket for kb-agent service.")
pflag.IntVar(&serverConfig.Port, "port", defaultPort, "The HTTP Server listen port for kb-agent service.")
pflag.IntVar(&serverConfig.Concurrency, "max-concurrency", defaultMaxConcurrency,
fmt.Sprintf("The maximum number of concurrent connections the Server may serve, use the default value %d if <=0.", defaultMaxConcurrency))
pflag.BoolVar(&serverConfig.Logging, "api-logging", true, "Enable api logging for kb-agent request.")
}

func main() {
Expand All @@ -66,26 +80,20 @@ func main() {
if strings.EqualFold("debug", viper.GetString("zap-log-level")) {
kopts = append(kopts, kzap.RawZapOpts(zap.AddCaller()))
}
ctrl.SetLogger(kzap.New(kopts...))
logger := kzap.New(kopts...)
ctrl.SetLogger(logger)

// init action handlers
err = handlers.InitHandlers()
// initialize kb-agent
services, err := kbagent.Initialize(logger, os.Environ())
if err != nil {
panic(errors.Wrap(err, "init action handlers failed"))
}

// start cron jobs
jobManager, err := cronjobs.NewManager()
if err != nil {
panic(errors.Wrap(err, "Cron jobs initialize failed"))
}
jobManager.Start()

// start HTTP Server
httpServer := httpserver.NewServer()
err = httpServer.StartNonBlocking()
server := server.NewHTTPServer(logger, serverConfig, services)
err = server.StartNonBlocking()
if err != nil {
panic(errors.Wrap(err, "HTTP server initialize failed"))
panic(errors.Wrap(err, "failed to start HTTP server"))
}

stop := make(chan os.Signal, 1)
Expand Down
55 changes: 14 additions & 41 deletions config/crd/bases/apps.kubeblocks.io_componentdefinitions.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1451,25 +1451,16 @@ spec:
command\nimplementation, or automatically by the database kernel
or a sidecar utility like Patroni that implements\na consensus
algorithm.\n\n\nThe container executing this action has access
to following environment variables:\n\n\n- KB_SERVICE_PORT:
The port used by the database service.\n- KB_SERVICE_USER: The
username with the necessary permissions to interact with the
database service.\n- KB_SERVICE_PASSWORD: The corresponding
password for KB_SERVICE_USER to authenticate with the database
service.\n- KB_PRIMARY_POD_FQDN: The FQDN of the primary Pod
within the replication group.\n- KB_MEMBER_ADDRESSES: A comma-separated
list of Pod addresses for all replicas in the group.\n- KB_NEW_MEMBER_POD_NAME:
The pod name of the replica being added to the group.\n- KB_NEW_MEMBER_POD_IP:
The IP address of the replica being added to the group.\n\n\nExpected
to following variables:\n\n\n- KB_JOIN_MEMBER_POD_FQDN: The
pod FQDN of the replica being added to the group.\n- KB_JOIN_MEMBER_POD_NAME:
The pod name of the replica being added to the group.\n\n\nExpected
action output:\n- On Failure: An error message detailing the
reason for any failure encountered\n during the addition of
the new member.\n\n\nFor example, to add a new OBServer to an
OceanBase Cluster in 'zone1', the following command may be used:\n\n\n```yaml\ncommand:\n-
bash\n- -c\n- |\n ADDRESS=$(KB_MEMBER_ADDRESSES%%,*)\n HOST=$(echo
$ADDRESS | cut -d ':' -f 1)\n PORT=$(echo $ADDRESS | cut -d
':' -f 2)\n CLIENT=\"mysql -u $KB_SERVICE_USER -p$KB_SERVICE_PASSWORD
-P $PORT -h $HOST -e\"\n\t $CLIENT \"ALTER SYSTEM ADD SERVER
'$KB_NEW_MEMBER_POD_IP:$KB_SERVICE_PORT' ZONE 'zone1'\"\n```\n\n\nNote:
bash\n- -c\n- |\n CLIENT=\"mysql -u $SERVICE_USER -p$SERVICE_PASSWORD
-P $SERVICE_PORT -h $SERVICE_HOST -e\"\n\t $CLIENT \"ALTER
SYSTEM ADD SERVER '$KB_POD_FQDN:$SERVICE_PORT' ZONE 'zone1'\"\n```\n\n\nNote:
This field is immutable once it has been set."
properties:
builtinHandler:
Expand Down Expand Up @@ -1807,26 +1798,17 @@ spec:
includes updating configurations and informing other group members
about the removal.\nData migration is generally not part of
this action and should be handled separately if needed.\n\n\nThe
container executing this action has access to following environment
variables:\n\n\n- KB_SERVICE_PORT: The port used by the database
service.\n- KB_SERVICE_USER: The username with the necessary
permissions to interact with the database service.\n- KB_SERVICE_PASSWORD:
The corresponding password for KB_SERVICE_USER to authenticate
with the database service.\n- KB_PRIMARY_POD_FQDN: The FQDN
of the primary Pod within the replication group.\n- KB_MEMBER_ADDRESSES:
A comma-separated list of Pod addresses for all replicas in
the group.\n- KB_LEAVE_MEMBER_POD_NAME: The pod name of the
replica being removed from the group.\n- KB_LEAVE_MEMBER_POD_IP:
The IP address of the replica being removed from the group.\n\n\nExpected
container executing this action has access to following variables:\n\n\n-
KB_LEAVE_MEMBER_POD_FQDN: The pod name of the replica being
removed from the group.\n- KB_LEAVE_MEMBER_POD_NAME: The pod
name of the replica being removed from the group.\n\n\nExpected
action output:\n- On Failure: An error message, if applicable,
indicating why the action failed.\n\n\nFor example, to remove
an OBServer from an OceanBase Cluster in 'zone1', the following
command can be executed:\n\n\n```yaml\ncommand:\n- bash\n- -c\n-
|\n ADDRESS=$(KB_MEMBER_ADDRESSES%%,*)\n HOST=$(echo $ADDRESS
| cut -d ':' -f 1)\n PORT=$(echo $ADDRESS | cut -d ':' -f
2)\n CLIENT=\"mysql -u $KB_SERVICE_USER -p$KB_SERVICE_PASSWORD
-P $PORT -h $HOST -e\"\n\t $CLIENT \"ALTER SYSTEM DELETE SERVER
'$KB_LEAVE_MEMBER_POD_IP:$KB_SERVICE_PORT' ZONE 'zone1'\"\n```\n\n\nNote:
|\n CLIENT=\"mysql -u $SERVICE_USER -p$SERVICE_PASSWORD -P
$SERVICE_PORT -h $SERVICE_HOST -e\"\n\t $CLIENT \"ALTER SYSTEM
DELETE SERVER '$KB_POD_FQDN:$SERVICE_PORT' ZONE 'zone1'\"\n```\n\n\nNote:
This field is immutable once it has been set."
properties:
builtinHandler:
Expand Down Expand Up @@ -2913,9 +2895,6 @@ spec:


- KB_POD_FQDN: The FQDN of the replica pod whose role is being checked.
- KB_SERVICE_PORT: The port used by the database service.
- KB_SERVICE_USER: The username with the necessary permissions to interact with the database service.
- KB_SERVICE_PASSWORD: The corresponding password for KB_SERVICE_USER to authenticate with the database service.


Expected action output:
Expand Down Expand Up @@ -3265,9 +3244,6 @@ spec:


- KB_POD_FQDN: The FQDN of the replica pod whose role is being checked.
- KB_SERVICE_PORT: The port used by the database service.
- KB_SERVICE_USER: The username with the necessary permissions to interact with the database service.
- KB_SERVICE_PASSWORD: The corresponding password for KB_SERVICE_USER to authenticate with the database service.


Expected action output:
Expand Down Expand Up @@ -3956,13 +3932,10 @@ spec:
Without this, services that rely on roleSelectors might improperly direct traffic to wrong replicas.


The container executing this action has access to following environment variables:
The container executing this action has access to following variables:


- KB_POD_FQDN: The FQDN of the Pod whose role is being assessed.
- KB_SERVICE_PORT: The port used by the database service.
- KB_SERVICE_USER: The username with the necessary permissions to interact with the database service.
- KB_SERVICE_PASSWORD: The corresponding password for KB_SERVICE_USER to authenticate with the database service.


Expected output of this action:
Expand Down
Loading

0 comments on commit 44edc0d

Please sign in to comment.