K8SPG-833 add "custom-envs" e2e test #1315

nmarukovich · 2025-10-08T12:20:55Z

CHANGE DESCRIPTION

Problem:
Add E2E test to verify custom environment variables across all components (instance, pgBouncer, pgBackRest )

Since we modified several independent parts of the code, I added repeated checks for the instance, pgBouncer, and pgBackRest types.

What this test covers:

Add env
Update env
Add secret (envFrom)
Update vars inside secret
Update secret name in cr -> we need to update secret and start to use variable from new secret
Delete env
Delete secret.

connected PRs
#1231
#1306

Cause:
Short explanation of the root cause of the issue if applicable.

Solution:
Short explanation of the solution we are providing with this PR.

CHECKLIST

Jira

Is the Jira ticket created and referenced properly?
Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

Is an E2E test/test case added for the new feature/change?
Are unit tests added where appropriate?

Config/Logging/Testability

Are all needed new/changed options added to default YAML files?
Are all needed new/changed options added to the Helm Chart?
Did we add proper logging messages for operator actions?
Did we ensure compatibility with the previous version or cluster upgrade process?
Does the change support oldest and newest supported PG version?
Does the change support oldest and newest supported Kubernetes version?

eleo007

I believe we should also add check for backup and restore pods.

eleo007 · 2025-10-10T10:33:58Z

e2e-tests/tests/custom-envs/04-add-env-secret-to-instance.yaml

+          kubectl create secret generic instance-env-secret \
+            --from-literal=DB_USER_TEST_ENV=myuser \
+            --from-literal=DB_PASSWORD_TEST_ENV='MyS3cretP@ss' \
+            -n "${NAMESPACE}" || true


Maybe it is better to fail a test if secret is not created? Without secret - the next step will fail anyway?

eleo007 · 2025-10-10T10:34:48Z

e2e-tests/tests/custom-envs/04-add-env-secret-to-instance.yaml

+
+          sleep 10  
+          wait_cluster_consistency custom-envs
+          check_env_in_pod add instance DB_USER_TEST_ENV myuser


We add 2 env vars, why do we check only 1 of them?

Add 2 vars for env and check both

eleo007 · 2025-10-10T10:46:15Z

e2e-tests/tests/custom-envs/06-update-secret-for-instance.yaml

+  
+          sleep 10  
+          wait_cluster_consistency custom-envs
+          check_env_in_pod add instance DB_USER_NEW myuser2


the same, pls add hceck for the seond var

eleo007 · 2025-10-10T11:22:29Z

e2e-tests/tests/custom-envs/05-update-env-in-secret-for-instance.yaml

+      sleep 10  
+      wait_cluster_consistency custom-envs  
+        
+      check_env_in_pod add instance DB_NEW_ENV new-value


Add check that old variables stay

eleo007 · 2025-10-10T11:40:51Z

e2e-tests/tests/custom-envs/06-update-secret-for-instance.yaml

+          kubectl create secret generic instance-env-secret-updated \
+            --from-literal=DB_USER_NEW=myuser2 \
+            --from-literal=DB_PASSWORD_NEW=NewS3cretP@ss \
+            --dry-run=client -o yaml | kubectl -n "${NAMESPACE}" apply -f -


It would be nice to unify the way we create secrets

…ostgresql-operator into K8SPG-833_add_e2e_test

eleo007 · 2025-10-14T14:25:31Z

e2e-tests/tests/custom-envs/08-delete-secret-for-instance.yaml

+        
+              sleep 10 
+              wait_cluster_consistency custom-envs
+              check_envs_for_component delete instance DB_USER_NEW=myuser2 DB_PASSWORD_NEW=NewS3cretP@ss


Passing params as this to test ends up in false result. Add env var 'DB_USER_NEW=myuser2'. This is we are passing full var definition as var_name to check_env_in_pod function.

Fetching environment variables for instance pod ... var_name DB_USER_NEW=myuser2 actual_value OK: DB_USER_NEW=myuser2 deleted in All environment checks passed for component 'instance'

It was fixed.
3134d14

github-actions · 2025-10-27T07:31:11Z

e2e-tests/functions

+  local namespace=$1
+  local pod=$2
+  local container=$3


[shfmt] _{reported by reviewdog 🐶}

Suggested change

local namespace=$1

local pod=$2

local container=$3

local namespace=$1

local pod=$2

local container=$3

github-actions · 2025-10-27T07:31:11Z

e2e-tests/functions

+  if [ -n "$container" ]; then
+    kubectl exec -n "$namespace" "$pod" -c "$container" -- printenv 2>/dev/null || true
+  else
+    kubectl exec -n "$namespace" "$pod" -- printenv 2>/dev/null || true
+  fi


[shfmt] _{reported by reviewdog 🐶}

Suggested change

if [ -n "$container" ]; then

kubectl exec -n "$namespace" "$pod" -c "$container" -- printenv 2>/dev/null || true

else

kubectl exec -n "$namespace" "$pod" -- printenv 2>/dev/null || true

fi

if [ -n "$container" ]; then

kubectl exec -n "$namespace" "$pod" -c "$container" -- printenv 2>/dev/null || true

else

kubectl exec -n "$namespace" "$pod" -- printenv 2>/dev/null || true

fi

github-actions · 2025-10-27T07:31:11Z

e2e-tests/functions

+  local check_type=$1
+  local pod=$2
+  local var_name=$3
+  local expected_value=$4
+  local env_content=$5
+
+  local actual_value
+  actual_value=$(echo "$env_content" | grep -E "^${var_name}=" | cut -d'=' -f2- || true)
+
+  if [[ "$check_type" == "add" ]]; then
+    if [ "$actual_value" != "$expected_value" ]; then
+      echo "ERROR: $var_name in $pod — expected '$expected_value', got '${actual_value:-<missing>}'"
+      return 1
+    else
+      echo "OK: $var_name=$actual_value in $pod"
+    fi
+  elif [[ "$check_type" == "delete" ]]; then
+    if [ -n "$actual_value" ]; then
+      echo "ERROR: $var_name exists in $pod (should not exist)"
+      return 1
+    else
+      echo "OK: $var_name deleted in $pod"
+    fi
+  else
+    echo "ERROR: unknown check type '$check_type'"
+    return 1
+  fi


[shfmt] _{reported by reviewdog 🐶}

Suggested change

local check_type=$1

local pod=$2

local var_name=$3

local expected_value=$4

local env_content=$5

local actual_value

actual_value=$(echo "$env_content" | grep -E "^${var_name}=" | cut -d'=' -f2- || true)

if [[ "$check_type" == "add" ]]; then

if [ "$actual_value" != "$expected_value" ]; then

echo "ERROR: $var_name in $pod — expected '$expected_value', got '${actual_value:-<missing>}'"

return 1

else

echo "OK: $var_name=$actual_value in $pod"

fi

elif [[ "$check_type" == "delete" ]]; then

if [ -n "$actual_value" ]; then

echo "ERROR: $var_name exists in $pod (should not exist)"

return 1

else

echo "OK: $var_name deleted in $pod"

fi

else

echo "ERROR: unknown check type '$check_type'"

return 1

fi

local check_type=$1

local pod=$2

local var_name=$3

local expected_value=$4

local env_content=$5

local actual_value

actual_value=$(echo "$env_content" | grep -E "^${var_name}=" | cut -d'=' -f2- || true)

if [[ $check_type == "add" ]]; then

if [ "$actual_value" != "$expected_value" ]; then

echo "ERROR: $var_name in $pod — expected '$expected_value', got '${actual_value:-<missing>}'"

return 1

else

echo "OK: $var_name=$actual_value in $pod"

fi

elif [[ $check_type == "delete" ]]; then

if [ -n "$actual_value" ]; then

echo "ERROR: $var_name exists in $pod (should not exist)"

return 1

else

echo "OK: $var_name deleted in $pod"

fi

else

echo "ERROR: unknown check type '$check_type'"

return 1

fi

github-actions · 2025-10-27T07:31:11Z

e2e-tests/functions

+  local check_type=$1      # add | delete
+  local component=$2       # instance | pgbouncer | repohost
+  local vars=("${@:3}")    # everything after the 2nd argument
+
+  case "$component" in
+    instance)
+      POD=$(kubectl get -n "${NAMESPACE}" pod -l postgres-operator.crunchydata.com/instance-set=instance1 -o 'jsonpath={.items[0].metadata.name}')
+      CONTAINER=""
+      ;;
+    pgbouncer)
+      POD=$(kubectl get -n "${NAMESPACE}" pod -l postgres-operator.crunchydata.com/role=pgbouncer -o 'jsonpath={.items[0].metadata.name}')
+      CONTAINER="pgbouncer"
+      ;;
+    repohost)
+      POD=$(kubectl get -n "${NAMESPACE}" pod -l postgres-operator.crunchydata.com/data=pgbackrest -o 'jsonpath={.items[0].metadata.name}')
+      CONTAINER="pgbackrest"
+      ;;
+    *)
+      echo "ERROR: unknown component '$component'"
+      return 1
+      ;;
+  esac
+
+  echo "Fetching environment variables for $component pod $POD..."
+  local env_content
+  env_content=$(get_envs_from_pod "${NAMESPACE}" "$POD" "$CONTAINER")
+
+  local errors=0
+  for var_entry in "${vars[@]}"; do
+    if [[ "$check_type" == "add" ]]; then
+      local var_name="${var_entry%%=*}"
+      local var_expected="${var_entry#*=}"
+      check_env_in_pod add "$POD" "$var_name" "$var_expected" "$env_content" || errors=$((errors+1))
+    else
+      check_env_in_pod delete "$POD" "$var_entry" "" "$env_content" || errors=$((errors+1))
+    fi
+  done
+
+  if (( errors > 0 )); then
+    echo "$errors environment check(s) failed for component '$component'"
+    return 1
+  else
+    echo "All environment checks passed for component '$component'"
+  fi
+}


[shfmt] _{reported by reviewdog 🐶}

Suggested change

local check_type=$1 # add | delete

local component=$2 # instance | pgbouncer | repohost

local vars=("${@:3}") # everything after the 2nd argument

case "$component" in

instance)

POD=$(kubectl get -n "${NAMESPACE}" pod -l postgres-operator.crunchydata.com/instance-set=instance1 -o 'jsonpath={.items[0].metadata.name}')

CONTAINER=""

;;

pgbouncer)

POD=$(kubectl get -n "${NAMESPACE}" pod -l postgres-operator.crunchydata.com/role=pgbouncer -o 'jsonpath={.items[0].metadata.name}')

CONTAINER="pgbouncer"

;;

repohost)

POD=$(kubectl get -n "${NAMESPACE}" pod -l postgres-operator.crunchydata.com/data=pgbackrest -o 'jsonpath={.items[0].metadata.name}')

CONTAINER="pgbackrest"

;;

*)

echo "ERROR: unknown component '$component'"

return 1

;;

esac

echo "Fetching environment variables for $component pod $POD..."

local env_content

env_content=$(get_envs_from_pod "${NAMESPACE}" "$POD" "$CONTAINER")

local errors=0

for var_entry in "${vars[@]}"; do

if [[ "$check_type" == "add" ]]; then

local var_name="${var_entry%%=*}"

local var_expected="${var_entry#*=}"

check_env_in_pod add "$POD" "$var_name" "$var_expected" "$env_content" || errors=$((errors+1))

else

check_env_in_pod delete "$POD" "$var_entry" "" "$env_content" || errors=$((errors+1))

fi

done

if (( errors > 0 )); then

echo "$errors environment check(s) failed for component '$component'"

return 1

else

echo "All environment checks passed for component '$component'"

fi

}

local check_type=$1 # add | delete

local component=$2 # instance | pgbouncer | repohost

local vars=("${@:3}") # everything after the 2nd argument

case "$component" in

instance)

POD=$(kubectl get -n "${NAMESPACE}" pod -l postgres-operator.crunchydata.com/instance-set=instance1 -o 'jsonpath={.items[0].metadata.name}')

CONTAINER=""

;;

pgbouncer)

POD=$(kubectl get -n "${NAMESPACE}" pod -l postgres-operator.crunchydata.com/role=pgbouncer -o 'jsonpath={.items[0].metadata.name}')

CONTAINER="pgbouncer"

;;

repohost)

POD=$(kubectl get -n "${NAMESPACE}" pod -l postgres-operator.crunchydata.com/data=pgbackrest -o 'jsonpath={.items[0].metadata.name}')

CONTAINER="pgbackrest"

;;

*)

echo "ERROR: unknown component '$component'"

return 1

;;

esac

echo "Fetching environment variables for $component pod $POD..."

local env_content

env_content=$(get_envs_from_pod "${NAMESPACE}" "$POD" "$CONTAINER")

local errors=0

for var_entry in "${vars[@]}"; do

if [[ $check_type == "add" ]]; then

local var_name="${var_entry%%=*}"

local var_expected="${var_entry#*=}"

check_env_in_pod add "$POD" "$var_name" "$var_expected" "$env_content" || errors=$((errors + 1))

else

check_env_in_pod delete "$POD" "$var_entry" "" "$env_content" || errors=$((errors + 1))

fi

done

if ((errors > 0)); then

echo "$errors environment check(s) failed for component '$component'"

return 1

else

echo "All environment checks passed for component '$component'"

fi

}

…ostgresql-operator into K8SPG-833_add_e2e_test

JNKPercona · 2025-10-29T15:24:07Z

Test Name	Result	Time
backup-enable-disable	passed	00:00:00
custom-envs	passed	00:18:30
custom-extensions	passed	00:00:00
custom-tls	passed	00:00:00
database-init-sql	passed	00:00:00
demand-backup	passed	00:00:00
finalizers	passed	00:00:00
init-deploy	passed	00:00:00
monitoring	passed	00:00:00
monitoring-pmm3	passed	00:00:00
one-pod	passed	00:00:00
operator-self-healing	passed	00:00:00
pgvector-extension	passed	00:00:00
pitr	passed	00:00:00
scaling	passed	00:00:00
scheduled-backup	passed	00:00:00
self-healing	passed	00:11:50
sidecars	passed	00:00:00
start-from-backup	passed	00:00:00
tablespaces	passed	00:00:00
telemetry-transfer	passed	00:00:00
upgrade-consistency	passed	00:00:00
upgrade-minor	passed	00:00:00
users	passed	00:00:00

Summary	Value
Tests Run	24/24
Job Duration	00:53:35
Total Test Time	00:30:21

commit: ffcc65e
image: perconalab/percona-postgresql-operator:PR-1315-ffcc65e02

K8SPG-833 add e2e test

17d6f03

nmarukovich requested review from eleo007, jvpasinatto and valmiranogueira as code owners October 8, 2025 12:20

Merge branch 'main' into K8SPG-833_add_e2e_test

b6f6bd0

hors changed the title ~~K8SPG-833 add e2e test~~ K8SPG-833 add "custom-envs" e2e test Oct 8, 2025

nmarukovich added 2 commits October 9, 2025 15:26

Merge branch 'main' into K8SPG-833_add_e2e_test

413370d

Merge branch 'main' into K8SPG-833_add_e2e_test

a785e80

eleo007 requested changes Oct 10, 2025

View reviewed changes

eleo007 reviewed Oct 10, 2025

View reviewed changes

nmarukovich and others added 4 commits October 13, 2025 11:48

fix PR comments

6219010

Merge branch 'K8SPG-833_add_e2e_test' of github.com:percona/percona-p…

feab0ae

…ostgresql-operator into K8SPG-833_add_e2e_test

Merge branch 'main' into K8SPG-833_add_e2e_test

09c40b5

Merge branch 'main' into K8SPG-833_add_e2e_test

3e34cdb

eleo007 requested changes Oct 14, 2025

View reviewed changes

Merge branch 'main' into K8SPG-833_add_e2e_test

57eb63a

github-actions bot reviewed Oct 27, 2025

View reviewed changes

nmarukovich added 2 commits October 27, 2025 10:13

Fix PR comments

3134d14

Merge branch 'K8SPG-833_add_e2e_test' of github.com:percona/percona-p…

a82d412

…ostgresql-operator into K8SPG-833_add_e2e_test

nmarukovich requested a review from eleo007 October 27, 2025 09:20

Merge branch 'main' into K8SPG-833_add_e2e_test

ffcc65e

eleo007 approved these changes Oct 29, 2025

View reviewed changes

hors approved these changes Oct 29, 2025

View reviewed changes

hors merged commit e0ac59d into main Oct 29, 2025
18 checks passed

hors deleted the K8SPG-833_add_e2e_test branch October 29, 2025 16:00

K8SPG-833 add "custom-envs" e2e test #1315

K8SPG-833 add "custom-envs" e2e test #1315

Uh oh!

Conversation

nmarukovich commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CHANGE DESCRIPTION

CHECKLIST

Uh oh!

eleo007 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

JNKPercona commented Oct 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

nmarukovich commented Oct 8, 2025 •

edited

Loading