How to configure WAL S3 archiving?

[yaroslavkasatikov]

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using?
  image: registry.opensource.zalan.do/acid/postgres-operator:v1.5.0

• Where do you run it - cloud or metal? Kubernetes or OpenShift?
  AWS. Openshift 4.5

• Are you running Postgres Operator in production?
  no

• Type of issue?
  question:

Hello team, I can't understand how to enable WAL S3 archiving and didn't find straight forward documentation for it.

Here is my steps:

I added into operatorconfigurations.acid.zalan.do this lines:

configuration:
  aws_or_gcp:
    aws_endpoint: s3.eu-west-1.amazonaws.com
    aws_region: eu-west-1
    wal_s3_bucket: zalando-bucketname

here is full output for my operatorconfgurations:

apiVersion: acid.zalan.do/v1                                                                                                                                                       [186/1894]
configuration:
  aws_or_gcp:
    aws_endpoint: s3.eu-west-1.amazonaws.com
    aws_region: eu-west-1
    wal_s3_bucket: zalando-bucketname
  connection_pooler:
    connection_pooler_default_cpu_limit: "1"
    connection_pooler_default_cpu_request: 500m
    connection_pooler_default_memory_limit: 100Mi
    connection_pooler_default_memory_request: 100Mi
    connection_pooler_image: registry.opensource.zalan.do/acid/pgbouncer:master-7
    connection_pooler_mode: transaction
    connection_pooler_number_of_instances: 2
  debug:
    debug_logging: true
    enable_database_access: true
  docker_image: registry.opensource.zalan.do/acid/spilo-12:1.6-p3
  etcd_host: ""
  kubernetes:
    cluster_domain: cluster.local
    cluster_labels:
      application: spilo
    cluster_name_label: cluster-name
    enable_init_containers: true
    enable_pod_antiaffinity: false
    enable_pod_disruption_budget: true
    enable_sidecars: true
    master_pod_move_timeout: 20m
    oauth_token_secret_name: postgresql-operator
    pdb_name_format: postgres-{cluster}-pdb
    pod_antiaffinity_topology_key: kubernetes.io/hostname
    pod_environment_configmap: zalando-operator/env
    pod_management_policy: ordered_ready
    pod_role_label: spilo-role
    pod_service_account_name: postgres-pod
    pod_terminate_grace_period: 1m
    secret_name_template: '{username}.{cluster}.credentials.{tprkind}.{tprgroup}'
    spilo_privileged: false
  kubernetes_use_configmaps: true
  load_balancer:
    enable_master_load_balancer: false
    enable_replica_load_balancer: false
    master_dns_name_format: '{cluster}.{team}.{hostedzone}'
    replica_dns_name_format: '{cluster}-repl.{team}.{hostedzone}'
  logging_rest_api:
    api_port: 8080
    cluster_history_entries: 1000
    ring_log_lines: 100
  logical_backup:
    logical_backup_docker_image: registry.opensource.zalan.do/acid/logical-backup:master-58
    logical_backup_s3_access_key_id: ******************
    logical_backup_s3_bucket: zalando-backup-bucketname
    logical_backup_s3_secret_access_key: *********************************
    logical_backup_s3_sse: AES256
    logical_backup_schedule: '*/2 * * * *'
  max_instances: -1
  min_instances: -1
  postgres_pod_resources:
    default_cpu_limit: "1"
    default_cpu_request: 100m
    default_memory_limit: 500Mi
    default_memory_request: 100Mi
  repair_period: 5m
  resync_period: 30m
  teams_api:
    enable_team_superuser: false
    enable_teams_api: false
    pam_role_name: zalandos
    protected_role_names:
    - admin
    team_admin_role: admin
    team_api_role_configuration:
      log_statement: all
  timeouts:
    pod_deletion_wait_timeout: 10m
    pod_label_wait_timeout: 10m
    ready_wait_interval: 4s
    ready_wait_timeout: 30s
    resource_check_interval: 3s
    resource_check_timeout: 10m
  users:
    replication_username: standby
    super_username: postgres
  workers: 4

I create a config map zalando-operator/env with the following content:

apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: **********************
  AWS_ENDPOINT: s3.eu-west-1.amazonaws.com
  AWS_INSTANCE_PROFILE: "true"
  AWS_REGION: eu-west-1
  AWS_SECRET_ACCESS_KEY: ********************************
  BACKUP_SCHEDULE: '*/2 * * * *'
  USE_WALG_BACKUP: "true"
kind: ConfigMap
metadata:
  creationTimestamp: "2020-08-26T21:02:41Z"
  managedFields:
  - apiVersion: v1
    operation: Update
    time: "2020-09-06T21:04:31Z"
  name: env
  namespace: zalando-operator

and all this variables have been applied and looped from pod:

# env|grep AWS
AWS_INSTANCE_PROFILE=true
AWS_ACCESS_KEY_ID=*******************
AWS_REGION=eu-west-1
AWS_ENDPOINT=s3.eu-west-1.amazonaws.com
AWS_SECRET_ACCESS_KEY=***************
# env|grep BACKUP
USE_WALG_BACKUP=true
BACKUP_SCHEDULE=*/2 * * * *

But... nothing happened, despite cluster is up and running.

Probably I missed something or did something incorrect. If there is a documentation for this point, please share it with me, but I haven't found it.

Many thanks,
Yaroslav

[Jan-M]

have you checked if the wal-e command is working?

have you checked if the backup cron job is executing?

log files?

error messages?

[memogarcia]

I'm interested in this topic as well, Is it a definitive documentation on how to accomplish this?

When I create a new cluster with WAL configuration, I'm expecting that the the WAL is being shipped but the reality is that the cluster try to bootstrap itself from a previous backup (which doesn't exits) and the creation fails.

The documentation on this topic is a little bit sparse.

[FxKu]

@yaroslavkasatikov your pod environment config map is missing some extra environment variables. I thought, I made this clear in the docs.

# env|grep AWS
AWS_INSTANCE_PROFILE=true
AWS_ACCESS_KEY_ID=*******************
AWS_REGION=eu-west-1
AWS_ENDPOINT=s3.eu-west-1.amazonaws.com
AWS_SECRET_ACCESS_KEY=***************
CLONE_AWS_ACCESS_KEY_ID=*******************
CLONE_AWS_REGION=eu-west-1
CLONE_AWS_ENDPOINT=s3.eu-west-1.amazonaws.com
CLONE_AWS_SECRET_ACCESS_KEY=***************
# env|grep BACKUP
USE_WALG_BACKUP=true
CLONE_USE_WALG_BACKUP=true
BACKUP_SCHEDULE=*/2 * * * *

[simonelbaz]

Hi,

I modified postgres-operator like shown above.

The pods have been deleted.

# env | grep AWS

returns nothing.

What am i missing ?

Thanks for your help

# grep configmap postgres-operator.yaml 
    pod_environment_configmap: default/pod-env-overrides
  kubernetes_use_configmaps: true
# cat pod-env-overrides.yaml 
apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: *****
  AWS_ENDPOINT: s3.GRA.cloud.ovh.net
  AWS_INSTANCE_PROFILE: "true"
  AWS_REGION: GRA
  AWS_SECRET_ACCESS_KEY: ****
  BACKUP_SCHEDULE: '*/2 * * * *'
  CLONE_AWS_ACCESS_KEY_ID: *****
  CLONE_AWS_ENDPOINT: *****
  CLONE_AWS_REGION: GRA
  CLONE_AWS_SECRET_ACCESS_KEY: *****
  CLONE_USE_WALG_BACKUP: "true"
  USE_WALG_BACKUP: "true"
  WAL_S3_BUCKET: pg-operator
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"AWS_ACCESS_KEY_ID":"****","AWS_ENDPOINT":"*****","AWS_INSTANCE_PROFILE":"true","AWS_REGION":"GRA","AWS_SECRET_ACCESS_KEY":"****","BACKUP_SCHEDULE":"*/2 * * * *","CLONE_AWS_ACCESS_KEY_ID":"******","CLONE_AWS_ENDPOINT":"******","CLONE_AWS_REGION":"GRA","CLONE_AWS_SECRET_ACCESS_KEY":"******","CLONE_USE_WALG_BACKUP":"true","USE_WALG_BACKUP":"true","WAL_S3_BUCKET":"pg-operator"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"pod-env-overrides","namespace":"default"}}
  creationTimestamp: "2022-02-15T14:05:49Z"
  name: pod-env-overrides
  namespace: default
  resourceVersion: "37860508697"
  uid: 24e37bc7-94e5-4a6c-a565-e0eb51ba9062

[devlifealways]

It

Hi,

I modified postgres-operator like shown above.

The pods have been deleted.

# env | grep AWS

returns nothing.

What am i missing ?

Thanks for your help

# grep configmap postgres-operator.yaml 
    pod_environment_configmap: default/pod-env-overrides
  kubernetes_use_configmaps: true
# cat pod-env-overrides.yaml 
apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: *****
  AWS_ENDPOINT: s3.GRA.cloud.ovh.net
  AWS_INSTANCE_PROFILE: "true"
  AWS_REGION: GRA
  AWS_SECRET_ACCESS_KEY: ****
  BACKUP_SCHEDULE: '*/2 * * * *'
  CLONE_AWS_ACCESS_KEY_ID: *****
  CLONE_AWS_ENDPOINT: *****
  CLONE_AWS_REGION: GRA
  CLONE_AWS_SECRET_ACCESS_KEY: *****
  CLONE_USE_WALG_BACKUP: "true"
  USE_WALG_BACKUP: "true"
  WAL_S3_BUCKET: pg-operator
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"AWS_ACCESS_KEY_ID":"****","AWS_ENDPOINT":"*****","AWS_INSTANCE_PROFILE":"true","AWS_REGION":"GRA","AWS_SECRET_ACCESS_KEY":"****","BACKUP_SCHEDULE":"*/2 * * * *","CLONE_AWS_ACCESS_KEY_ID":"******","CLONE_AWS_ENDPOINT":"******","CLONE_AWS_REGION":"GRA","CLONE_AWS_SECRET_ACCESS_KEY":"******","CLONE_USE_WALG_BACKUP":"true","USE_WALG_BACKUP":"true","WAL_S3_BUCKET":"pg-operator"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"pod-env-overrides","namespace":"default"}}
  creationTimestamp: "2022-02-15T14:05:49Z"
  name: pod-env-overrides
  namespace: default
  resourceVersion: "37860508697"
  uid: 24e37bc7-94e5-4a6c-a565-e0eb51ba9062

IMHO, this won't work, because of Spilo's configure_spilo python script, wich is triggered by launch.sh (Container's CMD); no matter what you do, at some point it will be overridden using a hard coded template, the only thing you can change is the region through AWS_REGION environment variable.

What I did was creating a new docker image with my won patch (changing template using other env variables with a fallback value to it's original form).

P.S: I know this is old, but who knows, it could help someone out there, and by the way I'm talking about 2.1-p5 tag only