Skip to content

K8SPG-437: merge upstream 5.4.2 changes #518

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 974 commits into from
Oct 5, 2023
Merged

K8SPG-437: merge upstream 5.4.2 changes #518

merged 974 commits into from
Oct 5, 2023

Conversation

pooknull
Copy link
Contributor

@pooknull pooknull commented Sep 28, 2023

K8SPG-437 Powered by Pull Request Badge

https://jira.percona.com/browse/K8SPG-437

DESCRIPTION

Problem:
We need to merge upstream changes.

Solution:
Merge the changes.

InstanceSidecars feature gate is disabled by default after the changes. This feature gate is necessary for PMM to work. This is why we should force it to be enabled, even if it is disabled in PGO_FEATURE_GATES env var.

CHECKLIST

Jira

  • Is the Jira ticket created and referenced properly?
  • Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
  • Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

  • Is an E2E test/test case added for the new feature/change?
  • Are unit tests added where appropriate?

Config/Logging/Testability

  • Are all needed new/changed options added to default YAML files?
  • Are the manifests (crd/bundle) regenerated if needed?
  • Did we add proper logging messages for operator actions?
  • Did we ensure compatibility with the previous version or cluster upgrade process?
  • Does the change support oldest and newest supported PG version?
  • Does the change support oldest and newest supported Kubernetes version?

benjaminjb and others added 30 commits April 27, 2022 14:54
TestReconcilerHandleDeleteNamespace was prone to flakes when run with
`envtest-existing`, and so is here replaced by a KUTTL test
with matching functionality.

Issue [sc-14273]
Adds better check logic to account for potential race conditions
that may be encountered in some environments due to delays in
garbage collection and ownership updating. Also fixed a comment
and harmonized filenames with existing patterns.
* Remove envtest-existing from upgradecheck

`envtest-existing` tests have been flaky and we are moving
towards KUTTL tests for e2e PostgresCluster behavior; several
tests in the `upgradecheck` package were originally written as
`envtest-existing` but are not really suitable as KUTTL tests,
so this PR changes them from `envtest-existing` to `envtest`

Issue [sc-14243]
This update allows the PGO controller image to be built without
CrunchyData specific RPMs. All existing make targets continue to
function in the same way as before, but the PGO controller image
no longer utilizes the base image. The base image is still used
by the Crunchy Postgres Exporter image.

Issue: [sc-14268]
This commit makes the following changes to the OLM bundle
generation logic:

- Update the version replacement value for OLM to 5.0.5
- Update the minimum supported Kubernetes version to 1.19
- Update logo files
- Update related images to exclude PG 12 and PG Upgrade (only in
   marketplace, removed to provide consistent images)
- Fix operator annotations for certified and marketplace
- Update README with information regarding issues encountered with
   5.1.0 bundles
- Update post bundle generation README instructions
- Update generation logic to match expected file, project and package
  names.
- Add a comment that minKubeVersion must support the related OCP
  version range.

Issue: [sc-13935]
As of Kubernetes v1.19, SecurityContext has a seccompProfile field
that can be set to RuntimeDefault to limit syscalls.

This PR adds that setting to the containers in order to
(a) limit syscalls from PGO-managed containers, while
(b) not preventing users from using other tools involving sidecars, etc.

Issue [sc-11286]
TestReconcileReplicaCreateBackup was flaking in envtest-existing
runs; experimentation revealed this was due to garbage collection.

Following current practice, this PR skips the test in envtest-existing
runs.

Issue [sc-14382]
Adds a script for updating the "monitoring" Kustomize installer in the
PGO examples repo using specific pgMonitor tag provided.

Issue: [sc-13611]
Issue: [sc-13940]
pgAdmin requires that the login username be formatted as an email. When
syncing PGO users with the pgAdmin database we add the `@pgo` suffix to
match this formatting. This change updates the documentation to match
this change.
This test creates two simple clusters with a single primary and a repo
host. In the first cluster we create data then increase the size of the
pvc. Then we check that the pvc size has changed, the size matches the
new expected side and the data is still present.

In the second cluster we attempt decrease the size of the volume and
expect the PersistentVolumeError.
* Pre-release update for v5.1.1

[sc-14408]
Revise update docs

(a) add note about potential automatic rollout of clusters when upgrading
(b) spin off separate upgrade section, with v4-v5 subsection
(c) tweak a little

Issue [sc-14467]
* updated from pg13 to pg14 in the update cluster instructions

* returned values to prior version to ensure images are present to
run k3d(s) tests
When migrating from v4 to v5, some legacy labels
may remain and cause unintended behavior. This PR
adds documentation around that issue and the manual
fix (done manually to avoid PGO having to remove
labels).

Issue [sc-14477]
* Revert "Enable seccomp on containers (#3193)"

This reverts commit 6193560.

* update Release notes
The releated images in the manager.yaml file now align with the related
images configured for OLM using related-images.yaml.

Issue: [sc-14517]
We do not set ownership on Patroni DCS Endpoints. These test should
verify that our controller is deleting them.

See: c13154e
PG 10 does not have stored procedures that support embedded transaction.
To get around this we use a bash and kubectl loop
Update general issue template to include necessary detail information
for incoming questions.

Issue [sc-14613]
The original implementation dynamically assigns functions that return
errors so we can swap them under test. Errors from these calls are
wrapped in sentinels so they can be identified at runtime. In practice,
however, these errors are never examined.

- Sentinel errors are removed. The "encoding/pem.Decode" function does
  not return errors, so we still generate our own in two places.

- All "Parse" functions are removed and replaced by their "Unmarshal"
  equivalents.

- Most "New" functions are removed. One remains to generate a fresh root
  CA certificate and private key pair.

- IP addresses are removed.

Fields on the "Certificate" and "PrivateKey" types are not exported,
making them opaque to consumers except for the PEM marshaling methods.
This provides a few benefits:

- The algorithms for keys and signatures can change without affecting
  callers.

- Certificates are parsed as they are generated and unmarshaled. Their
  values are always either zero or fully parsed.

- The root CA is parsed once per reconcile loop rather than once per
  leaf.

- Getter methods return copies so that certificate fields cannot change.

Issue: [sc-14620]
PostgreSQL, Patroni, pgBackRest, and PgBouncer all use certificates
through OpenSSL bindings. The format emitted by "MarshalText" is already
compatible with OpenSSL, so document that and add tests to enforce it.
tjmoore4 and others added 11 commits July 28, 2023 11:21
Adjusts the PGUpgrade logic to allow for easier recovery from a
missing image scenario. Specific Conditions are more clearly defined
and checking is added for the 'crunchy-upgrade' image.

A Kuttl test scenario is also added.

Issue: [sc-21130]
Move major-upgrade-missing-image test to e2e-other and create
shorter version, empty-image-upgrade.
gosec v2.17.0 detects more cases of pointers to loop variables.
Prior to 1.28.0, certain no-op server-side apply updates bumped
the resourceVersion value. For new Kubernetes versions this behavior
has been adjusted so that resourceVersion is not bumped.

This change adds an additional check for the server version to allow
the correct test to be executed.
The 'short' flag is now deprecated. The default output for
kubectl is now equivalent to the previous shortened output.

- https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.28.md#deprecation
…repo. Refactor postgres-operator to hold the setup.sql and queries.yml files used by the postgres_exporter. Add logic to postgres-operator to replace the functionality that was in the start.sh script that will be removed from the exporter image. Adjust testing accordingly.
@it-percona-cla
Copy link

it-percona-cla commented Sep 28, 2023

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 12 committers have signed the CLA.

✅ pooknull
❌ benjaminjb
❌ roberto-mello
❌ ValClarkson
❌ jmckulk
❌ tony-landreth
❌ dsessler7
❌ andrewlecuyer
❌ stemid
❌ tjmoore4
❌ dajeffers
❌ cbandy
You have signed the CLA already but the status is still pending? Let us recheck it.

@@ -19,7 +19,7 @@
directory=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[shfmt] reported by reviewdog 🐶

Suggested change
directory=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
directory=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)

@pooknull pooknull changed the title Merge upstream 5.4.2 changes K8SPG-437: merge upstream 5.4.2 changes Oct 2, 2023
@pooknull pooknull marked this pull request as ready for review October 2, 2023 17:02
Copy link
Contributor

@inelpandzic inelpandzic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pooknull this looks good, just address Ege's comments and we can merge it.

@JNKPercona
Copy link
Collaborator

Test name Status
demand-backup passed
init-deploy passed
monitoring passed
operator-self-healing passed
scaling passed
scheduled-backup passed
self-healing passed
start-from-backup passed
telemetry-transfer passed
users passed
We run 10 out of 10

commit: a5fc952
image: perconalab/percona-postgresql-operator:PR-518-a5fc9520a

@hors hors merged commit 0b7c3bf into main Oct 5, 2023
@hors hors deleted the dev/merge-upstream branch October 5, 2023 06:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.