[v1.28] Secrets Encryption V3

[dereknola]

Proposed Changes

• Implements https://github.com/k3s-io/k3s/blob/master/docs/adrs/secrets-encryption-v3.md, a new command k3s secrets-encrypt rotate-keys is now Experimental, and adds, rotates, and reencrypts secrets all in one command.
• Adds additional protections around attempting to k3s secrets-encrypt enable on HA clusters that do not have --secrets-encryption server flags.
• Makes a new E2E test with the old test called secretsencryption_old (likely will remove old test in future PR)

Types of Changes

Feature Improvement

Verification

Use the new Secrets Encryption E2E test
OR

1. Start k3s cluster with --secrets-encryption
2. Make a bunch of secrets

   echo "this is a file" > file.txt && for i in {1..500}; do echo test$i >> file.txt; kubectl create secret generic test$i --from-file=file.txt; done
   

3. Run k3s secrets-encrypt rotate-keys
4. Watch the server logs, or wait for reencrypt_finished to be the state in k3s secrets-encrypt status
5. Restart all servers

Testing

Changed existing E2E test

Linked Issues

#7760

User-Facing Change

Further Comments

[codecov]

Codecov Report

Patch coverage: 46.08% and project coverage change: +3.96% 🎉

Comparison is base (7f58a1c) 47.51% compared to head (834c49c) 51.48%.
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8111      +/-   ##
==========================================
+ Coverage   47.51%   51.48%   +3.96%     
==========================================
  Files         144      144              
  Lines       14614    14709      +95     
==========================================
+ Hits         6944     7573     +629     
+ Misses       6577     5937     -640     
- Partials     1093     1199     +106     

Flag	Coverage Δ
e2etests	48.80% <49.46%> (?)
inttests	44.55% <20.86%> (-0.28%)	⬇️
unittests	19.86% <0.00%> (-0.08%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
pkg/daemons/control/deps/deps.go	59.07% <0.00%> (+0.63%)	⬆️
pkg/util/file.go	22.50% <23.80%> (+1.44%)	⬆️
pkg/server/secrets-encrypt.go	32.12% <40.00%> (+31.40%)	⬆️
pkg/cli/secretsencrypt/secrets_encrypt.go	47.34% <53.12%> (+2.57%)	⬆️
cmd/server/main.go	100.00% <100.00%> (ø)
pkg/cli/cmds/secrets_encrypt.go	100.00% <100.00%> (ø)
pkg/daemons/control/server.go	73.29% <100.00%> (+1.21%)	⬆️
pkg/secretsencrypt/config.go	54.03% <100.00%> (+48.38%)	⬆️

... and 36 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[brandond]

minor whitespace nits but LGTM otherwise.

Do we want to put in any checks to confirm that all the apiservers are on a version that supports this before using it, or are we OK just assuming that the user won't do anything boneheaded?

[dereknola]

Its a bit of a chicken-and-egg problem. We can't find out which version the apiserver is before its running, and we cant know to not supply the flag without the version. I think this is just going to have to be "document it well".

[brandond]

I mean, we know that the local apiserver will be running, I am thinking about other apiservers in the cluster if the user for some reason tries to do this on a mixed-version cluster where some of the nodes support hot reloading and some do not. But that's probably a corner case not worth worrying about.

[dereknola]

@brandond The new commit should cover the mixed-version edge case.

[dereknola]

Rebase with 1.28 PR now in master, E2E should now pass as the semver check for 1.28 will no longer gatekeep the feature.