Change API availability to forward compatibility and add UnEmulatable…

…Features.

Signed-off-by: Siyuan Zhang <sizhang@google.com>

keps/sig-architecture/4330-compatibility-versions/README.md

@@ -90,6 +90,7 @@ tags, and then generate with `hack/update-toc.sh`.
   - [Changes to Feature Gates](#changes-to-feature-gates)
     - [Feature Gate Lifecycles](#feature-gate-lifecycles)
     - [Feature gating changes](#feature-gating-changes)
+    - [Un-Emulatable Features](#un-emulatable-features)
   - [Validation ratcheting](#validation-ratcheting)
     - [CEL Environment Compatibility Versioning](#cel-environment-compatibility-versioning)
   - [StorageVersion Compatibility Versioning](#storageversion-compatibility-versioning)
@@ -562,6 +563,13 @@ func ClientFunction() {
 
 ```
 
+#### Un-Emulatable Features
+In very rare cases, if the feature implementation history could not be preserved in the code base with reasonable maintenance cost, a feature could opt out of the compatibility version through the `UnEmulatableFeatures` list. 
+
+The feature has to be off by default for the emulation version range less than the binary version (`1.{binaryMinorVersion-3}..1.{binaryMinorVersion-1}`). 
+
+If a feature in the `UnEmulatableFeatures` list is enabled and the `EmulationVersion != BinaryVersion`, the server would fail to start. 
+
 ### Validation ratcheting
 
 All validationg ratcheting needs to account for compatibility version.
@@ -610,21 +618,25 @@ The storage version of each group-resource is the newest
 
 ### API availability
 
-Similar to feature flags, all APIs group-versions declarations will be modified
+Ideally, similar to feature flags, all APIs group-versions declarations should be modified
 to track which Kubernetes version the API group-versions are introduced (or
-removed) at.
+removed) at. But in practice, that would make code changes of API graduation intractable. 
+For example, if a controller is switching from Beta to GA API at the binary version, a full emulation would mean the whole controller code needs to be
+duplicated to select from the Beta or GA API based on the emulation version every time the API is used. That means a lot of extra work, complicated testing rules, and high maintenance cost even for simple API graduations.
 
-GA APIs should match the exact set of APIs enabled in GA for the Kubernetes version
-the emulation version is set to.
+Instead, we are proposing to keep forward compatibility of all AIPs in compatibility version mode: 
+All APIs group-versions declarations would be modified
+to track which Kubernetes version the API group-versions are removed at. APIs introduced between the emulation version and 
+the binary version are still available if the stability level is no less than the APIs introduced before or at the emulation version and enabled at the emulation version.
 
-All Beta APIs (both off-by-default and on-by-default, if any of those
-still exist) need to behave exactly as they did for the Kubernetes version
-the emulation version is set to. I.e. `--runtime-config=api/<version>` needs
-to be able to turn on APIs exactly like it did for each Kubernetes version that
-emulation version can be set to.
+If an API has GAed and has not been removed at the emulated version, it would be enabled by default at the emulation version. If a newer stable version of the GA API has been introduced between the emulation version and the binary version, the new GA API would also be enabled at the emulation version along with the old GA API.
+
+If an API is Beta at the emulation version (meaning the Beta API has been introduced and has not been removed by the emulation version), it can be enabled by `--runtime-config=api/<version>` or it can be on-by-default at the emulation version. If a Beta API is enabled at the emulation version, and it has GAed between the emulation version and the binary version, its GA version would also be enabled at the emulation version.
 
 Alpha APIs may not be enabled in conjunction with emulation version.
 
+The forward compatibility of APIs should not affect data compatibility because storage version is still controlled by the `MinCompatibilityVersion` regardless of whether the data are created through future versions of the API endpoints. Webhooks should also work fine if the matching policy is `Equivalent`.
+
 ### API Field availability
 
 API fields that were introduced after the emulation version will **not** be
@@ -831,9 +843,9 @@ Alpha feature graduated to Beta|off-by-default|on-by-default|feature enabled onl
 Beta feature graduated to GA|on-by-default|on|feature enabled unless `--feature-gates=<feature>=false`|feature always enabled, feature gate may not be set
 Beta feature removed|on-by-default|-|feature enabled unless `--feature-gates=<feature>=false`|feature does not exist
 Alpha API introduced|-|off-by-default|API does not exist|alpha APIs may not be used in conjunction with emulation version
-Beta API graduated to GA|off-by-default|on|API available only when `--runtime-config=api/v1beta1=true`|API `api/v1` available
-Beta API removed|off-by-default|-|API available only when `--runtime-config=api/v1beta1=true`|API `api/v1beta1` does not exist
-on-by-default Beta API removed|on-by-default|-|API available unless `--runtime-config=api/v1beta1=false`|API `api/v1beta1` does not exist
+Beta API graduated to GA|off-by-default|on|API `api/v1beta1` and `api/v1` available only when `--runtime-config=api/v1beta1=true`|API `api/v1` available
+Beta API removed|off-by-default|-|API `api/v1beta1` available only when `--runtime-config=api/v1beta1=true`|API `api/v1beta1` does not exist
+on-by-default Beta API removed|on-by-default|-|API `api/v1beta1` available unless `--runtime-config=api/v1beta1=false`|API `api/v1beta1` does not exist
 API Storage version changed|v1beta1|v1|Resources stored as v1beta1|Resources stored as v1
 new CEL function|-|function in StoredExpressions CEL environment|CEL function does not exist|Resources already containing CEL expression can be evaluated
 introduced CEL function|function in StoredExpressions CEL environment|function in NewExpressions CEL environment|Resources already containing CEL expression can be evaluated|CEL expression can be written to resources and can be evaluted from storage