Make it easier to enable/disable plugins without writing code and make it easier to add plugins

[shmuelk]

What would you like to be added:

1. The ability to enable/disable and configure plugins without touching the code. This should be done via a configuration yaml file or configuration yaml text (see how Envoy is configured)

2. The ability too add new plugins, without modifying existing code in the system.

Why is this needed:

1. In the existing system one needs to add/modify/remove various environment variables to configure the set of plugins that will be used by the scheduler. Sometimes one needs to change code for these sorts of efforts.

   The use of environment variables is not scalable as one needs to be careful about clashes and for scorers one needs two environment variables. Not to mention environment variables used to configure various plugins.

   A much better way would to use a configuration yaml file or configuration yaml text. The latter is very useful in K8S deployment yaml files, as the configuration can then be on-line without the use of a ConfigMap or something like that.

2. In the existing system to add a new plugin, in addition to writing the code one needs to modify the existing code that loads the scheduler config.

   A much better way would be to define a plugin registry that will contain a map of plugin name to plugin factory function.. The process would then be as follows:

   1. Every plugin in init() function will invoke the registry's static RegisterPlugin function.

   2. The main code reads in a configuration in which the user defines a set of references to configured plugins and a configuration that uses those plugin references. The purpose of the indirection is to support multiple concurrent scheduler profiles/configuration that share plugin instances if desired.

      The configuration file/text might look something like this:

      plugin_references:
      - name: "my-prefix-scorer"
        plugin: "prefix-cache"
        parameters:
           hash-block-size: 64
           max-prefix-blocks-to-match: 128
           lru-indexer-capacity: 50000
      - name: "my-load-filter"
         plugin: "load-aware-filter"
      
      scheduling_profiles:
      - name: default
         plugin-references:
         - name: "my-load-filter"
         - name: "my-prefix-scorer"
            weight: 20```
      

   I am proposing naming the set of scheduling plugins actually used as a scheduling profile, as that is a term has come up to the best of my knowledge in some of the re-architecting/refactoring work that is under way.

[shmuelk]

/assign

[ahg-g]

Regarding profiles, my view is to define "endpoint picking profiles", it is basically a named set of filters/scorers that an extension point named "ProfilePicker" selects from. With this, all forms of dynamically picking the profile depending on the request characteristics would now be modeled as a plugin that implements the profilePicker extension point.

[ahg-g]

For plugin-specific args, please take a look at how we did this for kube-scheduler, it is modeled as a raw object that the plugin itself knows how to parse (https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/apis/config/types.go#L207)

[shmuelk]

Thanks. My latest idea WRT to the parsing of the YAML was to have it be a Go json.RawMessage for the parameters field and pass it in that way to the plugin's factory function.

As far as calling these profiles "endpoint picking profiles", I need to think about somewhat, although I do like the idea of being to select the profile dynamically...

[elevran]

1. The ability to enable/disable and configure plugins without touching the code. This should be done via a configuration yaml file or configuration yaml text (see how Envoy is configured)

2. The ability too add new plugins, without modifying existing code in the system.

(1) can be done with a configuration file, as being proposed. Not all plugin's are the same (e.g., there can only be one picker in a profile. only scorers have weights, etc.), so we may want to define the configuration of a profile using separate sections (e.g., []filter, []{scorer, weight}, picker} and not a single []plugin.

(2) is a separate and orthogonal aspect. I would focus on (1) first and explore options for (2).

Configuration via some raw object that the plugin makes sense and keeps the configuration format relatively simple.

One aspect to highlight in @shmuelk suggestion is the ability to define plugin's outside the SchedulingProfile and then reference it inside the profile. This allows sharing the instance across different profiles and not creating a new instance for each profile (which should be the default).

[shmuelk]

To be clear about (2), I was talking about touching the existing code. This was not about the dynamic loading of plugins, perhaps using Golang plugins, which from experience is not the best idea.

Having said that, (2) is quite straight forward to implement,

We add a Registry (specifically plugins.Registry) which is a map of string to factory function. The factory function has the signature func(parameters json.RawMessage) (Plugin, error) (it returns error as there could be JSON parse errors). The parameters are passed in as json.RawMessage to defer the parsing of the parameters JSON to the factory function.

Each plugin simply adds an init() functions that calls the Registry plugin registration function to register itself.

[shmuelk]

(1) can be done with a configuration file, as being proposed. Not all plugin's are the same (e.g., there can only be one picker in a profile. only scorers have weights, etc.), so we may want to define the configuration of a profile using separate sections (e.g., []filter, []{scorer, weight}, picker} and not a single []plugin.

I have some issues with this:

1. It makes the configuration much more sensitive to name changes of the types of the plugins.
2. The user would be required to know that a specific plugin, for example the prefix aware plugin, is both a scorer and postSchedule plugin
3. As you mentioned in cases where one doesn't need to share plugins between profiles, one shouldn't need to use pluginReferences. If we explicitly separate the configuration by plugin type any plugin that spans types, will presumably also need to use plugin references.

You are correct, in that the code that "loads" the profiles, needs to validate there is only one picker, that weights have been specified for scorers, etc.

[ahg-g]

In kube-scheduler we supported both formats, configuring individual extension points and configuring plugins as a whole (with runtime code that detects and registers the plugin at the extension points it implements)

My recommendation is to adopt the plugin-focused format because from the user's perspective, the exact extensions that a plugin uses is an implementation detail of the plugin. Something like this:

endpointPickingProfiles
- prefill:
  - name: prefill-endpoints 
  - name: prefix-cache
     weight: 2
     args:
       LRUIndexerSize: 5000
  - name: server-queue-length
- decode
  - name: decode-endpoints
  - name: kv-cache-util

One thing we did in kube-scheduler config was allowing the user to disable extension points of a configured plugin to handle cases where a plugin implements more than on feature, like a prefix-cache plugin implementing score and filter, and we only want to enable score. I don't like that honestly, it complicates the configuration API and surfaces back the implementation detail of the plugin into the configuration. I think to address this, we should set a principle that a Plugin implements one feature.

[elevran]

... we should set a principle that a Plugin implements one feature.

Do we currently have examples of plugins that need to implement more than one hook?

I'm pretty sure some filters/scorers would also need response access (a different hook point, albeit not part of the scheduling profile cycle, so maybe a moot point).

[ahg-g]

Do we currently have examples of plugins that need to implement more than one hook?

For example prefix-aware scheduling implements a Score and PostSchedule extension points

[elevran]

For example prefix-aware scheduling implements a Score and PostSchedule extension points

I believe this would be better as Score and PostResponse as the target Pod is known in Post schedule, but no indication of the actual response context or even if the request actually succeeded.