gRPC configuration management control plane API #10

markmandel · 2020-01-28T20:18:27Z

(Placeholder for further discussion).

We'll need some kind of API surface for dynamic configuration values. Most likely to provide the capabilities for people to create their own control planes.

The way Envoy does it is quite well tested, and we can likely lift many ideas from it.
https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol

(Not the only API Envoy has, may also be worth looking at the whole thing: https://www.envoyproxy.io/docs/envoy/latest/api/api)

markmandel · 2020-02-20T17:57:39Z

Came up in Slack - we may want to do two builds of Quilkin. One that allows the YAML file, and one that only allows gPRC. Because the client side proxy essentially can't be trusted. To be explored more.

markmandel · 2020-07-29T15:24:56Z

Envoy xDS API references:

luna-duclos · 2020-07-29T15:49:41Z

Further references on Envoy xDS:

https://github.com/envoyproxy/go-control-plane
https://github.com/envoyproxy/go-control-plane/blob/master/internal/example/README.md

markmandel · 2020-08-12T16:38:35Z

Takeaways I had from xDS api chat with had with Googler/xDS friends:

The current xDS implementation could likely work for much of our use case, but we would likely ignore most of it for our use case
Let's do research to work out what the smallest sliver of xDS we would need to use to support our configuration model
- Sounds like we could maybe start with just filters and clusters+locality+endpoints?
Come up with a back-of-napkin design of how integration with Agones/Open Match might look like, just to verify the model
Biggest question I still have is how we represent our auth token routing in an xDS world.

How does that sound to anyone?

Work on #10

* Add xds proto code generation Work on #10

This adds the logic to processes responses from the XDS server and sends back acknowlegements according to the protocol. Implementation to talk to the server isn't included. Work on #10

* Add XDS logic for cluster and endpoints This adds the logic to processes responses from the XDS server and sends back acknowlegements according to the protocol. Implementation to talk to the server isn't included. Work on #10 * include xds modules when running doctest * exclude xds cluster in rustdoc, refactor

Adds an abstraction that knows about the different types of clusters/endpoints sources - handling both from static configuration file or XDS configuration Work on #10

* Add a ClusterManager abstraction Adds an abstraction that knows about the different types of clusters/endpoints sources - handling both from static configuration file or XDS configuration Work on #10 Co-authored-by: Mark Mandel <markmandel@google.com>

Integrates the XDS client into the cluster manager Work on #10

An endpoint will have associated metadata - provided by the XDS API that can contain e.g tokens for that endpoint. Work on #10

This includes the endpoint metadata provided by the XDS API as a JSON object internally. Work on #10

* Add dynamic proxy configuration Integrates the XDS client into the cluster manager * Do not store endpoints per cluster * update clone depth * update proxy config Work on #10

This includes the endpoint metadata provided by the XDS API as a JSON object internally. Work on #10

This includes the endpoint metadata provided by the XDS API as a JSON object internally. Work on #10 Co-authored-by: Mark Mandel <markmandel@google.com>

Parses generic yaml key values into a json value and in order to use as the endpoint's metadata. Values under the `quilkin.dev` key is reserved for quilkin usage. Extracts token into the endpoint object since we don't want to traverse the json value on each lookup. Removes name and connection ID from the static Endpoint config struct. Work on #10 Fixes #152

Rather than retry. Also pass a struct with all resource handlers to the client rather than individually. This allows us to easily add the listener handler later. Work on #10

* Endpoint metadata support from static config Parses generic yaml key values into a json value and in order to use as the endpoint's metadata. Values under the `quilkin.dev` key is reserved for quilkin usage. Extracts token into the endpoint object since we don't want to traverse the json value on each lookup. Removes name and connection ID from the static Endpoint config struct. Work on #10 Fixes #152 * Rename token key

* xds fail fast on invalid URI Rather than retry. Also pass a struct with all resource handlers to the client rather than individually. This allows us to easily add the listener handler later. Work on #10

Adds filter manager logic to listen for LDS resources and construct filter chains from them. This required moving some common setup out of the current CDS listener. Does not update the proxy to use the filter chain from the filter manager - follow up PR will do this. Work on #10

* Add XDS Filter implementation Adds filter manager logic to listen for LDS resources and construct filter chains from them. This required moving some common setup out of the current CDS listener. Does not update the proxy to use the filter chain from the filter manager - follow up PR will do this. Work on #10 Co-authored-by: Mark Mandel <markmandel@google.com>

Enables the filter manager which can either be static or dynamic depending on config, in place of the static filter chain. Also remove the `filters` field from `dynamic` section of static configs since we can now fetch those from the XDS server. Work on #10

* Enable XDS filter manager Enables the filter manager which can either be static or dynamic depending on config, in place of the static filter chain. Also remove the `filters` field from `dynamic` section of static configs since we can now fetch those from the XDS server. Work on #10 Co-authored-by: Mark Mandel <markmandel@google.com>

work on #10

* Add proto config for all filters work on #10 * use proto3

Work on #10

* Add xDS integration test Work on #10 * fix compilation Co-authored-by: Mark Mandel <markmandel@google.com>

markmandel · 2021-05-05T22:45:13Z

This should be able to be closed now, yes?

iffyio · 2021-05-06T08:55:00Z

Yes!

markmandel mentioned this issue Feb 20, 2020

Design and Implementation of Filters #1

Closed

iffyio mentioned this issue Sep 6, 2020

Add XDS configuration support for clusters and endpoints #97

Closed

iffyio added a commit that referenced this issue Sep 17, 2020

Add xds proto code generation

b1a7c9c

Work on #10

iffyio mentioned this issue Sep 17, 2020

Add xds proto code generation #108

Merged

iffyio added a commit that referenced this issue Oct 14, 2020

Add xds proto code generation (#108)

5997d4d

* Add xds proto code generation Work on #10

iffyio mentioned this issue Oct 14, 2020

Add XDS logic for cluster and endpoints #112

Merged

iffyio added a commit that referenced this issue Nov 7, 2020

Add a ClusterManager abstraction

f48f67b

Adds an abstraction that knows about the different types of clusters/endpoints sources - handling both from static configuration file or XDS configuration Work on #10

iffyio added a commit that referenced this issue Nov 7, 2020

Add a ClusterManager abstraction

05d6659

Adds an abstraction that knows about the different types of clusters/endpoints sources - handling both from static configuration file or XDS configuration Work on #10

iffyio added a commit that referenced this issue Nov 7, 2020

Add a ClusterManager abstraction

49507eb

Adds an abstraction that knows about the different types of clusters/endpoints sources - handling both from static configuration file or XDS configuration Work on #10

iffyio mentioned this issue Nov 7, 2020

Add a ClusterManager abstraction #128

Merged

iffyio added a commit that referenced this issue Dec 3, 2020

Add dynamic proxy configuration

526de44

Integrates the XDS client into the cluster manager Work on #10

iffyio mentioned this issue Dec 3, 2020

Add dynamic proxy configuration #150

Merged

iffyio added a commit that referenced this issue Dec 5, 2020

Include metadata for XDS endpoints

11cf681

An endpoint will have associated metadata - provided by the XDS API that can contain e.g tokens for that endpoint. Work on #10

iffyio added a commit that referenced this issue Dec 5, 2020

Include metadata for XDS endpoints

6c8ea4e

This includes the endpoint metadata provided by the XDS API as a JSON object internally. Work on #10

iffyio mentioned this issue Dec 5, 2020

Include metadata for XDS endpoints #154

Merged

markmandel pushed a commit that referenced this issue Dec 7, 2020

Add dynamic proxy configuration (#150)

3654bea

* Add dynamic proxy configuration Integrates the XDS client into the cluster manager * Do not store endpoints per cluster * update clone depth * update proxy config Work on #10

iffyio added a commit that referenced this issue Dec 8, 2020

Include metadata for XDS endpoints

da3a3de

This includes the endpoint metadata provided by the XDS API as a JSON object internally. Work on #10

markmandel added a commit that referenced this issue Dec 11, 2020

Include metadata for XDS endpoints (#154)

03350f8

This includes the endpoint metadata provided by the XDS API as a JSON object internally. Work on #10 Co-authored-by: Mark Mandel <markmandel@google.com>

iffyio mentioned this issue Dec 14, 2020

Endpoint metadata support from static config #160

Merged

iffyio added a commit that referenced this issue Dec 14, 2020

xds fail fast on invalid URI

d636273

Rather than retry. Also pass a struct with all resource handlers to the client rather than individually. This allows us to easily add the listener handler later. Work on #10

iffyio mentioned this issue Dec 14, 2020

xds fail fast on invalid URI #161

Merged

iffyio mentioned this issue Jan 22, 2021

Add XDS Filter implementation #171

Merged

iffyio mentioned this issue Feb 11, 2021

Enable XDS filter manager #190

Merged

iffyio added a commit that referenced this issue Feb 15, 2021

Add proto config for all filters

793e7b0

work on #10

iffyio added a commit that referenced this issue Feb 15, 2021

Add proto config for all filters

400974f

work on #10

iffyio added a commit that referenced this issue Feb 15, 2021

Add proto config for all filters

21ac398

work on #10

iffyio mentioned this issue Feb 15, 2021

Add proto config for all filters #193

Merged

iffyio added a commit that referenced this issue Feb 20, 2021

Add proto config for all filters (#193)

d5bf22a

* Add proto config for all filters work on #10 * use proto3

iffyio added a commit that referenced this issue Mar 10, 2021

Add xDS integration test

a8d8462

Work on #10

iffyio mentioned this issue Mar 10, 2021

Add xDS integration test #210

Merged

iffyio added a commit that referenced this issue Mar 14, 2021

Add xDS integration test (#210)

66f5c09

* Add xDS integration test Work on #10 * fix compilation Co-authored-by: Mark Mandel <markmandel@google.com>

iffyio closed this as completed May 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gRPC configuration management control plane API #10

gRPC configuration management control plane API #10

markmandel commented Jan 28, 2020

markmandel commented Feb 20, 2020

markmandel commented Jul 29, 2020 •

edited

Loading

luna-duclos commented Jul 29, 2020 •

edited

Loading

markmandel commented Aug 12, 2020 •

edited

Loading

markmandel commented May 5, 2021

iffyio commented May 6, 2021

gRPC configuration management control plane API #10

gRPC configuration management control plane API #10

Comments

markmandel commented Jan 28, 2020

markmandel commented Feb 20, 2020

markmandel commented Jul 29, 2020 • edited Loading

luna-duclos commented Jul 29, 2020 • edited Loading

markmandel commented Aug 12, 2020 • edited Loading

markmandel commented May 5, 2021

iffyio commented May 6, 2021

markmandel commented Jul 29, 2020 •

edited

Loading

luna-duclos commented Jul 29, 2020 •

edited

Loading

markmandel commented Aug 12, 2020 •

edited

Loading