feat: add file resource provider for EG standalone mode #3159

shawnh2 · 2024-04-10T04:06:53Z

What type of PR is this?

What this PR does / why we need it:

This PR implement two component for file resource provider: Notifier and ResourceStore.

Notifier monitors paths(files or dirs) on host via fsnotify, and send Write/Remove event to ResourceStore
ResourceStore load and store resources according to the received event

Which issue(s) this PR fixes:

related #1393

There're some TODOs in this PR, tracked by #3213 #3207

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

… provider Signed-off-by: shawnh2 <shawnhxh@outlook.com>

…rovider

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

…rovider

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

zhaohuabing

As discussed in today's meeting, I prefer to remove the EnvoyGatewayCustomProvider and add a flat EnvoyGatewayFileProvider(or a better name).

arkodg · 2024-04-24T00:24:43Z

As discussed in today's meeting, I prefer to remove the EnvoyGatewayCustomProvider and add a flat EnvoyGatewayFileProvider(or a better name).

Custom is needed to allow for different resource and infra provider

zhaohuabing · 2024-04-24T00:30:53Z

As discussed in today's meeting, I prefer to remove the EnvoyGatewayCustomProvider and add a flat EnvoyGatewayFileProvider(or a better name).

Custom is needed to allow for different resource and infra provider

Add it later when a use case comes out? The File/Host Provider is not a custom provider, it's a known one.

arkodg · 2024-04-24T00:47:36Z

As discussed in today's meeting, I prefer to remove the EnvoyGatewayCustomProvider and add a flat EnvoyGatewayFileProvider(or a better name).

Custom is needed to allow for different resource and infra provider

Add it later when a use case comes out? The File/Host Provider is not a custom provider, it's a known one.

use case exists today - file resource provider and host infra provider to implement standalone case, running EG in a docker container or bare metal directly w/o any API server

Xunzhuo · 2024-05-14T08:26:05Z

/retest

Xunzhuo

Do we plan on releasing this in v1.1.0 ?

cc @arkodg

arkodg · 2024-05-15T00:39:47Z

Do we plan on releasing this in v1.1.0 ?

cc @arkodg

yeah I'm a +1 for this, prioritizing other PRs right now, will get to this one soon

shawnh2 · 2024-06-04T03:42:47Z

will split changes of api/v1alpha1/validation/envoygateway_validate.go into a separate PR, since it's only a refactor and very easily to conflict.

github-actions · 2024-07-04T04:02:11Z

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. Please feel free to give a status update now, ping for review, when it's ready. Thank you for your contributions!

sky92zwq · 2024-07-30T11:25:26Z

Hi, does it set -c or --config-path to switch to file-mode? how to implement the config file?

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

codecov · 2024-08-13T14:53:44Z

Codecov Report

Attention: Patch coverage is 7.87270% with 550 lines in your changes missing coverage. Please review.

Project coverage is 66.54%. Comparing base (c109131) to head (3c5d0f1).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/provider/file/resources.go	0.00%	234 Missing ⚠️
internal/provider/file/notifier.go	0.00%	158 Missing ⚠️
internal/provider/file/file.go	0.00%	66 Missing ⚠️
internal/provider/runner/runner.go	0.00%	32 Missing ⚠️
internal/provider/file/store.go	0.00%	26 Missing ⚠️
internal/cmd/server.go	0.00%	15 Missing ⚠️
internal/infrastructure/manager.go	0.00%	9 Missing ⚠️
internal/infrastructure/runner/runner.go	0.00%	6 Missing ⚠️
internal/provider/file/path.go	88.88%	1 Missing and 1 partial ⚠️
internal/provider/kubernetes/kubernetes.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3159      +/-   ##
==========================================
- Coverage   67.99%   66.54%   -1.45%     
==========================================
  Files         190      195       +5     
  Lines       23130    23699     +569     
==========================================
+ Hits        15727    15771      +44     
- Misses       6284     6809     +525     
  Partials     1119     1119

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

api/v1alpha1/validation/envoygateway_validate.go

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

api/v1alpha1/envoygateway_types.go

sky92zwq · 2024-08-27T09:25:57Z

internal/provider/file/resources.go

+// - This issue is tracked by https://github.com/envoyproxy/gateway/issues/3207
+//
+// convertKubernetesYAMLToResources converts a Kubernetes YAML string into GatewayAPI Resources.
+func convertKubernetesYAMLToResources(str string) (*gatewayapi.Resources, error) {


should deal with ReferenceGrant type resource in this func?

Yes, it should. This PR only leverage the basic functionality for file-provider, as described in the comments here, this function will be revisited along with the one in egctl x translate.

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

shawnh2 · 2024-09-08T09:10:52Z

Hi, does it set -c or --config-path to switch to file-mode? how to implement the config file?

You can start file-provider mode like envoy-gateway server -c config.yaml, with config.yaml:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
gateway:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
provider:
  type: Custom
  custom:
    resource:
      type: File
      file:
        paths: ["test/"]
logging:
  level:
    default: info

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

…rovider

shawnh2 · 2024-09-08T12:42:50Z

/retest

arkodg

thanks !

Xunzhuo

LGTM. Thanks

) * add validations for envoy-gateway file resource type Signed-off-by: shawnh2 <shawnhxh@outlook.com> * improve eg validation and add resource provider interface for various provider Signed-off-by: shawnh2 <shawnhxh@outlook.com> * extract common gatewayapi layer translate logic in egctl translate Signed-off-by: shawnh2 <shawnhxh@outlook.com> * add notifier support Signed-off-by: shawnh2 <shawnhxh@outlook.com> * fix lint and move read yaml bytes function back to translate Signed-off-by: shawnh2 <shawnhxh@outlook.com> * add resources store support Signed-off-by: shawnh2 <shawnhxh@outlook.com> * fix lint Signed-off-by: shawnh2 <shawnhxh@outlook.com> * fix ci Signed-off-by: shawnh2 <shawnhxh@outlook.com> * update infra provider api and address comments Signed-off-by: shawnh2 <shawnhxh@outlook.com> * update custom provider comments and validate method test Signed-off-by: shawnh2 <shawnhxh@outlook.com> * restore extension manager and add health probe server for file provider Signed-off-by: shawnh2 <shawnhxh@outlook.com> * update envoy gateway helper functions Signed-off-by: shawnh2 <shawnhxh@outlook.com> * add some unit tests Signed-off-by: shawnh2 <shawnhxh@outlook.com> * properly handle the remove event for the file provider Signed-off-by: shawnh2 <shawnhxh@outlook.com> * fix lint Signed-off-by: shawnh2 <shawnhxh@outlook.com> * no default to k8s for infra provider Signed-off-by: shawnh2 <shawnhxh@outlook.com> * fix runner Signed-off-by: shawnh2 <shawnhxh@outlook.com> --------- Signed-off-by: shawnh2 <shawnhxh@outlook.com> Co-authored-by: Xunzhuo <bitliu@tencent.com> Signed-off-by: NCSham <sudoer@ncsham.in>

shawnh2 added 8 commits April 1, 2024 21:27

add validations for envoy-gateway file resource type

93e4dd3

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

improve eg validation and add resource provider interface for various…

3956911

… provider Signed-off-by: shawnh2 <shawnhxh@outlook.com>

Merge branch 'main' of github.com:envoyproxy/gateway into host-file-p…

9acf325

…rovider

extract common gatewayapi layer translate logic in egctl translate

7de91b3

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

add notifier support

3419af6

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

fix lint and move read yaml bytes function back to translate

e709eae

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

add resources store support

6f3b400

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

Merge branch 'main' of github.com:envoyproxy/gateway into host-file-p…

739c8dc

…rovider

shawnh2 marked this pull request as ready for review April 18, 2024 07:09

shawnh2 requested a review from a team as a code owner April 18, 2024 07:09

fix lint

5726ee5

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

zhaohuabing reviewed Apr 24, 2024

View reviewed changes

Xunzhuo reviewed May 14, 2024

View reviewed changes

github-actions bot added the stale label Jul 4, 2024

Xunzhuo removed the stale label Jul 25, 2024

arkodg added this to the v1.2.0-rc1 milestone Aug 1, 2024

shawnh2 mentioned this pull request Aug 4, 2024

refactor: clean envoygateway validate methods #3997

Merged

shawnh2 added 2 commits August 13, 2024 22:01

resolve conflicts

9460667

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

fix ci

2e1db1b

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

arkodg reviewed Aug 15, 2024

View reviewed changes

api/v1alpha1/validation/envoygateway_validate.go Outdated Show resolved Hide resolved

arkodg reviewed Aug 15, 2024

View reviewed changes

api/v1alpha1/validation/envoygateway_validate.go Outdated Show resolved Hide resolved

shawnh2 added 5 commits August 24, 2024 20:13

update custom provider comments and validate method test

21353e2

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

restore extension manager and add health probe server for file provider

59e9c59

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

update envoy gateway helper functions

be44b88

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

add some unit tests

ec357fa

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

properly handle the remove event for the file provider

4052618

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

shawnh2 dismissed stale reviews from zhaohuabing and arkodg via 4052618 August 25, 2024 08:21

shawnh2 removed the hold do not merge label Aug 25, 2024

shawnh2 requested review from zhaohuabing, arkodg, cnvergence and a team August 25, 2024 08:24

fix lint

381efd6

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

arkodg reviewed Aug 26, 2024

View reviewed changes

api/v1alpha1/envoygateway_types.go Outdated Show resolved Hide resolved

sky92zwq reviewed Aug 27, 2024

View reviewed changes

no default to k8s for infra provider

a30b1ea

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

shawnh2 requested a review from arkodg September 8, 2024 09:09

shawnh2 added 2 commits September 8, 2024 18:21

fix runner

a635d03

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

Merge branch 'main' of github.com:envoyproxy/gateway into host-file-p…

61638ee

…rovider

arkodg approved these changes Sep 9, 2024

View reviewed changes

arkodg requested review from a team September 9, 2024 19:03

Merge branch 'main' into host-file-provider

3c5d0f1

Xunzhuo approved these changes Sep 11, 2024

View reviewed changes

Xunzhuo merged commit f5693f6 into envoyproxy:main Sep 11, 2024
13 checks passed

shawnh2 deleted the host-file-provider branch September 11, 2024 02:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add file resource provider for EG standalone mode #3159

feat: add file resource provider for EG standalone mode #3159

shawnh2 commented Apr 10, 2024 •

edited

Loading

zhaohuabing left a comment •

edited

Loading

arkodg commented Apr 24, 2024

zhaohuabing commented Apr 24, 2024 •

edited

Loading

arkodg commented Apr 24, 2024

Xunzhuo commented May 14, 2024

Xunzhuo left a comment

arkodg commented May 15, 2024

shawnh2 commented Jun 4, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024

sky92zwq commented Jul 30, 2024

codecov bot commented Aug 13, 2024 •

edited

Loading

sky92zwq Aug 27, 2024

shawnh2 Sep 8, 2024

shawnh2 commented Sep 8, 2024

shawnh2 commented Sep 8, 2024

arkodg left a comment

Xunzhuo left a comment

feat: add file resource provider for EG standalone mode #3159

feat: add file resource provider for EG standalone mode #3159

Conversation

shawnh2 commented Apr 10, 2024 • edited Loading

zhaohuabing left a comment • edited Loading

Choose a reason for hiding this comment

arkodg commented Apr 24, 2024

zhaohuabing commented Apr 24, 2024 • edited Loading

arkodg commented Apr 24, 2024

Xunzhuo commented May 14, 2024

Xunzhuo left a comment

Choose a reason for hiding this comment

arkodg commented May 15, 2024

shawnh2 commented Jun 4, 2024 • edited Loading

github-actions bot commented Jul 4, 2024

sky92zwq commented Jul 30, 2024

codecov bot commented Aug 13, 2024 • edited Loading

Codecov Report

sky92zwq Aug 27, 2024

Choose a reason for hiding this comment

shawnh2 Sep 8, 2024

Choose a reason for hiding this comment

shawnh2 commented Sep 8, 2024

shawnh2 commented Sep 8, 2024

arkodg left a comment

Choose a reason for hiding this comment

Xunzhuo left a comment

Choose a reason for hiding this comment

shawnh2 commented Apr 10, 2024 •

edited

Loading

zhaohuabing left a comment •

edited

Loading

zhaohuabing commented Apr 24, 2024 •

edited

Loading

shawnh2 commented Jun 4, 2024 •

edited

Loading

codecov bot commented Aug 13, 2024 •

edited

Loading