feat: add security to edgex via openziti #2603

dovholuknf · 2024-02-01T02:30:36Z

EdgexFoundry has elected to adopt OpenZiti for additional security. This is an initial attempt at what integrating github.com/openziti/ziti into ekuiper for EdgexFoundry might look like. (I tried my best to follow the commit guidelines, but I'm sure I didn't do something quite right, apologies in advance)

This PR adds functionality to ekuiper. It allows ekuiper to be removed from the underlay network if desired. In this PR the restPort is no longer available on the underlay (IP-based) network as it is exclusively accessible via OpenZiti if enabled.

Considerations/Notes:

Logging - OpenZiti uses pfxlog which uses the logrus.StandardLogger() and had to be adapted to in order to allow OpenZiti logging in the ekuiper logs for debugging
VaultSecret - edgex uses vault to deliver vault tokens to services. this vault token can be exchanged for a jwt which is then usable as authentication to the OpenZiti overlay network and other edgex services. A small VaultSecret was created to keep the vault token fresh and obtain new jwt's on a regular interval
http clients have been adapted to allow connecting/sending data over the OpenZiti overlay network if enabled using the environment variables EDGEX_CREDENTIALS, EDGEX_CREDENTIAL_NAME, OPENZITI_CONTROLLER. If this is not desired, I'm happy to rework it how you see fit.
http server ListenAndServe/ListenAndServeTLS are changed to Serve(listener)/ServeTLS(listener) respectively. http server is able to be taken off the IP-based underlay network entirely (making it unattackable by conventional IP-based attacks)

ngjaying · 2024-02-01T02:40:29Z

@dovholuknf Thanks. Please follow the Details link in DCO check to sign the commit.

dovholuknf · 2024-02-01T02:41:22Z

@dovholuknf Thanks. Please follow the Details link in DCO check to sign the commit.

Ahhh shoot. I forgot. Thanks

dovholuknf · 2024-02-01T02:49:55Z

I have fixed the DCO issue. Thanks

ngjaying

When do you expect to merge this? We may help if available.

internal/conf/logger/logger.go

internal/io/http/client.go

internal/conf/vault/vault.go

internal/server/server.go

dovholuknf · 2024-03-19T11:55:36Z

sorry i've been away from this for a long long time... i'm nearly through the edgex work... i am hoping to return to this this week or next week.

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

dovholuknf · 2024-03-26T19:39:28Z

This should be ready to re-review now. If you would like to see instructions on testing with edgex - just let me know and i can provide a set of docker steps here

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

internal/server/server_edgex.go

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

dovholuknf · 2024-03-27T03:28:50Z

~~the linter failed to run but it looks like it was some kind of error? can you maybe rerun it?~~ I found the errors appear to be red herrings. ran gofumpt...

also is the test case failure due to something i did? I'm surprised to see that based on what i added?


--- FAIL: TestExtensions (0.03s)
    mock_topo.go:392: Drop stream fails: ext is not found.
    mock_topo.go:392: Drop stream fails: ext2 is not found.
    plugin_rule_test.go:81: The test bucket size is 2.

i'll try to fix the 'max commit len' issue -- 58 chars at the moment? ;/

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

ngjaying · 2024-03-27T03:40:15Z

the linter failed to run but it looks like it was some kind of error? can you maybe rerun it?

also is the test case failure due to something i did? I'm surprised to see that based on what i added?
--- FAIL: TestExtensions (0.03s)
    mock_topo.go:392: Drop stream fails: ext is not found.
    mock_topo.go:392: Drop stream fails: ext2 is not found.
    plugin_rule_test.go:81: The test bucket size is 2.
i'll try to fix the 'max commit len' issue -- 58 chars at the moment? ;/

OK， we can try to fix lint and ut problems

dovholuknf · 2024-03-27T03:41:28Z

OK， we can try to fix lint and ut problems

much appreciated. i found the linter error by looking at the raw logs. the "UI view" just showed all the errors and confused me. i ran gofumpt and hopefully appeased the linter.

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

dovholuknf · 2024-03-27T03:53:20Z

i think i figured out how to run gci and gofumpt and hopefully the linter will be happy this time

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

dovholuknf · 2024-03-27T04:03:06Z

plugin was built with a different version of package github.com/lf-edge/ekuiper/internal/conf/logger (previous failure)

I do think I somehow broke that test now that i see this. I changed this file. I'm definitely going to ask for help figuring out how to fix that. Also thanks for pushing the 'approve' button on all the linter commits. I sure hope I have them all sorted now :(

ngjaying · 2024-03-27T04:09:22Z

plugin was built with a different version of package github.com/lf-edge/ekuiper/internal/conf/logger (previous failure)

I do think I somehow broke that test now that i see this. I changed this file. I'm definitely going to ask for help figuring out how to fix that. Also thanks for pushing the 'approve' button on all the linter commits. I sure hope I have them all sorted now :(

Sure, I'll take a look later today but not now since I am busy with other things in hand. Once you have a commit merged, the checks will be run without a need of approval.

copyright should be in the beginning of each file Signed-off-by: Jiyong Huang <huangjy@emqx.io>

Signed-off-by: Jiyong Huang <huangjy@emqx.io>

By default, it is close. To use with EdgeX secure mode, it should be turned on. Signed-off-by: Jiyong Huang <huangjy@emqx.io>

ngjaying · 2024-03-27T08:12:13Z

Build packages error

The default build has edgex tag, the build package check will run the built kuiperd to check. Then the caErr cause a panic in openziti.go AuthenicatedContext.
I guess there are some dependency services which are not existed in the test env. Is EdgeX still have secure and non-secure mode? Maybe we need to have a configuration in eKuiper to switch between secure and non-secure and default to non-secure to pass the tests. Thus, I make a commit for that to add enableOpenZiti config.

UT error

It is caused by the logger adaptor. It seems to me that the adaptor is unneccessary to apply to the logger globally. We can decorate it only when calling open ziti functions. Thus, I make a commit for that.

Now all checks are passed except the coverage is too low. Is it possible to add ut? (Looks hard to me because of external dependencies).

codecov · 2024-03-27T08:13:17Z

Codecov Report

Attention: Patch coverage is 5.11811% with 241 lines in your changes are missing coverage. Please review.

Project coverage is 63.17%. Comparing base (d5bbd74) to head (ef120d3).
Report is 12 commits behind head on master.

Files	Patch %	Lines
internal/edgex/vault.go	0.00%	138 Missing ⚠️
internal/edgex/logger_adaptor.go	0.00%	35 Missing ⚠️
internal/edgex/openziti.go	0.00%	31 Missing ⚠️
internal/io/http/client_edgex.go	23.08%	19 Missing and 1 partial ⚠️
internal/server/server_edgex.go	18.18%	9 Missing ⚠️
internal/server/server.go	0.00%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2603      +/-   ##
==========================================
- Coverage   63.49%   63.17%   -0.32%     
==========================================
  Files         339      345       +6     
  Lines       38932    39283     +351     
==========================================
+ Hits        24717    24816      +99     
- Misses      12046    12295     +249     
- Partials     2169     2172       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

internal/io/http/client_edgex.go

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

internal/io/http/client_edgex.go

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

ngjaying · 2024-03-28T01:35:23Z

@dovholuknf Thank you. All looks good now. Is it possible to add more unit tests? If not, I think we can merge for now. And we'll need doc updates in another PR.

dovholuknf · 2024-03-28T01:38:44Z

@dovholuknf Thank you. All looks good now. Is it possible to add more unit tests? If not, I think we can merge for now. And we'll need doc updates in another PR.

Thank you for the help! I'd really like to say yes to adding tests but I'm really strapped trying to get this and all the other edgex work out the door. i will put it on my backlog though and maybe in the coming weeks I can add them? Unit tests aren't probably too useful here other than maybe making sure configuration is set and defaults processed since it really needs an openziti overlay -- if that makes sense? I'm happy to look at it in the future, when I can get to it. Right now it's tight. :(

dovholuknf · 2024-03-28T01:39:33Z

I'll need help with what kind of doc and 'how/where' to doc. If you have any pointers on that, please let me know! Maybe make an issue and assign it to me? -- same for test now that i think about it?

dovholuknf · 2024-03-28T01:41:17Z

and one related question - will a docker container be produced from this repo that I could reference in the edgex project? sorry for all the comments...

ngjaying · 2024-03-28T02:43:01Z

and one related question - will a docker container be produced from this repo that I could reference in the edgex project? sorry for all the comments...

Yes, we can trigger an alpha release, that would produce a docker image.

ngjaying · 2024-03-28T02:43:55Z

I'll need help with what kind of doc and 'how/where' to doc. If you have any pointers on that, please let me know! Maybe make an issue and assign it to me? -- same for test now that i think about it?

OK, I'll create an issue for that.

dovholuknf · 2024-03-28T17:22:47Z

Yes, we can trigger an alpha release

that would be much appreciated. I looked but it doesn't seem like one exists yet. Should I use GH Discussions for this request? Add a new issue? or just keep commenting here? :) thanks again

ngjaying · 2024-03-29T00:45:40Z

Yes, we can trigger an alpha release

that would be much appreciated. I looked but it doesn't seem like one exists yet. Should I use GH Discussions for this request? Add a new issue? or just keep commenting here? :) thanks again

@dovholuknf https://github.com/lf-edge/ekuiper/releases/tag/v1.14.0-alpha.2 tagged. Discussions thread is OK for me.

dovholuknf force-pushed the add-openziti branch from dc735bb to 4d78b12 Compare February 1, 2024 02:49

ngjaying reviewed Feb 1, 2024

View reviewed changes

internal/conf/logger/logger.go Outdated Show resolved Hide resolved

internal/io/http/client.go Outdated Show resolved Hide resolved

dovholuknf commented Feb 1, 2024

View reviewed changes

internal/conf/vault/vault.go Outdated Show resolved Hide resolved

ngjaying reviewed Feb 1, 2024

View reviewed changes

internal/server/server.go Outdated Show resolved Hide resolved

dovholuknf added 2 commits March 26, 2024 15:31

fix: refactor edgex/openziti integration

3c3ab79

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: add edgex build tag

c7c46d0

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

dovholuknf force-pushed the add-openziti branch from 4d78b12 to c7c46d0 Compare March 26, 2024 19:35

fix: missing import

c28676a

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

dovholuknf commented Mar 27, 2024

View reviewed changes

internal/server/server_edgex.go Outdated Show resolved Hide resolved

dovholuknf added 3 commits March 26, 2024 20:41

fix: warn->info

63a7b9e

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: linter

8e81fdc

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: add copyrights from NetFoundry Inc.

024662f

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

dovholuknf changed the title ~~feat: initial PR for adding security to edgex via openziti~~ feat: add security to edgex via openziti Mar 27, 2024

dovholuknf added 2 commits March 26, 2024 23:32

fix: fix comment language

a685352

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: ran gofumpt -l -w .

6712858

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: ran gci

4fb5f64

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: let's hope the final linter commit

8394fb0

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

ngjaying added 2 commits March 27, 2024 15:13

chore: fix format

2c6858a

copyright should be in the beginning of each file Signed-off-by: Jiyong Huang <huangjy@emqx.io>

fix: adapt logger only in openziti init

616b3be

Signed-off-by: Jiyong Huang <huangjy@emqx.io>

feat(conf): add enableOpenZiti switch

85c3e45

By default, it is close. To use with EdgeX secure mode, it should be turned on. Signed-off-by: Jiyong Huang <huangjy@emqx.io>

ngjaying force-pushed the add-openziti branch from d277afd to 85c3e45 Compare March 27, 2024 08:02

ngjaying reviewed Mar 27, 2024

View reviewed changes

internal/io/http/client_edgex.go Show resolved Hide resolved

fix: allow fallback dialer for split tunnel

585f65a

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

dovholuknf commented Mar 27, 2024

View reviewed changes

internal/io/http/client_edgex.go Show resolved Hide resolved

fix: gofumpt client_edgex.go

ef120d3

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

ngjaying approved these changes Mar 28, 2024

View reviewed changes

ngjaying merged commit bf3b679 into lf-edge:master Mar 28, 2024
58 of 59 checks passed

dovholuknf deleted the add-openziti branch April 3, 2024 17:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add security to edgex via openziti #2603

feat: add security to edgex via openziti #2603

dovholuknf commented Feb 1, 2024

ngjaying commented Feb 1, 2024

dovholuknf commented Feb 1, 2024

dovholuknf commented Feb 1, 2024

ngjaying left a comment

dovholuknf commented Mar 19, 2024

dovholuknf commented Mar 26, 2024

dovholuknf commented Mar 27, 2024 •

edited

Loading

ngjaying commented Mar 27, 2024

dovholuknf commented Mar 27, 2024

dovholuknf commented Mar 27, 2024

dovholuknf commented Mar 27, 2024

ngjaying commented Mar 27, 2024

ngjaying commented Mar 27, 2024 •

edited

Loading

codecov bot commented Mar 27, 2024 •

edited

Loading

ngjaying commented Mar 28, 2024

dovholuknf commented Mar 28, 2024

dovholuknf commented Mar 28, 2024 •

edited

Loading

dovholuknf commented Mar 28, 2024

ngjaying commented Mar 28, 2024

ngjaying commented Mar 28, 2024

dovholuknf commented Mar 28, 2024

ngjaying commented Mar 29, 2024 •

edited

Loading

feat: add security to edgex via openziti #2603

feat: add security to edgex via openziti #2603

Conversation

dovholuknf commented Feb 1, 2024

ngjaying commented Feb 1, 2024

dovholuknf commented Feb 1, 2024

dovholuknf commented Feb 1, 2024

ngjaying left a comment

Choose a reason for hiding this comment

dovholuknf commented Mar 19, 2024

dovholuknf commented Mar 26, 2024

dovholuknf commented Mar 27, 2024 • edited Loading

ngjaying commented Mar 27, 2024

dovholuknf commented Mar 27, 2024

dovholuknf commented Mar 27, 2024

dovholuknf commented Mar 27, 2024

ngjaying commented Mar 27, 2024

ngjaying commented Mar 27, 2024 • edited Loading

Build packages error

UT error

codecov bot commented Mar 27, 2024 • edited Loading

Codecov Report

ngjaying commented Mar 28, 2024

dovholuknf commented Mar 28, 2024

dovholuknf commented Mar 28, 2024 • edited Loading

dovholuknf commented Mar 28, 2024

ngjaying commented Mar 28, 2024

ngjaying commented Mar 28, 2024

dovholuknf commented Mar 28, 2024

ngjaying commented Mar 29, 2024 • edited Loading

dovholuknf commented Mar 27, 2024 •

edited

Loading

ngjaying commented Mar 27, 2024 •

edited

Loading

codecov bot commented Mar 27, 2024 •

edited

Loading

dovholuknf commented Mar 28, 2024 •

edited

Loading

ngjaying commented Mar 29, 2024 •

edited

Loading