Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: enhance transformer #305

Merged
merged 10 commits into from
Nov 23, 2022
Merged

feat: enhance transformer #305

merged 10 commits into from
Nov 23, 2022

Conversation

xdlbdy
Copy link
Contributor

@xdlbdy xdlbdy commented Nov 15, 2022

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary

What is changed and how does it work?

  • add transform function
  • transformer define use in function and template format change from "${var}" to "<var>"

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  1. vsctl eventbus create --name xdl
  2. create subscription
vsctl subscription create \
    --eventbus xdl \
    --sink 'http://localhost:18080' \
    --transformer '{
        "define": {
          "v1":"123",
          "v2":"456",
          "id":"$.id"
        },
        "pipeline":[
            {"command":["join","$.testtime","-","$.data.year","$.data.month","$.data.day"]},
            {"command":["math_add","$.data.math_add","<v1>","<v2>"]},
            {"command":["replace","$.data.id","<id>"]},
            {"command":["create","$.data.source","$.source"]},
            {"command":["delete","$.data.year"]},
            {"command":["delete","$.data.month"]},
            {"command":["delete","$.data.day"]}
        ]
      }'
  1. put event
vsctl event put xdl \           
  --data '{"year":2022,"month":11,"day":16,"id":"id1"}' \
  --id "1nd" \
  --source "transformer"
  1. sink receive event
specversion: 1.0
type: cmd
source: transformer
id: 1nd
time: 2022-11-16T07:32:21.9804Z
datacontenttype: application/json,
testtime: 2022-11-16
xvanuseventbus: xdl
xvanusstime: 2022-11-16T07:32:21.987Z\n
data:{
  "id": "1nd",
  "math_add": 579,
  "source": "transformer"
}

  • No code

@xdlbdy xdlbdy marked this pull request as draft November 15, 2022 13:33
@codecov
Copy link

codecov bot commented Nov 16, 2022

Codecov Report

Merging #305 (1faf2c6) into main (09f3748) will increase coverage by 0.14%.
The diff coverage is 73.67%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #305      +/-   ##
==========================================
+ Coverage   62.72%   62.86%   +0.14%     
==========================================
  Files         133      143      +10     
  Lines       12128    12737     +609     
==========================================
+ Hits         7607     8007     +400     
- Misses       3977     4147     +170     
- Partials      544      583      +39     
Impacted Files Coverage Δ
internal/controller/trigger/controller.go 38.78% <0.00%> (-0.33%) ⬇️
internal/store/io/file_linux.go 0.00% <0.00%> (ø)
internal/primitive/transform/arg/arg.go 41.17% <41.17%> (ø)
internal/primitive/transform/arg/event.go 50.68% <50.68%> (ø)
internal/trigger/transform/transformer.go 66.66% <65.38%> (-33.34%) ⬇️
...ernal/controller/trigger/validation/subscripton.go 70.24% <65.51%> (-0.65%) ⬇️
internal/trigger/util/event.go 58.39% <67.79%> (ø)
...nternal/primitive/transform/action/regex_action.go 71.11% <71.11%> (ø)
internal/primitive/transform/arg/define.go 75.00% <75.00%> (ø)
internal/primitive/transform/action/action.go 75.91% <75.91%> (ø)
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c87a1f...1faf2c6. Read the comment docs.

@xdlbdy xdlbdy marked this pull request as ready for review November 16, 2022 07:46
internal/controller/trigger/validation/subscripton.go Outdated Show resolved Hide resolved
internal/controller/trigger/validation/subscripton.go Outdated Show resolved Hide resolved
internal/primitive/transform/action/action.go Outdated Show resolved Hide resolved
internal/primitive/transform/arg/event.go Show resolved Hide resolved
internal/primitive/transform/arg/event.go Show resolved Hide resolved
internal/primitive/transform/arg/event.go Outdated Show resolved Hide resolved
internal/primitive/transform/arg/event.go Outdated Show resolved Hide resolved
@wenfengwang
Copy link
Contributor

Could you add at least one benchmark test for different string/regex/struct/time/math functions?

@xdlbdy
Copy link
Contributor Author

xdlbdy commented Nov 22, 2022

Could you add at least one benchmark test for different string/regex/struct/time/math functions?

goos: darwin
goarch: arm64
pkg: github.com/linkall-labs/vanus/internal/primitive/transform/action
BenchmarkAction
BenchmarkAction/delete
BenchmarkAction/delete-10 3875511 308.4 ns/op
BenchmarkAction/create
BenchmarkAction/create-10 1348846 889.0 ns/op
BenchmarkAction/replace
BenchmarkAction/replace-10 1889361 634.3 ns/op
BenchmarkAction/move
BenchmarkAction/move-10 670744 1634 ns/op
BenchmarkAction/rename
BenchmarkAction/rename-10 935678 1269 ns/op
BenchmarkAction/math_add
BenchmarkAction/math_add-10 1528915 799.0 ns/op
BenchmarkAction/math_sub
BenchmarkAction/math_sub-10 1506090 850.9 ns/op
BenchmarkAction/math_mul
BenchmarkAction/math_mul-10 1464643 765.0 ns/op
BenchmarkAction/math_div
BenchmarkAction/math_div-10 1580599 766.1 ns/op
BenchmarkAction/format_unix_time
BenchmarkAction/format_unix_time-10 942679 1256 ns/op
BenchmarkAction/format_date
BenchmarkAction/format_date-10 719745 1646 ns/op
BenchmarkAction/join
BenchmarkAction/join-10 1683962 717.5 ns/op
BenchmarkAction/upper_case
BenchmarkAction/upper_case-10 1815895 657.3 ns/op
BenchmarkAction/lower_case
BenchmarkAction/lower_case-10 1821562 657.7 ns/op
BenchmarkAction/add_prefix
BenchmarkAction/add_prefix-10 1773424 678.6 ns/op
BenchmarkAction/add_suffix
BenchmarkAction/add_suffix-10 1778683 695.2 ns/op
BenchmarkAction/replace_with_regex
BenchmarkAction/replace_with_regex-10 1271424 923.3 ns/op

Signed-off-by: delu <delu.xu@linkall.com>
Signed-off-by: delu <delu.xu@linkall.com>
Signed-off-by: delu <delu.xu@linkall.com>
Signed-off-by: delu <delu.xu@linkall.com>
Signed-off-by: delu <delu.xu@linkall.com>
Signed-off-by: delu <delu.xu@linkall.com>
Signed-off-by: delu <delu.xu@linkall.com>
Signed-off-by: delu <delu.xu@linkall.com>
Signed-off-by: delu <delu.xu@linkall.com>
Signed-off-by: delu <delu.xu@linkall.com>
Copy link
Contributor

@wenfengwang wenfengwang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wenfengwang wenfengwang merged commit d0eb300 into vanus-labs:main Nov 23, 2022
@xdlbdy xdlbdy deleted the transformer branch November 24, 2022 11:07
wenfengwang pushed a commit that referenced this pull request Mar 23, 2023
* feat: enhance transformer

Signed-off-by: delu <delu.xu@linkall.com>

* feat: transfrom function

Signed-off-by: delu <delu.xu@linkall.com>

* feat: transfrom function

Signed-off-by: delu <delu.xu@linkall.com>

* fix: fix golangci

Signed-off-by: delu <delu.xu@linkall.com>

* fix: fix ut

Signed-off-by: delu <delu.xu@linkall.com>

* fix: fix golangci

Signed-off-by: delu <delu.xu@linkall.com>

* fix: code review

Signed-off-by: delu <delu.xu@linkall.com>

* fix: code review

Signed-off-by: delu <delu.xu@linkall.com>

* test: add action benchmark

Signed-off-by: delu <delu.xu@linkall.com>

* fix: golangci-lint error

Signed-off-by: delu <delu.xu@linkall.com>

Signed-off-by: delu <delu.xu@linkall.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants