Skip to content

Commit

Permalink
[8.15](backport #39796) Upgrade azure-eventhub to the new Event Hub S…
Browse files Browse the repository at this point in the history 
…DK (#40455)

* Upgrade azure-eventhub to the new Event Hub SDK (#39796)

Restructure the `azure-eventhub` input, rebranding the current version as processor v1. Add a brand new processor v2, allowing users to select which version to use in the config:

- processor v1: uses the [legacy](https://github.com/azure/azure-event-hubs-go) Event Hub SDK (default processor, at least for 8.15)
- processor v2: uses the [modern](https://github.com/azure/azure-sdk-for-go/blob/main/sdk/messaging/azeventhubs/) Event Hub SDK

Why are we introducing a processor v2?

- processor v1 uses deprecated libraries
  - [github.com/Azure/azure-event-hubs-go](http://github.com/Azure/azure-event-hubs-go) (legacy)
  - [github.com/Azure/azure-storage-blob-go](http://github.com/Azure/azure-storage-blob-go) (legacy, [retiring](https://azure.microsoft.com/en-gb/updates/retirement-notice-the-legacy-azure-storage-go-client-libraries-will-be-retired-on-13-september-2024/) on Sep 2024)
- processor v1 does not support publishing acks (mostly due to lack of hooks; the legacy SDK is a black box)

---------

Co-authored-by: Tiago Queiroz <me@tiago.life>
Co-authored-by: Andrew Gizas <andreas.gkizas@elastic.co>
Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com>
  • Loading branch information
@mergify @belimawr
@gizas @shmsr
4 people authored Aug 12, 2024
1 parent cebcf1b commit c3c40c6
Show file tree
Hide file tree
Showing 25 changed files with 2,396 additions and 303 deletions.
105 changes: 79 additions & 26 deletions NOTICE.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1742,11 +1742,11 @@ SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/azcore
Version: v1.12.0
Version: v1.13.0
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.12.0/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.13.0/LICENSE.txt:

MIT License

Expand Down Expand Up @@ -1802,6 +1802,37 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE


--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/messaging/azeventhubs
Version: v1.2.1
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/messaging/azeventhubs@v1.2.1/LICENSE.txt:

Copyright (c) Microsoft Corporation.

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


--------------------------------------------------------------------------------
Dependency : github.com/elastic/azure-sdk-for-go/sdk/resourcemanager/consumption/armconsumption
Version: v1.1.0-elastic
Expand Down Expand Up @@ -1924,11 +1955,11 @@ SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources
Version: v1.1.1
Version: v1.2.0
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources@v1.1.1/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources@v1.2.0/LICENSE.txt:

MIT License

Expand All @@ -1954,11 +1985,11 @@ SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
Version: v1.3.2
Version: v1.4.0
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.3.2/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.0/LICENSE.txt:

MIT License

Expand Down Expand Up @@ -2215,11 +2246,11 @@ Contents of probable licence file $GOMODCACHE/github.com/!azure/go-autorest/auto

--------------------------------------------------------------------------------
Dependency : github.com/Azure/go-autorest/autorest/adal
Version: v0.9.21
Version: v0.9.24
Licence type (autodetected): Apache-2.0
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/go-autorest/autorest/adal@v0.9.21/LICENSE:
Contents of probable licence file $GOMODCACHE/github.com/!azure/go-autorest/autorest/adal@v0.9.24/LICENSE:


Apache License
Expand Down Expand Up @@ -24458,11 +24489,11 @@ THE SOFTWARE.

--------------------------------------------------------------------------------
Dependency : golang.org/x/crypto
Version: v0.24.0
Version: v0.25.0
Licence type (autodetected): BSD-3-Clause
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/golang.org/x/crypto@v0.24.0/LICENSE:
Contents of probable licence file $GOMODCACHE/golang.org/x/crypto@v0.25.0/LICENSE:

Copyright (c) 2009 The Go Authors. All rights reserved.

Expand Down Expand Up @@ -24569,11 +24600,11 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------------------------------
Dependency : golang.org/x/net
Version: v0.26.0
Version: v0.27.0
Licence type (autodetected): BSD-3-Clause
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/golang.org/x/net@v0.26.0/LICENSE:
Contents of probable licence file $GOMODCACHE/golang.org/x/net@v0.27.0/LICENSE:

Copyright (c) 2009 The Go Authors. All rights reserved.

Expand Down Expand Up @@ -24680,11 +24711,11 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------------------------------
Dependency : golang.org/x/sys
Version: v0.21.0
Version: v0.22.0
Licence type (autodetected): BSD-3-Clause
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/golang.org/x/sys@v0.21.0/LICENSE:
Contents of probable licence file $GOMODCACHE/golang.org/x/sys@v0.22.0/LICENSE:

Copyright (c) 2009 The Go Authors. All rights reserved.

Expand Down Expand Up @@ -29001,11 +29032,11 @@ Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-pipeline-g

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/internal
Version: v1.9.1
Version: v1.10.0
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/internal@v1.9.1/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/internal@v1.10.0/LICENSE.txt:

MIT License

Expand All @@ -29031,16 +29062,16 @@ SOFTWARE


--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal
Version: v1.1.2
Dependency : github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/eventhub/armeventhub
Version: v1.2.0
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/internal@v1.1.2/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/eventhub/armeventhub@v1.2.0/LICENSE.txt:

MIT License

Copyright (c) Microsoft Corporation.
Copyright (c) Microsoft Corporation. All rights reserved.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand All @@ -29058,8 +29089,7 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE

SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal/v2
Expand Down Expand Up @@ -29124,11 +29154,11 @@ SOFTWARE.

--------------------------------------------------------------------------------
Dependency : github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage
Version: v1.5.0
Version: v1.6.0
Licence type (autodetected): MIT
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage@v1.5.0/LICENSE.txt:
Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage@v1.6.0/LICENSE.txt:

MIT License

Expand Down Expand Up @@ -52967,11 +52997,11 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------------------------------
Dependency : golang.org/x/term
Version: v0.21.0
Version: v0.22.0
Licence type (autodetected): BSD-3-Clause
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/golang.org/x/term@v0.21.0/LICENSE:
Contents of probable licence file $GOMODCACHE/golang.org/x/term@v0.22.0/LICENSE:

Copyright (c) 2009 The Go Authors. All rights reserved.

Expand Down Expand Up @@ -55843,6 +55873,29 @@ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


--------------------------------------------------------------------------------
Dependency : nhooyr.io/websocket
Version: v1.8.11
Licence type (autodetected): ISC
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/nhooyr.io/websocket@v1.8.11/LICENSE.txt:

Copyright (c) 2023 Anmol Sethi <hi@nhooyr.io>

Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.


--------------------------------------------------------------------------------
Dependency : sigs.k8s.io/json
Version: v0.0.0-20221116044647-bc3834ca7abd
Expand Down
24 changes: 12 additions & 12 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ require (
code.cloudfoundry.org/rfc5424 v0.0.0-20180905210152-236a6d29298a // indirect
github.com/Azure/azure-event-hubs-go/v3 v3.6.1
github.com/Azure/azure-sdk-for-go v65.0.0+incompatible
github.com/Azure/azure-storage-blob-go v0.15.0
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
github.com/Azure/go-autorest/autorest v0.11.28
github.com/Azure/go-autorest/autorest/date v0.3.0
Expand Down Expand Up @@ -115,7 +114,6 @@ require (
github.com/lib/pq v1.10.3
github.com/magefile/mage v1.15.0
github.com/mattn/go-colorable v0.1.13
github.com/mattn/go-ieproxy v0.0.1 // indirect
github.com/miekg/dns v1.1.42
github.com/mitchellh/gox v1.0.1
github.com/mitchellh/hashstructure v1.1.0
Expand Down Expand Up @@ -146,13 +144,13 @@ require (
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/multierr v1.11.0
go.uber.org/zap v1.27.0
golang.org/x/crypto v0.24.0
golang.org/x/crypto v0.25.0
golang.org/x/lint v0.0.0-20210508222113-6edffad5e616
golang.org/x/mod v0.18.0
golang.org/x/net v0.26.0
golang.org/x/net v0.27.0
golang.org/x/oauth2 v0.18.0
golang.org/x/sync v0.7.0
golang.org/x/sys v0.21.0
golang.org/x/sys v0.22.0
golang.org/x/text v0.16.0
golang.org/x/time v0.5.0
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d
Expand All @@ -178,15 +176,17 @@ require (
cloud.google.com/go v0.112.1
cloud.google.com/go/compute v1.25.1
cloud.google.com/go/redis v1.14.2
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.12.0
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.13.0
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0
github.com/Azure/azure-sdk-for-go/sdk/messaging/azeventhubs v1.2.1
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/consumption/armconsumption v1.1.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v4 v4.6.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/costmanagement/armcostmanagement v1.1.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/monitor/armmonitor v0.8.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.1.1
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.3.2
github.com/Azure/go-autorest/autorest/adal v0.9.21
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.2.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.4.0
github.com/Azure/azure-storage-blob-go v0.15.0
github.com/Azure/go-autorest/autorest/adal v0.9.24
github.com/apache/arrow/go/v14 v14.0.2
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.1
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.16.15
Expand Down Expand Up @@ -237,7 +237,7 @@ require (
code.cloudfoundry.org/gofileutils v0.0.0-20170111115228-4d0c80011a0f // indirect
github.com/Azure/azure-amqp-common-go/v4 v4.2.0 // indirect
github.com/Azure/azure-pipeline-go v0.2.3 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.9.1 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 // indirect
github.com/Azure/go-amqp v1.0.5 // indirect
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest/to v0.4.0 // indirect
Expand Down Expand Up @@ -319,7 +319,6 @@ require (
github.com/jcmturner/gokrb5/v8 v8.4.2 // indirect
github.com/jcmturner/rpc/v2 v2.0.3 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/joho/godotenv v1.5.1 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/karrick/godirwalk v1.17.0 // indirect
Expand All @@ -332,6 +331,7 @@ require (
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/markbates/pkger v0.17.1 // indirect
github.com/mattn/go-ieproxy v0.0.1 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.9 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
Expand Down Expand Up @@ -373,7 +373,7 @@ require (
go.opentelemetry.io/otel/trace v1.28.0 // indirect
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
golang.org/x/exp v0.0.0-20231127185646-65229373498e // indirect
golang.org/x/term v0.21.0 // indirect
golang.org/x/term v0.22.0 // indirect
golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
google.golang.org/appengine v1.6.8 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240513163218-0867130af1f8 // indirect
Expand Down
Loading

0 comments on commit c3c40c6

Please sign in to comment.