Migrate to Netty 4.1 #4397
Merged
Migrate to Netty 4.1 #4397
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghost
assigned
This was referenced Dec 5, 2017
|
@joschi I am getting tons of these messages. I am running with a beats, netflow and GELF TCP input. |
|
@joschi I am also seeing leak detector messages. I am running with "paranoid". |
|
@joschi Using the following revisions I am still getting the errors below. For NetFlow I am using pmacct with the following configuration and another instance with NetFlow version |
|
|
@joschi Ah right, I rebased against master before the build. Fixed the value in my previous comment. |
|
@bernd Could you please record some pcaps of your pmacct traffic which I could replay to reproduce the problem? |
|
@joschi While working on a netflow packet capture, I noticed that I am getting the following error on two different setups when sending GELF TCP messages to the server. (using the Java gelfclient and the Ruby gelf-rb lib) Example code for sending GELF TCP messages: https://gist.github.com/bernd/330f794086afd1338255f0b5b08fed79 |
|
@joschi Another issue I noticed: When I am stopping my Beats input the listener is gone but active connections are still alive and sending new data. It looks like active connections aren't closed when stopping an input. |
|
@joschi I sent you the NetFlow pcap file privately. |
|
@joschi The input throughput counter does no work anymore:
Happens with a beats and raw input. |
bernd
reviewed
Jan 11, 2018
| @@ -64,7 +63,7 @@ protected void configure() { | |||
|
|
|||
| bind(ServiceManager.class).toProvider(ServiceManagerProvider.class).asEagerSingleton(); | |||
|
|
|||
| bind(HashedWheelTimer.class).toInstance(new HashedWheelTimer()); | |||
| // TODO: Dedicated scheduled executor | |||
There was a problem hiding this comment.
Is this a todo for this PR or a "nice to have"?
There was a problem hiding this comment.
It's a nice to have for ThroughputCounter.
bernd
reviewed
Jan 11, 2018
| LOG.debug("Setting receive buffer size to {} bytes", recvBufferSize); | ||
| bootstrap.setOption("receiveBufferSizePredictorFactory", new FixedReceiveBufferSizePredictorFactory(recvBufferSize)); | ||
| bootstrap.setOption("receiveBufferSize", recvBufferSize); | ||
| // TODO: Make number of channels configurable separately |
There was a problem hiding this comment.
Is this a todo for this PR or a "nice to have"?
added 10 commits
January 11, 2018 13:35
Ceterum censeo JavaScript esse delendam.
Closed
bernd
added a commit
that referenced
this pull request
Jan 15, 2019
This fixes an issue that has been introduced with the big Netty 4.1 update in PR #4397. In Netty 3.x we always passed a "MessageEvent" through the channel pipeline and got the remote address from that object. Since this object doesn't exist anymore in Netty 4.x, we only pass the message payload's "ByteBuf" through the pipeline and rely on the "Channel#getRemoteAddress" method to always return the remote address object. The problem is that this does indeed work for TCP based channels but it doesn't work for UDP based channels. For UDP channels the "#getRemoteAddress()" method always returns "null". This is probably due to the connection-less nature of UDP. For UDP transports Netty only creates a single channel. For TCP transports there is one channel per TCP connection To fix this we need to get our hands on the remote address when we create the "RawMessage" object at the very end of the Netty pipeline. Since we only pass the message payload "ByteBuf" through the Netty pipeline, we could previously reuse several classes for TCP and UDP transports because they were basically the same. For UDP transports we now need to carry the remote address through the pipeline by using a "AddressedEnvelope" (available in Netty) that takes a payload and a sender/receiver object. That means we have to create a few UDP specific - or rather "AddressedEnvelope" specific - pipeline handlers because the shared ones only know how to handle "ByteBuf" messages. This PR moves some shared code out of the "NettyTransport" class up to "AbstractTcpTransport" and "UdpTransport" so we can customize the pipeline for the two different payload types. It also creates new message aggregation handlers for the "AddressedEnvelope" objects. In the future we can probably refactor this to share some more code, but for 3.0 I tried to change as few as possible. The TCP pipeline is basically unchanged apart from the "AbstractTcpTransport" change. Fixes #5264 Fixes #5293
kroepke
pushed a commit
that referenced
this pull request
Jan 15, 2019
This fixes an issue that has been introduced with the big Netty 4.1 update in PR #4397. In Netty 3.x we always passed a "MessageEvent" through the channel pipeline and got the remote address from that object. Since this object doesn't exist anymore in Netty 4.x, we only pass the message payload's "ByteBuf" through the pipeline and rely on the "Channel#getRemoteAddress" method to always return the remote address object. The problem is that this does indeed work for TCP based channels but it doesn't work for UDP based channels. For UDP channels the "#getRemoteAddress()" method always returns "null". This is probably due to the connection-less nature of UDP. For UDP transports Netty only creates a single channel. For TCP transports there is one channel per TCP connection To fix this we need to get our hands on the remote address when we create the "RawMessage" object at the very end of the Netty pipeline. Since we only pass the message payload "ByteBuf" through the Netty pipeline, we could previously reuse several classes for TCP and UDP transports because they were basically the same. For UDP transports we now need to carry the remote address through the pipeline by using a "AddressedEnvelope" (available in Netty) that takes a payload and a sender/receiver object. That means we have to create a few UDP specific - or rather "AddressedEnvelope" specific - pipeline handlers because the shared ones only know how to handle "ByteBuf" messages. This PR moves some shared code out of the "NettyTransport" class up to "AbstractTcpTransport" and "UdpTransport" so we can customize the pipeline for the two different payload types. It also creates new message aggregation handlers for the "AddressedEnvelope" objects. In the future we can probably refactor this to share some more code, but for 3.0 I tried to change as few as possible. The TCP pipeline is basically unchanged apart from the "AbstractTcpTransport" change. Fixes #5264 Fixes #5293 **Note:** This needs to be cherry-picked into 3.0
bernd
added a commit
that referenced
this pull request
Jan 16, 2019
This fixes an issue that has been introduced with the big Netty 4.1 update in PR #4397. In Netty 3.x we always passed a "MessageEvent" through the channel pipeline and got the remote address from that object. Since this object doesn't exist anymore in Netty 4.x, we only pass the message payload's "ByteBuf" through the pipeline and rely on the "Channel#getRemoteAddress" method to always return the remote address object. The problem is that this does indeed work for TCP based channels but it doesn't work for UDP based channels. For UDP channels the "#getRemoteAddress()" method always returns "null". This is probably due to the connection-less nature of UDP. For UDP transports Netty only creates a single channel. For TCP transports there is one channel per TCP connection To fix this we need to get our hands on the remote address when we create the "RawMessage" object at the very end of the Netty pipeline. Since we only pass the message payload "ByteBuf" through the Netty pipeline, we could previously reuse several classes for TCP and UDP transports because they were basically the same. For UDP transports we now need to carry the remote address through the pipeline by using a "AddressedEnvelope" (available in Netty) that takes a payload and a sender/receiver object. That means we have to create a few UDP specific - or rather "AddressedEnvelope" specific - pipeline handlers because the shared ones only know how to handle "ByteBuf" messages. This PR moves some shared code out of the "NettyTransport" class up to "AbstractTcpTransport" and "UdpTransport" so we can customize the pipeline for the two different payload types. It also creates new message aggregation handlers for the "AddressedEnvelope" objects. In the future we can probably refactor this to share some more code, but for 3.0 I tried to change as few as possible. The TCP pipeline is basically unchanged apart from the "AbstractTcpTransport" change. Fixes #5264 Fixes #5293 **Note:** This needs to be cherry-picked into 3.0 (cherry picked from commit 375e618)
kroepke
pushed a commit
that referenced
this pull request
Jan 16, 2019
…5519) This fixes an issue that has been introduced with the big Netty 4.1 update in PR #4397. In Netty 3.x we always passed a "MessageEvent" through the channel pipeline and got the remote address from that object. Since this object doesn't exist anymore in Netty 4.x, we only pass the message payload's "ByteBuf" through the pipeline and rely on the "Channel#getRemoteAddress" method to always return the remote address object. The problem is that this does indeed work for TCP based channels but it doesn't work for UDP based channels. For UDP channels the "#getRemoteAddress()" method always returns "null". This is probably due to the connection-less nature of UDP. For UDP transports Netty only creates a single channel. For TCP transports there is one channel per TCP connection To fix this we need to get our hands on the remote address when we create the "RawMessage" object at the very end of the Netty pipeline. Since we only pass the message payload "ByteBuf" through the Netty pipeline, we could previously reuse several classes for TCP and UDP transports because they were basically the same. For UDP transports we now need to carry the remote address through the pipeline by using a "AddressedEnvelope" (available in Netty) that takes a payload and a sender/receiver object. That means we have to create a few UDP specific - or rather "AddressedEnvelope" specific - pipeline handlers because the shared ones only know how to handle "ByteBuf" messages. This PR moves some shared code out of the "NettyTransport" class up to "AbstractTcpTransport" and "UdpTransport" so we can customize the pipeline for the two different payload types. It also creates new message aggregation handlers for the "AddressedEnvelope" objects. In the future we can probably refactor this to share some more code, but for 3.0 I tried to change as few as possible. The TCP pipeline is basically unchanged apart from the "AbstractTcpTransport" change. Fixes #5264 Fixes #5293 **Note:** This needs to be cherry-picked into 3.0 (cherry picked from commit 375e618)
mpfz0r
added a commit
that referenced
this pull request
May 22, 2019
The Input configuration TLS Client Auth Trusted Certs used to support either a file, or a directory of certificates. This got broken in 3.0 with PR #4397 - Fix this by extending also handling directories in loadCertificates(). - Delete the old TrustManager based version that is not used anymore. - Extend the KeyUtilTest and use `Resources.getResource()` instead, which does not rely on the resources to exist in the target directory. Fixes #5939
mpfz0r
added a commit
that referenced
this pull request
May 22, 2019
The Input configuration TLS Client Auth Trusted Certs used to support either a file, or a directory of certificates. This got broken in 3.0 with PR #4397 - Fix this by extending loadCertificates() to also handle directories. - Delete the old TrustManager based version that is not used anymore. - Extend the KeyUtilTest and use `Resources.getResource()` instead, which does not rely on the resources to exist in the target directory. Fixes #5939
kmerz
pushed a commit
that referenced
this pull request
Jul 9, 2019
The Input configuration TLS Client Auth Trusted Certs used to support either a file, or a directory of certificates. This got broken in 3.0 with PR #4397 - Fix this by extending loadCertificates() to also handle directories. - Delete the old TrustManager based version that is not used anymore. - Extend the KeyUtilTest and use `Resources.getResource()` instead, which does not rely on the resources to exist in the target directory. Fixes #5939
kmerz
pushed a commit
that referenced
this pull request
Jul 9, 2019
The Input configuration TLS Client Auth Trusted Certs used to support either a file, or a directory of certificates. This got broken in 3.0 with PR #4397 - Fix this by extending loadCertificates() to also handle directories. - Delete the old TrustManager based version that is not used anymore. - Extend the KeyUtilTest and use `Resources.getResource()` instead, which does not rely on the resources to exist in the target directory. Fixes #5939
mpfz0r
pushed a commit
that referenced
this pull request
Jul 9, 2019
The Input configuration TLS Client Auth Trusted Certs used to support either a file, or a directory of certificates. This got broken in 3.0 with PR #4397 - Fix this by extending loadCertificates() to also handle directories. - Delete the old TrustManager based version that is not used anymore. - Extend the KeyUtilTest and use `Resources.getResource()` instead, which does not rely on the resources to exist in the target directory. Fixes #5939
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Migrate Graylog from Netty 3.10.x to Netty 4.1.x.
Refs #4226