Skip to content
This repository has been archived by the owner on Mar 21, 2023. It is now read-only.

Migrate to Netty 4.1 #28

Merged
merged 2 commits into from
Jan 18, 2018
Merged

Migrate to Netty 4.1 #28

merged 2 commits into from
Jan 18, 2018

Conversation

joschi
Copy link
Contributor

@joschi joschi commented Dec 5, 2017

@joschi joschi added this to the 3.0.0 milestone Dec 5, 2017
@bernd
Copy link
Member

bernd commented Dec 6, 2017

@joschi I am getting this when trying to ingest Netflow v9 data. (worked with master)

2017-12-06 15:55:01,965 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=6fbe6360-da95-11e7-95d9-024248
475c73, journalOffset=5044, codec=netflow, payloadSize=70, timestamp=2017-12-06T14:55:01.782Z}
java.lang.IndexOutOfBoundsException: null
        at io.netty.buffer.EmptyByteBuf.readUnsignedShort(EmptyByteBuf.java:550) ~[graylog.jar:?]
        at org.graylog.plugins.netflow.v9.NetFlowV9Parser.parseHeader(NetFlowV9Parser.java:93) ~[?:?]
        at org.graylog.plugins.netflow.v9.NetFlowV9Parser.parsePacket(NetFlowV9Parser.java:46) ~[?:?]
        at org.graylog.plugins.netflow.codecs.NetFlowCodec.lambda$decodeV9Packets$7(NetFlowCodec.java:186) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_151]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_151]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1380) ~[?:1.8.0_151]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_151]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_151]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_151]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_151]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_151]
        at org.graylog.plugins.netflow.codecs.NetFlowCodec.decodeV9Packets(NetFlowCodec.java:187) ~[?:?]
        at org.graylog.plugins.netflow.codecs.NetFlowCodec.decodeV9(NetFlowCodec.java:156) ~[?:?]
        at org.graylog.plugins.netflow.codecs.NetFlowCodec.decodeMessages(NetFlowCodec.java:134) ~[?:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:148) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

@joschi joschi force-pushed the netty-4 branch 2 times, most recently from d4a8500 to d830393 Compare December 11, 2017 11:51
@joschi joschi force-pushed the netty-4 branch 2 times, most recently from 70a681f to 74d123b Compare January 4, 2018 13:24
@bernd
Copy link
Member

bernd commented Jan 11, 2018

I am seeing the following error once after starting the server. So far it happened in the first 10 minutes after the server has been started. Could this be related to the template handling for NetFlow v9?

2018-01-11 15:33:07,083 ERROR: io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:

@joschi
Copy link
Contributor Author

joschi commented Jan 15, 2018

@bernd I was unable to locate the origin of that leak. Maybe you'll get lucky in your review.

@kroepke kroepke self-requested a review January 15, 2018 10:12
@kroepke
Copy link
Member

kroepke commented Jan 15, 2018

Since I'm the root of all evil^W^Wthis code, I'll take a look at the reported leak.

@kroepke
avoid creating a pooled buffer copy inside the aggregator method
5d6f565 
the buffer slice in question will be copied anyway, and for some reason
creating the copy sometimes triggered a buffer leak warning although nothing seems to be wrong
bernd
bernd approved these changes Jan 18, 2018
View reviewed changes
Copy link
Member

@bernd bernd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joschi Thanks for the good work! 👍

@bernd bernd merged commit 5249012 into master Jan 18, 2018
@bernd bernd deleted the netty-4 branch January 18, 2018 10:19
@bernd bernd mentioned this pull request Jan 18, 2018
Closed
joschi pushed a commit that referenced this pull request Jan 18, 2018
Replace usage of ChannelBuffer with ByteBuf in NetFlowV9HeaderTest
3b06dc0 
This occurrence of ChannelBuffer was missed in the original Netty 4 PR (#28)
@joschi joschi mentioned this pull request Jan 18, 2018
Merged
bernd pushed a commit that referenced this pull request Jan 18, 2018
@joschi @bernd
Replace usage of ChannelBuffer with ByteBuf in NetFlowV9HeaderTest (#29)
d03f677 
This occurrence of ChannelBuffer was missed in the original Netty 4 PR (#28)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@bernd bernd bernd approved these changes

@kroepke kroepke Awaiting requested review from kroepke

Assignees
No one assigned
Labels
ready-for-review
Projects
None yet
Milestone
3.0.0
Development

Successfully merging this pull request may close these issues.
3 participants
@joschi @bernd @kroepke