Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove call to Valid() during encoding to boost speed by about 13-18% #857

Merged
merged 1 commit into from
Apr 29, 2021

Conversation

fxamacker
Copy link
Member

@fxamacker fxamacker commented Apr 29, 2021

Closes #851

Description

  • eliminate call to Valid() in the encoder
  • increase max limits in decoder to eliminate one reason for encoder to call Valid().

Thanks @turbolent for suggesting this optimization last week.

Benchmark Comparisons

Speed improved by 13% to almost 18%. Memory use is unchanged.

name                                               old time/op    new time/op    delta
EncodeCBOR/comp_v2_96d7e06eaf4b3fcf_14850bytes-4      148µs ± 1%     121µs ± 1%  -17.97%  (p=0.000 n=10+10)
EncodeCBOR/comp_v3_99dc360eee32dcec_160bytes-4       4.33µs ± 1%    3.65µs ± 1%  -15.68%  (p=0.000 n=10+10)
EncodeCBOR/comp_v3_b52a33b7e56868f6_139bytes-4       3.22µs ± 1%    2.74µs ± 1%  -14.89%  (p=0.000 n=10+9)
EncodeCBOR/comp_v3_d99d7e6b4dad41e1_776155bytes-4    7.04ms ± 1%    6.12ms ± 0%  -13.09%  (p=0.000 n=10+10)

------

name                                            old time/op    new time/op    delta
EncodeCBOR/link_v3_2392f05c3b72f235_167bytes-4    3.30µs ± 2%    2.72µs ± 1%  -17.52%  (p=0.000 n=10+9)
EncodeCBOR/link_v3_3a791fe1b8243e73_192bytes-4    3.31µs ± 1%    2.75µs ± 2%  -17.04%  (p=0.000 n=9+10)

Benchmarks used Go 1.15.11 on linux_amd64.

Caveats

⚠️ WARNING: Max limits for array size, etc. are increased -- if adversaries can cause excessively large CBOR data to be decoded, they can launch resource exhaustion attacks.

Max limit for nested level is set to MaxInt16 because MaxInt32 would allow nested levels that would crash Go because of the 1 GB stack size limit of Go.


For contributor use:

  • Targeted PR against master branch
  • Linked to Github issue with discussion and accepted design OR link to spec that describes this work
  • Code follows the standards mentioned here
  • Updated relevant documentation
  • Re-reviewed Files changed in the Github PR explorer
  • Added appropriate labels

Also increase max limits in decoder to eliminate one reason
for encoder to call Valid().

Closes onflow#851
Copy link
Member

@turbolent turbolent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@@ -46,6 +47,9 @@ type DecoderV4 struct {
decodeCallback DecodingCallback
}

// maxInt is math.MaxInt32 or math.MaxInt64 depending on arch.
const maxInt = 1<<(bits.UintSize-1) - 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌

@turbolent turbolent merged commit 702fcd1 into onflow:master Apr 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Call to Valid() can be removed to improve encoding speed by 14-17%
2 participants