Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speedup the RAWX #2017

Merged
merged 26 commits into from
Apr 23, 2020
Merged

Speedup the RAWX #2017

merged 26 commits into from
Apr 23, 2020

Conversation

jfsmig
Copy link
Contributor

@jfsmig jfsmig commented Apr 16, 2020

SUMMARY

Speed-up the Golang RAWX with various techniques:

  • avoid systematically closing the connections
  • try to use the stack instead of the heap:
    • pass structures instead of pointers
    • remove log traces
    • avoid Printf and prefer strings.Builder
    • avoid memory copies
    • remove unnecessary calls
  • reuse structures and avoid create/destroy cycles
  • increase the queue to beanstalkd
  • parallelize on multiple beanstalkd connections
  • save syscalls when possible (pid cached)
  • trim unused code
ISSUE TYPE

enhancement

COMPONENT NAME

rawx

SDS VERSION

7.0.0.0a2-pouet-pouet

@codecov-io
Copy link

codecov-io commented Apr 21, 2020

Codecov Report

Merging #2017 into master will decrease coverage by 20.51%.
The diff coverage is 56.13%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master    #2017       +/-   ##
===========================================
- Coverage   77.49%   56.98%   -20.50%     
===========================================
  Files         335      228      -107     
  Lines       67771    50358    -17413     
===========================================
- Hits        52511    28691    -23820     
- Misses      15130    21494     +6364     
- Partials      130      173       +43     
Impacted Files Coverage Δ
rawx/buffer_pool.go 80.00% <ø> (-6.66%) ⬇️
rawx/chunkrepo.go 70.84% <ø> (ø)
rawx/conf_reader.go 75.61% <0.00%> (-5.96%) ⬇️
rawx/configuration.go 56.10% <ø> (ø)
rawx/hexa.go 87.50% <ø> (-12.50%) ⬇️
rawx/chunk_info.go 53.38% <25.31%> (-30.01%) ⬇️
rawx/logger.go 62.14% <45.24%> (-9.14%) ⬇️
rawx/handler_chunk.go 56.44% <62.75%> (-17.69%) ⬇️
rawx/rawx.go 50.00% <66.67%> (-18.75%) ⬇️
rawx/filerepo.go 63.82% <76.82%> (-2.99%) ⬇️
... and 262 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update be4bf24...694c8ac. Read the comment docs.

Copy link
Contributor

@murlock murlock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jfsmig jfsmig merged commit c447d7b into open-io:master Apr 23, 2020
@jfsmig jfsmig deleted the M-rawx-faster branch April 23, 2020 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants