Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory usage during gem statup #1090

Merged
merged 4 commits into from
Sep 21, 2020
Merged

Conversation

marcotc
Copy link
Member

@marcotc marcotc commented Jun 18, 2020

This PR introduces a few changes guide by the results from our new startup benchmarks: #1062.

The largest memory footprint we have as a gem is loading external dependencies (through require) and loading our own Ruby files.

Most of our dependencies will likely be used by the host application (e.g. thread, logger, sockets), so their overhead is shared with the host.

A few "unique" dependencies showed up, like msgpack, but they are required for the correct function of the tracer.

Another PR will follow with changes to load integration code dynamically, which should have the largest impact, based on my local testing.

This PR is an aggregation of a few changes that showed up as relevant enough in the benchmarks and that don't require large rework, nor introduce large risks.

This PR should have the following performance impact:

  • Retained memory:
    • Retained objects: 7,116 -> 7,030 (2% reduction)
    • Retained memory: 899,078 -> 895,471 (0.5% reduction)
  • Startup time: 137 ms -> 133 ms (3% reduction)
  • Total RubyVM RSS memory: 7320 KiB (no change)

@marcotc marcotc added core Involves Datadog core libraries performance Involves performance (e.g. CPU, memory, etc) labels Jun 18, 2020
@marcotc marcotc requested a review from a team June 18, 2020 19:49
@marcotc marcotc self-assigned this Jun 18, 2020
@@ -26,7 +26,8 @@ def option(name, meta = {}, &block)
builder = OptionDefinition::Builder.new(name, meta, &block)
options[name] = builder.to_definition.tap do
# Resolve and define helper functions
helpers = default_helpers(name).merge(builder.helpers)
helpers = default_helpers(name)
helpers = helpers.merge(builder.helpers) unless builder.helpers.empty?
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging with an empty hash creates an unnecessary copy of the original hash.

o.default false
o.on_set do |enabled|
# Enable rich debug print statements
require 'pp' if enabled
Copy link
Member Author

@marcotc marcotc Jun 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loading of pp is now lazy, as it is only used in production ddtrace when debug diagnostics are enabled.

Although pp is commonly used in testing and development, the host application is not guaranteed to need it in production mode.

Copy link
Contributor

@ericmustin ericmustin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work! My only feedback would be, maybe we don't need pretty print at all? probably not worth the change if it accidentally breaks some log parsing somewhere

@ericmustin
Copy link
Contributor

@marcotc None of these test failures seem related afaik

@marcotc marcotc merged commit 1ac9142 into master Sep 21, 2020
@marcotc marcotc deleted the feat/less-memory-startup branch September 21, 2020 19:14
@ericmustin ericmustin added this to the 0.41.0 milestone Sep 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Involves Datadog core libraries performance Involves performance (e.g. CPU, memory, etc)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants