Skip to content

benchmarking: add memory working set to server telemetry - #2386

Open
Nishanth Kotla (Nishanth29) wants to merge 4 commits into
agent-substrate:mainfrom
Nishanth29:working-set
Open

Nishanth Kotla (Nishanth29) wants to merge 4 commits into
agent-substrate:mainfrom
Nishanth29:working-set

Conversation

@Nishanth29

Copy link
Copy Markdown
Contributor

Depends on #2221. The first 3 commits are #2221's; this PR's change is only the last commit, 36d26077 ("benchmarking: add memory working set to server telemetry"). Will rebase once #2221 merges.

Part of #1590

Adds working_set to server telemetry:

  • node, ateom, atelet: from cAdvisor (container_memory_working_set_bytes).
  • actor, per_actor: reuse the atelet's existing actor stats metrics,
    ate_actor_stats_memory_working_set_bytes and ate_actor_stats_sampled_actors.
    actor is the actors' working set per atelet; per_actor divides it by
    the number of actors that atelet measured.

Both sources are already collected (cAdvisor and the atelet), so nothing changes in Substrate or the monitoring setup; this only adds queries.

Live check (gVisor, 4 users, 3m, through the telemetry-meter):

  • actor 1.13 GiB vs ateom 1.22 GiB at the same time

  • per_actor × count = actor (0.5628 × 2 = 1.1257)

  • 74 measurements, none null

  • Tests pass

  • Appropriate changes to documentation are included in the PR

@Nishanth29

Copy link
Copy Markdown
Contributor Author

@Nishanth29

Copy link
Copy Markdown
Contributor Author

Da Huang (@git286) Tim Bai (@baizhenyu), adding you as reviewers. For actors this only reads your existing ate.actor.stats.memory.working_set and ate.actor.stats.sampled_actors (no new metrics); node/ateom/atelet come from cAdvisor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant