Skip to content

[unified_analytics] Append to the log file instead of rewriting it - #2613

Merged
davidmorgan merged 2 commits into
mainfrom
append-only-log
Oct 1, 2026
Merged

davidmorgan merged 2 commits into
mainfrom
append-only-log

Conversation

@davidmorgan

Copy link
Copy Markdown
Contributor

LogHandler.save read the whole log file on every event and, once the log held kLogFileLength records, rewrote all of it. With a typical full log, about 1.7 MB, that costs about 16 ms of synchronous I/O per Analytics.send.

Long-lived tools feel this most: the analysis server sends about 130 events in a burst on its first periodic report and at shutdown, blocking its isolate for about 2.4 s. Processes rewriting the file concurrently also corrupt lines.

Now save stats the file and appends. When the file passes kLogFileTrimSize, 4 MiB, it is trimmed to the newest kLogFileLength records by writing a temporary file and renaming it over the log. logFileStats reads only the newest kLogFileLength records, so surveys see the same window as before.

Measured with a full log: send goes from 16.6 ms to 0.27 ms median; the analysis server burst from 2.42 s to 95 ms; a trim costs about 37 ms, once per roughly 3000 events.

Fixes #2611.

@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Package publishing

If you have publishing permissions, you can use the links below to publish the changes after merging this PR.

Package Version Status Publish tag (post-merge)
package:test_reflective_loader 0.6.0 ready to publish test_reflective_loader-v0.6.0
package:unified_analytics ⭐ 8.0.19-wip WIP (no publish necessary)
  • 19 already published.
  • 21 WIP (no publish necessary).

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request optimizes the log-saving mechanism in LogHandler by appending events directly to the log file instead of reading and rewriting it on every save. Trimming is now deferred until the log file exceeds 4 MiB, at which point it is trimmed to the newest records using a temporary file. The feedback suggests optimizing the trimming logic in _trim by using string length instead of utf8.encode to avoid unnecessary allocations in a loop.

Comment thread pkgs/unified_analytics/lib/src/log_handler.dart Outdated
@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

PR Health

Coverage ⚠️
File Coverage
pkgs/unified_analytics/lib/src/constants.dart 💔 Not covered
pkgs/unified_analytics/lib/src/log_handler.dart 💚 85 % ⬆️ 1 %

This check for test coverage is informational (issues shown here will not fail the PR).

This check can be disabled by tagging the PR with skip-coverage-check.

License Headers ✔️
// Copyright (c) 2026, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

Files
no missing headers

All source files should start with a license header.

Unrelated files missing license headers
Files
pkgs/html/test/differential_test.dart
pkgs/html/test/dom_parsing_test.dart
pkgs/html/test/mxss_test.dart
pkgs/html/test/noah_ark_clause_limit_test.dart

This check can be disabled by tagging the PR with skip-license-check.

Breaking changes ✔️
Package Change Current Version New Version Needed Version Looking good?
unified_analytics Non-Breaking 8.0.18 8.0.19-wip 8.0.19-wip ✔️

This check can be disabled by tagging the PR with skip-breaking-check.

Unused Dependencies ✔️
Package Status
unified_analytics ✔️ All dependencies utilized correctly.

For details on how to fix these, see dependency_validator.

This check can be disabled by tagging the PR with skip-unused-dependencies-check.

API leaks ⚠️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package Leaked API symbol Leaking sources
unified_analytics Condition survey_handler.dart::Survey::conditionList
survey_handler.dart::Survey::new::conditionList
unified_analytics PersistedSurvey survey_handler.dart::SurveyHandler::fetchPersistedSurveys
unified_analytics GAClient analytics.dart::Analytics::fake::gaClient
analytics.dart::AnalyticsImpl::new::gaClient
unified_analytics UserProperty analytics.dart::FakeAnalytics::userProperty

This check can be disabled by tagging the PR with skip-leaking-check.

Changelog Entry ✔️
Package Changed Files

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

LogHandler.save read the whole log file on every event and, once the log
held kLogFileLength records, rewrote all of it. With a typical full log,
about 1.7 MB, that costs about 16 ms of synchronous I/O per
Analytics.send.

Long-lived tools feel this most: the analysis server sends about 130
events in a burst on its first periodic report and at shutdown, blocking
its isolate for about 2.4 s. Processes rewriting the file concurrently
also corrupt lines.

Now save stats the file and appends. When the file passes
kLogFileTrimSize, 4 MiB, it is trimmed to the newest kLogFileLength
records by writing a temporary file and renaming it over the log. A
record another process appends during a trim can be lost; a failed trim
is retried on the next save. logFileStats reads only the newest
kLogFileLength records, so surveys see the same window as before.

Measured with a full log: send goes from 16.6 ms to 0.27 ms median; the
analysis server burst from 2.42 s to 95 ms; a trim costs about 37 ms,
once per roughly 3000 events.

Fixes #2611.
@keertip

keertip commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Overall looks good. I can't think of any major downsides to landing this, and if we do lose some entries in the local log file, that's the price for performance optimization. Thanks for doing this!

@davidmorgan
davidmorgan marked this pull request as ready for review September 30, 2026 09:29
@davidmorgan
davidmorgan requested a review from a team as a code owner September 30, 2026 09:29
@davidmorgan
davidmorgan requested a review from keertip September 30, 2026 09:37
@davidmorgan

Copy link
Copy Markdown
Contributor Author

Overall looks good. I can't think of any major downsides to landing this, and if we do lose some entries in the local log file, that's the price for performance optimization. Thanks for doing this!

Great--sent for your review, please.

@srawlins srawlins left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice tests

@davidmorgan
davidmorgan merged commit d523e1f into main Oct 1, 2026
21 checks passed
@davidmorgan
davidmorgan deleted the append-only-log branch October 1, 2026 09:09
copybara-service Bot pushed a commit to dart-lang/sdk that referenced this pull request Oct 7, 2026
Revisions updated by `dart tools/rev_sdk_deps.dart`.

core (https://github.com/dart-lang/core/compare/08e4022..4d80610):
  4d80610a  Thu Oct 1 22:03:38 2026 +0000  dependabot[bot]  Bump the github-actions group with 5 updates (dart-lang/core#1005)

dartdoc (https://github.com/dart-lang/dartdoc/compare/2a040a8..5eb0f2c):
  5eb0f2c7  Thu Oct 1 19:03:48 2026 +0000  dependabot[bot]  Bump github/codeql-action/upload-sarif from 4.37.9 to 4.38.2 in the github-actions group (dart-lang/dartdoc#4291)

http (https://github.com/dart-lang/http/compare/d8e8e9c..585d433):
  585d433  Mon Oct 5 16:12:54 2026 -0700  Brian Quinlan  fix(cupertino_http): fix a websocket memory leak (dart-lang/http#1991)
  337adf3  Mon Oct 5 16:03:25 2026 -0700  Brian Quinlan  fix(cupertino_http): add `autoReleasePool` use to `CupertinoClient` (dart-lang/http#1990)
  47c57df  Tue Sep 29 14:07:01 2026 -0700  Brian Quinlan  fix(ok_http): fix `NetworkOnMainThreadException` (dart-lang/http#1995)
  d8cde5d  Tue Sep 29 14:06:44 2026 -0700  Brian Quinlan  fix test (dart-lang/http#1996)
  31056fe  Tue Sep 29 13:57:20 2026 -0700  Brian Quinlan  chore(cronet_http): prepare to release 1.10.0 (dart-lang/http#1993)
  9bc3a91  Tue Sep 29 13:53:25 2026 -0700  Brian Quinlan  fix(cronet_http): one global reference per `onReadCompleted` (dart-lang/http#1992)
  49ecd36  Mon Sep 28 14:20:26 2026 -0700  Brian Quinlan  fix(conformance): `supportsAbort` not considered in response (dart-lang/http#1994)

native (https://github.com/dart-lang/native/compare/27f7678..a2a6179):
  a2a617910  Tue Oct 6 13:53:17 2026 +0200  Daco Harkes  [jni_flutter] Bump version for publishing (dart-lang/native#3730)
  d3a99d66b  Tue Oct 6 19:46:34 2026 +1100  Liam Appelbe  [infra] Fix ffigen and jni_flutter (dart-lang/native#3724)
  b66d0815a  Tue Oct 6 04:28:35 2026 +0300  Manar Elhabbal  [ffigen] add Support passing C++ classes by value  (dart-lang/native#3677)
  f7ffe9667  Fri Oct 2 11:33:49 2026 +0200  Moritz  [native_toolchain_c] Don't pass /INCLUDE per symbol with a generated .def on Windows (dart-lang/native#3718)
  77a5e4169  Thu Oct 1 10:56:45 2026 +0300  Hassnaa Mohamed  [ffigen] Fix cpp method filtering (dart-lang/native#3696)
  cca065ca6  Thu Oct 1 10:17:12 2026 +1000  Liam Appelbe  [infra] Prepare to publish everything (dart-lang/native#3716)
  f13623ee3  Thu Oct 1 10:16:39 2026 +1000  Liam Appelbe  [jnigen] API docs (dart-lang/native#3715)
  be618790c  Thu Oct 1 10:05:26 2026 +1000  Liam Appelbe  [objective_c] API docs (dart-lang/native#3708)
  a18dd1148  Thu Oct 1 10:04:38 2026 +1000  Liam Appelbe  [ffigen] API docs (dart-lang/native#3707)
  6fd87c3b2  Tue Sep 29 19:01:11 2026 -0700  Brian Quinlan  [jni] Clarify `JBuffer.asUint8List` behavior (dart-lang/native#3704)
  6d5bbcfad  Wed Sep 30 09:35:47 2026 +1000  Liam Appelbe  [jni] API documentation (dart-lang/native#3706)
  fbcbc894b  Wed Sep 30 09:35:34 2026 +1000  Liam Appelbe  [ffigen] Document autorelease issues (dart-lang/native#3705)
  f99449ec9  Tue Sep 29 09:39:27 2026 +1000  Liam Appelbe  [infra] Add swift2objc to the workspace (dart-lang/native#3700)
  31494fa08  Tue Sep 29 09:38:59 2026 +1000  Liam Appelbe  [infra] Add jni_util to the workspace (dart-lang/native#3698)

shelf (https://github.com/dart-lang/shelf/compare/e5c8dc6..b2cb503):
  b2cb503  Wed Sep 30 22:53:07 2026 -0700  Kevin Moore  Resolve shelf.io.connection_info lazily (dart-lang/shelf#542)
  b70fafa  Tue Sep 29 13:37:08 2026 -0700  Kevin Moore  Match parameterless shelf_router routes without running a regex (dart-lang/shelf#544)

test (https://github.com/dart-lang/test/compare/4f92d53..d749b10):
  d749b104  Mon Oct 5 15:42:46 2026 -0700  Nate Bosch  Fail isolate suites when the test isolate exits (dart-lang/test#2775)
  da2157e9  Mon Oct 5 11:12:43 2026 -0700  Nate Bosch  Wait longer for the Chrome test tab to appear (dart-lang/test#2771)
  00bd0c99  Mon Oct 5 18:50:03 2026 +0300  Yusuf İhsan Görgel  Fix broken links in architecture.md (dart-lang/test#2774)
  3be2f547  Thu Oct 1 03:46:29 2026 +0000  dependabot[bot]  Bump the github-actions group with 4 updates (dart-lang/test#2773)
  40a5ad90  Tue Sep 29 12:07:31 2026 -0700  Nate Bosch  Delete compilation artifacts once they are no longer needed (dart-lang/test#2764)
  171ad0e5  Mon Sep 28 17:29:48 2026 -0700  Jacob MacDonald  export additional types from package:test_api/backend.dart (dart-lang/test#2745)
  52b45064  Mon Sep 28 15:04:08 2026 -0700  Nate Bosch  Tag browser and node tests, relax flaky timeouts (dart-lang/test#2770)

tools (https://github.com/dart-lang/tools/compare/d87eaf7..a66d517):
  a66d517d  Tue Oct 6 09:41:32 2026 -0700  Phil Quitslund  [analytics] `DASH__SUPPRESS_ANALYTICS` env handling (dart-lang/tools#2641)
  00c6e4ad  Fri Oct 2 17:17:49 2026 +0530  shivytyahoo  Fix VersionConstraint.intersection throwing on disjoint ranges (dart-lang/tools#2622)
  d523e1f8  Thu Oct 1 11:09:16 2026 +0200  Morgan :)  [unified_analytics] Append to the log file instead of rewriting it (dart-lang/tools#2613)
  6993cbad  Wed Sep 30 14:09:32 2026 +0300  Yusuf İhsan Görgel  [yaml] Improve error messages when mixing list and key-value syntax (dart-lang/tools#2610)
  068119b4  Tue Sep 29 23:21:03 2026 +0200  Lukas Klingsbo  [pubspec_parse] Support tag_pattern for git dependencies (dart-lang/tools#2565)
  7d859ed5  Mon Sep 28 20:28:23 2026 -0700  Kevin Moore  [api_summary] Add expectApiSummaryClean and --write/--check CLI flags (dart-lang/tools#2603)

web (https://github.com/dart-lang/web/compare/ec71a4c..63e6662):
  63e6662  Sat Oct 3 00:23:21 2026 +0000  Natalie Weizenbaum  Add a JSPair type (dart-lang/web#586)
  2d68d90  Fri Oct 2 20:26:48 2026 +0000  Natalie Weizenbaum  Use `js_interop` APIs in `js_interop_gen` (dart-lang/web#585)
  037823c  Thu Oct 1 21:59:21 2026 +0000  Natalie Weizenbaum  Add JSMap, JSSet, and JSSetLike types (dart-lang/web#571)
  b2d1d6b  Thu Oct 1 06:57:03 2026 +0000  dependabot[bot]  Bump the github-actions group with 3 updates (dart-lang/web#584)
  8dc8f75  Wed Sep 30 08:12:18 2026 -0700  Kevin Moore  [js_interop_gen] Split union/intersection AST types and extract TypeTransformer (dart-lang/web#580)
  751ccdf  Wed Sep 30 08:12:18 2026 -0700  Kevin Moore  [js_interop_gen] Decompose `Transformer._transformType` and deduplicate type caching (dart-lang/web#579)
  6f3dd4e  Wed Sep 30 08:12:17 2026 -0700  Kevin Moore  [js_interop_gen] Deduplicate `UnionType` and `IntersectionType` via `UnionOrIntersectionType` (dart-lang/web#578)
  d9f095e  Tue Sep 29 12:17:29 2026 -0700  Kevin Moore  [js_interop_gen] Remove unused JS interop extension types and bindings (dart-lang/web#577)
  07111ee  Tue Sep 29 09:51:36 2026 -0700  Kevin Moore  fix(js_interop_gen): update js_type_supertypes for Dart 3.12 (dart-lang/web#582)

Change-Id: I4532049c45561fe8f8c57713ac8ec2153096fd9a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/560660
Commit-Queue: Nate Bosch <nbosch@google.com>
Reviewed-by: Brian Quinlan <bquinlan@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[unified_analytics] Full log rewrite on every event once log is full

3 participants