Query transforms return differently shaped data on aw-server and aw-server-rust (merge_events_by_keys, chunk_events_by_key, split_url_events, tag)

Found by the Python/Rust query parity suite (PR to follow in this repo). These transforms return differently shaped data on aw-server (aw-core 33c8528) and aw-server-rust (c68d32c) for the same input. Each needs a decision on which behaviour is the spec, so filing here rather than on one server.

merge_events_by_keys

Bucket a: (10:00, 10 s, {app: x, title: t1}), (10:01, 5 s, {app: x, title: t2}), (10:02, 7 s, {title: "no app"})

a = query_bucket("a"); RETURN = merge_events_by_keys(a, ["app"]);

Python: {app: x} 15 s, {} 7 s (only the merge keys are kept; events missing a key are grouped under the keys they do have)
Rust: {app: x, title: t2} 15 s (the first event's whole payload is kept, so an arbitrary title leaks into app-level results; events missing a key are dropped)

Also merge_events_by_keys(a, []) returns the input unchanged in Python and [] in Rust. aw-webui's multideviceQuery relies on the Rust behaviour of dropping events without a title.

chunk_events_by_key

Bucket a: (10:00:00, 10 s, {app: x, title: t1}), (10:00:10, 5 s, {app: x, title: t2})

a = sort_by_timestamp(query_bucket("a")); RETURN = chunk_events_by_key(a, "app");

Python: {app: x, subevents: [<both events>]} 15 s
Rust: {app: x, title: t1} 15 s (no subevents)

split_url_events

url = "https://x.org:8080/a;p?b=c#d"

Python: $domain: "x.org:8080", $path: "/a", $params: "p", $options: "b=c", $identifier: "d", $protocol: "https"
Rust: $domain: "x.org", $path: "/a;p", $params: "b=c", $protocol: "https" (no $options/$identifier)

For a non-URL ("not a url") Python adds all keys with empty values ($path: "not a url"), Rust leaves the event unchanged. $domain is what aw-webui groups browser time by, so a port in the URL splits a site into separate domains on Python only.

tag

tag(a, [[["Work"], {"type": "regex", "regex": "Code"}]]) (category-style list names) works in Python and fails in Rust with Expected function parameter of type String. String names ([["Work", {...}]]) work in both.

One more for tag: Rust sorts and dedups $tags (tags.sort_unstable(); tags.dedup() in aw-transform/src/classify.rs), Python keeps them in rule order. With rules [["Work", {"type": "regex", "regex": "Terminal"}], ["Comms", {"type": "regex", "regex": "Inbox"}]] an event {app: "Terminal", title: "Inbox"} gets $tags: ["Work", "Comms"] on aw-server and ["Comms", "Work"] on aw-server-rust.

Parity suite: #1467. The affected cases are strict xfails there, so they XPASS once the fix reaches the pinned submodules.

@ErikBjare

Proposal for @ErikBjare to decide. Nothing has been changed yet. This is based on what aw-webui (master 23c95ed) and aw-client (master) actually read, the docs, and the docstrings/TODOs on both sides.

`merge_events_by_keys`: change Python to match Rust (keep the first event's payload, drop events missing a key)

Payload: aw-webui reads non-key fields from merged events. top_titles is colored by data['$category'] on merge_events_by_keys(events, ["app", "title"]), and top_urls/top_browser_titles are colored by data.$domain on merges by ["url"]/["title"]. Python strips those fields, so on aw-server every row in these lists falls back to the same default color today.
Missing keys: aw-webui's canonicalEvents and multideviceQuery have comments explicitly written around "merge_events_by_keys drops events missing any requested key" (Android events without title). Python instead groups them under the keys they do have, which can also collide: {a: 1} and {b: 1} merged by ["a", "b"] both get the key (1,).
[] keys: no caller. Returning [] like Rust is fine.
Rust's doc comment (Example 1 keeps {b: 1}) contradicts its own implementation and should be fixed to describe the drop.
Blast radius: aw-core only. On aw-server, app-level results gain a stray title (the same "arbitrary first title" Rust already returns). Optional follow-up for both servers: keep only fields whose value is the same across the whole group. That keeps $category/$domain and drops the arbitrary title.

`chunk_events_by_key`: Python's shape is the spec, but I'd deprecate it rather than fix Rust

The aw-core docstring defines subevents, and Rust says it is incomplete: "Does not support sub-chunking which aw-server-python supports ... mostly just for backwards compatibility with older versions of aw-webui".
No consumer: aw-client doesn't call it, and aw-webui's only subevents reader (TimelineInspect.vue) is registered as a component but never used.
Python has its own bugs too: the gap check uses events[-1] (the last input event) instead of the previous one, and it breaks at the first event missing the key, where Rust skips it.
Blast radius: none in first-party code. Either deprecate it in both (document merge_events_by_keys instead), or have Rust add subevents and the pulsetime gap.

`split_url_events`: Python moves to Rust's `$domain`/`$path`/`$params`

$domain is the only derived field any first-party code reads (webui top domains, category/activity grouping). Python includes the port (and any user@), so x.org:8080 becomes its own domain. The Python TODO ("Parse user, port etc aswell") suggests that wasn't intended. Use hostname (minus www.).
$path/$params: Python's $params is the obsolete ;param path segment (RFC 1808) and $options is the query. Rust's $params = query is what people expect. Nothing first-party reads either.
Non-URLs: Rust leaves the event alone. Python adds empty keys and puts the whole string in $path. Rust's behavior is better.
$options/$identifier: Rust has a TODO: aw-server-python also has options and identifier, so the original intent was to add them. Since $params is being redefined anyway, I'd drop $options (it duplicates $params) and optionally add $identifier (fragment) to Rust.
Blast radius: hand-written queries on aw-server that use $params/$options, and domain totals on aw-server that merge across ports (for example localhost:3000 and localhost:8080 become localhost).

`tag`: Python moves to Rust (string names only, `$tags` sorted and deduplicated)

List names ([["Work"], rule]) only work in Python by accident: Tag = str, and the result is $tags: [["Work"]], which nothing can consume. Category-style list names belong to categorize.
Dedup matters, because two rules for the same tag give ["Work", "Work"] in Python. Sorting vs. rule order is arbitrary, but Rust's order is deterministic and already shipped.
Blast radius: none found. No aw-webui, aw-client or docs usage of tag/$tags.

Summary: Python changes for merge_events_by_keys, split_url_events and tag (one aw-core PR each, all small). chunk_events_by_key needs a call: deprecate it (my lean) or port subevents to Rust.

test(query-parity): attribute known failures to their measured causes

build(deps): bump aw-core to 7b24386 and aw-tauri to ec46d16

build(deps): bump aw-core to d6ec34c (aw-core#165)

Decided: aw-core (Python) adopts aw-server-rust's shape for merge_events_by_keys, split_url_events and tag, and chunk_events_by_key is deprecated in both (behavior kept for now).

feat(transform)!: match aw-server-rust's merge_events_by_keys, split_url_events and tag aw-core#173: the three transforms plus the Python deprecation
chore(transform): deprecate chunk_events_by_key aw-server-rust#758: Rust deprecation (#[deprecated] plus a warning when a query uses it)
fix(transform): return merge_events_by_keys groups in first-seen order aw-server-rust#757: merge_events_by_keys returned HashMap order in Rust, so order-sensitive queries (webui multidevice with Android merged first) changed from run to run. It now keeps first-seen order, like Python. Also fixes its doc comment.
fix(query): don't modify events that other query variables still hold aw-core#172 (found while measuring): Python transforms modified events that other query variables still held (period_union, categorize, tag, split_url_events), which broke aw-client's always_active_pattern query and the webui multidevice query.

test(query-parity): attribute known failures to their measured causes (…

build(deps): bump aw-core and aw-server-rust for the parity fixes (#1470

feat(transform)!: match aw-server-rust's merge_events_by_keys, split_…

chore(transform): deprecate chunk_events_by_key (#758)

feat(transform)!: match aw-server-rust's merge_events_by_keys, split_…

build(deps): bump aw-core for the query shape fixes, update parity kn…

Navigation Menu

Repository navigation

Query transforms return differently shaped data on aw-server and aw-server-rust (merge_events_by_keys, chunk_events_by_key, split_url_events, tag) #1466

merge_events_by_keys

chunk_events_by_key

split_url_events

tag

`merge_events_by_keys`: change Python to match Rust (keep the first event's payload, drop events missing a key)

`chunk_events_by_key`: Python's shape is the spec, but I'd deprecate it rather than fix Rust

`split_url_events`: Python moves to Rust's `$domain`/`$path`/`$params`

`tag`: Python moves to Rust (string names only, `$tags` sorted and deduplicated)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Query transforms return differently shaped data on aw-server and aw-server-rust (merge_events_by_keys, chunk_events_by_key, split_url_events, tag) #1466

Description

merge_events_by_keys

chunk_events_by_key

split_url_events

tag

Activity

ErikBjare commented on Sep 26, 2026

ErikBjare commented on Sep 26, 2026

ErikBjare commented on Sep 26, 2026

merge_events_by_keys: change Python to match Rust (keep the first event's payload, drop events missing a key)

chunk_events_by_key: Python's shape is the spec, but I'd deprecate it rather than fix Rust

split_url_events: Python moves to Rust's $domain/$path/$params

tag: Python moves to Rust (string names only, $tags sorted and deduplicated)

ErikBjare commented on Sep 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`merge_events_by_keys`: change Python to match Rust (keep the first event's payload, drop events missing a key)

`chunk_events_by_key`: Python's shape is the spec, but I'd deprecate it rather than fix Rust

`split_url_events`: Python moves to Rust's `$domain`/`$path`/`$params`

`tag`: Python moves to Rust (string names only, `$tags` sorted and deduplicated)