Repository navigation
dbserver: match rekordbox's menu protocol against the link-spec suite - #62
Merged
Merged
Conversation
Two crashes the link-spec conformance suite (evanpurkhiser/link-spec) surfaced in our dbserver, both reachable from the wire: - ParseDBMessage guarded len >= 11 but then slices the 12-byte tag array at data[7:19], so an 11-18 byte frame panicked the decoder — and a panic in a dbserver session goroutine takes down the process. Any peer could crash us with a short/truncated frame. Require the full 19-byte header. Covered by a malformed-input test (every length 0-32 plus oversized binary/string length fields; must never panic). - handleGetSongStructure (0x2504) dereferenced h.analysis without a nil check, unlike every other analysis use (which goes through the guarded lazyAnalyze). Nil-guard it so a song-structure request can't crash when no analysis store is wired. Found via the conformance report Evan produced against Vynull (malformed-framing and adjacent-payload suites); real robustness bugs independent of conformance.
The link-spec conformance report's root blocker: no suite matched because the setup exchange differed. Per the rekordbox-7.2.19 oracle (link-spec PROTOCOL_REFERENCE "Setup variants"), the reply shape depends on the request: legacy (1-arg): kind 0x4000, args [0, server_device] extended (2-arg): kind 0x0000, args [server_device, echo-of-arg1] handleSetup always sent the legacy shape (0x4000 [0, dev]), so a client using the extended setup (the conformance client; rb replied 0x0000 [17,20] while we replied 0x4000 [0,17]) never matched. Branch on arity: keep the legacy reply byte-identical (this is what real players we've tested, e.g. XDJ-RX3, send, so hardware behaviour is unchanged) and add the extended reply. Since setup is the first exchange, this should let many suites' setup match. HARDWARE NOTE: legacy path untouched; the extended branch only fires for 2-arg setups and mirrors rekordbox. Validate on a real CDJ before relying on it, per the usual handshake caution. Unit-tested both arities.
Stand Vynull's dbserver up on loopback against a fixture library so the
evanpurkhiser/link-spec conformance suite can replay the 287 rekordbox-7.2.19
golden suites against us (replay_vynull.py builds and drives this binary).
- cmd/conformance-server: --fixture-json (a tools/rekordbox_dump.py dump) is
loaded into a library, --second-column sets the detail column; serves the
dbserver on 127.0.0.1 and prints the readiness line
{status, listen_address, query_port} the adapter reads from stdout (all logs
go to stderr).
- library.ImportMasterDBDump: factored out of ImportRekordboxMasterDB so the
server can load an already-decoded dump without re-running the Python helper.
- dbserver: BindHost (loopback for conformance; empty = all interfaces as
before), a Ready channel + QueryPort accessor so the server learns its
dynamic port, and an exported DefaultRootMenu so the conformance menu can
reuse the built-in categories while supplying its own second column.
Smoke-tested: loads a fixture, binds loopback, emits correct readiness, and
accepts a dbserver handshake. Follow-up for fuller coverage: wire playlists /
cues / tags from the import bundle. Full suite green.
The link-spec client feeds the readiness query_port into its port-query (RemoteDBServer discovery) to learn the dynamic port, so query_port must be the discovery port (12523), not the dynamic one. Report 12523 and expose the dynamic port separately as dynamic_port for diagnostics. Verified end to end: the lab's remote_db client discovers the dynamic port, completes the greeting, and its strict extended-setup check accepts our reply (kind 0x0000 [17, 20]) — byte-identical to rekordbox's documented reply — then a root-menu query returns the 13 default categories.
The link-spec conformance harness runs the dump without putting the DB key on the command line, passing it in VYNULL_REKORDBOX_KEY instead. Accept the key from that env var when the second argument is absent (arg still wins when given); error if neither is present. Enables the harness to dump fixtures.
rekordbox's 0x4101 row has 16 arguments; a legacy row is the first 12. The 2-argument (extended) setup selects the 16-arg form. We always emitted 12, so args 12-15 were null in every row against the link-spec oracle. Track handler now carries a per-session 'extended' flag (set when the setup request had 2 args; the legacy 1-arg path real CDJs use is untouched). When set, the row builder appends the key/BPM tail: arg 12 = KeyID, 13 = UTF-16 byte length of arg 14, 14 = key display string, 15 = BPM*100 — plain string encoding throughout. Non-track rows use the empty tail [0, 2, "", 0]; track rows reuse the menuItem's sortKey/sortBPM (keyID is 0 until ID preservation lands). Verified via the link-spec harness: category rows' args 12-15 now match rekordbox exactly (genres etc. differ only on the row ID now), confirming the layout. Converting those to exact needs the complementary ID-preservation fix (we still serve hashed IDs, not the rekordbox lookup IDs). Layout per link-spec ROW_LAYOUT.md.
Two parts of the link-spec ID-preservation gap (finding B): - masterDBToLibrary now sets the library track ID to the rekordbox djmdContent.ID (addTrackLocked respects a preset ID; idMap still resolves identically). Served track rows now carry rekordbox's content IDs instead of our sequential/hash IDs — harness-verified: the tracks case's arg[1]/arg[8] content ID now matches the 7.2.19 oracle. - rekordbox_dump.py filters rb_local_deleted rows, which rekordbox omits from listings. Removes the stray 'Deleted Fixture Track' the harness surfaced. This is correct groundwork; it does not yet flip cases to exact on its own. The remaining needle-mover is category lookup-ID preservation (genre/artist/ album/label/key IDs), which the dump currently drops and the category builders replace with HashID — that is the next step.
Category rows (genre/artist/album/key/label) carried a stable hash of the value name as their ID. rekordbox serves the djmd* lookup-table row IDs, so decks drilling into a category sent back our hash and every ID-bearing row diverged from rekordbox's wire output. Preserve the real IDs end to end: the master.db dump now emits each track's GenreID/ArtistID/AlbumID/KeyID/LabelID, the importer records a value-name to ID map on the library, and a shared Handler.catID helper returns the real ID (falling back to the name hash for non-rekordbox libraries). Both row generation and drill-in matching route through catID, so they stay in lockstep. Track rows also carry the real KeyID in the extended-row tail. Album rows additionally repeat their AlbumID in arg[8] (rekordbox's album-art handle), unlike the other category rows which leave it zero. Lifts the link-spec conformance suite from 18 to 72 of 423 cases exact: genres, artists, albums, labels, label-artists, and label-albums now match rekordbox 7.2.19 byte for byte.
link-spec is the dbserver conformance suite that is surfacing the wire-format divergences from rekordbox we have been closing. Credit it alongside the other reverse-engineering work the project builds on.
Track menu rows carried several guessed flag values that disagreed with rekordbox's own output. Correct them against the link-spec ROW_LAYOUT.md, verified by the conformance oracle: - arg 7: song-tag bit (0x1) when the track is in the Tag List (djmdSongTagList), replacing a bogus constant 0x01000000. - arg 8: repeat djmdContent.ID, not the artwork ID. - arg 9: 0 for the direct track list (was a stray 2). - arg 10: new-CDJ compatibility bit (FLAC and off-rate WAV/AIFF) plus the hot-cue-auto-load bit, driven by rekordbox's FileType enum rather than the file extension (which can disagree) and the HotCueAutoLoad field. - arg 11: the converted Camelot 1-24 key index, derived from the musical key regardless of the configured display notation. The rekordbox FileType enum is now authoritative over the extension for the codec too, so a track stored as FileType 1 but named .wav decodes as MP3. The default (sort=0) all-track list now sorts alphabetically by title to match rekordbox; the library was returning ID order. Per-page menu headers carry the page's starting row offset instead of a constant 0. The dump gains FileType, HotCueAutoLoad, and djmdSongTagList membership. Lifts the link-spec conformance suite from 72 to 126 of 423 cases exact: tracks, bpm-tracks, year-tracks-all, color-tracks, time-tracks, and bitrate-tracks now match rekordbox 7.2.19 byte for byte. Hardware testing: the track-row flag changes affect what real CDJs receive when browsing a served library; validate browse/load on hardware before release.
The root LINK menu served 13 categories in our own order; rekordbox 7.2.19 serves 20 in a fixed order with per-category item types. Replace the default set with rekordbox's exact list (TRACK, KEY, BPM, GENRE, ARTIST, ALBUM, MATCHING, SEARCH, PLAYLIST, HISTORY, BITRATE, COLOR, FILE NAME, HOT CUE BANK, LABEL, ORIGINAL ARTIST, RATING, REMIXER, TIME, YEAR). A wired MenuSource (the web UI's configurable menu) still overrides it. GENRE was a flat genre-to-tracks jump; rekordbox drills genre -> artist -> album -> track with an [ALL] row at each level, like LABEL already did. Add the 0x1101/0x1201/0x1301 handlers mirroring the label hierarchy, and give the ARTIST drill (0x1102) its missing [ALL] row. Hierarchy track leaves now carry the packed context track type (TT<<24) in argument 7, which rekordbox adds for genre/artist/album/label leaves but not for the direct track list or scalar filters. Album-scope track leaves put the track number in argument 9. The artist->album->track handler (0x1202) now filters by artist and album, honouring the ALL sentinel at either level, so an artist's tracks across all albums resolve instead of coming back empty. Lifts the link-spec conformance suite from 135 to 198 of 423 cases exact: the root menu plus the full genre/artist/album/label drill-down now match rekordbox 7.2.19 byte for byte.
Both categories came back empty: the master.db dump never joined djmdContent.RemixerID / OrgArtistID, so the imported library had no remixer or original-artist names, and the list item types were wrong (0x08/0x0b instead of rekordbox's 0x29/0x28). REMIXER's request kind (0x1602) was also still wired to an old preflight-ack stub. Join the remixer and original artist (both djmdArtist rows) in the dump and carry their names plus IDs, register their category IDs on import, and serve the drill like an artist does: entry -> album -> track, with an [ALL] row at the album level and album-scope track numbers at the track level. Dispatch the remixer (0x1602/0x1702/0x1802) and original-artist (0x1302/0x1402/0x1502) hierarchies to the new handlers. The remixer list now returns a normal success in place of the preflight stub, which still satisfies the deck's ack. Lifts the link-spec conformance suite from 198 to 252 of 423 cases exact: all six remixer and original-artist cases now match rekordbox 7.2.19 byte for byte. Hardware testing: 0x1602 changed from a bare ack to the remixer list; confirm browsing REMIXER (and that SEARCH still opens its keyboard) on a deck.
Playlist menu rows carried no argument-9 sequence, playlist track leaves were force-sorted and lacked the playlist-scope conventions, and the conformance server served no playlist tree at all (it dropped the imported bundle). Serve the playlist tree in the conformance server straight from the dump (which keeps rekordbox's playlist IDs, parent IDs, Seq, and membership order), listing folders before playlists and each group in Seq order, as rekordbox does. Playlist and folder menu rows now carry the rekordbox display sequence in argument 9. Playlist track leaves keep playlist order for the default sort, number each row by its 1-based playlist position in argument 9, and carry the packed context track type in argument 7, like the other hierarchy leaves. The dump and PlaylistEntry gain the playlist Seq; argument 9 is now filled from the producing handler for every row kind, not just album track leaves. Lifts the link-spec conformance suite from 252 to 279 of 423 cases exact: all three playlist cases now match rekordbox 7.2.19 byte for byte. Hardware testing: playlist track rows now preserve playlist order on the default sort (was alphabetical) and carry position/context flags; confirm PLAYLIST browse and load on a deck.
…ime/bitrate) The BPM, RATING, YEAR (decade + year), TIME, and BITRATE category rows all carried a formatted label string and some were ordered or bucketed wrongly. rekordbox sends these numeric rows with an empty label (the deck draws the value from argument 1) and its own ordering and bucket boundaries: - RATING: always six rows, 5 down to 0 (unrated), descending. - YEAR decades and TIME/BITRATE buckets: descending. - Decades leave out year 0 (unset) and years >= 3000 (invalid). - TIME groups by whole minute and drops durations of 3 hours or more. - BITRATE keeps a 0 bucket for tracks with an unknown bitrate. Lifts the link-spec conformance suite from 279 to 333 of 423 cases exact: bpm, ratings, decades, years, times, and bitrates (and their drill-ins) now match rekordbox 7.2.19 byte for byte.
…labels Four more category fixes against the conformance oracle: - Rating: rekordbox's master.db stores djmdContent.Rating as the raw 0-5 star count (confirmed against a real rb6 library: values 0,2,3,4,5), not 0-100. The dump was dividing by 20, which silently zeroed every imported rating and made the RATING drill return nothing. Pass the value through unchanged. - COLOR: drop the leading "no colour" row (rekordbox lists only the eight named colours) and title-case the names, matching the 0x13+id row types. - FILE NAME: render each entry as a full track row (detail column and wire flags) labelled with the file name, not a bare row. - BPM distance ranges: send an empty label like the other numeric rows; the deck formats the +/-% from arg 1 (we were putting a track count there). Lifts the link-spec conformance suite from 333 to 369 of 423 cases exact.
The KEY list showed only the keys present in the library, each keyed by its djmd KeyID. rekordbox always lists all 24 harmonic positions in wheel order, keyed by the 1-24 "new key" index and labelled in the configured notation. Serve the fixed 24-entry list (Classic names, from link-spec's KEY_NOTATION_ORACLE.md); a served USB (PDB) keeps its own key list. Lifts the link-spec conformance suite from 369 to 378 of 423 cases exact: the keys list now matches rekordbox 7.2.19 byte for byte.
The KEY drill-ins still used the old djmd KeyID model; with the KEY list now keyed by the 1-24 harmonic index, they resolved nothing. Rework them around the index: - Key distances (0x1114): serve the three compatibility rings for the selected index, labelled in Classic notation (e.g. "Abm", "Abm, B", "Abm, B, Dbm, Ebm"), with the index in arg 1 and the distance in arg 0. - Key tracks (0x1214): rekordbox's KEY track view is not a harmonic filter; it returns the whole collection title-sorted as a scalar leaf (no context type). Match that rather than filtering by the ring. Adds keyIndexToCamelot / camelotToKeyIndex / classicKeyName helpers. Lifts the link-spec conformance suite from 378 to 396 of 423 cases exact: the key distance and key track views now match rekordbox 7.2.19 byte for byte. Only the matching, hot-cue-bank, and search cases remain.
The KEY drill rework no longer resolves a key name from an ID or counts tracks per key group, so these helpers are unreferenced.
MATCHING (0x1017) was unhandled. It is a relationship query over djmdRecommendLike, not a computed harmonic match: each row links two tracks, and the deck matches both directions. The dump now exports the (non-deleted) recommendation pairs, the importer records them as a symmetric related-track graph on the library, and the new handler returns a track's related tracks as a scalar leaf (no context type), numbered by track number in arg 9 and title-sorted. Lifts the link-spec conformance suite from 396 to 405 of 423 cases exact: the matching-forward-reverse case now matches rekordbox 7.2.19 byte for byte.
SEARCH matched a lowercased whole-query substring against title/artist/album only, and rendered results as forced-BPM rows. rekordbox's search (djeplGetNewSearchResult) splits the query into space-separated tokens, upper- cases ASCII only, and requires every token to occur as a substring; it spans four domains (artist, remixer, and original-artist names; album names; and tracks by Title or file name), listing matching artists and albums ahead of content rows. Rewrite handleSearch to tokenise and match across those domains. Content rows are now standard track rows labelled with whichever field matched (Title or file name) and carrying the Key secondary column (key alone), which is why a file-name match shows the file name. Artist rows cover remixers and original artists, not just the main artist. Lifts the link-spec conformance suite from 405 to 414 of 423 cases exact: the search case now matches rekordbox 7.2.19 byte for byte. Only hot-cue-bank remains.
The HOT CUE BANK root request is 0x1018 (we had only the NXS2 catalogue kind 0x2001 wired), so it fell through to the default success echo and returned an empty menu. rekordbox serves no catalogue at this root and instead replies with a menu-unavailable error (kind 0x4003) whose sole argument echoes the request kind. Add the 0x4003 reply kind and dispatch 0x1018 to it. Completes the link-spec conformance suite: 423 of 423 cases exact across all nine device models (rekordbox 7.2.19), every suite 47/47.
vynulldev
force-pushed
the
dbserver-robustness
branch
from
October 10, 2026 18:16
5c383a7 to
dcef176
Compare
vynulldev
marked this pull request as ready for review
October 10, 2026 18:38
vynulldev
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Stands Vynull's dbserver up against the evanpurkhiser/link-spec conformance suite and closes the gaps it surfaced. The suite replays rekordbox 7.2.19's own menu responses as goldens, so every divergence becomes a concrete, checkable diff instead of a guess. This branch takes the full suite from 18 to 423 of 423 cases exact across all nine device models (xdj-rx3, cdj-3000, cdj-2000nxs2, xdj-xz, xdj-az, xdj-1000mk2, the two unknown profiles, and the cdj-2000nexus status model), every suite 47/47.
Almost none of this was visible before: the decks were happy with our responses, so the only way to see where we diverged from rekordbox on the wire was to byte-diff a capture one request at a time. A replayable oracle is the automated backstop we did not have.
What changed, by area:
Two fixes in here matter independent of conformance:
Also adds cmd/conformance-server (a loopback harness entry point the suite's adapter drives) and the dump-tool/library plumbing it needs. No fixtures, databases, keys, or captures are committed; the harness builds its fixtures from the user's own rekordbox master.db at runtime.
Hardware testing
Matching rekordbox on the wire is not a reliable proxy for correctness, and this branch changes a lot of what real decks receive while browsing, so it was smoke-tested on hardware: connecting, loading and paging through the menus, browsing the categories, and loading tracks all work as expected.
Checklist
go build ./...,go vet ./..., andgo test ./...passgofmt -l .is cleanGPL-3.0-or-later)