Skip to content

Feat/delta prediction beta - #1095

Open
jimdroberts wants to merge 8 commits into
FirstGearGames:mainfrom
jimdroberts:feat/delta-prediction-beta
Open

jimdroberts wants to merge 8 commits into
FirstGearGames:mainfrom
jimdroberts:feat/delta-prediction-beta

Conversation

@jimdroberts

Copy link
Copy Markdown
Contributor

Problem

Prediction always sends replicates and reconciles with the full serializer. The delta path exists but cannot be turned on: every call site is under #if DO_NOT_USE, which is defined nowhere, and that branch no longer compiles. The delta primitives it would use also have bugs that only show once a delta serializer runs.

This came out of a production MMO project that has been running delta reconcile and delta replicate on a fork of FishNet 4.6.12. With the project's own serializers (tick rate 30, redundancy 3, walking character, measured by the project's benchmark test, not on this branch), reconcile went from 6750 to 980 B/s and the replicate packet from 85 to 51 B.

This PR puts delta prediction behind a beta option that is off by default, with a Beta menu entry.

Cause

  • NetworkBehaviour.Prediction.cs: the five delta sites (lines 409, 907, 990, 1080, 1468) are #if DO_NOT_USE. With the symbol defined, the branch does not compile: line 991 uses receivedReplicatesCount and arrBuffer, which no longer exist, and Writer/Reader.WriteDeltaReplicate/ReadDeltaReplicate take T : IReplicateData while the call sites now hold ReplicateDataContainer<T>. Line 1469 reads lastReconciledata, a casing typo. GetDeltaSerializeOption (line 854) returns FullSerialize before any of its logic.
  • Even if it compiled, the design would not hold up on an unreliable channel. Reconciles and replicates are sent unreliably, and PredictionManager also drops whole states without parsing them (dropReconcile at line 641 when reconciles are reduced with framerate, and older states at line 817). A delta decoded against a baseline the reader never received gives the wrong state, and nothing detects it. The replicate delta chained entry 0 to the previous packet, which makes RedundancyCount useless after a single loss.
  • Reader.ReadDeltaBoolean (Reader.Delta.cs:24) returns !valueA without reading, but WriteDeltaBoolean writes a byte. Each delta bool leaves the reader one byte behind, and a forced write of an unchanged value decodes inverted.
  • WriteDeltaSingle/Double/Decimal floor the difference (Writer.Delta.cs:191 and following lines), so the error always points toward the previous value and builds up along a chain.
  • WriteDeltaVector2/Vector3/TransformProperties call Skip(1) and then rewind only Position when nothing changed (lines 545, 608, 633). Skip grows Length (Writer.cs:175), so the skipped byte stays in GetArraySegment().
  • IsQuaternionChanged uses a fixed 0.0025 (line 571) and ignores the precision argument, which defaults to 0.0001. Any turn slower than about 0.29° per tick is never written.
  • QuaternionDeltaPrecisionCompression.Compress omits the component that changed most (line 83), and the reader rebuilds that component with a square root. Near identity a yaw mostly changes y, which is close to zero, so the reader rebuilds it from a w that has not changed at this precision. A slow turn from identity reads back as no rotation.
  • GenericWriter.SetWrite / GenericReader.SetRead (lines 32 / 34) say they unset a generated delta serializer. They test HasCustomSerializer without !, so they discard a custom one. If a type's delta serializer is registered before its regular serializer (the order of RuntimeInitializeOnLoadMethod calls is not defined), the delta serializer is dropped and cannot be registered again.

Fix

The PR has 8 commits. The first six fix the delta primitives. They only affect code that a delta serializer calls, so nothing changes on the stock path.

  1. ReadDeltaBoolean returns ReadBoolean().

  2. Round instead of floor in the three WriteDelta* writers. The precision type is chosen strictly below the width's maximum, so the rounded value still fits.

  3. Restore Length together with Position in the three writers that rewind.

  4. IsQuaternionChanged uses precision * 0.5, the smallest difference that compresses to a non-zero value.

  5. Invert the condition in GenericWriter.SetWrite / GenericReader.SetRead. FishNet generates no delta serializers today, so the corrected branch only clears a delegate that is already null.

  6. Compress omits the component with the largest magnitude, which is standard smallest-three and is always at least 0.5. Decompress is unchanged because the flags still identify the omitted component.

  7. The five sites become #if FISHNET_DELTA_PREDICTION. Design decisions:

    • Stock path unchanged. With the define off, Server_SendReconcileRpc, both replicate senders, the replicate reader and the reconcile reader run exactly the code they ran before. The #if only chooses between two call sites. The delta Writer/Reader methods and the NetworkBehaviour helpers are always compiled, so they cannot silently stop compiling again the way DO_NOT_USE did. They add 5 fields to NetworkBehaviour.
    • Replicate packets decode on their own. Entry 0 uses the regular serializer. Entries 1..n-1 are deltas against the entry before them in the same packet. A lost packet costs only that packet, and redundancy still works.
    • Reconcile deltas are written against the last full reconcile, not the previous reconcile. FishNet writes a one-byte header: a full flag and a 7-bit id of the full reconcile. Full reconciles use the regular serializer, because a delta serializer writes differences and needs a baseline even with FullSerialize. A delta against the previous reconcile would be about 20% smaller, but after any loss every later delta is unusable until the next full reconcile. The harness below measures both.
    • Built-in guard. The reader uses a delta only if it holds the full reconcile named by the delta's id. Otherwise the delta is discarded as if it had been lost. A tick check stops a wrapped id from matching a full reconcile written 128 or more ticks later. This works because at most one reconcile is written per tick and full reconciles are at most 60 ticks apart.
    • Restored GetDeltaSerializeOption. A full reconcile is written when none has been written yet, when an observer was added since the last full one, and after min(TickRate, 60) ticks. After a lost full reconcile, a reader recovers within one second.
    • Writer baseline. The writer's baseline is the full reconcile as readers decode it. The writer decodes its own output once per full reconcile, so a lossy full serializer cannot make the two sides diverge. The reader keeps its full reconcile in a separate field. lastReconcileData keeps its current meaning, the data passed to the user's reconcile.
    • Missing delta serializers. FishNet generates no delta serializers; users register GenericDeltaWriter<T>/GenericDeltaReader<T>. A prediction type without both is written exactly as without the define, with no header, and read the same way on both ends. It does not log per tick. ClearReplicateCache resets the delta state on both sides.
  8. Adds Tools/Fish-Networking/Beta/Enable|Disable Delta Prediction. This define changes the wire format. RemoveOrAddDefine only edits the selected build target group, and a dedicated server build has its own defines, so a Linux server and a Windows or WebGL client could end up disagreeing. The menu therefore uses a new DeveloperMenu.RemoveOrAddDefineForAllBuildTargets, which covers every non-obsolete BuildTargetGroup plus NamedBuildTarget.Server. The log message says servers and clients must be built with the same setting.

    A mismatch cannot be detected cheaply at runtime without changing the handshake. The stock format has no spare marker, and the first bytes of a reconcile are user data. The proper place for a check is a feature flag in the connection handshake, which I left out of a beta.

Primitives, upstream main vs this branch:

                               main                          branch
bool forced write        value False, next int 3160321   value True, next int 12345
unchanged Vector3        Length 1                         Length 0
5 m/s, 29 chained deltas 1.933 cm behind                  0.967 cm
0.1°/tick from identity  0/60 written, 6.00° error        60/60 written, 0.19° (0.04° with decoded baseline)
delta then regular reg.  delta writer/reader null         registered

Pipeline on the branch. 900 ticks at 30 Hz, a walking and turning character with a periodic jump, 10% random loss, 20 states dropped unread, and a 60-tick idle period:

branch                                  delivered 742, rejected 70, applied 672, bad 0, max error 0.07 cm / 0.04°
same encoding, reader without id check  applied 742, bad 70, max error 710 cm / 11°
bytes per reconcile: regular serializer 29.0 B, delta 15.0 B (header and full reconciles included)

usable/delivered  loss 0%   1%    5%    10%
vs last full      100%     100%   96%   93%   (15.0 B)
vs previous       100%      90%   46%   36%   (12.3 B)

replicate, 300 packets x 3 entries, 20% lost: 711 entries decoded from delivered packets, 0 differ; 30.9 B -> 15.2 B per packet
no delta serializers: reconcile and replicate bytes identical to the stock writers

Behavior change

None with the define off; the stock wire format is unchanged. GetDeltaSerializeOption still runs on the stock replicate path and its result is still ignored.

With the beta on:

  • The wire format changes for prediction types that have delta serializers registered, so every build must use the same setting.
  • Delta serializers must follow these contracts:
    • Write bytes if and only if they return true.
    • At the root (RootSerialize), always write something.
    • Consume the same number of bytes whatever the baseline. A rejected delta is still decoded to keep the reader aligned, because the reconcile length does not reach this code.
    • Never modify the baseline instance, which is reused until the next full reconcile.
  • Replicate deltas inside a packet use the writer's exact previous entry as the baseline, so a lossy replicate delta codec can differ from what the owner predicted. That is already true of lossy full serializers.
  • The replicate deltaOption parameter is now unused on both paths. I kept it to avoid changing the stock signature.

WriteDeltaBoolean writes the new value as a byte whenever it returns true, but
ReadDeltaBoolean returned !valueA without reading. Every delta boolean left the
reader one byte behind the writer, and a forced write of an unchanged value
(any option other than Unset) decoded as the opposite value.
WriteDeltaSingle, WriteDeltaDouble and WriteDeltaDecimal quantised a difference
with Math.Floor. For the unsigned writers that is the magnitude of the difference,
so every step loses up to one unit of precision in the same direction and the
decoded value trails the real one by an amount that grows along a delta chain.
Rounding halves the worst-case step error and removes the directional bias.
WriteDeltaVector2, WriteDeltaVector3 and WriteDeltaTransformProperties Skip(1) for
their flags byte and, when nothing changed, rewind Position only. Skip raises
Length and nothing lowers it, so when the unchanged value is the last thing
written GetArraySegment still includes the skipped byte. Inside a length-prefixed
RPC the reader does not consume that byte and the next packet in the same
buffer is parsed from it.
WriteDeltaQuaternion takes a precision (default QUATERNION_PRECISION, 0.0001) but
IsQuaternionChanged compared components against a fixed 0.0025. Any change below
that was reported as unchanged, about 0.29 degrees about a single axis, so a slow
turn was never written by a writer that advances its baseline each call. The
threshold is now half of precision, the smallest difference that compresses to a
non-zero value.
… after it

GenericWriter<T>.SetWrite and GenericReader<T>.SetRead are documented to unset a
generated delta serializer when a custom one is registered, but tested
HasCustomSerializer without negation, so they discarded the custom delta
serializer instead. Registering a type's delta writer/reader before its regular
writer/reader (the order of RuntimeInitializeOnLoadMethod callbacks is not
defined) left the delta serializer null, and because HasCustomSerializer stayed
true it could not be registered again.
QuaternionDeltaPrecisionCompression.Compress omitted the component whose value
changed the most, and Decompress rebuilds the omitted component from the other
three with a square root. Near identity a yaw changes y the most while y is close
to zero, so y was rebuilt as sqrt(1 - x^2 - z^2 - w^2) from a w that had not
changed at the compression precision: a slow turn from identity decoded as no
rotation at all. Smallest-three compression omits the largest-magnitude
component, which is always at least 0.5 and well conditioned. Decompress is
unchanged; the omitted component is still identified by the flags.
Every delta call site in NetworkBehaviour.Prediction.cs was under
`#if DO_NOT_USE`, which is defined nowhere and no longer compiled: the
replicate half referenced variables and container types that had since been
replaced, and the reconcile read used `lastReconciledata`. This replaces that
branch with a working one selected by FISHNET_DELTA_PREDICTION. Without the
define the wire format is unchanged.

Replicates: each packet is self-contained. The first entry is written with the
regular serializer and each following entry as a delta against the entry before
it in the same packet. Replicates are sent unreliably with RedundancyCount past
entries; a delta against an earlier packet would make every entry unreadable
once that packet was lost.

Reconciles: FishNet writes a one byte header, a full flag and a 7-bit id of the
full reconcile. A full reconcile uses the regular serializer, because delta
serializers write differences and cannot be read without a baseline even with
FullSerialize. Deltas are written against the last full reconcile, not the
previous reconcile: reconciles are sent unreliably and PredictionManager may also
drop whole states unread, and against the previous reconcile every loss would
make all later deltas unusable until the next full one. GetDeltaSerializeOption
is restored: full when none has been written, once per second (at most every 60
ticks), and when an observer was added since the last full reconcile.

The reader uses a delta only if it holds the full reconcile the delta names (id,
plus a tick check so the wrapped id cannot match a much later full reconcile),
and otherwise discards it as if it had been lost. The writer keeps as its
baseline the full reconcile as readers decode it, so a lossy serializer cannot
make the two sides diverge.

Types without both a registered GenericDeltaWriter and GenericDeltaReader are
written exactly as without the define (no header), on both ends.
Tools/Fish-Networking/Beta/Enable|Disable Delta Prediction toggles
FISHNET_DELTA_PREDICTION. RemoveOrAddDefine only edits the selected build target
group, and a dedicated server build has its own defines, so a server and its
clients could be built with different wire formats and fail silently. This
define is therefore set for every build target group and for the dedicated
server target, through a new DeveloperMenu.RemoveOrAddDefineForAllBuildTargets.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant