Skip to content

feat: snapshot. Adding RTT computation API. #963

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 19, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 96 additions & 0 deletions com.unity.multiplayer.mlapi/Runtime/Core/SnapshotRTT.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
using System;
using UnityEngine;

namespace MLAPI
{
internal class ConnectionRtt
{
internal const int RttSize = 5; // number of RTT to keep an average of (plus one)
internal const int RingSize = 64; // number of slots to use for RTT computations (max number of in-flight packets)

private double[] m_RttSendTimes; // times at which packet were sent for RTT computations
private int[] m_SendSequence; // tick, or other key, at which packets were sent (to allow matching)
private double[] m_MeasuredLatencies; // measured latencies (ring buffer)
private int m_LatenciesBegin = 0; // ring buffer begin
private int m_LatenciesEnd = 0; // ring buffer end

/// <summary>
/// Round-trip-time data
/// </summary>
public struct Rtt
{
public double BestSec; // best RTT
public double AverageSec; // average RTT
public double WorstSec; // worst RTT
public double LastSec; // latest ack'ed RTT
public int SampleCount; // number of contributing samples
}
public ConnectionRtt()
{
m_RttSendTimes = new double[RingSize];
m_SendSequence = new int[RingSize];
m_MeasuredLatencies = new double[RingSize];
}

/// <summary>
/// Returns the Round-trip-time computation for this client
/// </summary>
public Rtt GetRtt()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess one thing to think about - how many times do you think you might call GetRtt in a given frame? if you call it a bunch, you might have a dirty flag here and usually return a cached value. OR, if you need RTT almost every frame and call this multiple times, maybe the computation of the RTT could be in the NotifyAck? I guess that depends on how often you don't actually need the RTT data.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I lean this being "premature optimization" at the moment. I don't want to be cavalier about the feedback, it's good and very valid. But I'd wait until it shows in a profile. With the irons we already have in the fire, I'd wait before adding complexity here. Would that be workable to you?

{
var ret = new Rtt(); // is this a memory alloc ? How do I get a stack alloc ?
var index = m_LatenciesBegin;
double total = 0.0;
ret.BestSec = m_MeasuredLatencies[m_LatenciesBegin];
ret.WorstSec = m_MeasuredLatencies[m_LatenciesBegin];

while (index != m_LatenciesEnd)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll bet there is a Google interview question that asks how to do this operation in constant time :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm familiar with keeping a running sum and subtracting the element that are removed at the other end of the ring buffer. Then, you can divide by the number of elements.

However, I don't know of a good way to do it for the best and worst entries. I believe we can only do O(log(n)) but that would require an associative container for a handful of latencies. Seemed Overkill. Is it possible your comment forgot to consider min/max latency?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For min/max just keep track of the min/max and then compare < & > when adding / removing from the ring buffer?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, let's say stored latencies are [10, 60, 20, 60, 11, 80 ], min is 10, max is 80, a new latency of 15 needs to be stored. We now have [60, 20, 60, 11, 80, 15] (not in those positions, though). I can remove the min: 10, I can keep the max 80, but finding min: 11 requires a traversal.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess you could use 2 priority queues or heaps, one for min. But to your point we can save this for benchmarking

{
total += m_MeasuredLatencies[index];
ret.SampleCount++;
ret.BestSec = Math.Min(ret.BestSec, m_MeasuredLatencies[index]);
ret.WorstSec = Math.Max(ret.WorstSec, m_MeasuredLatencies[index]);
index = (index + 1) % RttSize;
}

if (ret.SampleCount != 0)
{
ret.AverageSec = total / ret.SampleCount;
// the latest RTT is one before m_LatenciesEnd
ret.LastSec = m_MeasuredLatencies[(m_LatenciesEnd + (RingSize - 1)) % RingSize];
}
else
{
ret.AverageSec = 0;
ret.BestSec = 0;
ret.WorstSec = 0;
ret.SampleCount = 0;
ret.LastSec = 0;
}

return ret;
}

internal void NotifySend(int sequence, double timeSec)
{
m_RttSendTimes[sequence % RingSize] = timeSec;
m_SendSequence[sequence % RingSize] = sequence;
}

internal void NotifyAck(int sequence, double timeSec)
{
// if the same slot was not used by a later send
if (m_SendSequence[sequence % RingSize] == sequence)
{
double latency = timeSec - m_RttSendTimes[sequence % RingSize];

m_MeasuredLatencies[m_LatenciesEnd] = latency;
m_LatenciesEnd = (m_LatenciesEnd + 1) % RttSize;

if (m_LatenciesEnd == m_LatenciesBegin)
{
m_LatenciesBegin = (m_LatenciesBegin + 1) % RttSize;
}
}
}
}
}
11 changes: 11 additions & 0 deletions com.unity.multiplayer.mlapi/Runtime/Core/SnapshotRTT.cs.meta

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 11 additions & 2 deletions com.unity.multiplayer.mlapi/Runtime/Core/SnapshotSystem.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,11 @@
using System.Collections.Generic;
using System.IO;
using MLAPI.Configuration;
using MLAPI.Messaging;
using MLAPI.NetworkVariable;
using MLAPI.Serialization;
using MLAPI.Serialization.Pooled;
using MLAPI.Transports;
using UnityEngine;
using UnityEngine.UIElements;

namespace MLAPI
{
Expand Down Expand Up @@ -275,9 +273,20 @@ public class SnapshotSystem : INetworkUpdateSystem, IDisposable
private NetworkManager m_NetworkManager = NetworkManager.Singleton;
private Snapshot m_Snapshot = new Snapshot(NetworkManager.Singleton, false);
private Dictionary<ulong, Snapshot> m_ClientReceivedSnapshot = new Dictionary<ulong, Snapshot>();
private Dictionary<ulong, ConnectionRtt> m_ClientRtts = new Dictionary<ulong, ConnectionRtt>();

private ushort m_CurrentTick = 0;

internal ConnectionRtt GetConnectionRtt(ulong clientId)
{
if (!m_ClientRtts.ContainsKey(clientId))
{
m_ClientRtts.Add(clientId, new ConnectionRtt());
}

return m_ClientRtts[clientId];
}

/// <summary>
/// Constructor
/// </summary>
Expand Down
75 changes: 75 additions & 0 deletions com.unity.multiplayer.mlapi/Tests/Editor/SnapshotRttTests.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
using MLAPI;
using NUnit.Framework;
using UnityEngine;

namespace MLAPI.EditorTests
{
public class SnapshotRttTests
{
private const double k_Epsilon = 0.0001;

[Test]
public void TestBasicRtt()
{
var snapshot = new SnapshotSystem();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about

  • a test that does > k_RingSize stores?
  • a test that ack's something that wasn't sent?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I'd rename this test to be something that communicates "send more, receive less".
also vice-versa option would be "a test that ack's something that wasn't sent?" aka "send less, receive more"

I'd also add:

  • send 2x more than k_RingSize
  • randomization: out-of-order send/recv variations (if ooo is ever possible, IDK)
  • randomization: sending/receiving (notify/ack) intentionally with varying artificatial latencies (WaitForSeconds)

at least, these are the test cases comes to my mind in a minute — perhaps more tests if applicable/sensible?

var client1 = snapshot.GetConnectionRtt(0);

client1.NotifySend(0, 0.0);
client1.NotifySend(1, 10.0);

client1.NotifyAck(1, 15.0);

client1.NotifySend(2, 20.0);
client1.NotifySend(3, 30.0);
client1.NotifySend(4, 32.0);

client1.NotifyAck(4, 38.0);
client1.NotifyAck(3, 40.0);

ConnectionRtt.Rtt ret = client1.GetRtt();
Assert.True(ret.AverageSec < 7.0 + k_Epsilon);
Assert.True(ret.AverageSec > 7.0 - k_Epsilon);
Assert.True(ret.WorstSec < 10.0 + k_Epsilon);
Assert.True(ret.WorstSec > 10.0 - k_Epsilon);
Assert.True(ret.BestSec < 5.0 + k_Epsilon);
Assert.True(ret.BestSec > 5.0 - k_Epsilon);

// note: `last` latency is latest received Ack, not latest sent sequence.
Assert.True(ret.LastSec < 10.0 + k_Epsilon);
Assert.True(ret.LastSec > 10.0 - k_Epsilon);
}

[Test]
public void TestEdgeCasesRtt()
{
var snapshot = new SnapshotSystem();
var client1 = snapshot.GetConnectionRtt(0);
var iterationCount = ConnectionRtt.RingSize * 3;
var extraCount = ConnectionRtt.RingSize * 2;

// feed in some messages
for (var iteration = 0; iteration < iterationCount; iteration++)
{
client1.NotifySend(iteration, 25.0 * iteration);
}
// ack some random ones in there (1 out of each 9), always 7.0 later
for (var iteration = 0; iteration < iterationCount; iteration += 9)
{
client1.NotifyAck(iteration, 25.0 * iteration + 7.0);
}
// ack some unused key, to check it doesn't throw off the values
for (var iteration = iterationCount; iteration < iterationCount + extraCount; iteration++)
{
client1.NotifyAck(iteration, 42.0);
}

ConnectionRtt.Rtt ret = client1.GetRtt();
Assert.True(ret.AverageSec < 7.0 + k_Epsilon);
Assert.True(ret.AverageSec > 7.0 - k_Epsilon);
Assert.True(ret.WorstSec < 7.0 + k_Epsilon);
Assert.True(ret.WorstSec > 7.0 - k_Epsilon);
Assert.True(ret.BestSec < 7.0 + k_Epsilon);
Assert.True(ret.BestSec > 7.0 - k_Epsilon);
}
}
}

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.