Skip to content

fix: Fixed a few memory leak cases when shutting down NetworkManager during Incoming Message Queue processing #1323

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Nov 12, 2021

Conversation

ShadauxCat
Copy link
Collaborator

Fixed a few memory leak cases when shutting down NetworkManager during Incoming Message Queue processing.

MTT-1440

(I'm not sure if I should update changelog.md since it still says [Unreleased] and I'm not sure if I should put this in that section or make a new one for 1.0.1)

Changelog

com.unity.netcode.gameobjects

  • Fixed: Fixed a few memory leak cases when shutting down NetworkManager during Incoming Message Queue processing.

Testing and Documentation

  • Includes integration tests.
  • No documentation changes or additions were necessary.

@@ -113,6 +114,16 @@ public void Dispose()
{
CleanupDisconnectedClient(kvp.Key);
}

// If there's a message currently being processed, it'll dispose itself, so we can't dispose here or that'll
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given we're single-threaded, is it possible we could be in the middle of processing a message and get here too?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So just to put this here even though we discussed it on slack... It's not an issue of concurrency, it's an issue of call stack.

Three messages come in: FooMessage, ClientRpc, BarMessage

We process FooMessage and FooMessage is Dispose()d

We start to process ClientRpc

ClientRpc message calls NetworkManager.Shutdown(), which calls MessagingSystem.Dispose()

After returning from ClientRpc, the ClientRpc message is Dispose()d.

Since MessagingSystem has now been Dispose()d as well, we can no longer process BarMessage, so we exit the loop.

Thus:

  • We can't Dispose() FooMessage because it was already Dispose()d
  • We can't Dispose() the current message being processed (ClientRpc) because HandleMessage is above us in the call stack and will Dispose() it before it returns
  • We must dispose BarMessage because HandleMessage will not get to that one as we're Dispose()ing the list that contains it.

That's what this comment means - it's not a threading concurrency, it's just that Shutdown() can be called from within HandleMessage(), and if that happens, the message currently being processed can't be Dispose()d.

@andrews-unity
Copy link
Contributor

So I have a question and this just for my understanding... Is there any reason why we just have some sort of flag that says currently we are processing messages and if you call dispose while we are processing these messages or Shutdown on NetworkManager, we will set m_Dispose to true and then once we are done processing all the messages we will dispose the NetworkManager ? Because I think in some cases you may want to process all the pending messages before you clean up stuff and then we don't have this magic index etc....

@unity-cla-assistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

var networkObject = m_NetworkManager.SpawnManager.CreateLocalNetworkObject(false,
spawnCommand.GlobalObjectIdHash, spawnCommand.OwnerClientId, (spawnCommand.ParentNetworkId == spawnCommand.NetworkObjectId) ? spawnCommand.NetworkObjectId : spawnCommand.ParentNetworkId, spawnCommand.ObjectPosition,
spawnCommand.ObjectRotation);
NetworkObject networkObject;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fix is technically not a part of this change, but the added tests won't pass without this change because it causes warnings to get logged.

@andrews-unity andrews-unity merged commit 5209fdc into develop Nov 12, 2021
@andrews-unity andrews-unity deleted the fix/fix_memory_leak_on_shutdown branch November 12, 2021 23:55
andrews-unity pushed a commit that referenced this pull request Nov 24, 2021
…during Incoming Message Queue processing (#1323)

* fix: Fixed memory leak on shutdown

* fix: Fixed a few memory leak cases when shutting down NetworkManager during Incoming Message Queue processing

* standards check

* Fixed test not cleaning up quite correctly.

* -Reworked the way shutdown is handled so it can't be processed while handling incoming messages.
-Added tests for both ways to shut down to make sure they both work correctly
-Fixed an issue in the snapshot system that was causing the new tests to fail

* standards

* Fixed broken test, slight naming change to be more accurate.
andrews-unity added a commit that referenced this pull request Nov 24, 2021
…during Incoming Message Queue processing (#1323) (#1466)

* fix: Fixed memory leak on shutdown

* fix: Fixed a few memory leak cases when shutting down NetworkManager during Incoming Message Queue processing

* standards check

* Fixed test not cleaning up quite correctly.

* -Reworked the way shutdown is handled so it can't be processed while handling incoming messages.
-Added tests for both ways to shut down to make sure they both work correctly
-Fixed an issue in the snapshot system that was causing the new tests to fail

* standards

* Fixed broken test, slight naming change to be more accurate.

Co-authored-by: Jaedyn Draper <284434+ShadauxCat@users.noreply.github.com>
@ashwinimurt
Copy link
Contributor

I will be updating the changelog.

mollstam pushed a commit to Keepsake-Games/com.unity.netcode.gameobjects that referenced this pull request Feb 13, 2023
…during Incoming Message Queue processing (Unity-Technologies#1323)

* fix: Fixed memory leak on shutdown

* fix: Fixed a few memory leak cases when shutting down NetworkManager during Incoming Message Queue processing

* standards check

* Fixed test not cleaning up quite correctly.

* -Reworked the way shutdown is handled so it can't be processed while handling incoming messages.
-Added tests for both ways to shut down to make sure they both work correctly
-Fixed an issue in the snapshot system that was causing the new tests to fail

* standards

* Fixed broken test, slight naming change to be more accurate.
mollstam pushed a commit to Keepsake-Games/com.unity.netcode.gameobjects that referenced this pull request Feb 13, 2023
…during Incoming Message Queue processing (Unity-Technologies#1323) (Unity-Technologies#1466)

* fix: Fixed memory leak on shutdown

* fix: Fixed a few memory leak cases when shutting down NetworkManager during Incoming Message Queue processing

* standards check

* Fixed test not cleaning up quite correctly.

* -Reworked the way shutdown is handled so it can't be processed while handling incoming messages.
-Added tests for both ways to shut down to make sure they both work correctly
-Fixed an issue in the snapshot system that was causing the new tests to fail

* standards

* Fixed broken test, slight naming change to be more accurate.

Co-authored-by: Jaedyn Draper <284434+ShadauxCat@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants