Spawning too many network objects at once

[Daniel4144]

Describe the bug
If too many network objects are spawned at once, they are spawned for the host, but not for the client (with Unity Transport).

To Reproduce
Steps to reproduce the behavior:

1. Create a Prefab with a NetworkObject component and add it to the NetworkPrefabs in the NetworkManager
2. Increase SendQueueBatchSize (tested with 6144000)
3. Start two instances (Host and Client) and spawn e.g. 1000 objects:

public void Spawn(int spawnCount)
{
    for (int i = 0; i < spawnCount; i++)
    {
        NetworkObject no = Instantiate(prefab);
        no.Spawn();
    }
}

4. The objects are spawned for the host, but not for the clients, no errors/warnings on both instances

Actual outcome

• The objects are spawned for the host, but not for the clients
• no errors/warnings on both instances
• existing NetworkObjects (e.g. with NetworkTransform) that were already spawned before are still synchronized from Client to Host, but not from Host to Client

Expected outcome
Ideally all objects are spawned on the client and everything is working.
If that is intended and NGO does not support that many objects, imo at least an exception should be thrown to indicate that something went wrong.

Environment (please complete the following information):

• OS: Windows 10
• Unity Version: 2021.2.7f1
• Netcode for GameObjects Version: 1.0.0-pre.4
• Unity Transport for Netcode for GameObjects Version: 1.0.0-pre.4

Additional context
If the Prefab contains only a NetworkObject, it works for up to 700 objects (750 does not).
If the Prefab also contains a Collider with gravity enabled (so all objects are moving at once when spawned), it only works for 200 objects (250 does not).
It does work with more objects, if I use the UNet Transport, but then the max MessageBufferSize (max 65535) is the limit and I cannot use the Relay Service with it.
I also considered spawning objects over time (e.g. max. 500 at once). It does work, if the client is already connected while spawning the objects, but I have the same problem, if a client connects afterwards (because all objects are spawned automatically when connecting).

[NoelStephensUnity]

@Daniel4144
I think that this could be similar to Github Issue 1564 but due to the number of objects this will need to be handled differently. This will need to be further explored as to how we will break up the initial Object synchronization as I think that this could get fixed by the snapshot system in a near future update.

[lcompton]

I've also run into this problem with Unity Transport. I was able to increase the number of network managed objects slightly by tweaking the three "queue size" parameters in the transport, but in my sample application the max was around 400. As I mentioned in #1564, I'm concerned about this number being so low. Is there a design goal for the minimum number of network managed objects NGO is intended to handle? What should we expect once the snapshot system is released?

[Daniel4144]

Hi @NoelStephensUnity
In version 1.0.0-pre.3 I encountered several different errors, but no KeyNotFoundException as in #1564 (but I haven't tested UNet Transport a lot).

Here is a sample project to reproduce my problem:
Netcode-issue-1570.zip
In this sample I am able to spawn thousands of objects (if I spawn max. 700 objects at once and the client conects before spawning)

Looking forward to try the snapshot system. How will the snapshot system work? Does it handle synchronization only when connecting or always when a lot of data is sent (to handle spawning during runtime)?
As a workaround I will try the ObjectVisibility to spawn objects manually over time.

@lcompton
Depending on your use case you could also use the ObjectVisibility to spawn objects only, if they are in range (assuming you don't need all objects at once).

[ezoray]

Daniel I've been having similar problems spawning too many network objects (I've been harping on about it on the forums under cerestorm). Object visibility appears to be broken in pre.4 so may not be an option right now, there's a raised issue here #1559.

[lcompton]

@Daniel4144
Thank you for the tip about object visibility. That will help.

[NoelStephensUnity]

@Daniel4144
I think this PR-1596 should help you, but it will require that you switch from UNet over to the UnityTransport (UTP).
The issue with high NetworkObject synchronization is still being discussed internally as to the various ways this can be addressed. The specifics are still being sorted out and so I am hesitant to provide any specifics, but I can say that we are aware of this issue and are currently investigating various ways to help mitigate it.
The next update will hopefully will include PR-1596 which will provide you with the ability to increase the maximum buffer size to whatever size you need. I downloaded your project and applied the changes in PR-1596 and had the host running as a stand alone build and the client in the editor. I basically made the default player prefab the same NetworkObject you were spawning so there is a +2 delta from the total NetworkObjects to spawn.

What the screenshot shows:

1. I started the host as stand alone and then spawned 700 objects
2. I then connected a client (from within the editor to see if the console showed any errors when the client first connects to a server with 700+ NetworkObjects spawned)
3. As you can see it yielded no errors and the end result was 702 objects (700 objects plus host player and client player).

I then shutdown the host and client and then started the process again with 1000 objects to spawn:

You will eventually reach the buffer size maximum, but you should be able to adjust it to fit your needs.

[Daniel4144]

Thank you for the update @NoelStephensUnity, that sounds great. Unfortunately I wasn't able to test it, but I am looking forward for the next version.

[Daniel4144]

This issue is fixed in the latest version 1.0.0-pre.5, I was able to spawn several thousand objects at once. Thanks for your help

[lcompton]

@Daniel4144 Would you share the transport parameters you used when you were able to successfully spawn several thousand objects?

[Daniel4144]

Hi @lcompton,
I can´t remember exactly, sorry. I just added one or two zeros to the buffer sizes until it worked (I think I did not change max packet queue size). If there are a lot of networked components such as Networktransforms on your objects, try to spawn empty NetworkObjects without any components first. Also, if spawning takes too long (heavy workload in Awake/Start, loading models etc.) and your application freezes temporarily, try to increase the timeout parameters, just add some zeros for testing.
I did not change anything that is not exposed in the inspector.