Implement debug to detect game state mismatch

[xezon]

Implement debug to detect game state mismatch.

Strategy 1

We can aggressively log out evolution of hashes every (logic) frame to see how they evolve, then take the log from 2 clients that mismatched and compare them to see where that mismatched hash came from. This potentially would also work for Replays, aka get 2 replays that had mismatched, run one after another, then compare the logs.

Reference on mismatch detection strategy:
https://springrts.com/wiki/Debugging_sync_errors

Additional note

Quote from Kris Morness via "John Wick":

It would be easy to disable the CRC mismatch. The highlighted code could be skipped by hooking up a new commandline arg like -ignoreDesyncs

If your playing with friends, then that would probably be fine, but you may still encounter situations where the game diverges and multiple people win. The longer a desynced game goes, the bigger the convergence.

I don't have the code compiled or anything but now with the open source, there are likely modders already planning to patch the exe, this is a very simple change!

[tintinhamans]

GeneralsGameCode/GeneralsMD/Code/GameEngine/Source/GameLogic/System/GameLogic.cpp

Lines 2555 to 2604 in 0a05454

	if (m_shouldValidateCRCs && !TheNetwork->sawCRCMismatch())
	{
	Bool sawCRCMismatch = FALSE;
	Int numPlayers = 0;
	DEBUG_ASSERTCRASH(TheNetwork, ("No Network!"));
	if (TheNetwork)
	{
	for (Int i=0; i<MAX_SLOTS; ++i)
	{
	if (TheNetwork->isPlayerConnected(i))
	++numPlayers;
	}
	if (m_cachedCRCs.size() < numPlayers)
	{
	DEBUG_CRASH(("Not enough CRCs!"));
	sawCRCMismatch = TRUE;
	}
	else
	{
	//DEBUG_LOG(("Comparing %d CRCs on frame %d\n", m_cachedCRCs.size(), m_frame));
	std::map<Int, UnsignedInt>::const_iterator crcIt = m_cachedCRCs.begin();
	Int validatorCRC = crcIt->second;
	//DEBUG_LOG(("Validator CRC from player %d is %8.8X\n", crcIt->first, validatorCRC));
	while (++crcIt != m_cachedCRCs.end())
	{
	Int validatedCRC = crcIt->second;
	//DEBUG_LOG(("CRC to validate is from player %d: %8.8X\n", crcIt->first, validatedCRC));
	if (validatorCRC != validatedCRC)
	{
	DEBUG_CRASH(("CRC mismatch!"));
	sawCRCMismatch = TRUE;
	}
	}
	}
	}
	if (sawCRCMismatch)
	{
	#ifdef DEBUG_LOGGING
	DEBUG_LOG(("CRC Mismatch - saw %d CRCs from %d players\n", m_cachedCRCs.size(), numPlayers));
	for (std::map<Int, UnsignedInt>::const_iterator crcIt = m_cachedCRCs.begin(); crcIt != m_cachedCRCs.end(); ++crcIt)
	{
	Player *player = ThePlayerList->getNthPlayer(crcIt->first);
	DEBUG_LOG(("CRC from player %d (%ls) = %X\n", crcIt->first,
	player?player->getPlayerDisplayName().str():L"<NONE>", crcIt->second));
	}
	#endif DEBUG_LOGGING
	TheNetwork->setSawCRCMismatch();
	}

[AdrianeYves]

Can you guys release a build of this mismatch logger once its up ? pweeaseeee -- we seriously need a logger

[xezon]

I can look into it if no one beats me to it. But currently still working on cmake setup with OmniBlade.

[AdrianeYves]

Ok, great! We're literally developing another co-op mission map, but there's just random mismatch issues, and we really can't figure it out. It's kinda killing our motivation to continue since it becomes completely unplayable at some points.

[xezon]

Can you share this coop map here? Could be good for testing then.

Also, if anyone has sets of replays were 2 players have mismatched, please share it. This might become useful for testing.

You can upload zip file into github comment.

[AdrianeYves]

ill probably will once you guys have the stuff, but i do prefer sending it out "privately" tho i dont really want to show the map at its unfinished state to public

[salem-gzh]

Attached 2 mismatch replays from a 1v1 game. Both clients running 1.04v with gentool
mismatched replays.zip

[nhatter]

Attached an instant mismatch from a 1v1 game (happens about 9 seconds in). Both running on same network, with v1.04 and Gentool.

Client 1: Windows 10 (Parallels Desktop VM on Intel-based Macbook, v1.04 + Gentool)
Client 2: Windows 11 (Parallels Desktop VM on Silicon-based Macbook, v1.04 + Gentool)

Might be helpful for debugging.

Instant Mismatch.zip