Dual OTR MQ and Vanilla Support (HarbourMasters#1694)

* Changes OTR Extraction to have specific mq and nonmq paths.

Also updates the game to load resources according to whether or not
Master Quest or Vanilla is loaded.

* Removes unneeded code from the last commit.

* Fixes some weird formatting in ZRom.c

* Loads oot-mq.otr and patches oot.otr on top, if both are present.

If only one or the other are present, it becomes the only and main OTR.

* Adds ImGui Logic for whether or an MQ Checkbox.

Checkbox checked only specifies whether new saves should be MQ or not.
Checkbox is disabled or force-enabled according to which OTRs are loaded.
Also as a necessity includes tracking what game versions have been loaded
from the OTRs.

* Adds MQ settings logic for Randomizer's ImGui menu.

* Writes Master Quest dungeons to the spoiler log

* Loads MQ Dungeons from spoiler, persists in save, and loads when appropriate.

* Adds logic to prevent loading or creating incompatible rando saves.

* Fixdes some linux build issues and new rando save issues

* Makes appimage create both vanilla and mq otrs

If either rom is present, it makes the corresponding OTR. If both are present,
it will make both. If one OTR is present but both roms are present, it will
create the missing OTR.

* Makes it so a randomized save file will not be marked as MQ.

* Refactors to load all OTRs from MainPath or a specific list.

Also adds the ability to take a std::unordered_set of hashes to
validate each OTR's version file against.

* Fixes a syntax error

* Makes ExtractAssets output Vanilla and MQ OTRs if both roms are present

* Fixes asset generation bug.

* Partially working fix for dual OTR extract_assets

Currently the cmake ExtractAssets target will return with a 1 if you
only end up exporting one type of OTR isntead of both. Haven't found
a great way to only attempt to copy a file if it exists from within
cmake. It does actually correctly copy the OTR that is generated,
despite the error from copying the other one.

Pushing as is for now but will keep investigating.

* Adds oot-mq.otr to the gitignore.

* Makes ExtractAssets not fail on only one rom/OTR.

* Removes PatchesPath from the constructors requiring OTRFiles vector.

* Renames OOT_UNKNOWN to just UNKNOWN to remove OOT specific reference.

* Removes randomizing MQ Dungeons and re-disables MQ rando.

Doing this so the PR can get merged quicker with just the Dual OTR
support and won't need to wait on rando logic to be updated. That
will happen in another PR directly after the merge.

* Update mac startup script for dual otr

* Update soh/macosx/soh-macos.sh

* Update soh/macosx/soh-macos.sh

* Update soh/macosx/soh-macos.sh

* Implements new BinaryReader to fix Linux build issue.

BinaryReader itself comes from https://github.com/Moneyl/BinaryTools
I added a wrapper to adapt it to the ABI from ZAPD's Binary Reader and
add Endianness checking. I also had to copy a handful of other bits and
pieces from ZAPD to make it all function as expected.

* A few edits to the updatream BinaryReader to compile it on Linux.

* Adds the Endianness to the first byte of the version file.

* Fixes Jenkins

* Addresses some of Kenix's comments

* Renames `ReadNullTerminatedString` to `ReadCString`

* Refactors Archive::LoadFile into a private method with more arguments.

* Removes BitConverter and extends existing endianness.h instead.

* Fixes an endianness issue with the version file.

Co-authored-by: Garrett Cox <garrettjcox@gmail.com>

.gitignore

@@ -410,6 +410,7 @@ ReleaseObj/*
 .tags
 tags
 oot.otr
+oot-mq.otr
 *.sav
 shipofharkinian.ini
 shipofharkinian.json

CMakeLists.txt

@@ -136,14 +136,13 @@ find_package(Python3 COMPONENTS Interpreter)
 add_custom_target(
     ExtractAssets
     # CMake versions prior to 3.17 do not have the rm command, use remove instead for older versions
-    COMMAND ${CMAKE_COMMAND} -E $<IF:$<VERSION_LESS:${CMAKE_VERSION},3.17>,remove,rm> -f oot.otr
-    COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/OTRExporter/extract_assets.py -z "$<TARGET_FILE:ZAPD>"
-    COMMAND ${CMAKE_COMMAND} -E copy oot.otr ${CMAKE_SOURCE_DIR}
-    COMMAND ${CMAKE_COMMAND} -E copy oot.otr ${CMAKE_BINARY_DIR}/soh
+    COMMAND ${CMAKE_COMMAND} -E $<IF:$<VERSION_LESS:${CMAKE_VERSION},3.17>,remove,rm> -f oot.otr oot-mq.otr
+    COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/OTRExporter/extract_assets.py -z "$<TARGET_FILE:ZAPD>" --non-interactive
+    COMMAND ${CMAKE_COMMAND} -DSOURCE_DIR=${CMAKE_CURRENT_SOURCE_DIR} -DBINARY_DIR=${CMAKE_BINARY_DIR} -P ${CMAKE_CURRENT_SOURCE_DIR}/copy-existing-otrs.cmake
     WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/OTRExporter
     COMMENT "Running asset extraction..."
     DEPENDS ZAPD
-    BYPRODUCTS oot.otr ${CMAKE_SOURCE_DIR}/oot.otr
+    BYPRODUCTS oot.otr ${CMAKE_SOURCE_DIR}/oot.otr oot-mq.otr ${CMAKE_SOURCE_DIR}/oot-mq.otr
 )
 
 if(CMAKE_SYSTEM_NAME MATCHES "Linux")

OTRExporter/OTRExporter/DisplayListExporter.cpp

@@ -882,8 +882,14 @@ std::string OTRExporter_DisplayList::GetPrefix(ZResource* res)
 	std::string prefix = "";
 	std::string xmlPath = StringHelper::Replace(res->parent->GetXmlFilePath().string(), "\\", "/");
 
-	if (StringHelper::Contains(oName, "_scene") || StringHelper::Contains(oName, "_room"))
+	if (StringHelper::Contains(oName, "_scene") || StringHelper::Contains(oName, "_room")) {
 		prefix = "scenes";
+        if (Globals::Instance->rom->IsMQ()) {
+            prefix += "/mq";
+        } else {
+            prefix += "/nonmq";
+        }
+    }
 	else if (StringHelper::Contains(xmlPath, "objects/"))
 		prefix = "objects";
 	else if (StringHelper::Contains(xmlPath, "textures/"))

OTRExporter/OTRExporter/Main.cpp

@@ -21,6 +21,7 @@
 #include <Utils/Directory.h>
 #include <Utils/MemoryStream.h>
 #include <Utils/BinaryWriter.h>
+#include <bit>
 
 std::string otrFileName = "oot.otr";
 std::shared_ptr<Ship::Archive> otrArchive;
@@ -69,13 +70,20 @@ static void ExporterProgramEnd()
 		std::string romPath = Globals::Instance->baseRomPath.string();
 		std::vector<uint8_t> romData = File::ReadAllBytes(romPath);
 		uint32_t crc = BitConverter::ToUInt32BE(romData, 0x10);
+        uint8_t endianness = (uint8_t)Endianness::Big;
 
 		// Write crc to version file
 		fs::path versionPath("Extract/version");
+        MemoryStream* versionStream = new MemoryStream();
+        BinaryWriter writer(versionStream);
+        writer.SetEndianness(Endianness::Big);
+        writer.Write(endianness);
+        writer.Write(crc);
 		std::ofstream versionFile(versionPath.c_str(), std::ios::out | std::ios::binary);
-		versionFile.write((char*)&crc, sizeof(crc));
+        versionFile.write(versionStream->ToVector().data(), versionStream->GetLength());
 		versionFile.flush();
 		versionFile.close();
+        writer.Close();
 
 		printf("Created version file.\n");
 

OTRExporter/extract_assets.py

@@ -16,7 +16,8 @@ def BuildOTR(xmlPath, rom, zapd_exe=None):
 
     exec_cmd = [zapd_exe, "ed", "-i", xmlPath, "-b", rom, "-fl", "CFG/filelists",
             "-o", "placeholder", "-osf", "placeholder", "-gsf", "1",
-            "-rconf", "CFG/Config.xml", "-se", "OTR"]
+            "-rconf", "CFG/Config.xml", "-se", "OTR", "--otrfile", 
+            "oot-mq.otr" if Z64Rom.isMqRom(rom) else "oot.otr"]
 
     print(exec_cmd)
     exitValue = subprocess.call(exec_cmd)
@@ -30,16 +31,18 @@ def main():
     parser = argparse.ArgumentParser()
     parser.add_argument("-z", "--zapd", help="Path to ZAPD executable", dest="zapd_exe", type=str)
     parser.add_argument("rom", help="Path to the rom", type=str, nargs="?")
+    parser.add_argument("--non-interactive", help="Runs the script non-interactively for use in build scripts.", dest="non_interactive", action="store_true")
 
     args = parser.parse_args()
 
-    rom_path = args.rom if args.rom else rom_chooser.chooseROM()
-    rom = Z64Rom(rom_path)
+    rom_paths = [ args.rom ] if args.rom else rom_chooser.chooseROM(args.non_interactive)
+    for rom_path in rom_paths:
+        rom = Z64Rom(rom_path)
 
-    if (os.path.exists("Extract")):
-        shutil.rmtree("Extract")
+        if (os.path.exists("Extract")):
+            shutil.rmtree("Extract")
 
-    BuildOTR("../soh/assets/xml/" + rom.version.xml_ver + "/", rom_path, zapd_exe=args.zapd_exe)
+        BuildOTR("../soh/assets/xml/" + rom.version.xml_ver + "/", rom_path, zapd_exe=args.zapd_exe)
 
 if __name__ == "__main__":
     main()

OTRExporter/rom_chooser.py

@@ -2,7 +2,7 @@
 
 from rom_info import Z64Rom
 
-def chooseROM():
+def chooseROM(non_interactive=False):
     roms = []
 
     for file in glob.glob("*.z64"):
@@ -14,7 +14,21 @@ def chooseROM():
         sys.exit(1)
 
     if (len(roms) == 1):
-        return roms[0]
+        return roms
+
+    if non_interactive:
+        romsToExtract = []
+        foundMq = False
+        foundOot = False
+        for rom in roms:
+            isMq = Z64Rom.isMqRom(rom)
+            if isMq and not foundMq:
+                romsToExtract.append(rom)
+                foundMq = True
+            elif not isMq and not foundOot:
+                romsToExtract.append(rom)
+                foundOot = True
+        return romsToExtract
 
     print(str(len(roms))+ " roms found, please select one by pressing 1-"+str(len(roms)))
 
@@ -34,4 +48,4 @@ def chooseROM():
 
         else: break
 
-    return roms[selection - 1]
+    return [ roms[selection - 1] ]

OTRExporter/rom_info.py

@@ -73,6 +73,11 @@ def __init__(self, file_path):
         if self.checksum == Checksums.OOT_UNKNOWN:
             self.is_valid = False
             return
+
+        if self.checksum in [Checksums.OOT_NTSC_JP_MQ, Checksums.OOT_NTSC_US_MQ, Checksums.OOT_PAL_GC_MQ_DBG, Checksums.OOT_PAL_MQ]:
+            self.isMq = True
+        else:
+            self.isMq = False
 
         # get rom version
         self.version = ROM_INFO_TABLE[self.checksum]
@@ -86,3 +91,7 @@ def readDmaEntry(self, entry):
     @staticmethod
     def isValidRom(rom_path):
         return Z64Rom(rom_path).is_valid
+
+    @staticmethod
+    def isMqRom(rom_path):
+        return Z64Rom(rom_path).isMq

ZAPDTR/ZAPD/ZResource.cpp

@@ -328,7 +328,7 @@ std::string ZResource::GetSourceOutputHeader([[maybe_unused]] const std::string&
 		std::string xmlPath = StringHelper::Replace(parent->GetXmlFilePath().string(), "\\", "/");
 
 		if (StringHelper::Contains(outName, "_room_") || StringHelper::Contains(outName, "_scene"))
-			prefix = "scenes";
+			prefix = "scenes/nonmq";
 		else if (StringHelper::Contains(xmlPath, "objects/"))
 			prefix = "objects";
 		else if (StringHelper::Contains(xmlPath, "textures/"))
@@ -342,8 +342,9 @@ std::string ZResource::GetSourceOutputHeader([[maybe_unused]] const std::string&
 		else if (StringHelper::Contains(xmlPath, "text/"))
 			prefix = "text";
 
-		if (prefix != "")
+		if (prefix != "") {
 			str += StringHelper::Sprintf("#define d%s \"__OTR__%s/%s/%s\"", name.c_str(), prefix.c_str(), outName.c_str(), nameStr.c_str());
+        }
 		else
 			str += StringHelper::Sprintf("#define d%s \"__OTR__%s/%s\"", name.c_str(), outName.c_str(), nameStr.c_str());
 

ZAPDTR/ZAPD/ZRom.cpp

@@ -64,7 +64,33 @@ namespace fs = std::filesystem;
 #define OOT_PAL_GC_MQ_DBG 0x917D18F6
 #define OOT_IQUE_TW 0x3D81FB3E
 #define OOT_IQUE_CN 0xB1E1E07B
-#define OOT_UNKNOWN 0xFFFFFFFF
+#define UNKNOWN 0xFFFFFFFF
+
+bool ZRom::IsMQ() {
+    int crc = BitConverter::ToInt32BE(romData, 0x10);
+    switch (crc) {
+        case OOT_NTSC_10:
+        case OOT_NTSC_11:
+        case OOT_NTSC_12:
+        case OOT_PAL_10:
+        case OOT_PAL_11:
+        case OOT_NTSC_JP_GC:
+        case OOT_NTSC_JP_GC_CE:
+        case OOT_NTSC_US_GC:
+        case OOT_PAL_GC:
+        case OOT_PAL_GC_DBG1:
+        case OOT_PAL_GC_DBG2:
+        case OOT_IQUE_CN:
+        case OOT_IQUE_TW:
+        default:
+            return false;
+        case OOT_NTSC_JP_MQ:
+        case OOT_NTSC_US_MQ:
+        case OOT_PAL_MQ:
+        case OOT_PAL_GC_MQ_DBG:
+            return true;
+    }
+}
 
 ZRom::ZRom(std::string romPath)
 {

ZAPDTR/ZAPD/ZRom.h

@@ -11,6 +11,7 @@ class ZRom
 	ZRom(std::string romPath);
 
 	std::vector<uint8_t> GetFile(std::string fileName);
+    bool IsMQ();
 
 protected:
 	std::vector<uint8_t> romData;

copy-existing-otrs.cmake

@@ -0,0 +1,11 @@
+if(EXISTS ${SOURCE_DIR}/OTRExporter/oot.otr)
+    execute_process(COMMAND ${CMAKE_COMMAND} -E copy oot.otr ${SOURCE_DIR})
+    execute_process(COMMAND ${CMAKE_COMMAND} -E copy oot.otr ${BINARY_DIR}/soh/)
+endif()
+if(EXISTS ${SOURCE_DIR}/OTRExporter/oot-mq.otr)
+    execute_process(COMMAND ${CMAKE_COMMAND} -E copy oot-mq.otr ${SOURCE_DIR})
+    execute_process(COMMAND ${CMAKE_COMMAND} -E copy oot-mq.otr ${BINARY_DIR}/soh/)
+endif()
+if(NOT EXISTS ${SOURCE_DIR}/oot.otr AND NOT EXISTS ${SOURCE_DIR}/oot-mq.otr)
+    message(FATAL_ERROR, "No OTR files found.")
+endif()

libultraship/libultraship/Animation.cpp

@@ -48,7 +48,7 @@ void Ship::AnimationV0::ParseFileBinary(BinaryReader* reader, Resource* res)
 			data.unk_02 = reader->ReadInt16();
 			data.unk_04 = reader->ReadInt16();
 			data.unk_06 = reader->ReadInt16();
-			data.unk_08 = reader->ReadSingle();
+			data.unk_08 = reader->ReadFloat();
 
 			anim->transformDataArr.push_back(data);
 		}