Support for Ikarus/LeGo

[Try]

Subject

Ikarus is a popular library that provides set of DMA(direct memory access) utilities, exploring volubility in vanilla script-interpreter.
LeGo is a game framework bases on Ikarus, that provides all sort of goods, like custom-views, painting, AI, and more.

Ikarus strongly relies on how exactly original game works: addresses of oGame/zTimer/oCInformationManager are hard-coded, pointer are stored in int32 - something what is not doable on x64 platform

Goal

Create good-enough compatibility layer, to enable most popular mods: the Chronicles of Myrtana, VarusBiker Edition and others

Design

We won't provide same memory layout for most of things, so support won't be bullet-prof. Instead, OpenGothic will give access to high-level functional.
To make int-pointers work OpenGothic will provide a fake 3GB heap structure, with layout like virtual memory in win32.
Mappings:

• MEM_Alloc/MEM_Free - just works; can be serialized
• script bytecode - cannot handle as-is, in theory we can align internal bytecode layout, or simply skip
• classes - VM doesn't have a support for user-defined classes; need to implement that
• objects - current version of VM provides a managed environment(with ref-counting). That won't work as-is, if script can emplace class in allocated memory
• Allocation should be padded to be multiple of 4 bytes, to ensure support of MEM_ReadInt/MEM_WriteInt

Memory layout should resemble Win32 heap, while be sealizable:

[0x00000000 .. 0x00001000] - nullptr page
[0x00001000 .. 0x80000000] - (2GB) user space; MEM_Alloc/MEM_Free
[0x80000000 .. 0xc0000000] - (1GB) engine mapped data
[0xc0000000 .. 0xffffffff] - (1GB) kernel space - not in use

Api(Ikarus):

Base on ikarus_Doc.d
JW - mean 'just works' - function does not require engine-level support.
Access will be reliable only for manually allocated memory(MEM_Alloc), engine-memory generally won't do, unless there is no alternative.

1. Memory

   • int MEM_ReadInt(var int address)
   • void MEM_WriteInt(var int address, var int val)
   • string MEM_ReadString(var int address)
   • void MEM_WriteString(var int address, var string val)
   • int MEM_ReadIntArray (var int arrayAddress, var int offset) [JW]
   • int MEM_WriteIntArray (var int arrayAddress, var int offset, var int value) [JW]
   • int MEM_ReadByte(var int adr) [JW]
   • void MEM_WriteByte (var int adr, var int val) [JW]

2. Parser

   • instance MEM_PtrToInst(var int ptr)
   • void MEM_AssignInst(var int inst, var int ptr)
   • instance MEM_CpyInst (var int inst)

3. Functions & Reflection

   • void MEM_CallByString(var string fnc)
   • void MEM_CallByID(var int ID)
   • void MEM_PushIntParam(var int param)
   • void MEM_PushStringParam(var string strParam)
   • void MEM_PushInstParam(var int instance)
   • int MEM_PopIntResult()
   • string MEM_PopStringResult()
   • instance MEM_PopInstResult()
   • int MEM_GetFuncID(var func function)
   • void MEM_Call(var func function)
   • int MEM_FindParserSymbol (var string inst)
   • int MEM_GetParserSymbol (var string inst)
     Note: Symbols do exist in OpenGothic, but do not match in terms of memory-layout

4. Jumps & Control flow

   • MEM_InitLabels
   • MEM_StackPos.position

5. Strings

   • int STR_GetCharAt(var string str, var int pos)
   • int STR_Len (var string str)
   • int STR_Compare (var string str1, var string str2)
   • int STR_ToInt(var string str)
   • string STR_SubStr(var string str, var int start, var int count)
   • string STR_Prefix (var string str, var int count) [JW]
   • string STR_Upper(var string str)
   • int STR_SplitCount(var string str, var string Seperator)
   • string STR_Split(var string str, var string Separator, var int index)
   • int STR_IndexOf(var string str, var string tok)

6. Menu

   • int MEM_GetMenuByString(var string menu)
   • int MEM_GetMenuItemByString(var string menuItem)

7. Global instances

   • instance MEM_Game (oCGame);
   • instance MEM_World (oWorld);
   • instance MEM_Timer (zCTimer);
   • instance MEM_WorldTimer (oCWorldTimer);
   • instance MEM_Vobtree (zCTree);
   • instance MEM_InfoMan (oCInfoManager);
   • instance MEM_InformationMan (oCInformationManager);
   • instance MEM_Waynet (zCWaynet);
   • instance MEM_Camera (zCCamera);
   • instance MEM_SkyController (zCSkyController_Outdoor);
   • instance MEM_SpawnManager (oCSpawnManager);
   • func void MEM_InitGlobalInst()

8. Ini file

   • string MEM_GetGothOpt (var string sectionname, var string optionname)
   • string MEM_GetModOpt (var string sectionname, var string optionname)
   • int MEM_GothOptSectionExists (var string sectionname)
   • int MEM_GothOptExists (var string sectionname, var string optionname)
   • int MEM_ModOptSectionExists (var string sectionname)
   • int MEM_ModOptExists (var string sectionname, var string optionname)
   • void MEM_SetGothOpt (var string section, var string option, var string value)

9. Keyboard settings

   • int MEM_GetKey(var string name)
   • int MEM_GetSecondaryKey(var string name)
   • int MEM_KeyPressed(var int key)
   • int MEM_KeyState(var int key)
   • void MEM_InsertKeyEvent(var int key)

10. Execute x86 code from memory

• void MemoryProtectionOverride (var int address, var int size) - won't do

• Only very limited subset of x86 instructions can be supported
• mem-protection call can be used as a performance hint

11. Calls to engine functions

• void CALL_IntParam(var int param)
• void CALL_FloatParam(var int param)
• void CALL_PtrParam(var int param)
• void CALL_zStringPtrParam(var string param)
• void CALL_cStringPtrParam (var string param)
• void CALL_StructParam(var int ptr, var int words) [JW]
• void CALL__stdcall(var int adr )
• void CALL__thiscall(var int this, var int adr)
• void CALL__cdecl(var int adr)
• void CALL__fastcall(var int ecx, var int edx, var int adr)
• int CALL_RetValAsInt/CALL_RetValAsPtr/CALL_RetValAsFloat [JW]
• instance CALL_RetValAsStructPtr()
• string CALL_RetValAszStringPtr()
  Note:

• only fixed subset of callable functions can be supported in individual manner
• CALL_* framework might be superseded by ASM support in future

12. External libraries

• int LoadLibrary(var string lpFileName)
• int GetProcAddress (var int hModule, var string lpProcName)
  Note: generally won't do, due to portability complexity.

13. Various

• int MEM_SearchVobByName (var string str)
• int MEM_SearchAllVobsByName (var string str)
• int MEM_InsertVob(var string vis, var string wp)
• void MEM_TriggerVob (var int vobPtr)
• void MEM_UntriggerVob (var int vobPtr)
• void MEM_RenameVob (var int vobPtr, var string newName)
• int Hlp_Is_oCMob(var int ptr)
• int Hlp_Is_oCMobInter(var int ptr)
• int Hlp_Is_oCMobLockable(var int ptr)
• int Hlp_Is_oCMobContainer(var int ptr)
• int Hlp_Is_oCMobDoor(var int ptr)
• int Hlp_Is_oCNpc(var int ptr)
• int Hlp_Is_oCItem(var int ptr)
• int Hlp_Is_zCMover(var int ptr)
• int Hlp_Is_oCMobFire(var int ptr)
• void MEM_SetShowDebug (var int on)
• string MEM_GetCommandLine()
• int MEM_MessageBox (var string txt, var string caption, var int type)
• void MEM_InfoBox (var string txt)
• int MEM_GetSystemTime()
• int MEM_BenchmarkMS(var func f)
• int MEM_BenchmarkPC(var func f)
• int MEM_BenchmarkMS_N(var func f, var int n)
• int MEM_BenchmarkPC_N(var func f, var int n)
• int MEM_ReadStatArr (var int array, var int offset)
• void MEM_WriteStatArr (var int array, var int offset, var int value)
• int MEM_GetStringAddress(var string s)
• int MEM_GetFloatAddress (var float f)
• int MEM_GetIntAddress (var int i)
• int MEM_ArrayCreate ()
• void MEM_ArrayFree (var int zCArray_ptr)
• void MEM_ArrayClear (var int zCArray_ptr)
• void MEM_ArrayInsert (var int zCArray_ptr, var int value)
• void MEM_ArrayRemoveIndex (var int zCArray_ptr, var int index)
• void MEM_ArrayRemoveValue (var int zCArray_ptr, var int value)
• void MEM_ArrayRemoveValueOnce (var int zCArray_ptr, var int value)
• void MEM_CopyBytes (var int src, var int dst, var int byteCount)
• void MEM_CopyWords (var int src, var int dst, var int wordcount) [JW]
• void MEM_SwapBytes (var int ptr1, var int ptr2, var int byteCount)
• void MEM_SwapWords (var int ptr1, var int ptr2, var int wordcount)
• int MEM_CompareBytes (var int ptr1, var int ptr2, var int byteCount)
• int MEM_CompareWords (var int ptr1, var int ptr2, var int wordcount)
• void MEM_InitAll()

14. Extra low-levels

• int MEM_GetFuncPtr(var func fnc)
• int MEM_GetFuncOffset(var func fnc) [JW]
• int MEM_GetClassDef (var int objPtr)
• string MEM_GetClassName (var int objPtr)
• int MEM_GetBufferCRC32 (var int buf, var int buflen)
• int MEM_GetStringHash (var string str)
• int MEM_Alloc (var int amount)
• int MEM_Realloc (var int oldptr, var int oldsize, var int newsize)
• void MEM_Free (var int ptr)
• void MEM_SetParser(var int parserID)

[lmichaelis]

Hi, just to let you know: I'll be looking into this as well. I'm about to take a look at Chronicles of Myrtana so see what kind of Ikarus/LeGo functions it uses so we can more accurately determine whether this is actually feasible. Do you know if its source code is publicly available somewhere? Otherwise I'll use my custom tools :)

[Try]

Do you know if its source code is publicly available somewhere? Otherwise I'll use my custom tools :)

Haven't seen a sources anywhere, so fell free ;)

[lmichaelis]

If anyone finds this helpful too, here's a decompilation of The Chronicles of Myrtana: Archolos :)

[hellozyemlya]

Just my 5 cents here: Is that really required to make OpenGothic compatible with all of that mem read mem write functions and anything that mess around with bytecode and/or internal class structures? Obviously due to many differences in native class structure mod code will not be able to produce same results as on original engine(and crash game in most cases).

Maybe it worth to focus on compatibility layer that will allow to dynamically replace/hook/patch VM code and implement all required functionality directly in C++?

For example, items picking while swimming in CoM: locate all places that messing with engine from VM and stub/hook them, implement feature as c++ plugin for opengothic, enjoy feature.

So it will be possible to run some of the biggest and most noticeable mods with manual patching & using original mod distro. And maybe provide normal modding api for modmakers to ensure they are creating their mods on modern engine without messing with memory patching.

[lmichaelis]

Maybe it worth to focus on compatibility layer that will allow to dynamically replace/hook/patch VM code and implement all required functionality directly in C++?

Yeah, I was looking at that, but for now I concluded that it isn't that straightforward since we would have to replace mod-internal functions. We would have to replace all functions in CoM which do any kind of non-trivial direct memory access using Ikarus, which iirc were quite a lot, though I haven't been able to compile a full list.

Additionally, any update to the mod then has the potential to break in OpenGothic, so it would likely require constant updates to the implementation.

A more general implementation would be better, I think. I'm trying to figure out a good solution looking at CoM. Some things are trivial (like memory allocation) while others seem very hard (like mocking the ZenGin class structures).

[Try]

Hi, @lmichaelis , thank for bumping this up.

I haven't been able to compile a full list.

OK, but can you go into a few examples to illustrate problematic cases?

A more general implementation would be better, I think

I was thinking about emulating memory-mapping-unit in engine, to mock ZenGin classes (at least to some extent) and implement only a few Ikarus api's. Basically implement layer that is close-most to DMA/Winapi and assume that rest will 'just work'

[lmichaelis]

Some examples:

const int OCLOGMANAGER_PTR = 11191608;
func void LOG_MOVETOTOP(var string TOPICNAME) {
    LIST = _^(OCLOGMANAGER_PTR);
    NEWLISTPTR = LIST_CREATE(0);
    NEWLIST = _^(NEWLISTPTR);
    PREVLIST = _^(OCLOGMANAGER_PTR);
    WHILE(LIST.NEXT);
    LIST = _^(LIST.NEXT);
    if (LIST.DATA) {
        if (HLP_STRCMP(MEM_READSTRING(LIST.DATA), TOPICNAME)) {
            NEWLIST.DATA = LIST.DATA;
            PREVLIST.NEXT = LIST.NEXT;
        };
    };
    PREVLIST = _^(_@(0x5446));
    if ((NEWLIST.DATA) == (0)) {
        if ((GAME_RUNTIMECONFIG) != (TARGET_SHIPPING)) {
            MEMINT_FORCEERRORBOX = TRUE;
            MEM_SENDTOSPY(ZERR_TYPE_FAULT, CS3("B_LogEntry: Error! Probably topic '", TOPICNAME, "' does not exist. Probably B_LogEntry has been called before Log_CreateTopic."));
            MEMINT_FORCEERRORBOX = FALSE;
        };
    };
    END;
    LIST = _^(OCLOGMANAGER_PTR);
    NEWLIST.NEXT = LIST.NEXT;
    LIST.NEXT = NEWLISTPTR;
}

https://github.com/lmichaelis/tcom-decompilation/blob/ef9bd1034d3e44b28f3cd9173cf52122d6c9d84e/Source/Raw/115.d#L11-L37

func void TORCHLOADFIX_CHECKTORCH() {
    if (!(HLP_ISVALIDNPC(HERO))) {
        return;
    };
    PTR = TORCH_GETSLOTITEM();
    if (PTR) {
        ITM = _^(PTR);
        if ((HLP_GETINSTANCEID(ITM)) == (0x87b7)) {
            TORCHLOADFIX_GIVETORCH = TRUE;
        };
    };
    if ((!(PTR)) || ((HLP_GETINSTANCEID(ITM)) != (0x87b7))) {
        TORCHLOADFIX_GIVETORCH = FALSE;
    };
}

(and more) https://github.com/lmichaelis/tcom-decompilation/blob/ef9bd1034d3e44b28f3cd9173cf52122d6c9d84e/Source/Raw/343.d#L94-L108

func int CATINV_NEXTNONACTIVEITEM(var int LIST, var int MAX) {
    I = 0;
    WHILE((LIST) && ((I) < (MAX)));
    L = _^(LIST);
    if (HLP_IS_OCITEM(L.DATA)) {
        ITM = _^(L.DATA);
        if (((ITM.FLAGS) & (ITEM_ACTIVE)) != (ITEM_ACTIVE)) {
            BREAK;
        };
        I += 1;
    };
    LIST = L.NEXT;
    END;
    return I;
}

(and more) https://github.com/lmichaelis/tcom-decompilation/blob/ef9bd1034d3e44b28f3cd9173cf52122d6c9d84e/Source/Raw/99.d#L1-L15

instance ITGR_MEATFRIED(C_ITEM) {
    NAME = FOODNAME_MEATFRIED;
    DESCRIPTION = NAME;
    HP = _@(ITEMGROUP_MEAT[0]);
    VALUE = MAX_IG_MEATFRIED;
}

instance ITGR_VEGETABLE(C_ITEM) {
    NAME = FOODNAME_VEGETABLES;
    DESCRIPTION = NAME;
    HP = _@(ITEMGROUP_VEGETABLE[0]);
    VALUE = MAX_IG_VEGETABLE;
}

https://github.com/lmichaelis/tcom-decompilation/blob/ef9bd1034d3e44b28f3cd9173cf52122d6c9d84e/Source/Raw/181.d#L26-L38

There are a lot more direct uses of _^, _@s and _@ everywhere in CoM. I believe it would be very hard to accurately map everything the mod might access.

This is why we can't easily just replace Ikarus and LeGo functions since usage of these _^ etc. requires a direct memory access routine to be present.