Reversing BG3 "NewAge" #127
Comments
|
I found a WebP in my NewAge, and it contains an image of Gale. Not sure why, since Gale was not the active player at the time of saving. There are no other WebP files, the WebP is also not an animated version with more than 1 frame. I would expect to find the images of my other party members to, though. EDIT: I was wrong, there is more than one RIFF header. |
|
Ahh, I'm beginning to see some logic in this. So the list of block 0 entries appear to be references to components. The unknown bit of the entry references a type and an offset (not exactly sure what it's relative to). They are also sequential, so the next "component" starts where the last one ended. For example if you take a look at the last few entries (enumeration values like EPriority, EState), there are flags describing 40, 32, 16 size, etc. The components then store data about, for example, character creation. EDIT: |
|
@LennardF1989 I can't provide anything meaningful here and it's unlikely the best place to ask but can you recommend any resources to learn this type of RE or even some keywords I might lookup to find the right type of content? |
|
RE-ing files is quite hard to explain, haha:P Get a dump of a save-game (using LSLib), extract the NewAge portion, base64 decode it, then stare at hex data in a Hex Editor (I'm using ImHex now for the template feature instead of the commercial 010 Editor) and try to make sense of what you are seeing. Right now I found the name table for the components (see above), I found the WebP files for the avatars. I think I found something that resembles your base stats (STR/DEX/etc). I have a feeling there is a bit of LSV format in this as well (see LSLib's LSFReader.cs on how that is generally read) when it comes to node-structures (it's a returning pattern in their files, so why not in this one too). This is my ImHex pattern so far (it's pretty much a port of the above 010 one): For simplicity, I've attached the NewAge file I'm using (saves you from trying to get one - extract it first): |
|
Managed to map another whole range of strings, found it by accident as I was scanning for repeating patterns. Offset into my NewAge file. Lot's of duplicate key/valye pairs, so the unknown bits probably have something interesting in it. |
|
Things are starting to fall in place now that I know that every offset is probably minus/plus 48. I found a way to map a large portion of strings (also "referenced strings"). These include the character names in your party, your profile id, etc. Output: |
|
Before I head to bed, last bit of info I have on the first bit of the file (after the huge chunk of unknown data): It's WIP as I'm still looking for the logic how it decides how many substrings follow after a header. I think Test5Header and Test2SubStrings should be combined based on the values it reads, and instead of all kinds of small chunks, it's actually a list of 6000-ish structures. There is a lot of FF FF FF FF, so they could also be separate arrays that have a particular size pre-reserved. Who knows! We'll find it :) EDIT: Also, I found the block that describes where to find the WebP files, see offset 0x00110868. It's a list of start/end offsets. |
|
Have you tried modifying some of this and loading it back into your game to see if it sticks? I've meticulously gone through every LSV/LSX attempting to remove a custom character from my party, recompile the save, and they're still there as if nothing changed... So I assume it's all thanks to NewAge keeping the "state" of things. Would be interesting to take NewAge from a new save with 1 player, then another after adding a second custom player and diff the two strings... |
|
I haven't yet, this is all by just analyzing. Changing stuff other than some stats without touching the integrity of the file is going to be hard. Most of the format will have to be digested before we can even go as far as re-saving it. There is so much stuff pointing at other stuff. Eg. Say you want to rename your character to a name with more characters than you originally had, everything that points at this particular string will need to know the new size. And everything that shifts will need to have their reference position updated. |
|
Some context for NewAge: Even though D:OS2 had an entity-component system, most of the logic was historically packed into two giga-components, esv::Character and esv::Item (which had their own LSF nodes). |
|
Good to know! We can probably get away with not understanding every byte to modify some portions of it. I've manually "deserialized" some of the components now. I think people will be mostly interested (at least I am), to modify your character model, stats and maybe some other bits, that (probably) doesn't require reserializing the whole file. I have almost all bytes referenced by something now (I've pretty much figured out how the header finds the components, how the components find their data). Out of 4MB, only 1MB is left untouched (but I also know why). Other than that, I like the challenge of solving these kinds of things, whether or not it will lead to something useful, haha. |
|
@LennardF1989 just wondering if you've had any luck with this! I'm also interested in parsing out character stats (mostly to recreate characters between saves but also out of interest in seeing what can be modified), but haven't had any luck in trying my own hand at this. |
|
I have parked it for a moment to actually stop tinkering and enjoy the game for a bit, haha. As Norbyte mentioned, being able to modify data is going to be hard, reading it should be possible to a certain extend. I have made a bunch of different saves with slight alterations between the characters, and the NewAge data is only slightly different. Those should be the parts that the determine how the character looks. I will resume and share findings somewhere this week :) |
|
Haha fair enough! When you say you made a bunch of different saves with slight alterations between the characters, did you just write down or roughly remember what you did for each character, or do you know if there's a way at all to see what options were chosen at character creation? |
I remembered what I did and made subtle changes like eye color, hair style, nothing too fancy. I have not found anything conclusive yet to say "Oh, this means gold blond 5, and this is deep blue, or anything yet." But I'll get there :) |
|
Someone @ Nexus Mods managed to fiddle with similar appearance items, but I don't believe they are doing anything with NewAge: https://www.nexusmods.com/baldursgate3/mods/899 Did NewAge change at all after the hotfixes Larian made to allow save files to be larger? |
|
This is by starting the character creation again. I've played with that, but evident by the large description in that mod, but it's far from ideal. The NewAge format doesn't change, but the contents can change. There are components in there that describe stuff, but they are versioned. So a component that's v1 one now, can have a new layout in v2. I haven't tested my theory yet, but some of these components Norbyte has already mapped out in his bg3se (see |
How would you go about approaching modifying these components? My first thought would be the ComponentHandle or Entity classes? |
|
just to provide some data to compare, here's a NewAge that I pulled from my save with my partner. Two Custom Characters handful of mods. I'm not sure that our progress is going to impact this data, but our game is just progressed past the tutorial |
Is this correct @LennardF1989 ? I'm stuck trying to fin the start/end of each component. Tried to use |
|
It is correct, it's a list of GUIDs - most likely with all items spawned into the world. I'm slowly picking up reversing this again and will share my results over this week. |
|
I modified the for-loop a bit. Seems to get the correct offset for each component now. I also noticed that core.v0.Level overlaps with most of Block1Header. |
It's correct :) Block1Header is near the bottom of the file, so I think you misread. |
|
I started with your template above. For some reason the Block1Header doesn't offset below the Block0Entries.
|
|
I'm waiting/working on a fix (crash-fix) for a new feature in ImHex to use dynamic groups in the pattern editor to get something like this: |
@LennardF1989 Would you be willing to share the pattern that you have as of now? I found this issue after losing some days trying to figure out the "NewAge" structure and it seems you are way ahead of me. |
|
I'm not quite as far as Lennard, but here goes: The primary header was more or less covered in the initial post already, but since I've renamed quite a few of them, once again in full. Apart from naming, the only thing that I added was stuff in the ComponentInfo (formerly Block0Entry) struct. As such, the loop becomes Two utility functions used for component alignment. The second important thing as far as structure goes is the game.v0.Level component. The first two numbers bound a range of ids, the second an array of structs in the 'Heap' (The section of the file past the named components). These are serialized somewhat strangely, there are more of them than components, but most of them are invalid (hence the check for the component being u64_max) to the point that not every component has an associated owner list. Based on the respective value ranges ranges, I interpret these lists as being read in parallel to their component (They always have the same length). Then the owner of the i-th entry of a component would be For convenience I've also made a lookup for the OwnerList of each component, though there's the obvious weakness that you have to make sure their sizes match in case you get 0. I simply check manually beforehand, but you could obviously fill empty entries with a value that'd error if used. This roughly divides the components into 'top level' ones that directly have owners for their entries and the rest, which have their data referred to by others. I'm not entirely sure what the purpose of the latter (as opposed to just putting them on heap) is yet, since everything I have come across for now (apart from the owner lists) retrieves its data by address rather than index. On the other hand, something I noticed while trying to map the party characters to their names is that the name components introduce seemingly unused entries with each 'layer', i.e. from The character names were one of the consistently affected, the only way I have found for programmatically retrieving them for now is to go through Edit: The ordering of the components in the index seems to be mutable, so identification by index (like done by the utility functions above) might break between saves and mustn't be relied on for fully automatic applications. |
|
The old link is dead due to a refactor, but @LennardF1989 's idea to cross compare against the findings of the script extender has proven extremely helpful. The appearance component for example seems to be a serialization of Along with the material settings struct from the same file, the save entry would likely be read like this: So the next major issue would be the nature of the hash/checksum, a topic I am anything but familiar with. Comparing the component index across saves from different patches has reinforced my guess that the last unknown 8 byte section is related to the component name, since it always changes when a component gets renamed, for example because it got a new version. Given that they have the same size as the file checksum and were probably developed alongside the main header, I think it's plausible they'd use the same algorithm, which would mean much more manageable data pairs to work with. |
|
For some reason, newage also partially contains some WPF code? This was taken from one of my recent saves (patch 3).
Not sure if WPF specifically, but this is definitely XAML, including indentation (all the 0x20). lication:,,,/GustavNoesisGUI;component/Assets/CharacterSheet/btn_round_medium_h.png"/>
</Trigger>
<Trigger Property="IsPressed" Value="True">
<Setter TargetName="bg" Property="Source" Value="pack://application:,,,/GustavNoesisGUI;component/Assets/CharacterSheet/btn_round_medium_p.png"/>
</Trigger>
<Trigger Property="IsChecked" Value="True">
<Setter TargetName="bg" Property="Source" Value="pack://application:,,,/GustavNoesisGUI;component/Assets/CharacterSheet/btn_round_medium_p.png"/>
</Trigger>
</ControlTemplate.Triggers>
</ControlTemplate>
</ls:LSToggleButton.Template>
</ls:LSToggleButton>
<Popup IsOpen="{Binding IsChecked, ElementName=ToggleBtn}" Placement="Bottom" StaysOpen="False" HorizontalOffset="-40">
<ls:LSNineSliceImage x:Name="PopularFiltersHolder" Margin="0,-12,0,0" HorizontalAlignment="Stretch" ImageSource="pack://application:,,,/GustavNoesisGUI;component/Assets/CharacterPanel/sorting_bg.png" Slices="60" Padding="10,40">
<StackPanel Margin="30,0,40,0" IsItemsHost="True" MinWidth="200" MinHeight="200"/>
</ls:LSNineSliceImage>
</Popup>
</Grid>
</ControlTemplate>
</ComboBox.Template>
<ComboBox.ItemContainerStyle>
<Style TargetType="ComboBoxItem">
<Setter Property="Template">
<Setter.Value>
<ControlTemplate TargetType="ComboBoxItem">
<Grid x:Name="ButtonRoot" Background="Transparent">
<Image x:Name="HLBG" Source="pack://application:,,,/GustavNoesisGUI;component/Assets/CharacterPanel/selector_listitem_d.png" StrFor those who have not worked with WPF, this is a sample of code for defining the style and behaviour of a ComboBox ( i.e drop-down menu) and the items it lists. Looking at the namespace references (e.g: Now, why this source code sample is in NewAge, including indentation, and incomplete... ??? |
|
Interesting, it is part of the game file |
|
Sorry this took way too long to setup, but here is my work on all of this so far: https://github.com/LennardF1989/BG3-ImHex-Patterns I had to wait until ImHex included my changes to make all this possible. I would like to invite everyone to combine their research with mine through PRs so we have one location to crack this thing :) EDIT: Have to add this is a cleaned up version of everything I have shared so far. Some things are not in here (yet), because I found better ways to handle them. |
Thanks! This last week and weekend I've been a bit busy with work, but I'll take a look next week to see if we can crack this thing once and for all.
That's weird. Maybe they implemented something quickly as a dynamic change into a XAML template that is then saved directly into the save file? Or maybe it was an accident? 🤔 |
|
I have no idea... Actually I'm wondering if there might have happened something wrong when I extracted the data from the LSV file with ConverterApp - I'm running the released version in Wine on Linux. |
|
My repository has a tool to extract the NewAge portion from a save: https://github.com/LennardF1989/BG3-ImHex-Patterns/releases/tag/v1.0.0 |
|
I saw, it's pretty useful! 😃 I wrote a port of LSLib in Rust (only the small part used for extracting LSV files) which I used to extract the NewAge LSMF that contained the unexpected XAML mentioned earlier, and used your CLI tool to extract the same save - both extracted LSMF had the same SHA512 checksums and the XAML was there in both cases, so it truly is random stuff that's part of the actual save. It's not much help towards figuring out actual useful data unfortunately, just that we know there's some odd stuff in there... |
Any chance you'd be willing to share the work you did for the Rust port? I'm a Linux user and I've started getting the itch to muck around with my Baldur's Gate 3 save files (which seem to be plain .lsv files?), and Rust is always my go-to language. I was going to resign myself to trying to port it myself before stumbling upon your comment here, so I looked through your repos but couldn't find it there. No worries if you'd prefer not to (although I'd love any advice/pointers to resources you might be able to suggest since I probably will end up doing it myself otherwise)! |
|
@saghm Sure, I'll just need a bit of free time to clean up the project. |
|
@saghm there you go: https://github.com/clemarescx/bg3d |
|
Any updates on this? Can I extract, change things, then repack everything back to save file? |
|
It has grown stale - do you still need to with the Magic Mirror? The whole point of reversing this was to be able to change appearance, mostly. |
|
I need to change my character's background. Somehow I totally missed the background selection screen when creating character 😂. |
|
@RichardLuo0 The inspiration screen showing old background is one of the reasons I removed that from AEE. Realistically, I can add it in but it's not perfect, it's tied in a bunch of places in a weird way. Maybe in Patch 7, I'll look into it. To be honest, having to reverse NewAge for background isn't worth it, it can probably be done with SE just fine if more time is put into it. |
|
Hope you can add it soon, I really need this |
Heya!
Since many hands make light work, I thought I'd share my first draft of reversing the NewAge LSMF format (Which I think stands for Larian Studios Metadata File).
It's a 010 Editor template, but it reads like C/C++ really.
Obviously work in progress, but as this file gets shaped more and more, I reckon a lot of tech-savvy players can pitch in their thoughts on the unknown bits.
With this version, Block1 is still very WIP. Just by looking at the data, the current size seems alright. But I'm pretty sure it's not the full block size. As between the Block0 and Block 1, there is a chunk of space (about 1/4 of the NewAge data in size), that contains your character names (among other things). I'm pretty sure if we look for a similar string-lookup structure for Block0, that we will find pointers back to other parts in the file.
EDIT: By just looking at some of the data and recognizing certain structures from other games, I'm pretty sure there are is actually 3D model data in this file. Give-aways are usually the large repeated structures repeating the alphabet (IndexBuffer data). Like these:
The text was updated successfully, but these errors were encountered: