Headless mode #600

Infernio · 2021-05-27T09:14:48Z

A headless mode for Bash seems a laudable goal:

people are interested in using bash record libraries (fi https://discord.com/channels/537706885965676554/537710082755133460/895557710400094208 )
internally it will guide/finalize the separation of the UI and business logic code
naming idea for the issue from 9a87dda, although I should probably call this Boot :P

Codewise this is about boot (bash.py, initialization.py, bosh.initBosh, etc). There is already a lot of ground covered and as of 170ad99 we have the tools to complete this. What's involved is:

Start from bash.py - see how early up we plug wx in ;)

Blocked by #190, #570
Blocks #554, #568, #250

The text was updated successfully, but these errors were encountered:

This was motivated by the CsvListPatcher._update_patcher_factories. I could not be sure that the srcs attribute was not set to some list of mods by mistake - so I had finally to add validation to the patcher. Of course the logic was there but as seen in various bits and places, most notably in gui_patchers - splitting responsibilities between the APatcher and PatcherPanel was a long journey that came home finally - PatcherPanel has now the responsibilities it should have, namely config (as in pickled config) handling and GUI panels creation, and delegates to the patcher for the business logic as it should. The patcher guards its init method (via _process_sources) from being fed junk, which is a step towards making the patcher headless (see #600) and therefore testable. Note: - autoKey (now patcher_tags) really belongs to ListPatcher. Certainly did not belong to the gui patchers, hence some ugly get_cls_vars dropped or simplified - list_patches_dir centralized and the ugly global became a beautiful instance attribute - _csv_key belongs also to the ListPatcher but really to the parser - what stands in the way here is CsvListPatcher - all_tags cache not really needed (few calls of getBashTags really it' more for clarity) - some bush game vars uses reduced - another sign autoKey belonged to the model - we could change the logic of the validation to print some warnings (for instance for mods tagged with add *and* remove spells) - and the validation itself could use some more items but the tough part is done, rest is nit - game_specific_import_patchers did not seem to have any csv support ever - keep it that way. Note _AListPanelCsv only remaining purpose is listLabel - get rid of that class (done) - in @@ -474,5 +473,4 @@ def _getConfig(self, configs): #--Verify file existence - self.configItems = [srcPath for srcPath in self.configItems if ( - srcPath in or (srcPath.fn_ext == '.csv' and - srcPath in patches_set()))] + self.configItems = self.patcher_type.get_sources(self._bp, + self.configItems) we were checking for bosh.modInfos membership - this seems wrong. Anyway decoupling gui_patchers from bosh is a laudable goal. - _ListsMergerPanel._get_auto_items escapes me a bit - actually the checks in MergePatchesPatcher._validate_mod are essentially encapsulated in PatchFile.set_active_arrays (bp_mergeable) as seen - well. Mopy/bash/patcher/patch_files.py: I don't see why making the dict to a set - rework the arrays a bit.

This merge finalizes 6659eb3 by finally decoupling bosh from balt/gui. This was a major goal of the parameters (keep them at a min and ideally move them to basher somehow) and a few changes in progress, that we will revisit at some point. This also lead to: - aligning the default for ask_confirm between windows and shutil implementations - this will result in some file operation dialogs previously shown in windows to not be shown anymore. We can always fine tune. - saving sizes for real (most of them were saved for the session - the ones that were not will reset their sizes for the users that had saved them, but just once, no backwards compat needed) The branch finishes with some more work around the status bar and app_buttons, see the merge cited above for the whys. Under #600 - no headless mode (aka Bash as a library) possible while the data model was coupled to wx.

This merge finalizes 6659eb3 by finally decoupling bosh from balt/gui. This was a major goal of the #190 refactoring - was achieved at the expense of some callback parameters (keep them at a min and ideally move them to basher somehow) and a few changes in progress, that we will revisit at some point. This also lead to: - aligning the default for ask_confirm between windows and shutil implementations - this will result in some file operation dialogs previously shown in windows to not be shown anymore. We can always fine tune. - saving sizes for real (most of them were saved for the session - the ones that were not will reset their sizes for the users that had saved them, but just once, no backwards compat needed) The branch finishes with some more work around the status bar and app_buttons, see the merge cited above for the whys. Under #600 - no headless mode (aka Bash as a library) possible while the data model was coupled to wx.

This was meant to be more granular - I wanted to centralize image initialisation including setting dirs['image'] - turned out that constants were imported too early (#600, control import order), (mopy) dirs were not initialised and this necessitated moving the image definitions in initStatusBar - however to make this simpler I renamed the images based on the key in tooldirs (ini backwards compat). We could: - turn all these to svgs? - add a sensible default to the image dict? (we could do this instead of the CI hack) - further centralise get_*** from _gui_globals especially get_image_dir - CI Hack - move to caller or add a param Next commit finalizes this - for now bye staticBitmap, au revoir, auf wiedersehen etc. Also see the GuiImage imports and dirs['images'] go down and imgDirJoin (make ugly code look ugly) centralised - getting somewhere: """ Thou hast committed bugs, but that was in another language, and besides, the critters are dead. """ Note cause initially _icons were defined in `import balt` time and images were not initialized yet the imports would crash cause: if not os.path.isabs(img_path): img_path = os.path.join(get_image_dir(), img_path) would produce a relative (non-existing) path and we would land in _tkinter_error_dial - which crashes bigtime on mac: 2023-10-10 14:49:27.496 Python[11010:375752] -[wxNSApplication macOSVersion]: unrecognized selector sent to instance 0x7fcf3c738aa0 2023-10-10 14:49:27.503 Python[11010:375752] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[wxNSApplication macOSVersion]: unrecognized selector sent to instance 0x7fcf3c738aa0' ... libc++abi: terminating due to uncaught exception of type NSException Process finished with exit code 134 (interrupted by signal 6: SIGABRT) To fix `import balt` crash I moved Colorchecks images init to gui. This now crashed on importing basher in Mopy.bash.bash._main - when constants is imported still get_image_dir() returns '' Traceback (most recent call last): ... File "/Users/.../Mopy/bash/basher/constants.py", line 458, in <listcomp> return [GuiImage.from_path(template % x) for x in (16, 24, 32)] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/.../Mopy/bash/gui/images.py", line 90, in from_path return _BmpFromPath(img_path, iconSize, img_type, quality) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/.../Mopy/bash/gui/images.py", line 56, in __init__ raise ArgumentError(f'Missing resource file: {self._img_path}.') bash.exception.ArgumentError: Missing resource file: tools/isobl16.png. To fix this I had to finally ---> Simplify tools init: Here is the map of filenames that differed: {'ISRMG': "insanity'sreadmegenerator", 'ISRNG': "insanity'srng", 'ISRNPCG': 'randomnpc', 'OBFEL': 'oblivionfaceexchangerlite', 'OBMLG': 'modlistgenerator', 'BSACMD': 'bsacommander', 'Tes4FilesPath': 'tes4files', 'BlenderPath': 'blender', 'GmaxPath': 'gmax', 'MayaPath': 'maya', 'MaxPath': '3dsmax', 'FastStone': 'faststoneimageviewer', 'PaintNET': 'paint.net', 'PaintShopPhotoPro': 'paintshopprox3', 'PhotoshopPath': 'photoshop', 'PixelStudio': 'pixelstudiopro', 'MAP': 'interactivemapofcyrodiil', 'NPP': 'notepad++', 'RADVideo': 'radvideotools'} I hope no one relied on these. Now as seen having to deal with two image formats introduces complexity - there was a merge about that no ? <--- RRR There is also the matter of game dependencies - and settings saving - and profiles per game - that's # RRR launchers Accept Path in GuiImage.from_path: os.path.splitext(img_path) works fine for path - yey for os.pathlike! inline _init_tool_buttons - note that TesVGecko was added twice things like renames = { 'OblivionBookCreatorPath': 'oblivionbookcreator%s.png', 'Tes4GeckoPath': 'tes4gecko%s.png', 'Tes5GeckoPath': 'tesvgecko%s.png', 'Tes4ViewPath': 'tes4view%s.png', 'Tes4TransPath': 'tes4trans%s.png', 'Tes4LodGenPath': 'tes4lodgen%s.png', 'NifskopePath': 'nifskope%s.png', } tooldir = '/Users/.../Mopy/bash/images/tools' for k, v in renames.items(): for pix in (16,24,32): try: shutil.move(_j(tooldir, v % pix), _j(tooldir, f'{k.lower()}{pix}.png')) except Exception as e: print(k,v,e) break One (1) use of app_buttons_factory - yes! This is geared towards # 570 and initTooldirs The [!] for image renames - this was 99% of the perspiration - dict used: { 'nifskope16.png': 'nifskopepath16.png', 'nifskope24.png': 'nifskopepath24.png', 'nifskope32.png': 'nifskopepath32.png', 'oblivionbookcreator16.png': 'oblivionbookcreatorpath16.png', 'oblivionbookcreator24.png': 'oblivionbookcreatorpath24.png', 'oblivionbookcreator32.png': 'oblivionbookcreatorpath32.png', 'tes4gecko16.png': 'tes4geckopath16.png', 'tes4gecko24.png': 'tes4geckopath24.png', 'tes4gecko32.png': 'tes4geckopath32.png', 'tes4lodgen16.png': 'tes4lodgenpath16.png', 'tes4lodgen24.png': 'tes4lodgenpath24.png', 'tes4lodgen32.png': 'tes4lodgenpath32.png', 'tes4trans16.png': 'tes4transpath16.png', 'tes4trans24.png': 'tes4transpath24.png', 'tes4trans32.png': 'tes4transpath32.png', 'tes4view16.png': 'tes4viewpath16.png', 'tes4view24.png': 'tes4viewpath24.png', 'tes4view32.png': 'tes4viewpath32.png', 'tesvgecko16.png': 'tes5geckopath16.png', 'tesvgecko24.png': 'tes5geckopath24.png', 'tesvgecko32.png': 'tes5geckopath32.png', 'blender16.png': 'blenderpath16.png', 'blender24.png': 'blenderpath24.png', 'blender32.png': 'blenderpath32.png', 'bsacommander16.png': 'bsacmd16.png', 'bsacommander24.png': 'bsacmd24.png', 'bsacommander32.png': 'bsacmd32.png', 'faststoneimageviewer16.png': 'faststone16.png', 'faststoneimageviewer24.png': 'faststone24.png', 'faststoneimageviewer32.png': 'faststone32.png', 'gmax16.png': 'gmaxpath16.png', 'gmax24.png': 'gmaxpath24.png', 'gmax32.png': 'gmaxpath32.png', "insanity'sreadmegenerator16.png": 'isrmg16.png', "insanity'sreadmegenerator24.png": 'isrmg24.png', "insanity'sreadmegenerator32.png": 'isrmg32.png', "insanity'srng16.png": 'isrng16.png', "insanity'srng24.png": 'isrng24.png', "insanity'srng32.png": 'isrng32.png', 'randomnpc16.png': 'isrnpcg16.png', 'randomnpc24.png': 'isrnpcg24.png', 'randomnpc32.png': 'isrnpcg32.png', 'interactivemapofcyrodiil16.png': 'map16.png', 'interactivemapofcyrodiil24.png': 'map24.png', 'interactivemapofcyrodiil32.png': 'map32.png', '3dsmax16.png': 'maxpath16.png', '3dsmax24.png': 'maxpath24.png', '3dsmax32.png': 'maxpath32.png', 'maya16.png': 'mayapath16.png', 'maya24.png': 'mayapath24.png', 'maya32.png': 'mayapath32.png', 'notepad++16.png': 'npp16.png', 'notepad++24.png': 'npp24.png', 'notepad++32.png': 'npp32.png', 'oblivionfaceexchangerlite16.png': 'obfel16.png', 'oblivionfaceexchangerlite24.png': 'obfel24.png', 'oblivionfaceexchangerlite32.png': 'obfel32.png', 'modlistgenerator16.png': 'obmlg16.png', 'modlistgenerator24.png': 'obmlg24.png', 'modlistgenerator32.png': 'obmlg32.png', 'paint.net16.png': 'paintnet16.png', 'paint.net24.png': 'paintnet24.png', 'paint.net32.png': 'paintnet32.png', 'paintshopprox316.png': 'paintshopphotopro16.png', 'paintshopprox324.png': 'paintshopphotopro24.png', 'paintshopprox332.png': 'paintshopphotopro32.png', 'photoshop16.png': 'photoshoppath16.png', 'photoshop24.png': 'photoshoppath24.png', 'photoshop32.png': 'photoshoppath32.png', 'pixelstudiopro16.png': 'pixelstudio16.png', 'pixelstudiopro24.png': 'pixelstudio24.png', 'pixelstudiopro32.png': 'pixelstudio32.png', 'radvideotools16.png': 'radvideo16.png', 'radvideotools24.png': 'radvideo24.png', 'radvideotools32.png': 'radvideo32.png', 'tes4files16.png': 'tes4filespath16.png', 'tes4files24.png': 'tes4filespath24.png', 'tes4files32.png': 'tes4filespath32.png' }

They key observation here is this is done once on boot. Let's decide also that people should put their tool paths in the Tools Options (sic!) section. So, instead of populating a bass tooldirs structure check for an ini defined path JIT when we initialize the launchers, using the bashIni that's in scope. It would seem that initOptions would add each setting in bash.ini either in inisettings or tooldirs but actually would add it to unknownSettings, effectively discarding it. Things like: { 'oblivionmods': '/Volumes/.../GAMES/Skyrim Special Edition Mods', 'oblivionpath': '/Volumes/.../GAMES/SteamLibrary/steamapps/common/Skyrim Special Edition', 'keeplog': '2', 'autosizelistcolumns': '1' } But then tooldirs would be only queried for the keys specified in initTooldirs - so we could move the ini gets in InitStatusBar. To reach these conclusions I had to find my way out of the single deepest rabbit hole in this branch. initOptions was probably *the* most complex piece in initialization so correct me if wrong - notes from intermediate commits follow: Track keys between constants/bash_default.ini/initTooldirs Regex galore pathlist$u?'([a-zA-Z0-9. _-]+)'\, ?u?'([a-zA-Z0-9. _-]+)'$ -> '$2', {'subfolders': '$1'} pathlist$u([a-zA-Z0-9. _()+\[\]\-', ]+)u'([a-zA-Z0-9. +_\[\]\- ]+\.exe)'$ -> '$2', {'subfolders': ($1)} missing_constants {'SSEEditPath', 'TES3EditPath', 'TES5VREditPath', 'SF1EditPath', 'OblivionBookCreatorPath', 'NifskopePath', 'Tes4EditPath', 'Tes4LodGenPath', 'Tes4GeckoPath', 'Fo4EditPath', 'EnderalEditPath', 'Tes5EditPath', 'Fo3EditPath', 'Tes5GeckoPath', 'FnvEditPath', 'FO4VREditPath', 'SoftimageModTool'} sorted(m for m in missing_constants if 'Edit' not in m) ['NifskopePath', 'OblivionBookCreatorPath', 'SoftimageModTool', 'Tes4GeckoPath', 'Tes4LodGenPath', 'Tes5GeckoPath'] Missing are also Tes4ViewPath/Tes4TransPath that are added by hand in initOptions (...) Tool Options will be dropped from the ini but for now you can probably add OBMM in there as sOBMM as well as boss and LOOT -> I added keys for LOOT, OBMM, boss -> BOSS in the ini just for completeness but the plan is to remove those all - only keep the few settings we need inn boot Note I pass a uid for the anonymous App_buttons (the app_key) that may shuffle saved settings but uid should not be set to the tip as this is translatable (I mean to the tuple (class.__name__,tip) duh) Mopy/bash/balt.py: removed the except AttributeError but kept the warning in app_buttons for imageKey (still looking for a more elegant way of defining images). At this point an error here is a compile time bug, let it blow. Tes4FilesPath was there - Tes4LODGen necessitated adding exeArgs Tes4View had to be set first due to weird App_xEdit.allow_create, cause 'Tes4ViewPath' needed to be set - but what all this was probably about, was setting the path to the Tes4Edit path Get me that bashIni: I had to take some decisions: - add a key to App folder launchers - I chose the .lnk filename as this should be unique. uid is used in 'order' so the order will be reset on renaming the .lnk, also users that had saved those will have their order reset - but this uid = (self.__class__.__name__, self._tip) had to go - not least as __class__._tip was translatable, or cause made no sense to not specify an id from the caller always - same with the tooldir launchers - I added the app_key as uid - should be unique - therefore made uid argument required - instead of adding different items for different game/settings we add all possible buttons and disable/enable them accordingly. Note this is done once on boot - a more dynamical approach would be preferred, so make _display_launcher a property and refreshing on toggling settings like bass.inisettings['ShowTextureToolLaunchers'] -> then remove the settings from the ini - towards #250 Bye tooldirs - the crown jewel here - under #178 #600 AppLauncher encapsulates the find_launcher logic, merging App_Button overrides - this last one gained its class factory :) I did not think much about linux - see next commits where I override AppLauncher, still WIP but the APIs are (will be) there. initOptions: instead of scanning the ini for settings which were not present in inisettings and would never be used we should just update the inisettings unconditionally from the ini if present. TODO: we should track ini uses and collect all the valid settings in inisettings -> done later in this branch Refactor obse handling - obse_tip was only passed for the CK/Game Remove _run_exe

This finally pins it and bins initOptions. I went ahead and collected all the settings we query the ini for, added SkipWSDetection and stopped passing the ConfigParser around - adieu bashIni, bashIni_, bash_ini_ Mopy/bash/initialization.py: we should make sure that we have an absolute path and drop the rest of bass.dirs['app'].join Nit in find_launcher sig Under #178, #600

Part one of using lazy wrappers for the rest of the wx classes that were not wx.Window instances. Turns out the sought after images API is a Lazy wx.Object that puts Lazy's _native widget property to the test. By wrapping images we were able to address the API part of #570, which evolved into refactoring large parts of initialisation (#600). Under #190. Signed-off-by: MrD <the.ubik@gmail.com>

I dropped the unicode error tests, let's adress it if we have a report (if anyone uses this). Removed the u'/re.U Under #600

This decouples belt from the rest of Bash, by moving the non Bash specific parts into PreParser and belt itself into basher - hereby closing #163 after 9+ years of work - gui imports are confined in balt/bash/basher, bolt is thinned nd decoupled from belt and belt code is ready to be deduplicated (gui_fomod shares some gui logic). Some (!) work on #219, but mainly under #600. Signed-off-by: MrD <the.ubik@gmail.com>

@Utumno

On the commit side, we managed to keep this one smaller (~250 commits rather than the ~500 commits that made up 311). On the complexity side, we probably failed, introducing a couple huge refactorings. A lot of Fallout 4 refactoring and patcher porting, Starfield support, native Linux support and refactoring WB's refresh handling are the biggest contributors to line count and commit number. All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### FileInfo(s) (#336) #336 came back with a vengeance this time around. It started with fbb1925, which reworked the AFile API to cover installers and netted us some very nice performance gains and simplifications of refresh APIs - we'll some more work on refreshes later on. ----------------------------------------------------------------------- ### Native Linux Support (#243) Since Windows crapped out on me recently, I decided to ditch it and use Linux full-time. As a result, native Linux support for WB suddenly became a much more pressing issue. 312 improves it to the point where we now mark it as supported, with only some QOL issues remaining (and launchers, but those are a complete WIP anyways, see the later Launchers section in this commit message). There were a ton of random commits that made up this improved support. Basically, whenever I noticed something broken or in need of improvement, I would end up committing the fix, which means the commits are scattered like crazy. Nevertheless, here are some highlights: - e86e939 reworked temporary files handling, but this deserves its own section, which it will get below. - 1993f9b reworked parent handling in wx menus. We were always using Link.Frame.show_popup_menu, but this broke the Bashed Patch menus on Linux entirely. This turned into a mini-refactoring under #190 to track down and do this properly. - b762cc6 made the date and time dialog work on Linux (at the cost of no longer using locale-specific date/time formatting for it). - 20dd955 rewrote Steam game detection entirely to avoid the Windows registry. This was prompted by Starfield not creating a registry key at all, but I was planning to do this anyways, because Linux obviously does not have a registry and because it means users no longer need to launch the game once to generate a registry key. - 61d4d87 is what prompted me to actually mark Linux as supported. This merge added: - Proton detection, meaning the out of the box experience on Linux is now comparable to the one on Windows (in terms of it simply detecting your installed games with no manual intervention needed). - A warning when the Data folder is mounted case-sensitively. - Various fixes for race conditions and more, exposed by Linux's filesystems (perhaps specifically by btrfs?). - Functional Ctrl+Tab handling for cycling through WB's tab and some other misc improvements. - Also worth mentioning here is the File Operations (#241) section, see below for more on that. Linux was then finally marked as supported in c855882. Shortly before 312 was released, @BGazotti also contributed a whole host of small fixes and improvements for WB's Linux support in 00381da. Many thanks! ----------------------------------------------------------------------- ### Temporary Files (#665) This was born out of three needs: - On Linux, the global temporary directory (/tmp) is generally mounted in RAM. This means one can easily run out of space here when e.g. extracting a giant couple BSAs. And even worse, if the system has no swap configured, it can completely lock up when this happens. Wrye Bash should, in fact, *not* lock up the entire system. - We can get a decent speed boost by making sure the temporary directory we're using sits on the same device/filesystem as the Data folder. That way, the OS only has to rename some file paths rather than copying data around. - And lastly, our temporary file APIs were all over the place. There were three distinct types of temp file handling, and various pieces of code seemingly randomly chose which one to use: - bass.getTempDir/newTempDir/rmTempDir - Path.temp and Path.untemp - Just use Path.baseTempDir/tempDir or even tempfile directly and do it completely manually See the commit that introduced this refactoring (e86e939) for a full breakdown of the problems these APIs had. So we designed a new API that can cover all use cases and makes it hard to get wrong. Because, as it turns out, correct temporary file handling is *hard*. And another huge advantage of this design is that it will work with multiple instances of WB running in parrallel, which is an eventual goal. See the aforementioned e86e939 for the full lowdown. ----------------------------------------------------------------------- ### Fallout 4 (#482, #525) The refactoring towards full FO4 Bashed Patch support is still ongoing. The goal was to get it done for 312, but then Starfield released and took the title of this version, plus we didn't want to drag out the release of 312 even further. Still, 312 brings the BP for FO4 very far. Work began in 2327ef4, which cleaned up header flags and implemented the next ten record types. Next up, 548bce5 tackled two difficult record types (NPC_ and OMOD, see the referenced commit for details on their problems) and implemented the next seven record types. With so many record types done, it was time to test them properly. To that end, d941cae ported the first batch of patchers over. In total, we now have 20/33 patchers available for FO4, though not all of them support all the record types they could support yet, simply because those record types aren't implemented yet. 28fb000 continued the records refactoring, though with the added complication that now, each time we implement a record type that an already-ported patcher can target, we also add support for that record type to the patcher in question. In that vein, this merge implements the next thirteen record types and adds them to patchers where appropriate. ----------------------------------------------------------------------- ### Starfield (#667) The star of the show (no pun intended). Note that this is early support, meaning that we don't actually support reading or writing plugins for Starfield yet. The main reason for that is Starfield's... *poor* design, when it comes to the plugin format. You can see the merge commit for more details (ec30f02), but basically, Bethesda clearly did not take moddability into account when designing the plugin format for Starfield. Unless they *drastically* rework the engine before releasing the Creation Kit, the Bashed Patch might not happen, period. ----------------------------------------------------------------------- ### wx (#190) It never ends (okay, maybe soon, but no guarantees). ea96e99 at least took some good steps towards closing off #190 for good by decoupling bosh (the data model) from balt/gui (the GUI backend). An important step towards #600 (headless mode) as well. Some more work landed in 170ad99, where @Utumno introduced gui.Lazy, a wrapper class for components that need to lazily create their native widgets. This led to heavy refactoring of the status bar and its buttons, taking down lots of dodgy code in the process. Also contained in that merge is a sub-merge, bd0a897, which utilized the new lazy classes to really tackle WB's image handling. Especially useful in preparation for the high DPI improvements that will be elaborated on later in this commit message and the launchers (see Launchers section below). ----------------------------------------------------------------------- ### Overlay Plugins (#668) The actual reason to be excited for Starfield's potential is shown in 9d21b40. Overlay plugins are a new type of plugin that does not take up a plugin slot at all, meaning you can have an unlimited number of them(!), with the downsides that such plugins can't contain new records (since they don't have a plugin slot to place the new records into) and must have at least one master. Unfortunately, due to the aforemnentioned massive problems with Starfield's plugin format, overlay plugins and ESLs aren't usable right now. Let's hold out hope that Bethesda can fix this, if only so the ~1 day of engineering I put into supporting them won't go to waste :P But jokes aside, the refactoring in this merge is very much worthwhile on its own. It makes supporting multiple mergeability checks for one game possible (since Starfield supports both ESLs and Overlay plugins - well, in theory it does) and thus opens the door for #596. ----------------------------------------------------------------------- ### Refreshes (#265, #353) We already mentioned fbb1925, which tackled refreshes from the perspective of BAIN and AFile. But midway through 312's development, I wanted to try my hand at implementing a new tab, namely the xSE Plugins tab (#456). That made me run headfirst into a wall, namely the fact that BAIN only knows about plugins and INI tweaks when it comes to trackin and updating other data stores. BSAs were kind of in there too, but not fully. My new tab needed BAIN to keep proper track of it as well, which meant refactoring. That bloomed into a very nice merge in c6ec399, which took down a bunch of old and rusty APIs (e.g. RefreshUIMods, saveListRefresh, _is_ini_tweak, etc.). The refactoring eventually culminated in us centralizing cross-tab communication via bass.Store and taking down a few hundred lines of really annoying, duplicate code. Some followup refactorings grew out of this as well. 70fe061 took down lots of complicated code related to ghost handling, paving the way for much more refresh handling work to come in 313+. Similarly, b8d9e0a refactored ModInfos.refresh, slowly unraveling the complicated stack of refreshes we've got going on. 387c9df tackled refreshes from the perspective of load order handling, attempting to simplify the latter significantly. One big goal here was to preserve as much information about the changes in load order throughout any given LO operation as possible, since that is crucial in order to properly refresh only the changed files - think elegance *and* performance. ----------------------------------------------------------------------- ### File Operations (#241) d897347 reworked our file operations backend significantly, prompted by two things: - Linux support, since we can't delegate to ifileoperation like we do on Windows (duh). - The last couple bits of FOMOD support (fingers crossed). Specifically, it turned out that StarUI, a Starfield mod, would fail to install correctly in WB when run via FOMOD. The reason turned out to be that it wanted to install a single file to two destinations. You can see the linked commit for all the details. BAIN did not have much trouble handling this, since it generally stores everything as destination -> source dicts, but our file operations backends did not support this at all. As part of this commit, we also introduced support for reflinks/file cloning to Wrye Bash. This currently has to be done via a less-than-ideal third-party depedency. An upstream request to the CPython project has stalled. As a result, we only support reflinks/file cloning on Linux and macOS filesystems right now (not a major problem, since ReFS has shown no signs of making it to mainstream Windows deployment yet). But what are reflinks? They behave like regular file copies, except that the backing data is shared between both copies. This is generally enabled by copy-on-write (COW) filesystems, since this feature is pretty much implemented for free on such filesystems. This lets all of WB's copy operations become much faster on Btrfs, XFS, ZFS, APFS, etc. ----------------------------------------------------------------------- ### Auto-Splitting the BP (#533) A long-standing feature request (open for more than three years at this point) has been addressed in 312: automatically splitting the Bashed Patch into multiple files once it ends up with too many masters for a single file. The advantage is obvious, especially compared to the previous behavior (throwing all work away at the end). 6ef2198 introduced the feature, though it turned out to be simultaneously much less work than I expected *and* much more complicated than I expected. Which is weird. ----------------------------------------------------------------------- ### Scattered Packages (#670) I have had this idea for more than three years, at least since I completed work on #380 back in 2020. The last big problem with FOMOD support in Wrye Bash was that a whole bunch of weirdly packaged FOMODs could not be installed in BAIN without a silly workaround (creating an empty plugin and putting it somewhere in the package to trick BAIN into thinking it's a simple package, then using its FOMOD support to remap files and thereby skip the empty plugin, which won't be referenced by the FOMOD's ModuleConfig). The plan was to model the properly, then see what happens. That meant creating an entirely new type of BAIN package. Previously, we had three types: - Simple packages, which contain some files to be placed in the Data folder. - Complex packages, which contain multiple sub-packages. Each sub-package behaves like a simple package. - Invalid packages, whose layout BAIN can't make sense of. Granted, from an end-user perspective, there are more types of 'packages': - Simple/complex packages, which behave like simple packages but have some folders in between the package root and the actual files that need to be placed in the Data folder. In WB's code, these are treated as simple packages. - Markers, which don't exist in the filesystem and are only an aid for users to better organize their packages. In WB's code, these aren't technically packages at all. 759055c introduces a new type of package: - Scattered packages, which have an unrecognized layout, but also come with instructions that tell BAIN how to make sense of the *scattered* mess of files that it otherwise can't parse - hence the name. Combined with some more refactoring to make BAIN recognize scattered packages by the presence of an 'fomod' folder and the aforementioned multi-destination file installation support, this finally tackles the last remaining FOMOD issues (again, fingers crossed). ----------------------------------------------------------------------- ### High DPI (#511) One big goal for 313 is #511, i.e. the ability to customize checkbox colors. But this is much more than just that - it's also about simplifying the resources for checkboxes by using a single SVG instead of a ton of different PNGs, making it possible to have more overlays over the icons (the current implementation of the wizard overlay is just terrible, it's literally a matter of copy-pasting all the checkbox icons, adding a wand over them and then loading all of them manually. This won't scale if we want to, e.g., add an FOMOD overlay). And, of course, this is also about using an SVG to make these scale properly on screens with >100% scaling. To that end, ff276b1 addresses high DPI handling of images. Yes, we already had support for high DPI images via SVGs since #557 in 313, but it turns out that implementation had lots of holes - most notably at 200% scaling, where wxPython decided to scale all our icons *again*, making them absolutely massive and really ugly. With the aforementioned commit, nothing should stop us anymore from tackling #511 in 313. ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @lojack5, @sibir-ine, @BGazotti and many more that GitHub's contribution tracker doesn't list.

@Utumno

On the commit side, we managed to keep this one smaller (~250 commits rather than the ~500 commits that made up 311). On the complexity side, we probably failed, introducing a couple huge refactorings. A lot of Fallout 4 refactoring and patcher porting, Starfield support, native Linux support and refactoring WB's refresh handling are the biggest contributors to line count and commit number. All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### FileInfo(s) (#336) #336 came back with a vengeance this time around. It started with fbb1925, which reworked the AFile API to cover installers and netted us some very nice performance gains and simplifications of refresh APIs - we'll some more work on refreshes later on. ----------------------------------------------------------------------- ### Native Linux Support (#243) Since Windows crapped out on me recently, I decided to ditch it and use Linux full-time. As a result, native Linux support for WB suddenly became a much more pressing issue. 312 improves it to the point where we now mark it as supported, with only some QOL issues remaining (and launchers, but those are a complete WIP anyways, see the later Launchers section in this commit message). There were a ton of random commits that made up this improved support. Basically, whenever I noticed something broken or in need of improvement, I would end up committing the fix, which means the commits are scattered like crazy. Nevertheless, here are some highlights: - e86e939 reworked temporary files handling, but this deserves its own section, which it will get below. - 1993f9b reworked parent handling in wx menus. We were always using Link.Frame.show_popup_menu, but this broke the Bashed Patch menus on Linux entirely. This turned into a mini-refactoring under #190 to track down and do this properly. - b762cc6 made the date and time dialog work on Linux (at the cost of no longer using locale-specific date/time formatting for it). - 20dd955 rewrote Steam game detection entirely to avoid the Windows registry. This was prompted by Starfield not creating a registry key at all, but I was planning to do this anyways, because Linux obviously does not have a registry and because it means users no longer need to launch the game once to generate a registry key. - 61d4d87 is what prompted me to actually mark Linux as supported. This merge added: - Proton detection, meaning the out of the box experience on Linux is now comparable to the one on Windows (in terms of it simply detecting your installed games with no manual intervention needed). - A warning when the Data folder is mounted case-sensitively. - Various fixes for race conditions and more, exposed by Linux's filesystems (perhaps specifically by btrfs?). - Functional Ctrl+Tab handling for cycling through WB's tab and some other misc improvements. - Also worth mentioning here is the File Operations (#241) section, see below for more on that. Linux was then finally marked as supported in c855882. Shortly before 312 was released, @BGazotti also contributed a whole host of small fixes and improvements for WB's Linux support in 00381da. Many thanks! ----------------------------------------------------------------------- ### Temporary Files (#665) This was born out of three needs: - On Linux, the global temporary directory (/tmp) is generally mounted in RAM. This means one can easily run out of space here when e.g. extracting a giant couple BSAs. And even worse, if the system has no swap configured, it can completely lock up when this happens. Wrye Bash should, in fact, *not* lock up the entire system. - We can get a decent speed boost by making sure the temporary directory we're using sits on the same device/filesystem as the Data folder. That way, the OS only has to rename some file paths rather than copying data around. - And lastly, our temporary file APIs were all over the place. There were three distinct types of temp file handling, and various pieces of code seemingly randomly chose which one to use: - bass.getTempDir/newTempDir/rmTempDir - Path.temp and Path.untemp - Just use Path.baseTempDir/tempDir or even tempfile directly and do it completely manually See the commit that introduced this refactoring (e86e939) for a full breakdown of the problems these APIs had. So we designed a new API that can cover all use cases and makes it hard to get wrong. Because, as it turns out, correct temporary file handling is *hard*. And another huge advantage of this design is that it will work with multiple instances of WB running in parrallel, which is an eventual goal. See the aforementioned e86e939 for the full lowdown. ----------------------------------------------------------------------- ### Fallout 4 (#482, #525) The refactoring towards full FO4 Bashed Patch support is still ongoing. The goal was to get it done for 312, but then Starfield released and took the title of this version, plus we didn't want to drag out the release of 312 even further. Still, 312 brings the BP for FO4 very far. Work began in 2327ef4, which cleaned up header flags and implemented the next ten record types. Next up, 548bce5 tackled two difficult record types (NPC_ and OMOD, see the referenced commit for details on their problems) and implemented the next seven record types. With so many record types done, it was time to test them properly. To that end, d941cae ported the first batch of patchers over. In total, we now have 20/33 patchers available for FO4, though not all of them support all the record types they could support yet, simply because those record types aren't implemented yet. 28fb000 continued the records refactoring, though with the added complication that now, each time we implement a record type that an already-ported patcher can target, we also add support for that record type to the patcher in question. In that vein, this merge implements the next thirteen record types and adds them to patchers where appropriate. ----------------------------------------------------------------------- ### Starfield (#667) The star of the show (no pun intended). Note that this is early support, meaning that we don't actually support reading or writing plugins for Starfield yet. The main reason for that is Starfield's... *poor* design, when it comes to the plugin format. You can see the merge commit for more details (ec30f02), but basically, Bethesda clearly did not take moddability into account when designing the plugin format for Starfield. Unless they *drastically* rework the engine before releasing the Creation Kit, the Bashed Patch might not happen, period. ----------------------------------------------------------------------- ### wx (#190) It never ends (okay, maybe soon, but no guarantees). ea96e99 at least took some good steps towards closing off #190 for good by decoupling bosh (the data model) from balt/gui (the GUI backend). An important step towards #600 (headless mode) as well. Some more work landed in 170ad99, where @Utumno introduced gui.Lazy, a wrapper class for components that need to lazily create their native widgets. This led to heavy refactoring of the status bar and its buttons, taking down lots of dodgy code in the process. Also contained in that merge is a sub-merge, bd0a897, which utilized the new lazy classes to really tackle WB's image handling. Especially useful in preparation for the high DPI improvements that will be elaborated on later in this commit message and the launchers (see Launchers section below). ----------------------------------------------------------------------- ### Overlay Plugins (#668) The actual reason to be excited for Starfield's potential is shown in 9d21b40. Overlay plugins are a new type of plugin that does not take up a plugin slot at all, meaning you can have an unlimited number of them(!), with the downsides that such plugins can't contain new records (since they don't have a plugin slot to place the new records into) and must have at least one master. Unfortunately, due to the aforemnentioned massive problems with Starfield's plugin format, overlay plugins and ESLs aren't usable right now. Let's hold out hope that Bethesda can fix this, if only so the ~1 day of engineering I put into supporting them won't go to waste :P But jokes aside, the refactoring in this merge is very much worthwhile on its own. It makes supporting multiple mergeability checks for one game possible (since Starfield supports both ESLs and Overlay plugins - well, in theory it does) and thus opens the door for #596. ----------------------------------------------------------------------- ### Refreshes (#265, #353) We already mentioned fbb1925, which tackled refreshes from the perspective of BAIN and AFile. But midway through 312's development, I wanted to try my hand at implementing a new tab, namely the xSE Plugins tab (#456). That made me run headfirst into a wall, namely the fact that BAIN only knows about plugins and INI tweaks when it comes to trackin and updating other data stores. BSAs were kind of in there too, but not fully. My new tab needed BAIN to keep proper track of it as well, which meant refactoring. That bloomed into a very nice merge in c6ec399, which took down a bunch of old and rusty APIs (e.g. RefreshUIMods, saveListRefresh, _is_ini_tweak, etc.). The refactoring eventually culminated in us centralizing cross-tab communication via bass.Store and taking down a few hundred lines of really annoying, duplicate code. Some followup refactorings grew out of this as well. 70fe061 took down lots of complicated code related to ghost handling, paving the way for much more refresh handling work to come in 313+. Similarly, b8d9e0a refactored ModInfos.refresh, slowly unraveling the complicated stack of refreshes we've got going on. 387c9df tackled refreshes from the perspective of load order handling, attempting to simplify the latter significantly. One big goal here was to preserve as much information about the changes in load order throughout any given LO operation as possible, since that is crucial in order to properly refresh only the changed files - think elegance *and* performance. ----------------------------------------------------------------------- ### File Operations (#241) d897347 reworked our file operations backend significantly, prompted by two things: - Linux support, since we can't delegate to ifileoperation like we do on Windows (duh). - The last couple bits of FOMOD support (fingers crossed). Specifically, it turned out that StarUI, a Starfield mod, would fail to install correctly in WB when run via FOMOD. The reason turned out to be that it wanted to install a single file to two destinations. You can see the linked commit for all the details. BAIN did not have much trouble handling this, since it generally stores everything as destination -> source dicts, but our file operations backends did not support this at all. As part of this commit, we also introduced support for reflinks/file cloning to Wrye Bash. This currently has to be done via a less-than-ideal third-party depedency. An upstream request to the CPython project has stalled. As a result, we only support reflinks/file cloning on Linux and macOS filesystems right now (not a major problem, since ReFS has shown no signs of making it to mainstream Windows deployment yet). But what are reflinks? They behave like regular file copies, except that the backing data is shared between both copies. This is generally enabled by copy-on-write (COW) filesystems, since this feature is pretty much implemented for free on such filesystems. This lets all of WB's copy operations become much faster on Btrfs, XFS, ZFS, APFS, etc. ----------------------------------------------------------------------- ### Auto-Splitting the BP (#533) A long-standing feature request (open for more than three years at this point) has been addressed in 312: automatically splitting the Bashed Patch into multiple files once it ends up with too many masters for a single file. The advantage is obvious, especially compared to the previous behavior (throwing all work away at the end). 6ef2198 introduced the feature, though it turned out to be simultaneously much less work than I expected *and* much more complicated than I expected. Which is weird. ----------------------------------------------------------------------- ### Scattered Packages (#670) I have had this idea for more than three years, at least since I completed work on #380 back in 2020. The last big problem with FOMOD support in Wrye Bash was that a whole bunch of weirdly packaged FOMODs could not be installed in BAIN without a silly workaround (creating an empty plugin and putting it somewhere in the package to trick BAIN into thinking it's a simple package, then using its FOMOD support to remap files and thereby skip the empty plugin, which won't be referenced by the FOMOD's ModuleConfig). The plan was to model the properly, then see what happens. That meant creating an entirely new type of BAIN package. Previously, we had three types: - Simple packages, which contain some files to be placed in the Data folder. - Complex packages, which contain multiple sub-packages. Each sub-package behaves like a simple package. - Invalid packages, whose layout BAIN can't make sense of. Granted, from an end-user perspective, there are more types of 'packages': - Simple/complex packages, which behave like simple packages but have some folders in between the package root and the actual files that need to be placed in the Data folder. In WB's code, these are treated as simple packages. - Markers, which don't exist in the filesystem and are only an aid for users to better organize their packages. In WB's code, these aren't technically packages at all. 759055c introduces a new type of package: - Scattered packages, which have an unrecognized layout, but also come with instructions that tell BAIN how to make sense of the *scattered* mess of files that it otherwise can't parse - hence the name. Combined with some more refactoring to make BAIN recognize scattered packages by the presence of an 'fomod' folder and the aforementioned multi-destination file installation support, this finally tackles the last remaining FOMOD issues (again, fingers crossed). ----------------------------------------------------------------------- ### High DPI (#511) One big goal for 313 is #511, i.e. the ability to customize checkbox colors. But this is much more than just that - it's also about simplifying the resources for checkboxes by using a single SVG instead of a ton of different PNGs, making it possible to have more overlays over the icons (the current implementation of the wizard overlay is just terrible, it's literally a matter of copy-pasting all the checkbox icons, adding a wand over them and then loading all of them manually. This won't scale if we want to, e.g., add an FOMOD overlay). And, of course, this is also about using an SVG to make these scale properly on screens with >100% scaling. To that end, ff276b1 addresses high DPI handling of images. Yes, we already had support for high DPI images via SVGs since #557 in 313, but it turns out that implementation had lots of holes - most notably at 200% scaling, where wxPython decided to scale all our icons *again*, making them absolutely massive and really ugly. With the aforementioned commit, nothing should stop us anymore from tackling #511 in 313. ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @lojack5, @sibir-ine, @BGazotti and many more that GitHub's contribution tracker doesn't list.

I have not implemented the whole backported ESL thing for regular Skyrim. First off, I dislike supporting older game versions. But more to the point, xEdit does not implement anything like it either, so I'm not sure the whole description of it being HEDR-version-independent is actually true. Let's just operate off the assumption that SSE always supports 1.71 and the expanded range and if that breaks anything for anyone it's a PEBKAC, with the solution being either to upgrade to the latest game version (recommended) or to install the backport plugin. Also includes some minor fixes for to_os_path usages, Enderal SE updates (latest Enderal SE on modpub supports HEDR 1.71 and expanded range, so same situation as SSE now) and adding support for the limit fixer plugin detection on Linux. Introduce GameInfo.post_init Turns out bass.dirs['mods'] isn't set by the time init() runs yet, which makes perfect sense in hindsight. The exact point at which we post-init is an open question - plus, in the future we'd like to use bosh.se_infos to check for xSE Plugin existence etc., so that's another can of worms. Ut: that's #600 - we 'd need to define boot stages (see initialization.bash_dirs_initialized and co), but this needs the dust to settle from #336 (includes initialization work). For now keep post_init use count to 1 :) Closes #673

At long last, we can close the oldest still-open issue in the WB repo. With the attention translations have been getting in 313's development, a persistent language setting is more important than ever. However, its implementation has always been blocked by one fundamental problem: all of Wrye Bash's settings are stored in .dat files in the My Games subfolder for the game. Which means that a game must be chosen before we can actually read or write settings. But in order to get to that point, we need to have already set up wx and the locale, which means it's far too late to read a language setting. The result is that I had to create a new configuration file for WB's early boot. I chose a TOML file, so that it's easily editable by end users as well, and placed it per user (using XDG_CONFIG_HOME if that environment variable is set, otherwise in %AppData% on Windows, ~/.config on Linux and ~/Library/Application Support on macOS). With that, we can tackle this issue (and as a bonus, remember the last-launched game and default to that when showing the game selection popup, which is another feature that has been requested for years and was blocked by the exact same problem). A side effect is that early boot has further crystallized and some more documentation has been added to make dependencies between parts and stages clearer. Closes #26 Under #500 and #677

I thought let's inline those two static methods as they're not as huge as in the old days. Turns out 'Start application' was not accurate - BashApp did nothing much anymore - the thin class so to speak. The boot code is nowhere near to where it was 10 years ago, looks almost boring now, used to be a perilous mess of wx and weird unpredictable stuff with queer names like bosh. Hundreds of tracebacks. Encapsulated the splash thing. What I like most is further untangling the main modules responsibilities. Under #600

Move plugin types handling at top level - we have enough variety now to need a centralized model. Note I had to move the mod_files import in local scope - this is #600 territory, but this commit already simplifies stuff as GameInfo can freely import isPBashMergeable and MasterFlag. The former will allow making mergeability checks a dict reducing flags handling in client code. I added comments and rearranged method definitions - with the help of the IDE's marking of overrides this makes it easier to see what are the responsibilities of each (sub)class. Note _pbash_mergeable_no_load is a quick check for MERGE

We need to do that - anyway were there be called before refresh we would blow on accessing modInfos=None - an explicit (to be) boot stage is better than local imports - plus I need to pass the correct MasterFlag.ESM in there. Under #600

@Ortham

SF introduced new plugin types - current plugin types handling was ad hoc - there are 174/146 in code (324/165 in code, case insensitive) occurrences for `(?<!\.)esl` (that is the str esl not following a dot, which excludes mostly vanilla files listed in game/ modules) in python files, including comments and strings. Mostly those were ad hoc handling of esl stuff (rinse and repeat for overlay) - so this should be repeated for each new type. Enters PluginFlag, and some new game attributes that store info about the plugin types a game supports, and the code now iterates available flags instead of checking has_esl and co. Occurrences of `(?<!\.)esl` went down to 35/14 (174/44 case insensitive), most of them unrelated stuff - very few has_esl remain as a curiosity. This permitted us implementing the rest of SF support that depended on mid/blueprint plugins - I followed @Ortham's posts, especially: - https://blog.ortham.net/posts/2024-06-28-load-order-in-starfield/ - https://blog.ortham.net/posts/2024-07-30-blueprint-plugins/ and loot issues linked there - for saves mainly: https://github.com/Nexus-Mods/StarfieldSaveTool/blob/1270ecf4009d21d807d4caeda5a53e9427f25d3d/DatFile.cs Eventually this fits well with the ongoing work on #336/#353. ModInfo lost 10 methods out of ~60 (gained a few helpers but these do not add any complexity), while ModInfos stopped caching mergeability following the line of moving responsibilities to the info - refresh/rescan mergeable lost half of their lines and mergeability is essentially cached in 'mergeInfo' table attribute (where it is stored anyway). This was one of the trickiest parts in ModInfos.refresh so now we are ready to address #353 core. There is some #600 related work here also, in initialization and decoupling of top level modules (especially Loot). Closes #681. Closes #688.

Utumno changed the title ~~[Reserved]~~ Headless mode Oct 12, 2021

Utumno added this to the 311 milestone Oct 12, 2021

Utumno added A-boot Area: Early Boot (bash.py, bush.py and initialization.py) C-goal Category: Long term goal. May be code-related or a meta goal. C-refactoring Category: Refactoring. A purely internal refactoring, with no user-facing changes labels Oct 12, 2021

Utumno mentioned this issue Oct 16, 2021

Various locale settings crash WB on boot #610

Closed

Infernio modified the milestones: 311, 312 Jul 4, 2022

Infernio modified the milestones: 312, 313 Jan 5, 2023

Utumno mentioned this issue Oct 19, 2023

Boot sequence #244

Closed

4 tasks

Utumno mentioned this issue Nov 4, 2023

Configuring paths via settings #572

Open

Utumno added a commit that referenced this issue Nov 4, 2023

Move genHtml higher in boot:

2c9e69a

I dropped the unicode error tests, let's adress it if we have a report (if anyone uses this). Removed the u'/re.U Under #600

Infernio modified the milestones: 313, 314 Dec 16, 2023

Utumno modified the milestones: 314, 315 Jul 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Headless mode #600

Headless mode #600

Infernio commented May 27, 2021 •

edited by Utumno

Loading

Headless mode #600

Headless mode #600

Comments

Infernio commented May 27, 2021 • edited by Utumno Loading

Infernio commented May 27, 2021 •

edited by Utumno

Loading