Make MapParser.cs more robust for patches and additional map formats

[ballerburg9005]

I wanted to add Patch support to Qodot 4, but when I had a look at the map parser it didn't seem very robust in design. Basically anything makes it fail that is not formatted correctly or that it does not understand.

For example if it encounters patches, it fails. If it encounters some extension to the Quake 1 format, it fails. There are like a dozen extensions beyond Quake 1 that add stuff to the map format. Not sure how many, but it is extensive.

I was wondering if it was just provisionary thing? Are you working on a total rewrite or would welcome someone to do it?

As the .map format is complicated, I am not sure if I am up to the task. But I could rewrite it with regex, so it reads at least as much as it can without totally failing, and that it simply ignores stuff it doesn't understand.

With Trenchbroom Loader, you could load Quake3 or Xonotic maps if they were previously converted to Valve 220 format in Netradiant Custom. Just the patches were missing. A lot of maps were playable this way, and some didn't even look that much different. Plus you can convert any bsp file with q3map2 to map format, with little to no issues.

I think you only get real value from able to load Radiant maps, if it can actually load existing maps from other games. And there are literally like tens of thousands of maps out there.

So I believe a rewrite and Patch support would be a major priority.

[EMBYRDEV]

The current version I ported to C# for Godot 4.x is a direct port of the libmap parser. It has unfortunately been a bit of a headache and I'd love to rewrite it with a more robust design, however I have fairly limited time so I'm currently focused on maintaining and polishing existing behaviour.

That being said, I fully welcome any contribution like this and would be happy to merge it if this is something you decide to work on!

[DeerTears]

@ballerburg9005 Would you mind explaining what you mean by patches? I'm super intrigued and would love to see Qodot become more capable. It seems most active community members roll their own maps, but I also see no downside to trying to include more map formats by making the parser more flexible.

[ballerburg9005]

Ok thanks. I will work on it the next weeks.

@DeerTears Patches are basic geometry, like Brushes, but you can make more complex shapes with them and they are meant for more decorative purposes. Examples would be bent pipes, arches, fabric or flags, barrels, cone roofs, pillars, round floor platforms, and all sorts of things. They are extensively used in a lot of old maps (esp. for door arches), because the polygons are calculated based on render distance. I think they were first introduced in Quake 3.

Community mappers may not really make use of patches, which is why their maps are still playable without patch support. But then if patches are heavily utilized, like in original Q3 maps, you would need actual patch support to make them playable.

I think Trenchbroom has some limited support to load Patches from existing maps, but no GUI tools to create them. Last time I tried it was not really working.

[DeerTears]

How are they defined? I can't find documentation for them, which is what would help me the most.

In Qodot there's no need to declare a brush as decorative vs functional, unless we're trying to toggle-off the collisions or script it to do something, since that's already covered in our solid class system. And I'm assuming all brushes in patches are still convex. So to me this sounds like a new solid class for porting patches would be helpful in the default fgd? I'd be happy to write a definition to support these patches too if Quake 3 stores them separately from worldspawn.

[ballerburg9005]

Since people can use netradiant-custom and convert maps simply by copy & pasting into a new Valve 220 formatted map, this only leaves patchDef2 for all maps that are not Doom3 and beyond (which has patchDef3, brushDef3, etc).

Here is an example of a patch.

// brush 358
{
patchDef2
{
exomorphx/panel-metal_panel01a
( 9 3 0 0 0 )
(
( ( 504 220 -552 0 0 ) ( 528 220 -552 0 -0.09375 ) ( 552 220 -552 0 -0.1875 ) )
( ( 504 216 -552 0.015625 0 ) ( 528 216 -552 0.015625 -0.09375 ) ( 552 216 -552 0.015625 -0.1875 ) )
( ( 504 216 -556 0.03125 0 ) ( 528 216 -556 0.03125 -0.09375 ) ( 552 216 -556 0.03125 -0.1875 ) )
( ( 504 216 -560 0.046875 0 ) ( 528 216 -560 0.046875 -0.09375 ) ( 552 216 -560 0.046875 -0.1875 ) )
( ( 504 220 -560 0.0625 0 ) ( 528 220 -560 0.0625 -0.09375 ) ( 552 220 -560 0.0625 -0.1875 ) )
( ( 504 224 -560 0.078125 0 ) ( 528 224 -560 0.078125 -0.09375 ) ( 552 224 -560 0.078125 -0.1875 ) )
( ( 504 224 -556 0.09375 0 ) ( 528 224 -556 0.09375 -0.09375 ) ( 552 224 -556 0.09375 -0.1875 ) )
( ( 504 224 -552 0.109375 0 ) ( 528 224 -552 0.109375 -0.09375 ) ( 552 224 -552 0.109375 -0.1875 ) )
( ( 504 220 -552 0.125 0 ) ( 528 220 -552 0.125 -0.09375 ) ( 552 220 -552 0.125 -0.1875 ) )
)
}
}

Here you see various types of Patches inserted in Netradiant. You can manipulate the vertices mostly by moving, but sometimes also rotating (which gives the curves a different arch). Note that you don't manipulate polygons, but rather various curves and shapes with those vertex points. I think bevels, cylinders and mesh are the most used types.

Here is a video demo of the scene:

2023-07-14_01-07-45.out.mp4

I wasn't aware that they actually have collision, but apparently they do.

I'm not sure exactly how it works, it doesn't seem that trivial. I guess you would have to go through the source code and just replicate it. Personally I would first redo the MapParser.cs, so it can also deal with brushDef etc. in the future and at least ignores it without failing, then write PatchDef2 support.

Not sure if you want to jump in an do something as well?

[ballerburg9005]

I have been working on it the last days and just reached the milestone of replicating the functionality of the old parser.

I have programmed it such that it is extremely easy to adapt to the zillion variations of format, thanks to crazy regex usage.

The new parser will very soon be able to read ANY map format from Q1 to Q4. This means we can use more advanced tools like Netradiant Custom and for example use patch meshes for terrain, rather than splitting up hundreds of brushes.

It does not fail under any circumstance, and ignores alien information.

[EMBYRDEV]

Oh this would bre a really neat first step and opens the doors to a lot of awesome features! Excited to check out the PR!

[HK47196]

Since people can use netradiant-custom and convert maps simply by copy & pasting into a new Valve 220 formatted map, this only leaves patchDef2 for all maps that are not Doom3 and beyond (which has patchDef3, brushDef3, etc).

Is this documented somewhere? I did some searching but couldn't find any authoritative specification on .map & its various incarnations.

[ballerburg9005]

It is not really documented anywhere.

I am currently using output from QuArK to produce the various exotic variations from Quake 1 - Quake 3, plus Doom 3 and such. We are maybe looking at close to twenty thousand lines of code here in total how the whole thing is implemented. I believe the patch code from Netradiant is about 5k or so.

Luckily there is a quite terse Quake 3 bsp loader for Unity in C# called uQuake, which I am currently taking as reference to implement patches in Qodot.

Basically the story of the .map format is this: Quake 1 was the first to come. A brush is a collection of faces, each face is defined by three 3 vectors + a texture + various modifiers such as scale and rotation. But this produced some shortcomings with texture alignment, which games solved "inofficially" with various additions, such as a texture plane in the Valve format or the "// TX" comment. In Quake 3 they added patches and also brushDef, which also uses a texture plane but in a different notation. Then in Doom 3 they used 4D Vectors for everything and someone said online some later games even use 5D vectors. Quake 2 also uses semicolons instead of slashes for comments and little stuff like that is different.

Here is my reference file:

omnidefs.txt

There are more variations still, but its fair to say no one will ever use it.

I am not sure currently how I could best fit loading Patches into the pipeline of Qodot. I mean in theory it should be possible to just "sell" it to Qodot as a brush after it is tesselated and such, instead of cloning the whole pipeline for brushes. But probably this will break some assumptions made about brushes somewhere, and then maybe result in a mess. I have to dive more into the Qodot source code. It is a bit over my head, so I am taking a break currently.

Here is the new parser (work in progress): https://github.com/ballerburg9005/Qodot-new-map-parser/blob/main/addons/qodot/src/core/MapParserRegex.cs

Update: Progress is imminient!

[ballerburg9005]

Another milestone reached: Patches work now.

Code is still kind of ugly in places (repo is here).

I reused the faceGeo pipeline, not sure if that results in tons of duplicate vertices? Will leave it for now.

Next up, some code cleaning, then brushDef2. Then I make a PR, and after that make it load all the other exotic formats.

2023-10-19_21-24-10.out.mp4

Xonotic map Bloodyrun:

2023-10-19_23-49-00.out.mp4

[HK47196]

Since people can use netradiant-custom and convert maps simply by copy & pasting into a new Valve 220 formatted map, this only leaves patchDef2 for all maps that are not Doom3 and beyond (which has patchDef3, brushDef3, etc).

Is this documented somewhere? I did some searching but couldn't find any authoritative specification on .map & its various incarnations.

Didn't think of it at the time, but a reference implementation for the later .map format is Doom 3 itself:
https://github.com/id-Software/DOOM-3/blob/master/neo/idlib/MapFile.cpp
https://github.com/id-Software/DOOM-3/blob/master/neo/tools/radiant/EditorBrush.cpp

Relevant Trenchboom issues & PRs with regards to post-Quake map formats:
TrenchBroom/TrenchBroom#2425 <- lots of good info from an Unvanquished developer posted
TrenchBroom/TrenchBroom#2427
TrenchBroom/TrenchBroom#3806

[ballerburg9005]

I just noticed that the C# was switched back to GDscript ... this is a huge problem for my map parser as it was using very advanced regex specific to C#, with lookahead and lookbehind features, plus capture groups, stuff that's only going to work reliably in highly standartized regex implementations ... which I believe Godot does not have.

So it not only needs conversion away from C#, but the new parser basically needs a total rewrite with simpler regex, on a line-by-line basis.

The big benefit of using excessive regex was, that it made it extremely robust to not fail from all the different variations in the .map format, while it was very maintainable and easy to adapt. It is unfortunate, but with GDscript I highly suspect that it just can't turn out as rock-solid and agile as it was before...

Well enough whining, I think switching away from C# is an overall good thing. But the additional work that it has caused for this issue is very substantial. I don't know when I will have time for this.