Naive blending compatibility

[illwieckz]

Work-in-progress attempt to implement compatibility for naive blended assets in the linear pipeline.

It includes a patch to delay the loading of textures from pre-collapsed stages to the end of the stage parsing, after the blend mode is known, because it is required to select the right compatibility code for the given stage.

[illwieckz]

For now I haven't implemented a keyword to enforce blended stages to be processed the linear way.

The compatibility mode simply makes the texture and the colorGen not being converted from sRGB, making the colors painted as linear, like the naive pipeline did before the linear pipeline was implemented.

[slipher]

In what scenario is the code intended to be used? For example rendering legacy maps in linear blendspace? Rendering new game models in naive blendspace?

[illwieckz]

In what scenario is the code intended to be used?

Rendering maps recompiled for the linear pipeline but without any asset change.

You take a legacy map, you rebuild it, you modify nothing else, it would be “good enough”.

To complete the port one would have to modify the blending shaders, but this compatibility mode removes that step as a hard requirement.

Especially, maps using only the filter blend mode like forlorn can even avoid any tweak.

[illwieckz]

Map forlorn: 3716 2523 226 123 -3

Point of interest: the background door, behind two windows in a row.

Naive blend shaders, naive build, naive renderer:

Naive blend shaders, linear build, linear renderer:

Naive blend shaders, linear build, linear renderer with compatibility blend mode:

Map atcshd, viewpos 720 -384 296 150 0

Point of interest: the force field.

Naive blend shaders, naive build, naive renderer:

Naive blend shaders, linear build, linear renderer:

Naive blend shaders, linear build, linear renderer with compatibility blend mode:

The glass in forlorn uses blend filter and it looks just right with the compatibility mode. There is nothing to edit, we just have to rebuild the map and call it a day.

The force field in atcshd uses blend add and then the compatibility is much more perfectible, but that's still better with the compatibility mode. It's better to edit the asset, but that's already a good start that doesn't look broken. Without the compatibility mode the waving grid on the force field is almost invisible, now we see the wave effect as expected. If you don't compare the maps side by side before and after the rebuild, with the compatibility mode you may believe it's what's intended, as it would be harder to notice the color isn't exactly as the one expected, while without the compatibility mode you immediately notice that something is broken without needing any comparison reference.

[illwieckz]

One that suffers a lot is the plat23 map, that is also the map for which the lighting itself is the most affected, we may even consider to tweak the lighting, not just the blended forcefield shader.

Map plat23, viewpos 0 2688 192 -90 13

Point of interest: the force field.

Naive blend shaders, naive build, naive renderer:

Naive blend shaders, linear build, linear renderer:

Naive blend shaders, linear build, linear renderer with compatibility blend mode:

We notice that without the compatibility mode, we cannot see the grid of the forcefield at all, and the whole surface is too much transparent, while with the compatibility mode, we see the grid and the surface is opaque enough, that fixes an element of gameplay.

Of course the color would better be tweaked, and like with atcshd, we can make it a bit less opaque, but the gameplay is fixed by the compatibility mode.

[illwieckz]

Alongside forlorn, a map that can be re-released just by rebuilding it once the compatibility mode is implemented is vega.

Map vega, viewpos: -181 -151 133 35 13

Points of interest: the various glass surfaces on doors and cylindrical models on the floor.

Naive blend shaders, naive build, naive renderer:

Naive blend shaders, linear build, linear renderer:

Naive blend shaders, linear build, linear renderer with compatibility blend mode:

Something we can touch-up a bit in vega are the corridor lamp glow maps (the ones I discuss aren't visible in those screenshots), some would look better if more brighter, but they are not an obvious problem like the door glass is. This glow map tweak is only wanted because since the light attenuation curve is more physically correct, the ambiant light is brighter, and then the contrast between usual surfaces and glow maps is less obvious. Tweaking the glow maps would be just a perceptual work to improve the looking of the map, there is no bug in them. It just happens that the mapper never had the opportunity to test his glow maps with a physically correct lighting, so he couldn't know they weren't bright enough.

[illwieckz]

Hmm, one that looks ugly with that compatibility mode for add is yocto, with the linear mode some yocto glass looks correct on some surfaces and some other kind of glass looks almost correct. The add blend mode can't have a fully automated compatibility mode anyway, while the compatibility mode will probably always look correct with filter.

[illwieckz]

Hmm, while for filter it's always good, with add it's a hit or miss. Half of the time it's better, half the time it's worst.

That's annoying because I was hoping for a quirk that would be better in average.

If we can't be better in average for blend modes as popular as add, then it means we require a manual knob for half of them. And if we require a manual knob for half of them, it is better that this manual knob is only used to fix thing, not to cancel the fix.

And if a blend mode cannot be automated and we better default to the “non fix” for it, then for consistency we better stick to the default behavior for all blend modes.

What this experiment demonstrated though, is that switching to either linear or to naive mode is always a good way to give a better result in a very quick way, without editing the actual files.

So I plan to do that:

• rely on the code already implemented in this branch to always know the blend options when loading a texture
• rely on the code already implemented in this branch to be able to switch the light mode per blended stage
• don't automate that switch
• add a keyword like blendMode naive and blendMode linear to annotate stages
• add a cvar to force a blend mode to easily test which blend mode better suits a given scene (for the mapper to test things)

All the maps I have tested from stock Unvanquished assets and InterStellarOasis ones were able to immediately get something usable just by selecting one mode or another. I assume we can blindly enable the linear mode in all the blend filter stages, and for the other ones, we have to check.

Selecting one mode or another per blend stage is enough to be able to re-release a map just by rebuilding it, without breaking the gameplay. Tweaking the assets to make them look better is only an improvement over that.

It's probable that only the quoted force fields may really benefit from manual asset tweaking outside of just flipping a blend mode in a shader (like actually editing an image).

[slipher]

1. How will this compatibility mode be controlled? A worldspawn option to turn it on for all map surfaces? But then what if texture packs used by the map have already been optimized for linear blendspace?
2. The heuristic-based color transformation here (compared to the default behavior, it converts some colorgen and texture colors from linear to SRGB) is basically fudging things to be brighter; there is no real connection to correct sRGB conversion. We could just as well consider a function like color *= 1.3 for making stuff brighter. So for the colorgen stuff at least, I suggest renaming pStage->convertColorFromSRGB to something more generic like adjustColorForBlendspace so that we give ourselves freedom to use different heuristic functions later.
3. filter is the one blend function that behaves almost identically between linear and naive blending. I believe we shouldn't need to do anything to it. But that spot on Forlorn indeed comes out anomalously dark... I will try to investigate.
4. The compatibility heuristic should only act on an approved and tested list of blend functions, since with other ones it may make things worse. With naive blending, blendfunc add systematically makes things brighter than they should be, so it may help to apply a compatibility heuristic making things brighter. But with naive blending, alpha blending systematically makes things darker than they should be, so the existing heuristic would make things worse. Looks like it has just been tested on add and filter blending so far.
5. I've been intending to add a cvar r_forceBlendspace which would allow us to decouple the blending regime used by the renderer, and the sRGB-ness of the lightmaps. It would be useful for debugging purposes: having those two things tied together makes it hard to tell whether changes in a map are caused by different lightmaps or by different blending. It would help investigate things like #3 in this list. With the cvar you would be able to use naive blending with a map built with sRGB lightmaps or linear rending with a legacy map. I already implemented this before on a branch while evaluating other sRGB-related stuff, so I just need to port/rebase the code. Hmm, I like the term "blending regime", maybe I will call it r_forceBlendRegime instead.

[illwieckz]

1. How will this compatibility mode be controlled? A worldspawn option to turn it on for all map surfaces? But then what if texture packs used by the map have already been optimized for linear blendspace?

If we do a compatibility mode, it would be the default mode, and would use heuristics to somewhat fix legacy assets in the linear pipeline.

And then q3shaders fixed for the linear blending would have a special keyword to tell they're fully linear and skip the heuristics. That was my idea when I first started to work on this.

My first idea was that the blend something or blendFunc something syntax would automatically use the compatibility mode with heuristics, and then we would have something like linearBlend something for non-legacy assets to skip the heuristics.

Now, the results of my tests with blend add seem to geopardize such default compatibility mode as I now know no acceptable heuristics for blend add.

I don't see any advantage of making a compatibility mode that doesn't work out of the box, if it requires to flag something to enable the compatibility mode, then it's not a compatibility mode.

Now, what we can do is to not have such compatibility mode, but to add compatibility keywords to the shader lexicon.

Basically one would do:

blend add
blendMode naive

This requires every shader that are known to work better the naive way to be tagged this way. With legacy maps it would do nothing, it's already naive, with maps rebuilt for the naive pipeline, it would upload the image as is without transformation, the naive way, because we would have tested we prefer that look.

If we can find good heuristics that works “good enough” like 95% of the time, we can consider such automatic compatibility mode, otherwise no. Right now with blend add it's fifty-fifty so with my current knowledge such automatic compatibility mode isn't possible.

Now, one thing I need to say is that I want to guarantee mappers and map porters they wont have to edit all the blend images in an image editor to benefit from the linear pipeline. I want to reduce the map migration cost as much as possible, ideally just requiring a rebuild with the new map compilation profile, otherwise we will suffer maps rebuilt the legacy way forever, and this is not good because we have unfixable bugs with maps built the legacy way, especially when we start adding specular maps, etc.

If we cannot avoid requiring the edit of some legacy assets when rebuilding map the linear way, then the cost of such edit should be the lowest possible. If switching from linear to naive a blend shader fixes it in an acceptable way, that cost looks small enough for me. Maps usually only have one or two blend shaders, with luck, some of them will already in be rendered in a “good enough way” without doing anything. Using a trick once or twice per map by adding a keyword to a shader is fine. Also I don't mind if such trick hasn't any mathematical meaning.

The correct way to fix a blend shader is to edit the image, but if a low-cost stupid and wrong trick provides an acceptable result, we should provide it. In other words: I want the mappers to be able to port the maps first and procrastinate the perfect solution for the only one shader that doesn't work well by default. If we don't give such option, mappers may keep the whole map the legacy way while only one shader is broken the linear way, that would be a shame…

[illwieckz]

And by mappers I include myself, I want to rebuild all the Unvanquishes stock maps the linear way as soon as possible, and I want to do the same for the InterstellarOasis maps. I don't want to edit images. I don't want to allocate time to run a trial and error dance while editing images on every step to fix a shader for the linear build. One force field to edit? Why not, but not more than that.

[illwieckz]

It's also important to know that once an Unvanquished release is published with the stock maps built the linear way, I will make that map build profile the default in Urcheon and NetRadiant and I will re-release NetRadiant. They will provide a legacy build profile, but the linear one will become the default.

That's also why I want to reduce the cost of porting maps to the linear pipeline to zero if possible, almost zero otherwise. If heuristics can do that, let's go. If heuristics can't do, I want the port effort to only require one or two keywords added to shaders, per map, and for not all the maps.

We need that starting from a given day, very soon, every newly built map, brand new or rebuilt, will use the linear pipeline, by default.

[slipher]

It seems wrong to enable compatibility heuristics by default. This means that a mapper starting a new map from scratch will be burdened by compatibility cruft, unless they know about the hidden option to turn it off.

I don't see any advantage of making a compatibility mode that doesn't work out of the box, if it requires to flag something to enable the compatibility mode, then it's not a compatibility mode.

Supposed you have implemented the magical compatibility heuristic that makes everything look good 100% guaranteed. If the compatibility mode is off by default, all you have to do to re-release a map is add the sRGB flags for q3map2, enable the compatibility option, and rebuild. That's hardly more difficult than just changing the q3map2 flags and rebuilding. For artists, the more common pattern is to work on the map for a relatively brief period of time, then stop. Legacy map re-releases are likely to be done by a more technical person such as yourself. It doesn't make sense to hobble artists starting a new map just to make recompiling a legacy map 1% easier.

[illwieckz]

It seems wrong to enable compatibility heuristics by default. This means that a mapper starting a new map from scratch will be burdened by compatibility cruft, unless they know about the hidden option to turn it off.

Unless newly made shaders use a special syntax that never rely on heuristics. My idea was that the historical blendFunc function would do heuristics, and newly assets would use another keyword and skip the heuristics. This way the engine would always have the knowledge of the shader being naive or not. But for that we need heuristics that work good enough, something we don't have.

So I don't consider such heuristics anymore anyway, unless someone comes with some magic by the end of the next week.

[slipher]

Unless newly made shaders use a special syntax that never rely on heuristics. My idea was that the historical blendFunc function would do heuristics, and newly assets would use another keyword and skip the heuristics.

Well, the Q3A shader manual still seems to be state-of-the-art for shader documentation... the likely result is that 90% of new maps would keep using the old blendFunc. Only the most intrepid mappers like Matth who reverse-engineer Unvanquished assets to find out how to make cool effects would have a chance of using it 😆

[illwieckz]

So,

• I rewrote a bit the delayed image loader,
• I removed the compatibility heuristics and added the naiveBlend shader keyword.

Here is how looks the forcefield shader on plat23 when I set that naiveBlend keyword on the forcegrid stage, and only on it:

Here is the modified shader:

textures/plat23_custom/forcefield
{
	qer_editorImage textures/plat23_custom_src/forcefield_p
	surfaceparm trans
	surfaceparm nomarks
	surfaceparm nolightmap
	cull none
	{
		map textures/plat23_custom_src/forcefield_a
		tcMod Scroll .1 0
		blendFunc add
	}
	{
		map textures/plat23_custom_src/forcegrid_a
		tcMod Scroll -.01 0
		blendFunc add
		naiveBlend
		rgbgen wave sin .2 .2 0 .4
	}
	{
		map textures/plat23_custom_src/forcefield_heathaze
		stage heathazeMap
		deformMagnitude 1
		tcmod scale 1 1
		tcmod scroll .03 .03
	}

It's a very easy way to workaround a shader designed for the naive pipeline when rendering with the linear pipeline.

I like it.

[illwieckz]

That may be ready.

I now believe it's a bad idea to have a cvar to force a blend mode for quickly testing how shaders in a scene render, because there can be multiple blended stages in a single shader, and people may want to only tweak on of them. Such cvar would tweak all of them at once and then would be useless.

[illwieckz]

I also implemented the delaying of animation textures.

[illwieckz]

This also improves over existing code (even without using the feature):

• Invalidate the stage when the colormap is missing when it's not a colormap stage (like diffuse map)
• Do not attempt to load a colormap after an animMap.

[slipher]

So we need a way to make an image brighter. The solution proposed here is to load the image with deliberately wrong colorspace management. That works, but it has the drawback of not being adjustable - it's a checkbox rather than a slider. It would be nice to be able to just multiply the colors by some freely chosen value, as if you doing rgbGen const ( 1.3 1.3 1.3 ), and imagining that our engine didn't clamp colors to [0, 1]. Then you could adjust the value from 1.3 to whatever looks good.

By the way is the design goal of this to make a shader that can load differently depending on the blend regime, or is it just for one-way migrations to linear blending?

[slipher]

Or perhaps a gamma curve type adjustment like sRGB conversions would indeed be better than simple multiplication, but it would be nice to have an adjustable exponent. Let the user do pow(color, 1 / gamma) for any gamma, not restricting to only 1 or 2.4.