Terrain feathering

[CassidyPrather]

Summary

Content "Improves the blending of forests into adjacent biomes."

Purpose of change

Addresses some key points of #47563.

The current "forest" (i.e. natural biome) generation code attempts to feather forests, swamps, and etcetera into adjacent terrain types, but the algorithm for doing so needs to be improved.

First, adjacent and separate biomes would not feather their groundcover into each other- only their furniture. This resulted in a harsh, unnatural uniform distribution of foliage separating plains and forests/swamps. Even when accounting for terrain feathering, the border between biomes was always an exact straight line with insufficient random noise to obscure that fundamental nature. On a similar note, translating along the border of a biome spanning several overmap tiles had seams by which the underlying grid could be visually identified, particularly at the corners between plains and forests/swamps. The feathering between biomes (e.g. forest and forest_water, or "swamps") was not considered separately from feathering between biomes and biome-less terrain; notably, the critical assumption for where the border between the terrain lies varies based on whether the involved terrains intend to implement feathering on their maps or not.

There were also several bugs in the original biome generation code, from inconsistent reference to the construct of "biome scarcity" to inconsistent handling of northern directions. Furthermore, while the original code recommended biomes for forest-esque terrain, such as woodland roads and trails, be defined, this was never implemented and there was no JSON infrastructure for indicating these relationships beyond that supplied by the process of synthesizing a brand new biome for each overmap terrain id.

Describe the solution

• Blends terrain groundcover between adjacent biomes, instead of just furniture.
• Considers cardinal corner terrains during feathering to eliminate outcroppings along long, straight biomes bordering other biomes (or biome-less terrain). See figure 4 in additional context for details.
• Replaces the flat margin between biomes with a randomly generated curve. See figure 5 in additional context for details.
  • The curve is continuous across adjacent terrains.
  • The curve is parameterized with constants which may be json-ized.
  • Adjusts the feathering weights as needed when feathering into biome and biome-less terrains. See figure 6 in additional context for details.
• Fixes old JSON oversights regarding the attribution of biomes to forest trails.
  • Added a new optional json field defining the terrain types associated with biomes.

Describe alternatives you've considered

Instead of using constants to define the properties of the margin curve, it would be preferable to json-ize it at the regional or biome setting levels. However, this would require a change in definition to a ubiquitous, standardized json schema, and I would not want to incur such a dramatic change without first having the core algorithm reviewed and appropriately criticized; certainly not in my first pull request.

As a side effect of fixing the original algorithm to incorporate groundcover, blending will now be visually obvious when adjacent to man-made overmap terrains (e.g. roads, houses, etcetera). Previously, the flat and abrupt transition from the clear-cut area along the edge of the road was realistic and possibly desirable, even if this was only the result of an oversight in the original algorithm attempting to feather into these terrains, but neglecting groundcover. For example, consider this top-down satellite photo of a highway through a forest; there is no "feathering" between the highway and the forest, it is a sharp, straight edge:

Figure 1: Satellite image of a highway through a forest. From Google Maps.

However, as was written as a comment in the original code, "As with the sparseness adjacency factor, it's possible to define non-forest biomes in the regional settings so that they provide neighbor features". In turn, I believe that the best solution to this would be for tiles to define their biomes to create a sharp edge between itself and adjacent forests/swamps, rather than having this behavior be the default. An alternative would be to define a biome for every natural terrain, e.g. plains, and to modify this code to not feather into adjacent biome-less terrains. This does not seem to be desirable, as it would require manual intervention in every common case (i.e. terrains for which no biome is defined) rather than intervention in the exceptions.

Instead of generating a curve along the border between biomes to use for during feathering, it may instead be desirable to use a flat margin as was originally present behind the scenes (but not working for groundcover). The addition of the curve was primarily an aesthetic choice to provide noise and prevent the visual of the natural woodlines from being uniform. However, it may be that attempting to curve the woodline while working within the constraints of a square-tile based overmap draws more attention to the underlying phenomenon and looks worse overall. As this is largely an aesthetic consideration, feedback would be appreciated. The curve can be made to be a straight line by setting the depth_deviation to 0. If these constants are json-ized, the properties of the curve can be customized on a biome-by-biome level, or in JSON mods.

The concept of "forced sparsity" is an artifact from the previous forest generation algorithm which was intended to make forests surrounded by other forests more dense than those on their own, layered on top of the core feathering system. It is maintained in this version, although reworked to be somewhat more uniform and to also apply to groundcover. Overall, the idea behind forced sparsity may be appropriate to re-visit in future work, and perhaps removed altogether in favor of another mechanism at the overmap generation level for making forests thicker the further they are from open terrain.

It would be fine to use any other algorithm for the curves along biome boundaries, and I would be open to suggestions. I chose this one with the hope that it was legible and conformed to the "weighted list" structure so frequently used by the previous algorithm, but it also has some conformance to the underlying map grid (which of course a natural forest would not).

The existing infrastructure for defining biomes for terrain types required a lot of repeated json. For example, to define the same terrain type for forest_thick and forest_trail, the entire biome entry for both would have to be repeated both times. By adding a new optional field defining all of the overmap terrain types associated with a biome, the need for this repitition can be removed, and changes to the biome can be applied to all terrains that adopt it. An alternative would be to implement json inheritance for biomes; however, this would still result in the allocation of a large number of identical biomes in memory- the point of this change is applying biomes to multiple terrain types, not improving the developer tools available for biome derivation. Another alternative would be to define the biomes associated with the overmap terrain in the terrain's definition, rather than vice-versa. While I would prefer this approach, this seems like it would be a more disruptive change, and would not be good for a first pull request- certainly not without extensive prior discussion of the architectural considerations. This would also help in the case of rotating terrain, such as forest trail. In that case, rather than enumerating each possible rotation of the terrain in the biome definition, the biome could be listed once in the template for the rotatable terrain.

Testing

Primarily tested using visual inspection, both with and without special debug indicators of critical points in the generation algorithm to verify the curve characteristics and feathering. I tested with a variety of parameters to generate what I felt were nice looking and natural forest biomes and borders, but this is highly subjective and as such feedback and additional experimentation in this regard is especially valuable. Examples of the change working as intended are listed at the beginning of the additional context section, as follows:

Additional context

Testing Results

Label	Before	After
Corner of forest terrain surrounding plains.
Vertical border of forest terrain, zoomed out for detail.

Table 1: Before and After comparison for forest-orientations of the same form.

Figure 2: Visualization of the "curve/perimeter" between biomes. Feature feathering only, does not show groundcover feathering. Legend: Yellow Pavement = Margin curve, Purple Carpet = Contribution from own biome, Yellow Carpet = Contribution from adjacent biome, Red Carpet = Contribution from adjacent biomeless terrain (in this case, plains). Observe the correction in figure 4 in effect along the southern border of the three captured columns in this image.

Figure 3: Visualization of the curve between biomes, with the addition of "forced sparsity" as described in more detail in the additional context section. Does not show groundcover feathering. Legend is the same as figure B, with the addition that Green Carpet = Forced Sparsity. Zoomed out to demonstrate effect over the course of transition from forest into forest_thick.

Further Context

This PR only addresses a subset of #47563, which is a wish-list of several features related to the biome system. In that issue, feathering is referred to as "fuzzing", and in the original code, it was referred to as "blending". The underlying phenomenon is the same, although I adopt the graphical terminology for its association with well-known algorithms.

The exact desired behavior with respect to the sparsity factor remained somewhat opaque following a static analysis of the previous algorithm, where it appeared in a variety of locations serving different purposes. Originally, biomes with higher sparsity factors were both less likely to have adjacent terrains feather into them (having higher self-weights in the calculation of contributions to each coordinate), were more likely to feather into adjacent terrains (ibid.) and were more likely to be overridden entirely by the regional default terrain in every direction except for north (i.e. east, west, and south). To evaluate the intended effect of the sparsity factor, I read the regional settings and found that forest_water (swamp) type terrain had a sparsity factor of 2, forest had a sparsity factor of 3, and forest_thick had a sparsity factor of 4. "Sparseness", as it appears in the "sparseness_adjacency_factor" field, is a misnomer; the value instead corresponds to density, rather than sparseness. It may be worth renaming this field, or at least updating its description in REGION_SETTINGS.md. The consequences of this error in nomenclature are evident in the aftershock exoplanet region, where the more sparse biomes have higher sparsity factors than the dense ones, when the opposite case is intended.

From this, I concluded that the intended effect of the sparsity factor was just for the latter effect- that is, the replacement of biome terrain with regional terrain. I also applied the new weighting algorithm in this regard as well, so that sparsity also feathered along the borders of biomes alongside groundcover and furniture. In the long run, it may be appropriate to consider removing the sparsity/density factor altogether, as the solution of "override the biome terrain with regional terrain" may make less sense as biomes grow more diverse and further removed from the regional default as development continues.

Forest trails previously attempted to drive their biome to forest_thick, even when they were replacing regular forest tiles. This anomaly is fixed in this PR and forest_trails are now registered with the correct biome, but this anomaly may be worth revisiting in future work.

Additional details on many of the key biome and regional settings discussed in this PR and utilized in the code can be found in regional_map_settings.json.

Figure 5: Description of the formation of contiguous forest borders, before and after these changes. The red line indicates the point of transition, and random noise throws elements from either biome to either side. Previously, F_b1 would attempt to blend into F_b2 and vice versa along the north tip of the west-east edge; now, the code checks to verify that both F_a1 and F_a2 constitute the same biome border, and do not attempt to blend into each other at the north tip of the east-west edge. This does not include a visualization of the curve contributed by this PR. This phenomenon is not as clear in screenshots of the original version on account of the aforementioned lack of biome groundcover feathering.

Figure 6: Description of the curve separating biomes. Fairly primitive, and is presently parameterized in terms of peaks and valleys. The end-points of the curve line up with those of adjacent overmap tiles by using a hash of the overmap tile's coordinates.

Figure 7: Description of the borders between two biomes compared to biome and "biome to biomeless" (e.g. plains) transitions. The red line indicates the point where the involved biomes are contributing equally, and random noise throws elements from either biome to either side.

Some things to consider regarding the specific code modifications:

• Although the function has "forest" in its title, this method is used by all natural biomes, including swamps, and possibly those in mods. It is a bit of a misnomer, but this anomaly is maintained for the purpose of this PR.
• Many other map generation functions call this to set the base terrain, and then modify it (e.g. forest trails), so these changes should apply everywhere needed.
• dat.dir() defines a mapping between integers and cardinal directions which is used for the purpose of indexing and reference in this function.
• I prefer to use integers wherever possible due to the style guidelines; however, many values, especially those related to weighting, would produce results with better granularity as floats, which may be a good consideration for future work.
• The current algorithm, like the previous one, does not feather into corners (ne, nw, se, and sw), although it considers them for the deduction of contiguous biome borders. However, it may be desirable to implement a solution which also feathers into corner biomes. The previous algorithm did not consider corners.
• regional_map_settings.json does not adhere to the JSON style guidelines; I only formatted my contribution, and left the existing entries as-were.

This is my first pull request, so thank you for your time and patience. Feedback is extraordinarily appreciated.

[actual-nh]

Ping: @ralreegorganon?

[ralreegorganon]

Cool! I'm the guilty party for the implementation there, which JSONized the whole thing in #25425 from it's previously completely hardcoded version. A couple general comments, based on reflecting on what has happened since then:

There's a lot of weird stuff you've maintained here that I also maintained from the previous version (e.g. the whole sparseness thing) that I'd suggest you just do whatever you think appropriate (e.g. removing it entirely). My reflection on the years since I did this is that honestly no one is going to notice, and making something easier and clearer to maintain is going to be a net positive. There are really no sacred cows when it comes to the procedural mapgen here, so rather than trying to just nibble at the edges of the problem, I'd encourage you to think about how you'd like it to actually be, and do that.

Large swathes of "functionality" in this whole forest mapgen have honestly never been used, and other parts of it have been superseded by better solutions. For example, the entire concept of having the big dictionary of terrain and furniture with weights in here so that you could easily modify it via JSON both in the base game and mods is mostly a moot point after I implemented regional terrain and furniture in #35627, since that has a similar result but is game-wide. I wouldn't be afraid of losing those features if you do a different wholesale implementation.

Another thing to note is that in the time since I did this, our mapgen "tools" available via JSON have gotten much better, to the point where, aside from this feathering issue, I'd probably just nuke the entire C++ driven forest/forest_thick/forest_water from orbit and replace it with standard JSON mapgen that leverages nested mapgen, map extras, and regional terrain/furniture. The upside of doing that is the content then is much more easily modifiable via mods and these handful of locations cease to be the special cases that they currently are. There are some serious flaws with the current implementation (my implementation) in the sense of the randomness of the core algorithm (pick a weighted feature and plop it down for each tile) versus something more truly organic (e.g. clustering), and again I'd personally just go back to the drawing board on it--it was better than what was there before, but can certainly be improved.

Leading from that, you might consider if you can extract out the useful features you've added here in terms of providing mechanisms for feathering features in mapgen and make them broadly available to all mapgen via JSON. I don't know what that would look like, but I do know it would be appreciated by the content developers and further us down the path of being more data driven. This comment shouldn't preclude merging what you've got here, but I do think some of this is basically building on shaky foundation rather than just blowing the whole thing down and starting on a new foundation.

Lastly, since it's your first PR:

regional_map_settings.json does not adhere to the JSON style guidelines; I only formatted my contribution, and left the existing entries as-were.

Always style any file you touch using the tooling included in the project, and don't ever waste your time trying to do it by hand. I'm guessing you think it doesn't look like it conforms to the style guidelines (if it really doesn't, it's because a change that breaks the build snuck in), but that's mostly because the document doesn't cover all the rules the tooling implements (it gets a little weird around breaks depending line length). Invalid JSON styling will be a build failure.

[CassidyPrather]

@ralreegorganon Thank you so much for your insights! It's good to hear about a lot of the context regarding how this method came to be, and I'm glad to have your support with respect to some of the strange elements regarding the weighted biome furniture list and so on.

I was very hesitant to do anything sweeping since it was my first PR, so doing things like throwing out weighted lists for furniture/terrain selection on a per-tile basis and getting rid of the "sparseness" construct was a little bit out of scope. But now that I have this very valuable context, I would be glad to begin work on a broader contribution that's applicable to more cases and exposes more of the nuts and bolts of the generation algorithm to the json level. I agree with you, for instance, that it would be cool to have groups and patterns of groundcover and furniture as opposed to a random speckling of items from a very large weighted list; just needs the time and elbow grease, I reckon. If there are no "scared cows", as you put it, I'll be a lot less reluctant for next time! I am definitely not done exploring the map generation, and I'm quite interested in these new tools; but I just wanted to get started with something (hopefully) unobtrusive. It's been a good chance for me to learn about the workflow.

Regarding JSON formatting, I ran it through the "tooling" (read: the nifty online JSON converter instance), and got a huge git diff, but if there was no build failure in the JSON analysis then I suppose that tooling in-of itself was out of date or that I have some fundamental misunderstanding at some point in the process. Thanks for the insight, I'll double check that I'm using the tools correctly.

You've given me plenty to read up on and think about; thanks for the support.

[I-am-Erk]

I strongly agree that we should be leveraging JSON mapgen more here. Rather than listing ingredients in a forest, specify a map ID like "forest_deciduous_thin", and use weighting and random selection if there are multiple maps of that ID, like current mapgen specials work. That would allow mappers to design the forest tiles and their frequency and include unique features that appear randomly. If you want to have dithered borders, I believe there are map extras that can do that by only appearing on omts that border each other

The role of hard coded stuff should be, basically, just to select which map and set of extras to use.

[lgtm-com]

This pull request introduces 13 alerts when merging 1aeeb18 into 611fa07 - view on LGTM.com

new alerts:

• 13 for Use of c-style math functions

[kwl01skz]

My map became a mess because of a retrograde version. I am planning to empty them all and rebuild them, so I am looking forward to this merger.

[CassidyPrather]

I have been stuck on the latest issue with the unit tests. In commit
98fc6bd (above), all tests were passing just fine, and this PR was ready to be merged. However, in the weeks since that commit, a change was made to the function above mapgen_forest in the source code which in turn caused a merge conflict. I was able to resolve this by deleting a doxygen comment in commit f691112 (above).

However, there is now a unit test failing related to calories and food consumption that did not occur in commit 98fc6bd (above). I have been struggling to determine the source of this issue and have not been able to make any progress- I am particularly puzzled since, stepping through the failing unit test, I could not find interaction with any of the three files this PR touches. Furthermore, the only change I made in f691112 to resolve the merge conflict was to delete a comment; no source code was changed to get rid of it.

The comment I deleted was:

/**
* Generates a natural, default, and biome-appropriate forest.
*
* In addition to randomly populating using biome (or region) appropriate
* features, also blends into adjacent tiles using biome or region data.
* 
* @param dat The map to apply the generated forest to.
*/

This was the doxygen documentation of the function mapgen_forest.

The unit test failing is:

(~[slow] ~[.])=> ../tests/comestible_test.cpp:110
(~[slow] ~[.])=> ...............................................................................
(~[slow] ~[.])=> 
(~[slow] ~[.])=> ../tests/comestible_test.cpp:146: FAILED:
(~[slow] ~[.])=>   CHECK( lower_bound <= mystats.calories.avg() )
(~[slow] ~[.])=> with expansion:
Error: (~[slow] ~[.])=>   673.69562f <= 650.2235294118
(~[slow] ~[.])=> 
(~[slow] ~[.])=> 
(~[slow] ~[.])=> 
(~[slow] ~[.])=> RecipeID: choc_pancakes, default is 868 Calories,
(~[slow] ~[.])=> Current recipe range: 529-993, Average 650
(~[slow] ~[.])=> Average recipe Calories must fall within this range, derived from default Calories: 674-1062

I am unable to reproduce this failure on my local machine; the failure only occurs in the automated unit tests provided by GitHub.

Any advice as to how to proceed would be greatly appreciated.

[StStep]

So I'm starting to look at this to see if I can help. As far as the unit-test failing, has it happened once or multiple times? Some tests can randomly fail, especially if they involve averages of randomly generated values. If you can't reproduce it locally it might only rarely fail. It might be worth trying to trigger another check on the PR to see if it fails again.

[StStep]

I ran the unit test recipe_permutations 100 times using your previous commit and didn't get a failure. This was on windows, so it either only happens on linux builds or it's very rare.

[actual-nh]

That particular error has been fixed; you will need a maintainer to actually run the tests again, unfortunately... sigh. EDIT: I have posted this to the dev discord.

[CassidyPrather]

Thank you very much for the assistance, @StStep ! It has only failed twice: GCC 9 and Clang 12 / Ubuntu. I have attempted to trigger the tests again by making a trivial commit. As always, this ran and passed all tests on my local machine:

All tests passed (3846213 assertions in 722 test cases)


18:26:32.736 INFO : Ended test at Mon Sep 13 18:26:32 2021

18:26:32.739 INFO : The test took 191.834 seconds

The test failing was precisely as you described: An average of randomly generated values.

I have been waiting for a maintainer to approve re-running these tests, but because StStep and actual-nh have both provided updates, here is my progress so far in this regard for the sake of completeness.

[CassidyPrather]

Seems that triggering the unit tests to re-run has solved the issue. No changes were necessary to the code, just pushing another set of commits seems to have done the trick. Thanks StStep for the insight as to the nature of the problem and encouraging me to try this. This PR is now back to being ready.

[kevingranade]

Sorry for the exceedingly long delay here, I've had very limited bandwidth and this has been a challenging PR to review.