Follow-ups for draping imagery on 3D Tiles

[javagl]

The functionality for draping imagery on 3D Tiles had been requested several times. A first version of this has been implemented in a pull request that has just been merged, meaning that the functionality will be part of the June 2025 release.

While the hope is that the current state already supports some of the most important use-cases, there are also some known limitations for that draping in the current state.

1. (Update: Extracted into Distorted textures for draped imagery near antimeridian #12769 ) : There are issues with the computation of the texture coordinates when the tilesets cross the antimeridian.
   • This is particularly noticable for global tilesets, causing the textures to look distorted in this area. The exact location of the issue (in the code... not on the globe) and possible solutions have to be identified and tested, but the hope is that this will only be a minor change that can happen soon in a follow-up PR
   • Similarly, there are limits for the applicability near the north and south pole, for imagery using WebMercator projections. There is no silver bullet for solving that.
2. (Update: Fixed via Update draped imagery visibility #12743 ) : The draping visualization is not updated when toggling imageryLayer.show between true/false. This should be an easy fix. As a workaround for now, setting alpha=1.0 and alpha=0.0 can achieve the desired effect.
3. Upsampling is not implemented.
   • This means that objects that cover a large geographic area can only be covered with relatively low-resolution imagery. It also means that when using multiple imageries, it can happen that the number of available texture units is exceeded, and some of the textures are omitted
   • Some additional details and considerations can be found in the discussion of the pull request. A quick summary: When there is geometry that covers a large geographical area, and high-resolution imagery should be draped on that, then many textures have to be mapped to the geometry. Due to limitations of the graphics card, the number of textures that can be used "at once" is limited (to ~16 textures). This means that in order to drape these many textures, the geometry has to be split into smaller parts, so that each part only needs a few (less than 16) textures. There currently is no infrastructure for "splitting geometry" in CesiumJS, because once the geometry data is loaded, it is usually directly uploaded to the GPU.
4. Draping does not work for tilesets where the content uses the CESIUM_primitive_outline extension. The reason for that is, roughly speaking, that this extension causes the geometry of the glTF data to be modified to take into account the outline rendering information (modifying indices, duplicating vertices), and it's not clear how this can be aligned with the computation of the texture coordinates for the draping
5. The concept of a model being "ready" has to be reviewed. When a tileset with draped imagery is loaded, then the question is roughly: Should the tileset be displayed with its original texture, which later (when the required imagery is loaded) is replaced with the imagery texture, or should the tileset only be displayed when both the geometry and the imagery are loded? Both approaches can cause ~"undesirable effects" for certain usage scenarios. Right now, there is the asynchronouslyLoadImagery=false option for the tileset constructor to offer both options, but some details have to be reviewed. For example, what this should mean when an imagery layer is added to a tileset that already is visible.
   • Update: One specific aspect of that is Allow passing tileset options to reality data tilesets #12709
6. Credit handling. Right now, the credits for the imagery are not displayed when the imagery is only draped on a tileset. The exact path for passing on these credits has to be determined. It might be possible to simply pass the credits from the imagery to the model.
7. Memory statistics: The model carries a bunch of statistics for the memory consumption, including that of textures. It is not clear where and how imagery textures should be accounted for. (And when the same imagery is also draped over terrain, then tracking that in the model as well might be misleading...)
8. The test coverage has to be improved. - A first set of ("integration-level") test cases has been created, and a first set of specs/unit tests has been added. But there have to be tests for more corner cases. Of course, this should include a dedicated test at the antimeridian, but also tilesets with a mix of very large and very small objects, and other tests that aim at identifying limitations or drawbacks of the current approach. (A few profiling runs could also be worthwhile, but until now, the draping did not seem seem to have a significant performance impact)Further points will be added here, also based on the feedback from users who try out this new feature.

[jo-chemla]

Really nice to see this draping feature ported from globe/terrain to 3d-tiles tilesets, congrats! Very nice as well to support from the start all ImageryLayers cesiumjs already supports (TMS, WxS, ESRI etc).

Just had a quick look at the tiled pointcloud and meshes examples, and noticed there is a small offset especially visible on the cesium hq - either from the 3d dataset or the draped bing imagery - and thought it might be a welcome (and maybe not so hard) addition to allow for simple texture transformation, probably in scaled uv space. Letting the user control an offset of say X meters East and Y meters North relative to the imagery, that could be added as a delta to the computed textureTranslationAndScale from ImageryInput/Configuration could be the way to go.

From experience, I've encountered a non negligible number of cases where small misalignments within imagery or 3d datasets can be corrected with such an offset relative to one dataset chosen as the ground truth - usually editing scale is not required.

[javagl]

There indeed is a small offset. And when looking at the original 3D Tiles Photogrammetry Sandcastle, one can see that the position of the tileset is slightly off: The roads (at the edges) do not align perfectly with the underlying terrain.

One could now argue about what the "truth" is. The texture coordinates for the draped imagery are computed from the actual cartographic positions of the tileset vertices. So these are "correct" in a narrow, technical sense. And one could ""fix"" this with a small transform of the tileset itself, inserting something like

const initialRootTransform = Cesium.Matrix4.clone(tileset.root.transform);
tileset.root.transform = Cesium.Matrix4.clone(Cesium.Matrix4.IDENTITY);
tileset.modelMatrix = Cesium.Matrix4.clone(Cesium.Matrix4.IDENTITY);
const m = Cesium.Matrix4.clone(Cesium.Matrix4.IDENTITY);
Cesium.Matrix4.multiply(m, initialRootTransform, m);
const translation = Cesium.Matrix4.fromTranslation(
  new Cesium.Cartesian3(2, -8, 0), new Cesium.Matrix4());
Cesium.Matrix4.multiply(m, translation, m);
tileset.modelMatrix = m;

after creating the tileset (not perfect, but close - just quickly made up).

The other option would be to add some "texture translation" to the imagery, as you suggested.

This might raise a few questions: When the imagery that is draped over the tileset has that translation, but the imagery of the underlying terrain does not have this translation, then these two will be misaligned.

But in general (and be it only for the case that different imageries are draped on the terrain and the tileset), I think that offering such an offeset is something that we should definitely consider.

(It might be relatively easy to add this. You already identified one place for that: The ImageryCoverage is computed from the tileset geometry. And from that, the ImageryInput objects (which are passed to the shader) are computed. Adding such a translation e.g. here should be easy. But there are a few degrees of freedom about where this is coming from. One could just throw in some tileset.imageryTranslation = new Cartesian2(x,y), but we'll have to check what makes most sense here. Eventually, someone might want different translations for different imagery layers and such ...)

[javagl]

When the draping implementation is trying to find imagery to drape on a given geometry, then it tries to find a "matching" one, which means that it will try to determine an imagery level so that (roughly) one imagery tile is mapped on the given geometry. Until now, much of the discussion has been about upsampling, where higher-level imagery should be draped on the geometry. But the current approach also does not do downsampling, so to speak: Many imagery providers don't provide sufficient information about the levels that they actually provide, meaning that, for example, trying to obtain an imagery tile of level 15 will result in an error message, but the parent at level 14 might be available, and could safely be used instead.

Some of that has been discussed in the context of ~"error handling". But given that many imagery providers seem to provide only level 12 or less, and this results in "(apparently) missing imagery", I'd be leaning towards calling this a "bug" in the current implementation. Going up the hierarchy until a valid imagery is found should be relatively easy, and I'll do some tests to see how that could be fixed quickly.

[jo-chemla]

Thanks for the suggestion above regarding local shift of imagery vs tileset. Indeed, depending on what is considered the truth, offsetting the tileset itself is already a good idea.

Regarding downsampling, some imagery providers return 404 when a missing is requested, while others return a blank image like esri - so it might be harder to automatically tell if the tile exists before going up one level in the hierarchy. Wouldn't you want to simply rely on the user-defined CesiumJS ImageryProvider maximum-zoom like TileMapServiceImageryProvider maximumLevel or WebMapTileServiceImageryProvider maximumLevel? Those are usually known and available via services like NextGIS QMS for eg Google, ESRI.

[javagl]

Wouldn't you want to simply rely on the user-defined CesiumJS ImageryProvider maximum-zoom like [...] maximumLevel or [...] maximumLevel? Those are usually known and available

Admittedly: One reason for why I added this comment was that I used an imagery provider where this maximum level was not configured correctly, and this caused really odd behavior. But I didn't delete the comment even after I figured that out, because the broader point still stands: When a certain imagery texture cannot be loaded for whatever reason, then this should be handled in a sensible way. Some approaches for dealing with errors could very roughly be:

• Retry n times...
• Print an error message to the console
• Use a dummy 'error indicator' texture
• Use a parent imagery texture (iff it is available...)

The behavior should preferably be configurable (or at least, be prepared for different options).

(Somehow related: #12442 . I think that there may be imagery providers where certain imagery tiles are ("randomly") not available)

[jo-chemla]

Indeed, the described behavior and rough approaches could be helpful for downsampling imagery when the higher resolution is not available for any reason.

On a side-note that might be out-of-scope of the follow-ups for draping imagery over 3d-tiles: This draping imagery could at some point be extended to arbitrary ImagerySources (not only ellipsoidal projection), for example to perspective projection pinhole camera models. See this interactive example of the Sacré-Coeur public demo where we used a custom shader equivalent of lightMaps to project archive photos onto a potree pointcloud - basically compute uv screenspace coordinates of each point from the projected camera, to read texture from these coordinates and falllback to vertex or uniform color.

Note this effect is applied in the vertex shader rather than fragment shader, so each point of the pointcloud can have only a single color - even if they are large disks of say 10px, while doing this to meshes or in the fragment shader could improve the effect and result in getting the full photo resolution even on coarse pointclouds.

Slight offset	Matching viewpoint

[javagl]

OK, that 'Projective texturing' is pretty cool. While there certainly is an overlap on an abstract, technical level (~"compute texture coordinates for points", "color points by texture"), the structures in CesiumJS and the draping are pretty much focussed on the existing *Imagery* classes, which are inherently the 2D imagery tiles. But even though this is several steps beyond that is achievable right now with reasonable effort, it's certainly worth mentioning here. Having such use cases at least on the radar can be helpful.