Implementing streamed tiles system for big georeferenced images #205

Fletterio · 2025-07-24T02:17:56Z

Run on this branch:

https://github.com/Devsh-Graphics-Programming/Nabla/tree/geotex_streaming

…d to VkBufferImageCopy used to upload tiles)

Erfan-Ahmadi · 2025-07-30T07:38:31Z

62_CAD/main.cpp

+			tiledGridParams.imageID = 6996;
+			tiledGridParams.geoReferencedImage = bigTiledGrid;
+
+			DrawResourcesFiller::StreamedImageManager tiledGridManager(std::move(tiledGridParams));


Seems reasonable to have a manager that knows everything about the image placement and info, useful for tile calculation.
Just keep in mind that we want to separate the idea of a "loader" and "tileCaclulator".
One actually does loads from disks or wherever, the other calculates which tiles and which mip levels are needed.
One has access to image data, One just knows the resolution and basic information
One doesn't have info about image placement in the worldspace and the view projection, one does.
One is loader, one is tile calculator or SteamedImageManager or whatever you want to call it

Erfan-Ahmadi · 2025-07-30T09:48:24Z

62_CAD/DrawResourcesFiller.cpp

 				cachedImageRecord->state = ImageState::CREATED_AND_MEMORY_BOUND;
 				cachedImageRecord->lastUsedFrameIndex = currentFrameIndex; // there was an eviction + auto-submit, we need to update AGAIN
 				cachedImageRecord->allocationOffset = allocResults.allocationOffset;
 				cachedImageRecord->allocationSize = allocResults.allocationSize;
 				cachedImageRecord->gpuImageView = allocResults.gpuImageView;
-				cachedImageRecord->staticCPUImage = nullptr;
+				cachedImageRecord->staticCPUImage = manager.georeferencedImageParams.geoReferencedImage;


this should stay nullptr, georeferenced images should have nothing to do with staticCPUImage. the cacheImageRecord only holds the status of the gpu allocated georef image.
then you just copy your regions with buffers into it.

i.e. the cpu image should only be part of the loader and not the draw resources filler OR the tile calculator

Erfan-Ahmadi · 2025-07-30T09:55:06Z

62_CAD/DrawResourcesFiller.cpp

@@ -879,11 +878,21 @@ void DrawResourcesFiller::addGeoreferencedImage(image_id imageID, const Georefer
 		return;
 	}

+	// Generate upload data
+	auto uploadData = manager.generateTileUploadData(NDCToWorld);


Correct/Good decision to calculate the tiling and which sections to load here in addGeoreferencedImage

Erfan-Ahmadi · 2025-07-30T10:06:50Z

62_CAD/DrawResourcesFiller.cpp

@@ -631,20 +631,19 @@ bool DrawResourcesFiller::ensureMultipleStaticImagesAvailability(std::span<Stati
 	return true;
 }

-bool DrawResourcesFiller::ensureGeoreferencedImageAvailability_AllocateIfNeeded(image_id imageID, const GeoreferencedImageParams& params, SIntendedSubmitInfo& intendedNextSubmit)
+bool DrawResourcesFiller::ensureGeoreferencedImageAvailability_AllocateIfNeeded(StreamedImageManager& manager, SIntendedSubmitInfo& intendedNextSubmit)


Ok so I have a few requests about this.
This function should only care about the GPU allocation and the image itself and decide which resolution to create it with. (i.e. decide to go full res or streamed with tiles)

And you can know these with only 1. OriginalImageResolution 2. ViewportResolution 3. TileSize and nothing more

I don't see any need for the manager to go through here.
If this function is doing more than just figuring out what resolution to allocate the image in the vram with, then it's a sign something is wrong.

then when you're in addGeorefImage function, you can look up the cached image and see if it was decided to be "Stremead" or "FullRes" depending on that, you either request loading the whole shit from the loader OR you request tiles (through your tile manager or whatever)

Erfan-Ahmadi · 2025-07-30T10:08:46Z

62_CAD/DrawResourcesFiller.cpp

 	{
 		outImageParams.extent = { georeferencedImageParams.imageExtents.x, georeferencedImageParams.imageExtents.y, 1u };
 	}
 	else
 	{
-		// TODO: Better Logic, area around the view, etc...
-		outImageParams.extent = { georeferencedImageParams.viewportExtents.x, georeferencedImageParams.viewportExtents.y, 1u };
+		// Pad sides to multiple of tileSize. Even after rounding up, we might still need to add an extra tile to cover both sides.


continuing from https://github.com/Devsh-Graphics-Programming/Nabla-Examples-and-Tests/pull/205/files#r2242153374
I believe these calculations is better if done inside addGeorefImage function

Erfan-Ahmadi · 2025-08-18T04:42:50Z

62_CAD/shaders/main_pipeline/vertex_shader.hlsl


-            float2 corner = float2(bool2(vertexIdx & 0x1u, vertexIdx >> 1));
-            float2 uv = corner; // non-dilated
+            const bool2 corner = bool2(vertexIdx & 0x1u, vertexIdx & 0x2u);


vertexIdx >> 1 makes more sense than vertexIdx & 0x2u and cast to bool imo

Erfan-Ahmadi · 2025-08-18T07:36:20Z

62_CAD/DrawResourcesFiller.h

@@ -120,6 +122,117 @@ struct DrawResourcesFiller
 				geometryInfo.getAlignedStorageSize();
 		}
 	};
+
+	// Used to load pieces of an ECW from disk - currently just emulated
+	struct ImageLoader


ImageLoader is User space (n4ce) code, and for examples, somewhere in main.cpp
So here you have multiple options, here are 2:

The whole thing is a single load function, just register a lambda, same as setGlyphMSDFTextureFunction and setHatchFillMSDFTextureFunction

I have a hunch the ImageLoader might have more than a load function, it may have functions that give info about the georeferenced image that are embedded in ECW/GeoTiff:

Make the Abstract IGeoreferencedImageLoader here with pure virtual functions, then registering a class to DrawResourcesFiller

Erfan-Ahmadi · 2025-08-18T07:45:40Z

62_CAD/DrawResourcesFiller.h

+		std::vector<std::vector<bool>> currentMappedRegionOccupancy = {};
+		// Converts a point (z = 1) in worldspace to UV coordinates in image space (origin shifted to topleft of the image)
+		float64_t2x3 world2UV = {};
+		ImageLoader loader;


The loader should be part of DrawResourcesFiller, here is why:

n4ce will have only 1 global georeferenced loader that handles georeferenced images. we don't want to specialize loader per image

In Cache&Replay mode, we won't go through the usual ensureGeoreferencedImageAvailability_AllocateIfNeeded --> addGeoreferencedImage. and actually no drawresources that add objects will be called, it will go into replay mode where everything is already cached. BUT for ECWs it's a little more complicated the tiles need constant check for updates since they depend on where the camera/viewport is; (other kinds of objects such as lines, curves, static images, are just there in the cache and they don't depend on the viewport)
If you're curious about the replaying, see createReplayCache and pushAllUploads.
For ECWs (unlike line work) we need to check all the georeferenced images, potentially load new tiles and update their data, we will do that with the already registered loader.

Erfan-Ahmadi · 2025-08-18T08:15:11Z

62_CAD/DrawResourcesFiller.h

+	};
+
+	// @brief Used to load tiles into VRAM, keep track of loaded tiles, determine how they get sampled etc.
+	struct StreamedImageManager


Looking at this, it seems like StreamedImageManager is partly functionality to compute tiles and partly a cache for the georeferenced image since it's used to store things like currentMappedRegionOccupancy

This all makes sense, BUT :

it shouldn't be user's job to keep this thing alive and give you the correct streamedImageManager.

what if we internally evicted the georeferenced image due to vram pressure, but we get old tile residency info and cache from the user

It makes more sense if each imageId is matched with a StreamedImageManager internally in DrawResources filler, and user doesn't pass this to our functions
How might we do that?

Well we already have a CachedImageRecord for our static and georeferenced images. (see CachedImageRecord* cachedImageRecord = imagesCache->insert in ensureGeoreferencedImageAvailability_AllocateIfNeeded)

What you need to do is just store it there as a new member variable, it will be invalid for static images.

also it doesn't "manage" the image per-se, it computes tiles to load and stores residency info.

here is what I suggest, feel free to play around with the idea:

rename to GeoreferencedImageStreamingState and only store context/cache requred

Move the functions that load and compute tiles to DrawResourcesFiller as protected member function. (all the functions will take the State/Context struct for computation)

This make more sense to me because this will become just a struct that holds important info we need to cache for streamed images, and drawResourcesFiller is the class which actually "manages/caches/evicts" georeferenced images.

Erfan-Ahmadi · 2025-08-18T08:19:44Z

62_CAD/DrawResourcesFiller.h

+		constexpr static float64_t3 bottomRightViewportNDCH = float64_t3(1.0, 1.0, 1.0);
+
+		// Measured in tile coordinates in the image, and the mip level the tiles correspond to
+		struct TileLatticeAlignedObb


nice job using the word Lattice but don't use fancy names when we can use plain language. (imagine 4-5 years later somebody trying to read the code and wants to understant it as soon as possble to make edits)
It's just a TileAlignedOBB, it's not even an OBB in image space
you could just name this TexelRange or something.

Simple is always better

Erfan-Ahmadi · 2025-08-18T08:21:53Z

62_CAD/DrawResourcesFiller.h

@@ -351,7 +464,7 @@ struct DrawResourcesFiller
 	void addImageObject(image_id imageID, const OrientedBoundingBox2D& obb, SIntendedSubmitInfo& intendedNextSubmit);

 	// This function must be called immediately after `addStaticImage` for the same imageID.
-	void addGeoreferencedImage(image_id imageID, const GeoreferencedImageParams& params, SIntendedSubmitInfo& intendedNextSubmit);
+	void addGeoreferencedImage(StreamedImageManager& manager, const float64_t3x3& NDCToWorld, uint32_t2 viewportExtents, SIntendedSubmitInfo& intendedNextSubmit);


If/When you apply:; https://github.com/Devsh-Graphics-Programming/Nabla-Examples-and-Tests/pull/205/files#r2281633149

No need to take StreamedImageManager, it will be cached and created (if not cached already) internally.
You can just take imageId and the other params here

Erfan-Ahmadi · 2025-08-18T08:23:29Z

62_CAD/DrawResourcesFiller.cpp

+				continue;
+
+			auto tile = loader.load(uint32_t2(tileX * TileSize, tileY * TileSize), uint32_t2(TileSize, TileSize));


seems like you only need to use the buffer result of the load, just make the loader return an ICPUBuffer instead of ICPUImage and use that insread of all the null resource cpu buffer creation stuff.

Erfan-Ahmadi · 2025-08-21T06:29:08Z

62_CAD/main.cpp

+	system::ILogger* m_logger = {};
+	video::IPhysicalDevice* m_physicalDevice = {};
+	// We're going to fake it in the example so it's easier to work with, but the interface remains
+	core::vector<smart_refctd_ptr<ICPUImage>> mipLevels = {};


Unurgent + Unimportant (but I have to ask):
an ICPUImage can have mip levels inside it.
and why load images separately? I think .exr can hold multiple mip levels by itself and our loader can handle that.

Erfan-Ahmadi · 2025-08-21T06:29:44Z

62_CAD/main.cpp

@@ -494,10 +664,11 @@ class ComputerAidedDesign final : public nbl::examples::SimpleWindowedApplicatio

 		// Static Image Sampler
 		{
+			constexpr auto wrapMode = mode == ExampleMode::CASE_12 ? IGPUSampler::E_TEXTURE_CLAMP::ETC_REPEAT : IGPUSampler::E_TEXTURE_CLAMP::ETC_MIRROR;


no need for this if it didn't mess up other examples

Erfan-Ahmadi · 2025-08-21T09:34:12Z

62_CAD/DrawResourcesFiller.cpp

+				cachedImageRecord->georeferencedImageState = GeoreferencedImageStreamingState::create(std::move(params));
+
+				// This is because gpu image is square
+				cachedImageRecord->georeferencedImageState->maxResidentTiles = cachedImageRecord->gpuImageView->getCreationParameters().image->getCreationParameters().extent.width / GeoreferencedImageTileSize;


This is assuming width = height (due to rotation)?

Fletterio added 6 commits July 23, 2025 20:38

Checkpoint 1 is close

6a74ab1

Off by one error fix

e523868

Fix tile offsets for upload

a54f6c6

Skeleton done but currently bugged, some byte offset is wrong (relate…

c3f7d04

…d to VkBufferImageCopy used to upload tiles)

Fix square bytes computation

7a5e948

Checkpoint 1!

52d947d

Erfan-Ahmadi reviewed Jul 30, 2025

View reviewed changes

Fletterio added 7 commits July 31, 2025 10:34

Save before merge

665559b

Merge + incorporate obb shrinking at the edges

a24281a

Bug: using uploaded uvs seems to stretch/not shrink along v direction

fc2d504

Fixed y-axis bug

e61e389

Diagonal computation

258920c

Some names are wrong here, but the example still works

6a1b76e

Tile tracking done

0ed564e

Erfan-Ahmadi reviewed Aug 18, 2025

View reviewed changes

Fletterio added 2 commits August 19, 2025 00:34

Cleaning up the code following PR review

f638ca6

Checkpoint for Phase 2

89af347

Erfan-Ahmadi reviewed Aug 21, 2025

View reviewed changes

		continue;

		auto tile = loader.load(uint32_t2(tileX * TileSize, tileY * TileSize), uint32_t2(TileSize, TileSize));

Implementing streamed tiles system for big georeferenced images #205

Are you sure you want to change the base?

Implementing streamed tiles system for big georeferenced images #205

Uh oh!

Conversation

Fletterio commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Erfan-Ahmadi Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Erfan-Ahmadi Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Erfan-Ahmadi Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Fletterio commented Jul 24, 2025 •

edited

Loading

Erfan-Ahmadi Jul 30, 2025 •

edited

Loading

Erfan-Ahmadi Aug 18, 2025 •

edited

Loading

Erfan-Ahmadi Aug 18, 2025 •

edited

Loading