Current gdalwarp 10-15% (?) slower than 3.10

[pjonsson]

What is the bug?

I ran my testsuite in the ubuntu-full image built on latest master towards the end of last week, possibly commit c4a2e0b, and the runtime for the testsuite has increased from 4m11s with a GDAL 3.10.2-based image to 4m32s. I don't have a setup for measuring performance beyond total runtime, but I ran the test suite several times in both images and the numbers are quite stable, there is a ~20s difference between the two.

Trying to dig into the cause, I looked at things in top and observed that the slowest gdalwarp (NUM_THREADS=1) process ran for 42 CPU seconds with 3.10.2, and 47 seconds with 3.11. Top is imprecise, but the testsuite runs 3 rounds of gdalwarp so if each run is ~7 seconds slower that would neatly sum up to the 21 seconds of observed time difference.

My machine is not starved of memory, but I also noticed the gdalwarp processes have a slightly larger VIRT with my 3.11 build according to top, from 128xxx (~1.2G) with 3.10.2 to 140xxx (~1.3G). I considered whether that might be debug info, but I did not pass any debug flags to build.sh when building the image, so the only difference I'm aware of is using the ubuntu-full instead of ubuntu-small image and the missing --release parameter.

Looking at the release notes for 3.11, I get the impression that gdalwarp should have gotten slightly faster (or at least not slower), so I'm filing this as a bug. Feel free to close this if the slowdown is expected, 10-15% is not make-or-break for me.

While working on this I also noticed that running the commands (only 3.11 for this one, don't have a 3.10 reference point) but changing compression and -of COG to -of Zarr gives a different number of decimals in the second and last component of the tuple when running gdalcompare -dumpdiffs Oa01_reflectance.tif Oa01_reflectance.zarr which surprised me a bit, should I make a separate issue for that?

GeoTransforms Differ:
  Golden: (-31.560815811157227, 0.0058750907369288004, 0.0, 73.37567138671875, 0.0, -0.0058750907369288004)
  New:    (-31.560815811157227, 0.0058750907369288, 0.0, 73.37567138671875, 0.0, -0.0058750907369288)

Steps to reproduce the issue

It's the same image and steps as #10809, but using Float32 for latitude.vrt and longitude.vrt.

Versions and provenance

The ubuntu-full image built on latest master towards the end of last week, possibly commit c4a2e0b.

Additional context

No response

[rouault]

which surprised me a bit, should I make a separate issue for that?

no that part is kind of expected. GeoTIFF encodes the top-left corner origin and the resolution, whereas Zarr encodes X and Y arrays with the coordinate of all X and all Y values. So small differences due to numerical precision issues are expected.

[rouault]

On my machine, "gdalwarp -q -geoloc -r cubic -of COG -co COMPRESS=DEFLATE -co NUM_THREADS=1 --config GDAL_NUM_THREADS 1 ../../gdal/build_cmake/oa02.vrt out.tif -overwrite" runs on about 29.9 seconds on 3.10.3 and 30.4 on master, so a small difference but almost in the measurement noise (less than 2%). Trying to bisect...

[pjonsson]

On my machine, [..] runs on about 29.9 seconds on 3.10.3 and 30.4 on master, so a small difference but almost in the measurement noise (less than 2%). Trying to bisect...

I agree, that is most likely a noise level difference, and quite far from my 10+%. I ran things several times on a mostly idle machine, and even went back to the 3.11 image to verify the numbers, so I believe my measurements are solid.

Is there anything I could have done wrong when building the image that could explain the performance difference?

[pjonsson]

@rouault I have to run now, but I don't want to waste your time since this was difficult to reproduce. Let me build a fresh image from latest master on Monday and try this again, hopefully the performance is restored, and otherwise I'll give you a reproducer that can be cut&pasted.

[rouault]

so the only difference I'm aware of is using the ubuntu-full instead of ubuntu-small image

ubuntu-small doesn't include the netCDF driver , so how can you run gdalwarp on your test image with it ?

Is there anything I could have done wrong when building the image that could explain the performance difference?

nothing comes to mind. I've tried my bisection session, but I'm deeply perplexed because now I find master to be a tiny bit faster than 3.10.0 or 3.10.3. I've disabled TurboBoost to get more predictable timings but that doesn't show anything obvious. I've nothing better to propose to you that you run a git bisect session on your own environment...

I've also just tried the official Docker images, with script.sh running twice "time gdalwarp -q -geoloc -r cubic -of COG -co COMPRESS=DEFLATE -co NUM_THREADS=1 --config GDAL_NUM_THREADS 1 /home/even/gdal/gdal/build_cmake/oa02.vrt out.tif -overwrite", I get (with TurboBoost disabled)

with 3.10.3:
$ docker run --rm -it -v $HOME:$HOME ghcr.io/osgeo/gdal:ubuntu-full-3.10.3 bash $PWD/script.sh
Warning 1: Too many points (528 out of 529) failed to transform, unable to compute output bounds.
Warning 1: Unable to compute source region for output window 0,0,6443,2543, skipping.

real 0m54.647s
user 0m53.317s
sys 0m1.316s
Warning 1: Too many points (528 out of 529) failed to transform, unable to compute output bounds.
Warning 1: Unable to compute source region for output window 0,0,6443,2543, skipping.

real 0m55.189s
user 0m53.774s
sys 0m1.388s

and master:
$ docker run --rm -it -v $HOME:$HOME ghcr.io/osgeo/gdal:ubuntu-full-latest bash $PWD/script.sh
Warning 1: Too many points (528 out of 529) failed to transform, unable to compute output bounds.
Warning 1: Unable to compute source region for output window 0,0,6443,2543, skipping.

real 0m55.525s
user 0m54.319s
sys 0m1.200s
Warning 1: Too many points (528 out of 529) failed to transform, unable to compute output bounds.
Warning 1: Unable to compute source region for output window 0,0,6443,2543, skipping.

real 0m55.908s
user 0m54.611s
sys 0m1.284s

so master here being slightly slower. I'm perplexed by this

[pjonsson]

ubuntu-small doesn't include the netCDF driver , so how can you run gdalwarp on your test image with it ?

I have a local build of the images, and that adds the required netcdf packages to the Dockerfile before building.

Is there anything I could have done wrong when building the image that could explain the performance difference?

nothing comes to mind. I've tried my bisection session, but I'm deeply perplexed because now I find master to be a tiny bit faster than 3.10.0 or 3.10.3. I've disabled TurboBoost to get more predictable timings but that doesn't show anything obvious. I've nothing better to propose to you that you run a git bisect session on your own environment...

I wasn't aware of the development images being available, but that eliminates the issue of me potentially doing something wrong when building the image. I just tested with ghcr.io/osgeo/gdal:ubuntu-full-latest pulled ~30 minutes ago, and the testsuite takes 2m50s 4m15s in that. I ran it twice just to be sure and the time is consistent (the runtime varies +-3s, but I consider that noise on a 3 4 minute total). The version inside the image presents itself as:

$ gdal-config --version
3.11.0
$ gdalinfo --version
GDAL 3.11.0beta1 "Eganville", released 2025/04/17

The 2m50s is quite a lot better than the 4m30s I mentioned earlier, so I ran the testsuite with 3.10.2 just to see if my computer was having a heatstroke during my previous benchmarks. The time for that is 4m13s, which I would say is spot on with what I initially wrote in this issue. There must have been something bad with my locally built 3.11 ubuntu-full image, but I have no explanation of how I could mess that up since I cut&pasted the command that builds the 3.10.2 image and removed the release parameter. I'm also puzzled with how the image could be bad because I gdalcompare'd the images with the 3.10.2 result and for the ones that did have some valid pixels, they were all identical.

A 2x performance improvement on top of the improvement with latest master is awesome, thank you!

With the current master that doesn't contain the optimization PR you just made, the mentioned gdalwarp command is now faster than the gdal_translate commands that I use for making COGs of the JPEG Sentinel-2 S2MSI2A images. Does gdal_translate do the same kind of double initialization that you just fixed in gdalwarp so there's a 2x improvement available for gdal_translate as well?

Edit: I'm running with GDAL_CACHEMAX=256 and I arrived at that number when still using GDAL 3.4.3, so I tried bumping CACHEMAX to 512 just in case the cache requirements had changed, but that just increased the memory consumption, it did not make a difference for the runtime.

Edit 2: I measured Zarr runtimes for 3.11 in the first revision of this comment. My 3.11 image was still running, so I have re-run the tests with COGs this time, and the times are comparable to 3.10.2. Bumping CACHEMAX does still not make a difference in performance. I still have no idea how I got the poor performance with my locally built image initially.

Is there some way I can get memory stats from gdalwarp? I would love if I could set a large GDAL_CACHEMAX and tell gdalwarp to keep a high watermark for the different phases and print the GDAL_CACHEMAX I should have used at the end of the run.

[pjonsson]

I know the Github mail notifications don't behave well with edits, so writing a new comment for any watchers. The initial version of my previous comment was wrong because I was measuring Zarr runtimes with 3.11. I have updated the comment with measurements of COGs and added an "Edit 2" at the end.

[rouault]

Does gdal_translate do the same kind of double initialization that you just fixed in gdalwarp so there's a 2x improvement available for gdal_translate as well?

no, that was specific to gdalwarp with COG output. The 2x is not necessarily always a 2x. Here it is a 2x because the time to compute the inverse geolocation matrix is almost all the computing time. The warping itself is almost instantaneous once that's done.