Make perspective transformations composable and invertable

[yakir12]

Currently it is not possible to invert a perspective transformation:

julia> inv(PerspectiveMap())
ERROR: Inverse transformation for PerspectiveMap has not been defined.

and composition is problematic when combined with a projective transformation (see on slack or on zulip).

The MWE in those links is detailed here as well:

using Images, CoordinateTransformations, Makie, TestImages, StaticArrays
using Makie.AbstractPlotting.MakieLayout

fliprotate(img) = rotr90(img)[:, end:-1:1] # images needed to be rotated and vertically flipped to show correctly on Makie

img = fliprotate(testimage("mandrill"))
point = SVector(175.0, 59.0) # the left eye

push1(x) = push(x, 1) # see Tim Holy's answer for why this is needed: https://stackoverflow.com/questions/45772848/perspective-warp-an-image-in-julia/45782741#45782741

T = Translation(SVector(-100,0,0)) # this is the additional correction I need
M = LinearMap(SMatrix{3,3}([1.3 0.1 0.4; 0.1 1.22 0.33; 0.001 0.0 1]))
tform = PerspectiveMap() ∘ inv(M) ∘ inv(T) ∘ push1
itform = PerspectiveMap() ∘ M ∘ T ∘ push1
indices = ImageTransformations.autorange(img, tform)
imgw = warp(img, itform, indices);
pointw = tform(point)
scene, layout = layoutscene(0)
ax1 = layout[1, 1] = LAxis(scene, aspect = DataAspect(), yreversed = true)
image!(ax1, img)
opts = (; color = :transparent, strokecolor = :white, markersize = 10px, strokewidth = 2)
scatter!(ax1, point; opts...)
ax2 = layout[1, 2] = LAxis(scene, aspect = DataAspect(), yreversed = true)
image!(ax2, UnitRange.(axes(imgw))..., parent(imgw))
scatter!(ax2, pointw; opts...) # note how the marker is a bit off
linkaxes!(ax1, ax2)
scene

[jenkspt]

Currently it is not possible to invert a perspective transformation:

The PerspectiveMap implemented as follows

function (::PerspectiveMap)(v::AbstractVector)
    scale =  1/v[end]
    return [v[i] * scale for i in 1:length(v)-1]
end

This isn't invertable since you need to know v[end] to apply the inverse transform

[yakir12]

Would it be useful to define the PerspectiveMap struct with a scale field? It can default to 1, have zero negative effect on the current implementation (I think?), and it will be useful when inverting it.

To be clear though, I'm not sure if I'm missing some main point here.

[jenkspt]

Are you trying to invert a perspective camera projection?

[yakir12]

Anyways.

I think I meant to write "Always.", I apologize for the mistake, the tone might have sounded off putting, which was not at all my intention. Sorry again.

[SamRodkey]

I think when you take a projective transformation from 3D to 2D, such as for a intrinsic model of a camera or for a "camera calibration" model, you cannot invert the transformation and get a unique point? Typically these camera models have 4-5 coefficients which describe how to perform the linear transformation from 3D to 2D coordinates. The scaling /focal distance in the principal camera directions, shear, center point locations are all encoded in these coefficients.

See Eq. 7 here: https://learnopencv.com/geometry-of-image-formation/

I think if you do the inversion of the projective transform, you'll kind of see that the system is underdetermined. A single camera coordinate is often not enough to determine a unique point in 3D space? You can get a "minimum distance" solution using something like pseudoinverse or some other technique, but I don't think there is a single "unique" solution to the problem, even with a specified projective transformation?

[yakir12]

Thank you @SamRodkey, those are really good documentations! This is a good explanation for what is missing. I'll close this for now.