[bug]: Transform-upscale produces clipped edges when resizing layers with smoothing enabled

[DustyShoe]

Is there an existing issue for this problem?

• I have searched the existing issues

Install method

Invoke's Launcher

Operating system

Windows

Version number

v6.11.1.post1

---

What happened

When resizing a transparent layer whose visible content nearly touches its bounding box, the resulting image appears to have its edge clipped at the most protruding areas.

This occurs when transform smoothing is enabled (e.g. bicubic or lanczos). It does not occur when using nearest neighbor.

---

Observed Behavior

• The edge looks slightly flattened or clipped at the most protruding parts.
• If the layer’s bounding box is manually enlarged (e.g. by adding tiny pixels in the corners), the issue disappears.
• The issue does not occur if upscaling opaque rectangular image/layer.

What you expected to happen

• Resizing with smoothing should produce consistent smooth edges, especially on curved convex contours.
• The result should not depend on how tightly the alpha bounds match the layer rectangle.
• Curvature should remain visually smooth and not appear geometrically flattened.

How to reproduce the problem

Reproduction Steps

1. Use a layer with transparency where the visible content is very close to the layer bounds.
2. Ensure the shape includes convex or rounded protruding areas.
3. Enable transform smoothing (e.g. bicubic or lanczos).
4. Apply a scale transform (x2-x4).
5. Zoom in significantly (e.g. 500–800%).

Additional context

[DustyShoe]

I've spent two days trying to pinpoint the culprit, but haven't been able to identify it.