layered: tolerate non-facing-side external port dummies in boundary layers

[novazembla]

Relating to issue #1192.

The hierarchical layer sweep assumed that when the first node of a nested graph's boundary layer is an external port dummy, the WHOLE layer consists of dummies for hierarchical ports on the sweep-facing side (WEST/EAST). That assumption does not hold: dummies for NORTH/SOUTH ports carry no layer constraint (LGraphUtil.createExternalPortDummy only sets an in-layer constraint for them), so the layerer may place them in the first or last layer too. The count mismatch then threw IllegalStateException: Expected N hierarchical ports, but found only 0.

sortPortDummiesByPortPositions now sorts only the dummies that belong to hierarchical ports on the sweep-facing side and leaves all other nodes of the layer in their current relative order, instead of throwing.

The inverse write-back, sortPortsByDummyPositionsInLastLayer, had the same latent assumption and could silently write a NORTH/SOUTH port into an EAST/WEST port slot, corrupting the port order. It now writes back only the dummies whose origin port lies on the side facing the end of the sweep, permuting exactly the ports whose dummies were found in that layer.

On graphs where the old invariant held, both methods behave exactly as before.

I'm happy to have my contribution licenced under GPLv3 (#1185)

ELK Test Graph

This graph fails before the fix (see issue #1192) and does now pass

algorithm: layered
direction: RIGHT
hierarchyHandling: INCLUDE_CHILDREN

node core {
    layout [ size: 120, 60 ]
    portConstraints: FIXED_ORDER
    port p_north_1 { ^port.side: NORTH }
    port p_north_2 { ^port.side: NORTH }
    port p_east_unused { ^port.side: EAST }   // referenced by NO edge — yet load-bearing
    port p_east_2 { ^port.side: EAST }

    node boxA {
        layout [ size: 120, 60 ]
        node leafA { layout [ size: 80, 40 ] }
    }
    node boxB {
        layout [ size: 120, 60 ]
        node leafB { layout [ size: 80, 40 ] }
    }
    node plain { layout [ size: 80, 40 ] }

    // edges fully inside core
    edge boxA -> boxB
    edge boxA.leafA -> plain
}

// edges crossing core's boundary, contained at the root graph (load-bearing)
edge core.p_north_1 -> core.boxA
edge core.p_north_2 -> core.boxB
edge core.boxB.leafB -> core.p_east_2

[soerendomroes]

Thank you, should you also post the test elkt graph here?

It will take a while to look into this, so please do not be alarmed.

[novazembla]

I've added the graph. Thank you!