Running list: potential discrepancies in preprint timings #183

Summary

Follow-up work on the differentiable-solver benchmarks (dolfinx-adjoint#81, #179, #182) has surfaced several timing discrepancies between Table 3 of the paper (arXiv:2606.27895) and what the current benchmark suite reproduces. This issue collects them in one place and tracks their resolution.

These look like artifacts of the earlier benchmark harness and the environment the paper's numbers were produced in, not fundamental problems with the solvers or the physics. The forward and gradient accuracy results are unaffected. Only the wall-clock cost columns are in question. We intend to re-run the affected timing measurements on the current suite and hardware, incorporating the first wave of third-party solver fixes, and update the table accordingly.

The paper is still at the preprint stage, so correcting these now is straightforward. To be clear, this is a one-time reconciliation while the work is pre-publication, not an open-ended commitment to chase every measurement indefinitely.

Discrepancies

1. FEniCS structural forward time (~140 s) does not reproduce

Table 3 lists the legacy-FEniCS structural forward at ~140 s (N=128). On the current suite (#179 extended the cost sweep to the paper's 128×2×64 resolution), FEniCS forward is ~5.5 s and VJP ~16.3 s on CI. After #182 (UMFPACK→mumps plus tape caching) it drops to ~3.7 s / ~11.1 s. That is a ~25–38× gap from the published number.

• The FEniCS solver code is essentially unchanged since it was written, so this is not a solver regression that was later fixed.
• Instrumentation (see fix: tune FEniCS Tesseracts (better linear solver, avoid redundant forward calls, only setup once) #182) shows ~99.6% of the structural time is the linear solve, with the adjoint costing ~1.0× the forward. There is no path from ~11 s to ~140 s on this solver.
• The most likely cause is the paper-repro environment (thread count, solver defaults, no warm-tape reuse) rather than the solver. An end-to-end re-run is needed to confirm.

2. Sub-1.0 VJP/forward ratios are not physically reachable

Several Table 3 rows report VJP cheaper than a standalone forward:

The VJP column times jax.grad, which dispatches one apply RPC followed by one vector_jacobian_product RPC, and most VJP endpoints re-run the forward internally. A VJP measurement is therefore about two forward solves plus one adjoint, which cannot come out below 1× a forward. JAX-FEM and Firedrake sit where this predicts (~2.0–2.4). The sub-1.0 rows should not be reachable. (Credit: @jorgensd, #182 §3.)

The current suite confirms this directly. Structural cost at nx=128 (paper resolution) on our CI shows every solver's VJP/forward ratio at 1.7 or above, including the ones the paper puts below 1.0:

Firedrake and JAX-FEM match the paper. FEniCS and TopOpt.jl flip from sub-1.0 to above 1.0 under re-measurement.

3. torch-fem row: re-measured at ~1.6×, not 0.33

We reproduced the torch-fem thermal endpoints in a container (torch-fem 0.6.2, CPU, which is the tesseract's actual numeric path since it hardcodes _SPARSE_SOLVE_DEVICE="cpu"), instrumenting every sparse_solve:

• VJP/forward is ~1.6 at the paper's N=256, not 0.33. The VJP always does one extra adjoint solve on top of the forward, exactly as the source implies. This is consistent with @jorgensd's in-process ~1.5.
• The preconditioner reuse the paper describes is real. The adjoint reuses the AMG hierarchy, making it ~0.77× a forward. That reduces the adjoint cost but cannot invert the ratio below 1.
• Our absolute forward at N=256 is single-digit seconds against the published 33 s. Like the FEniCS 140 s, that large gap points to the earlier environment, so the torch-fem row is worth re-running end-to-end rather than just checking its ratio.
• Caveat on our numbers: CPU, in-process (not over the tesseract RPC), on a laptop, so absolute times carry run-to-run variance. The ratio stays above 1 at every size across repeats.

4. TopOpt.jl (0.19) — not yet checked

The listed strategy is "Analytic." For self-adjoint compliance minimisation the λ=−u shortcut means no separate adjoint solve, so a genuinely cheap VJP is plausible. It still needs a forward solution, so the same "cannot be far below a forward" question applies. Not yet independently measured.

Plan / tracking

• Re-run FEniCS structural and heat timings end-to-end on the current suite and hardware (see fix: extend cost sweep to the paper's nx=128 resolution #179/fix: tune FEniCS Tesseracts (better linear solver, avoid redundant forward calls, only setup once) #182). The mumps+cache curve still needs to be captured as the published baseline.
• Re-run torch-fem thermal end-to-end over the tesseract RPC (confirm the ~1.6 ratio and resolve the absolute-time gap against 33 s)
• Independently measure TopOpt.jl VJP/forward
• Apply the analogous Firedrake fixes (@dham, same pyadjoint core, noted in fix: tune FEniCS Tesseracts (better linear solver, avoid redundant forward calls, only setup once) #182)
• Decide the intended semantics of the VJP column (marginal gradient cost vs. full forward+adjoint) and state it explicitly in the paper. tesseract_core's jax_recipes residual cache could make it "marginal cost of a gradient" if enabled suite-wide (see fix: tune FEniCS Tesseracts (better linear solver, avoid redundant forward calls, only setup once) #182 §3)
• Regenerate Table 3 and post a corrected version / preprint revision

References

• dolfinx-adjoint#81, the original observation
• fix: extend cost sweep to the paper's nx=128 resolution #179, extended the structural cost sweep to the paper resolution (merged)
• fix: tune FEniCS Tesseracts (better linear solver, avoid redundant forward calls, only setup once) #182, FEniCS caching + mumps, and the methodology analysis (open)