In MLJIteration InvalidValue() is not triggering a stop when training losses go to NaN:
using MLJFlux
using MLJIteration
using Flux
using MLJBase
using IterationControl
model = NeuralNetworkRegressor(optimiser=Descent(100), rng=123)
imodel = IteratedModel(model=model,
controls=[Step(1),
InvalidValue(),
WithTrainingLossesDo(),
NumberLimit(5)])
X, y = make_regression();
mach = machine(imodel, X, y) |> fit!
julia> mach = machine(imodel, X, y) |> fit!
[ Info: Training Machine{DeterministicIteratedModel{NeuralNetworkRegressor{Linear,…}},…}.
[ Info: training: [3.826121942447169e21]
[ Info: training: [2.8339361370466164e49]
[ Info: training: [NaN]
[ Info: training: [NaN]
[ Info: training: [NaN]
[ Info: training: [NaN]
[ Info: final loss: 1.7113521901845545
[ Info: final training loss: NaN
[ Info: Stop triggered by NumberLimit(5) stopping criterion.
[ Info: Total of 6 iterations.
The reason is that IterationControl only grabs the training loss for feeding into a stopping criterion if needs_training_loss(c) = true, which is not the case for c = InvalidValue(). The naive fix which re-defines this trait in StoppingCriterion.jl may not be the best fix. For it may have unintended consequences for models that don't support training losses. In MLJIteration, this might not matter as training_losses returns nothing as fallback, but for other clients of EarlyStopping.jl this could lead to unexpected behaviour.
In MLJIteration
InvalidValue()is not triggering a stop when training losses go toNaN:The reason is that IterationControl only grabs the training loss for feeding into a stopping criterion if
needs_training_loss(c) = true, which is not the case forc = InvalidValue().The naive fix which re-defines this trait in StoppingCriterion.jl may not be the best fix. For it may have unintended consequences for models that don't support training losses. In MLJIteration, this might not matter astraining_lossesreturnsnothingas fallback, but for other clients of EarlyStopping.jl this could lead to unexpected behaviour.