Skip to content

InvalidValue is not triggering a stop for invalid training losses #36

Description

@ablaom

In MLJIteration InvalidValue() is not triggering a stop when training losses go to NaN:

using MLJFlux
using MLJIteration
using Flux
using MLJBase
using IterationControl

model = NeuralNetworkRegressor(optimiser=Descent(100), rng=123)
imodel = IteratedModel(model=model,
                       controls=[Step(1),
                                 InvalidValue(),
                                 WithTrainingLossesDo(),
                                 NumberLimit(5)])
X, y = make_regression();

mach = machine(imodel, X, y) |> fit!

julia> mach = machine(imodel, X, y) |> fit!
[ Info: Training Machine{DeterministicIteratedModel{NeuralNetworkRegressor{Linear,…}},…}.
[ Info: training: [3.826121942447169e21]
[ Info: training: [2.8339361370466164e49]
[ Info: training: [NaN]
[ Info: training: [NaN]
[ Info: training: [NaN]
[ Info: training: [NaN]
[ Info: final loss: 1.7113521901845545
[ Info: final training loss: NaN
[ Info: Stop triggered by NumberLimit(5) stopping criterion. 
[ Info: Total of 6 iterations. 

The reason is that IterationControl only grabs the training loss for feeding into a stopping criterion if needs_training_loss(c) = true, which is not the case for c = InvalidValue(). The naive fix which re-defines this trait in StoppingCriterion.jl may not be the best fix. For it may have unintended consequences for models that don't support training losses. In MLJIteration, this might not matter as training_losses returns nothing as fallback, but for other clients of EarlyStopping.jl this could lead to unexpected behaviour.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions