Repeated iterations in callback

[DerWeh]

I tried using the callback to provide progress indicated. But using a simple print functions, I am quite confused by the output: Why do I get repeated output for the same step and back, as soon as the progress halts?

Example code:

from interpret.glassbox import ExplainableBoostingRegressor
from interpret.utils import make_synthetic
from sklearn.model_selection import train_test_split

seed = 42

X, y, names, types = make_synthetic(classes=None, n_samples=50000, missing=False, seed=seed)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=seed)


def logging_callback(bag_idx: int, n_steps: int, has_progressed: bool, best_score: float) -> bool:
    if n_steps % 10000 == 0:
        print(f"{bag_idx=:2}, {n_steps=:5}, {has_progressed=!s:>5}, {best_score=:.6}")
    return False


ebm = ExplainableBoostingRegressor(names, types, interactions=3, smoothing_rounds=5000, reg_alpha=10.0, callback=logging_callback)
ebm.fit(X_train, y_train)

Some output:

bag_idx= 6, n_steps=100000, has_progressed= True, best_score=0.313481
bag_idx= 6, n_steps=100000, has_progressed=False, best_score=0.313481
bag_idx= 6, n_steps=100000, has_progressed=False, best_score=0.313481
bag_idx= 6, n_steps=100000, has_progressed=False, best_score=0.313481
bag_idx= 6, n_steps=100000, has_progressed=False, best_score=0.313481
bag_idx= 6, n_steps=100000, has_progressed=False, best_score=0.313481
bag_idx= 6, n_steps=100000, has_progressed=False, best_score=0.313481
bag_idx= 6, n_steps=100000, has_progressed=False, best_score=0.313481
bag_idx= 6, n_steps=100000, has_progressed=False, best_score=0.313481
bag_idx= 6, n_steps=100000, has_progressed=False, best_score=0.313481
bag_idx= 6, n_steps=100000, has_progressed=False, best_score=0.313481

[paulbkoch]

The current EBM algorithm mostly operates via greedy boosting, but with a catch. We don't re-calculate the gain for each possible term on each boosting step. We use slightly stale gain values because it's more computationally efficient. Periodically though, it can be beneficial to re-calculate the gains. The time it takes to calculate the gain is almost identical to the time required for boosting the term, so during the gain recalculation we call the user's calllback function with has_progressed=False to indicate that no progress is being made while updating the gain values.

[DerWeh]

Thanks for the clarification, so to indicate progress I should more appropriately use the callback

def logging_callback(bag_idx: int, n_steps: int, has_progressed: bool, best_score: float) -> bool:
    if has_progressed and n_steps % 10000 == 0:
        print(f"{bag_idx=:2}, {n_steps=:6}, {best_score=:.6}")
    return False

Restring the output to a single bag for clarity, we see that after 190,000 steps, the counter resets and some more iterations are done; but the score is around 6 times smaller:

bag_idx= 7, n_steps= 10000, best_score=0.802715
bag_idx= 7, n_steps= 20000, best_score=0.642954
bag_idx= 7, n_steps= 30000, best_score=0.568031
bag_idx= 7, n_steps= 40000, best_score=0.524845
bag_idx= 7, n_steps= 50000, best_score=0.49645
bag_idx= 7, n_steps= 60000, best_score=0.332974
bag_idx= 7, n_steps= 70000, best_score=0.321133
bag_idx= 7, n_steps= 80000, best_score=0.315527
bag_idx= 7, n_steps= 90000, best_score=0.312416
bag_idx= 7, n_steps=100000, best_score=0.310612
bag_idx= 7, n_steps=110000, best_score=0.309533
bag_idx= 7, n_steps=120000, best_score=0.308807
bag_idx= 7, n_steps=130000, best_score=0.308336
bag_idx= 7, n_steps=140000, best_score=0.308004
bag_idx= 7, n_steps=150000, best_score=0.307764
bag_idx= 7, n_steps=160000, best_score=0.307627
bag_idx= 7, n_steps=170000, best_score=0.307522
bag_idx= 7, n_steps=180000, best_score=0.307436
bag_idx= 7, n_steps=190000, best_score=0.307367
bag_idx= 7, n_steps= 10000, best_score=0.0744886
bag_idx= 7, n_steps= 20000, best_score=0.0735434
bag_idx= 7, n_steps= 30000, best_score=0.0734163
bag_idx= 7, n_steps= 40000, best_score=0.073368

Can you please also explain this effect?
('score' is likely a misnomer from my site, I think it is called metric in the code. Should be a loss as smaller=better.)

[paulbkoch]

After the mains we do the pairs, that's the source of the reset in n_steps. We probably need to include more information in the callback like n_stage or something like that.

[paulbkoch]

I'm also thinking we should change the api so that the callback uses named parameters instead of positional ones and suggest the use of kwargs to allow for future additions. So far, I think, the only consumer of this interface is autogluon, and they shouldn't break if we change it since they used args & kwargs in their callback.