Quantized Grad Randomly Fails `best_split_info.left_count` check

[bbstats]

Description

When using the new 'use_quantized_grad' in python, lightgbm will randomly fail, giving this error:

LightGBMError: Check failed: (best_split_info.right_count) > (0) at /__w/1/s/lightgbm-python/src/treelearner/serial_tree_learner.cpp, line 855 .

Even when locking in the random state, it will fail randomly. Run the below code multiple times and you'll see it occasionally passes but fails maybe 1/2 the time?

Reproducible example

In google colab,

%pip install -U lightgbm
from lightgbm import LGBMRegressor
from sklearn.datasets import make_regression
X,y=make_regression()
while True:
    model = LGBMRegressor(use_quantized_grad=True, random_state=4, verbose=-1)
    model.fit(X,y)
    print('pass')

Environment info

Google Colab
Python 3.10.6
LightGBM version or commit hash: 4.0.0

Command(s) you used to install LightGBM

%pip install -U lightgbm

[jameslamb]

Linking this related issue: #5982

[jameslamb]

Even LightGBM's CI is sometimes randomly failing with this error. e.g., I just saw this on Windows: https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=14958&view=logs&j=67ae2ff8-4b33-578b-0813-9c0440091667&t=aed7750d-29a7-50d7-d7f0-fc7145577059

FAILED ..\tests\python_package_test\test_engine.py::test_quantized_training
= 1 failed, 527 passed, 33 skipped, 2 xfailed, 284 warnings in 112.45s (0:01:52) =

[LightGBM] [Fatal] Check failed: (best_split_info.left_count) > (0) at C:\Users\VssAdministrator\AppData\Local\Temp\pip-req-build-7q_w26kz\src\treelearner\serial_tree_learner.cpp, line 845 .

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.