Sklearn operators end the run on an empty cell, with an error from inside scikit-learn

[kz930]

What happened?

Every Sklearn operator ends the execution when any cell it reads is empty. The error comes from inside scikit-learn's input validation rather than from the operator: ValueError: Input X contains NaN.

No operator in the family looks at missing values. Searching sklearn/ and machineLearning/ for dropna, fillna, isna or SimpleImputer returns nothing. The generated Python takes the columns and fits: SklearnTrainingOpDesc.scala:45 is X = table.drop(target, axis=1) followed by .fit(X, Y), and SklearnClassifierOpDesc.scala:45, SklearnTestingOpDesc.scala:74 and SklearnAdvancedBaseDesc.scala:158 all do the same.

An empty value is ordinary input here. A blank CSV cell arrives as null, since univocity returns null for an empty field and AttributeTypeUtils.parseField passes it through by design.

Two things make this worse than a strict estimator refusing bad input. The message names nothing the user can act on, since X is a variable inside generated code and identifies neither the column nor the row. And the training and classifier operators feed every column except the target into the estimator, so a blank in a note or source column that has nothing to do with the model ends the run just the same. Only the Advanced Sklearn operators read a named list of features, and they crash on those.

Two neighbouring cases fail the same way, for the same reason. A blank in the target column gives ValueError: Input y contains NaN. A blank in the text column with Count Vectorizer enabled gives AttributeError: 'NoneType' object has no attribute 'lower' from inside CountVectorizer.

Expected: a blank cell is skipped, which is what the rest of the codebase does with a value that is not there. Twenty-four visualization operators open their generated Python with dropna(subset=[...]) #remove missing values, COUNT(column) counts only non-null rows, CONCAT and MIN pass over them, and FilterPredicate answers false for every condition but IS_NULL / IS_NOT_NULL. The Sklearn family is the only one with no answer at all. What skipping should mean differs by what each operator emits, since the prediction operator adds a column to the user's rows rather than emitting a model.

How to reproduce?

Upload a CSV with a blank cell:

x1,x2,y
0.1,0.2,0
0.3,,0
0.5,0.6,1
0.7,0.8,1

Build CSV File Scan to Training: Bernoulli Naive Bayes from the Sklearn Training group, set Target Attribute to y, and run. The execution stops and the operator's console shows the ValueError. Every other operator in the Sklearn, Sklearn Training and Advanced Sklearn groups behaves the same way on the same file.

The training and classifier operators are blocking, so the whole table is consumed before the estimator is fitted. The error arrives at the end, after every upstream operator has done its work.

Version/Branch

1.3.0-incubating-SNAPSHOT (main)

Relevant log output

Traceback (most recent call last):
  File "core/runnables/data_processor.py", line 77, in process_internal_marker
    self._set_output_tuple(executor.on_finish(port_id))
  File "core/runnables/data_processor.py", line 125, in _set_output_tuple
    for output in output_iterator:
  File "core/models/operator.py", line 280, in on_finish
    yield from self.process_table(table, port)
  File "udf-v1.py", line 12, in process_table
    model = make_pipeline(  BernoulliNB()).fit(X, Y)
  File "sklearn/pipeline.py", line 663, in fit
    self._final_estimator.fit(Xt, y, **last_step_params["fit"])
  File "sklearn/naive_bayes.py", line 735, in fit
    X, y = self._check_X_y(X, y)
  File "sklearn/utils/validation.py", line 169, in _assert_all_finite_element_wise
ValueError: Input X contains NaN.
BernoulliNB does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values.