Ignore hidden columns in AutoML schema checks of validation data #4490
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
daholste
added
the
AutoML.NET
| @@ -183,14 +183,19 @@ private static void ValidateValidationData(IDataView trainData, IDataView valida | |||
|
|
|||
| const string schemaMismatchError = "Training data and validation data schemas do not match."; | |||
|
|
|||
| if (trainData.Schema.Count != validationData.Schema.Count) | |||
| if (trainData.Schema.Count(c => !c.IsHidden) != validationData.Schema.Count(c => !c.IsHidden)) | |||
| { | |||
| throw new ArgumentException($"{schemaMismatchError} Train data has '{trainData.Schema.Count}' columns," + | |||
| $"and validation data has '{validationData.Schema.Count}' columns.", nameof(validationData)); | |||
| } | |||
|
|
|||
| foreach (var trainCol in trainData.Schema) | |||
There was a problem hiding this comment.
Could use a comment...
Suggested change
| foreach (var trainCol in trainData.Schema) | |
| // Also indirectly checks for new columns in the validation datasets as we above enforce the column counts are equal | |
| foreach (var trainCol in trainData.Schema) |
I was otherwise going to suggest we check the reverse direction. A comment helps future readers to not have to think through it.
Codecov Report
@@ Coverage Diff @@
## master #4490 +/- ##
=========================================
Coverage ? 74.85%
=========================================
Files ? 908
Lines ? 159880
Branches ? 17215
=========================================
Hits ? 119678
Misses ? 35384
Partials ? 4818
|
ghost
locked as resolved and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #4491
When the AutoML API consumes data, it validates schema consistency between the train and validation data.
There are two bugs in this logic:
The API asserts that the count of columns in the train and validation data must be equal. This throws an exception if the two data views have the same number of active columns but a different number of hidden columns. This PR updates to assert that the # of active (not hidden) columns in the train and validation data are equal.
If either the train or validation data has a hidden column with a type that differs from an active column of the same name, an exception is thrown. This PR restricts type consistency checks to active columns only.