Skip to content

[WIP] Avoid propagating some input columns when applying Onnx models #4971

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 22 commits into from
Closed

[WIP] Avoid propagating some input columns when applying Onnx models #4971

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
7be5229
Merged with Harish commit with partial solution:
antoniovs1029 Feb 20, 2020
0ac2da1
Merge remote-tracking branch 'upstream/master' into is25onnxColSelect…
antoniovs1029 Mar 25, 2020
00a26b2
Revert "Merged with Harish commit with partial solution:"
antoniovs1029 Mar 25, 2020
944b066
Revert "Revert "Merged with Harish commit with partial solution:""
antoniovs1029 Mar 25, 2020
5a6a1b0
Modified test, because now output schema should be the same as onnx m…
antoniovs1029 Mar 25, 2020
4dbdc26
Actually check that the output schema has dropped the columns
antoniovs1029 Mar 25, 2020
a1b2b6e
Further modifications to make this work with all the existing tests
antoniovs1029 Mar 25, 2020
dfed82e
Remove unnecessary outputschema property on OnnxTransformer
antoniovs1029 Mar 25, 2020
816c66f
Move OutputSchema logic to OnnxDataTransform instead of Mapper
antoniovs1029 Mar 30, 2020
e928b3b
Added the use of ColumnBindings on OnnxDataTransform
antoniovs1029 Mar 30, 2020
d604cf4
Added MYTODO to comments
antoniovs1029 Mar 30, 2020
e9f4def
Still not working. GetActive() can't return inputcolumns from 2 diffe…
antoniovs1029 Apr 2, 2020
357648e
* Removed ColumnSelectingTransformer from OnnxDataTransform
antoniovs1029 Apr 7, 2020
0687f52
Drop columns inside OnnxDataTransformer.Bindings and added comments
antoniovs1029 Apr 7, 2020
9191ea9
Revert changes in OnnxTransformTests
antoniovs1029 Apr 7, 2020
88bd905
Added comment
antoniovs1029 Apr 7, 2020
183ba26
Added test for the different possible cases
antoniovs1029 Apr 9, 2020
315aa52
Added comments
antoniovs1029 Apr 9, 2020
6759324
Added test for drop columns
antoniovs1029 Apr 9, 2020
0fd1557
Fixed mistakes in ColumnSelectingOnnxTestColumnPropagation
antoniovs1029 Apr 9, 2020
f73e097
Added side effect test of dropping input columns
antoniovs1029 Apr 9, 2020
26e72ec
Updated comments
antoniovs1029 Apr 9, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions src/Microsoft.ML.Data/Transforms/RowToRowTransformerBase.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@ private protected RowToRowTransformerBase(IHost host)

bool ITransformer.IsRowToRowMapper => true;

IRowToRowMapper ITransformer.GetRowToRowMapper(DataViewSchema inputSchema)
IRowToRowMapper ITransformer.GetRowToRowMapper(DataViewSchema inputSchema) => GetRowToRowMapperCore(inputSchema);

protected virtual IRowToRowMapper GetRowToRowMapperCore(DataViewSchema inputSchema)
{
Host.CheckValue(inputSchema, nameof(inputSchema));
return new RowToRowMapperTransform(Host, new EmptyDataView(Host, inputSchema), MakeRowMapper(inputSchema), MakeRowMapper);
Expand All @@ -37,16 +39,20 @@ IRowToRowMapper ITransformer.GetRowToRowMapper(DataViewSchema inputSchema)
[BestFriend]
private protected abstract IRowMapper MakeRowMapper(DataViewSchema schema);

public DataViewSchema GetOutputSchema(DataViewSchema inputSchema)
public DataViewSchema GetOutputSchema(DataViewSchema inputSchema) => GetOutputSchemaCore(inputSchema);

protected virtual DataViewSchema GetOutputSchemaCore(DataViewSchema inputSchema)
{
Host.CheckValue(inputSchema, nameof(inputSchema));
var mapper = MakeRowMapper(inputSchema);
return RowToRowMapperTransform.GetOutputSchema(inputSchema, mapper);
}

public IDataView Transform(IDataView input) => MakeDataTransform(input);
public IDataView Transform(IDataView input) => MakeDataTransformCore(input);

[BestFriend]
private protected virtual IDataView MakeDataTransformCore(IDataView input) => MakeDataTransform(input);

[BestFriend] // MYTODO: Since this is "BestFriend" here, should I also make the MakeDataTransformCore in OnnxDataTransform a "BestFriend"? What's the purpose?
private protected RowToRowMapperTransform MakeDataTransform(IDataView input)
{
Host.CheckValue(input, nameof(input));
Expand Down
Loading