Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

[v1.x] Onnx Support for Dropout #19837

Merged
Zha0q1 merged 6 commits into from
Feb 8, 2021
Merged

[v1.x] Onnx Support for Dropout #19837

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1abdf28
Dropout
Zha0q1 Feb 3, 2021
cadfb1a
Update _op_translations.py
Zha0q1 Feb 4, 2021
97fd88c
Update _op_translations.py
Zha0q1 Feb 4, 2021
f106308
Update _op_translations.py
Zha0q1 Feb 5, 2021
049fe77
Merge branch 'v1.x' into onnx_dropout_conv
Zha0q1 Feb 5, 2021
cb078d6
Update _op_translations.py
Zha0q1 Feb 5, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions python/mxnet/contrib/onnx/mx2onnx/_op_translations.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1122,19 +1122,19 @@ def convert_dropout(node, **kwargs):
"""
from onnx.helper import make_node
name, input_nodes, attrs = get_inputs(node, kwargs)
opset_version = kwargs["opset_version"]

probability = float(attrs.get("p", 0.5))
_ = float(attrs.get("p", 0.5))
_ = convert_string_to_list(attrs.get("axes", "None"))
mode = attrs.get('mode', 'training')

if opset_version >= 12:
# opset >= 12 requires the ratio to be an input
nodes = [
create_const_scalar_node(name+"_ratio0", np.float32(probability), kwargs),
make_node("Dropout", [input_nodes[0], name+"_ratio0"], [name], name=name)
]
return nodes
else:
return [make_node("Dropout", input_nodes, [name], ratio=probability, name=name)]
if mode != 'training':
Copy link
Contributor

@josephevans josephevans Feb 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to check and preserve the Dropout nodes if mode is training? Otherwise if inference mode, return identity op.

Copy link
Contributor Author

@Zha0q1 Zha0q1 Feb 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think dropout is almost never used in inference because the output values will be randomly dropped and theoretically it's designed for training only. I think not supporting 'both' mode, at least for now, can confirm this while we export the models in the model zoo.

Copy link
Contributor

@josephevans josephevans Feb 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, just trying to understand the mode parameter and how it applies to these training-only ops. Is mode set to 'training' in the models or are you just depending on the default from attrs.get() above?

Just wondering for future plans to export models for training (if needed.) Do we need to specify whether we are exporting for training or for inference in the export call, or can we make it operator dependent?

Copy link
Contributor

@josephevans josephevans Feb 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to merge, we should just continue this conversation for future uses.

Copy link
Contributor Author

@Zha0q1 Zha0q1 Feb 6, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, just trying to understand the mode parameter and how it applies to these training-only ops. Is mode set to 'training' in the models or are you just depending on the default from attrs.get() above?

Just wondering for future plans to export models for training (if needed.) Do we need to specify whether we are exporting for training or for inference in the export call, or can we make it operator dependent?

mode is a parameter to MXNet operator Dropout only; it's now a parameter for the model :). I think since we targeting inference we might be able to set it to Identity for now

raise NotImplementedError("Dropout does not currently support mode!=\'training\'")

nodes = [
make_node('Identity', [input_nodes[0]], [name])
]

return nodes


@mx_op.register("Flatten")
Expand Down
9 changes: 9 additions & 0 deletions tests/python-pytest/onnx/test_operators.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -896,6 +896,15 @@ def test_onnx_export_broadcast_mul(tmp_path, dtype):
op_export_test('broadcast_mul', M, [x, y], tmp_path)


@pytest.mark.parametrize('dtype', ['float16', 'float32', 'float64'])
@pytest.mark.parametrize('shape', [(3, 4, 5), (1, 2, 3, 2, 1)])
@pytest.mark.parametrize('p', [0, 0.1, 0.5, 1])
def test_onnx_export_dropout(tmp_path, dtype, shape, p):
x = mx.random.uniform(-100, 100, shape=shape).astype(dtype)
M = def_model('Dropout', p=p)
op_export_test('Dropuout', M, [x], tmp_path)


@pytest.mark.parametrize('dtype', ['float32'])
@pytest.mark.parametrize('shape', [(1, 3, 64, 64), (2, 6, 60, 60)])
@pytest.mark.parametrize('num_filter', [2, 4, 32])
Expand Down