rfc: dropout primitive attribute

[itaraban]

This is proposal to support dropout operation in oneDNN via primitive attribute
Link to rendered document
Link to PoC
Performance data for DGL (with PyTorch backend) GNN training benchmarks using Icelake server machine:

benchmark	dataset	default	with oneDNN dropout	difference
bench_gat	cora	0.012357805	0.009989643	124%
	pubmed	0.029577794	0.021833464	135%
bench_sage	cora	0.007244702	0.00432033	168%
	pubmed	0.017254775	0.009778418	176%

[mgouicem]

Thanks for the RFC and prototype. Could you clarify the parameters for the attribute. For example is the memory descriptor for output masks or input mask ? I guess I see two ways:

• user might pass probability and the primitive computes the mask
• user might passes the mask and the primitive just applies it.

If you have training in mind, I wonder if we need to support both (on forward pass, generate the mask, on backward pass consume it).

Other questions:

• what kind of randomness is required on oneDNN implementations (how much entropy?, what distribution? ...)?
• what would be the plan to validate this attribute (the masking part should be fine, talking about randomness)
• we might want to add a knob for user to set the random seed, in order to be able to reproduce runs.

[mgouicem]

Do I understand correctly that, if convolution has the dropout attribute set, on forward we would:

• compute the dropout mask
• apply the mask to the destination
• and write the mask to a new output memory

The question is how would that mask be applied on backward? For example, we have Conv -> Dropout -> Relu.
Do we expect user to pass the dropout mask to the Relu backward computation or the Conv backward computation? I would think the former would be simpler to implement.

[itaraban]

Yes, mentioned algorithm is what we expect from this primitive attiribute - compute and apply mask.
On backward pass mask from forward will be multiplied by input(see native_dropout_backward from PyTorch)
The main idea was to move mask generation inside of oneDNN. Otherwise user can just use binary_mul post op to apply existing mask.

[mgouicem]

Thanks. That clarifies the scope of the rfc (the new attribute is for fwd only).

[mgouicem]

BTW, what datatypes is oneDNN expected to support for the mask?

[itaraban]

Currently, we test only float32. I think in future, probably bfloat16 support will be also needed.

[mgouicem]

is the probability needed at creation time, or can we take it at execution time?
In particular, if this parameter changes for each execution of a given primitive, passing probability at execution time would allow to increase primitive cache hit rate.

[itaraban]

In our models/benchmarks we have same probability for all layers, but I think, that it is possible to make this runtime parameter.

[mgouicem]

From the POC, it seems that in ref path the dropout is applied after post-ops and not before. Is that intentional?

[itaraban]

From the POC, it seems that in ref path the dropout is applied after post-ops and not before. Is that intentional?

No, I'll change it. Thank you for finding it!

[igorsafo]

I would suggest to remove enable_drop from both APIs:

• In getter rely on md returned from the API call. If it is a zero md, then dropout is not set by user.
• In setter -- once user called dnnl_primitive_attr_set_dropout(attr, mask) dropout is set.

The current API is a bit confusing when user calls dnnl_primitive_attr_set_dropout(attr, **false**, mask_desc)

[itaraban]

@igorsafo , thanks for you suggestion!
I updated README and PoC

[igorsafo]

(random spot): We might need to introduce a standalone Dropout primitive to support frrameworks like ONNX that registers operations supported by the backend. In the current proposal Dropout functionality will be limited to a few patterns, but the rest patterns will not be able to implement Dropout functionality using oneDNN. Please double check with the frameworks if the solution works for them as well.
+@georgen117

[mgouicem]

I guess the question here is "is there a benefit to support dropout primitive in oneDNN"?
For the case of ONNX, isn't it ok if the oneDNN provider provides the dropout operation without using oneDNN, but uses oneDNN when dropout fusion occurs?

[igorsafo]

Suggested change

	uint8_t enabled_u8;
	error::wrap_c_api(
	dnnl_primitive_attr_get_dropout(get(), &enabled_u8, &cdesc),
	"could not get parameters of a dropout attribute");
	dnnl_memory_desc_t cloned_md = nullptr;
	error::wrap_c_api(dnnl_memory_desc_clone(&cloned_md, cdesc),
	"could not clone a memory descriptor");
	mask_desc = memory::desc(cloned_md);
	enabled = enabled_u8;
	error::wrap_c_api(
	dnnl_primitive_attr_get_dropout(get(), &cdesc),
	"could not get parameters of a dropout attribute");
	dnnl_memory_desc_t cloned_md = nullptr;
	error::wrap_c_api(dnnl_memory_desc_clone(&cloned_md, cdesc),
	"could not clone a memory descriptor");
	mask_desc = memory::desc(cloned_md);

[itaraban]

Thanks, changed