[Relay][Frontend][Tensorflow] Fix GatherV2, Add StopGradient

[trevor-m]

These changes are needed to parse a BERT TF model into relay.

1. GatherV2
   TensorFlow recently added batch_dims as an attribute of the GatherV2 op to support batched gather. This would cause the parser the fail since it didn't expect that attribute. I added a check to make sure the attribute doesn't change the behavior of the op if it happens to exist and also added it to the ignore list. We can support parsing of batched gather at a later date.

2. StopGradient
   This is a training op meant to stop gradient flow over certain edges in the graph. It has no purpose in inference so it can be safely mapped to identity.

[kevinthesun]

@trevor-m Thank you for adding this. Can you take a look at the CI failure?

[trevor-m]

@trevor-m Thank you for adding this. Can you take a look at the CI failure?

Thanks @kevinthesun, the link http://ci.tvm.ai:8080/job/tvm/job/PR-4238/1/display/redirect isn't working for me.

Edit:
Looks like unrelated issue with runner:

docker: failed to register layer: Error processing tar file(exit status 1): write /usr/local/VulkanSDK/1.0.65.0/source/shaderc/src/build/third_party/spirv-tools/test/val/test_val_validation_state: no space left on device.

[yzhliu]

please kick ci later.

[MarisaKirisame]

I object mapping it to identity directly, because we have an ad algorithm. Can it be replaced with an operator that map to identity, and has grad of 0?

[trevor-m]

I object mapping it to identity directly, because we have an ad algorithm. Can it be replaced with an operator that map to identity, and has grad of 0?

That makes sense to me. How do you see this being implemented? Do you mean creating a new op in relay or is there a way to do it just in the parser?

Looks like nnvm had an op block_grad which is equivalent to StopGradients.

[MarisaKirisame]

You have to create an op in relay.
Then, it will have a grad function, which will return [zeros_like(orig.args[0])].
It wont be a big change (probably ~30 lines, most of which will be copied), and it will help you learn a bit about the stack.
I can do it if you found it too difficult though.

[trevor-m]

That sounds good, thanks for the description!

I’ll update this PR with the change shortly.