[Discussion] When to utilize tf.function

[seanpmorgan]

Are there any best practices for when to decorate functions with tf.function? The TF2 documentation has a nice example where a keras Model is decorating the call method.

@karmel @martinwicke Should we use this extensively in our repo anytime there is a method without python side-effects? Is it safer to only utilize it when there are clear performance gains by exploiting parallelism or other optimizations? The latter seems like a vague criteria without specifying what patterns to look for.

[martinwicke]

I think as a general rule, we should use it wherever possible unless there's a clear reason not to.

[karmel]

Some reasons not to, off the top of my head:

• Variables need to be created each time the function is called.
• You want dynamic Python control flow inside of the function (ie, that should change each time you call the function based on criteria not passed to the function as args/kwargs)

Barring those, function-ify away. And, try to avoid the above scenarios if possible. (For example, in Keras, we have the object store a "built" attr such that we only create vars on the first call, and subsequent calls don't recreate attrs.)

[karmel]

(To clarify the second point there, Python control flow in general is fine, but there are certain edge cases where, once we trace the tf.function, it will stop being dynamic each time you call the function. For example, if you made a decision based on time or some Python object outside the function and not passed into the function. In those cases, the function would great a bit-of-graph based on the first calling, but would not regenerate with the most up-to-date Python values at each calling. That said, different values of inputs cause the function to be re-traced, so if the dynamic value in question were an input to the tf.function, you would be okay.)

[seanpmorgan]

Thanks! Closing issue as it can be re-opened if there are more thoughts/questions

[seanpmorgan]

So re-opening this issue for a bit just to continue the discussion. Scott has mentioned that hes hesitant to decorate some seq2seq stuff with tf.function for fear of debug-ability loss (See #72 review) .

We have noticed a possible issue in #94 that may be related to the liberal decorating, but I wanted to discuss our strategy a bit. It's certainly possible that in #94 we're seeing just a bug or some improper setup of the function for using autograph... but it does seem to have the potential to make troubleshooting more difficult.

@karmel @martinwicke @facaiy Should we consider letting the user decide when to decorate addon functions? That might make it easier in the short term, but I expect we'll field a lot of issues once users try to do just that.

[martinwicke]

@alextp FYI

Debugging alone shouldn't be grounds for not decorating. for debugging etc. you can use tf.config.experimental_run_functions_eagerly (which temporarily disables running functions as functions).

[qlzh727]

As the API author, my take on this is that we shouldn't annotation python function with tf.functions by default, instead, we should let user annotation their function and call ours. With that our function will proper inherit the context (eager/graph) based on user's current context.

@alextp, is there any reason to enable tf.function by default from API perspective?

[alextp]

@qlzh727 I think using tf.function in an API implementation, if safe, is a good way to help people get performance by default (both for graph building, as graphs with functions are smaller and quicker to build, and for eager execution).

[qlzh727]

Ack, one of side effect I am thinking of is that for any mathematical computation related python function, user kind of losses the ability to debug the tensor value if the function is wrapped by tf.function by default.

[alextp]

Which is why we have the execute_functions_eagerly thing.

…

On Tue, Mar 19, 2019 at 3:43 PM Qianli Scott Zhu ***@***.***> wrote: Ack, one of side effect I am thinking of is that for any mathematical computation related python function, user kind of losses the ability to debug the tensor value if the function is wrapped by tf.function by default. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAATxbAGUbJzbwZ6j9bSN-YKGnSKoLYMks5vYWgUgaJpZM4Z_QTu> .

-- - Alex

[qlzh727]

Oh, do u mean "tf.experimental.function_executor_type"? I didn't find "tf.execute_functions_eagerly" in the API.