Skip to content

Separate train an eval for estimator #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: resnet_perf_tweaks
Choose a base branch
from
Open

Separate train an eval for estimator #22

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ef7ba43
Changed a flag so that it is possible to do train without eval
Nov 7, 2018
efab22b
Change the flag do_eval to skip_eval
Nov 7, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 18 additions & 18 deletions official/resnet/resnet_run_loop.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -523,8 +523,8 @@ def input_fn_eval():
num_epochs=1,
dtype=flags_core.get_tf_dtype(flags_obj))

if flags_obj.eval_only or not flags_obj.train_epochs:
# If --eval_only is set, perform a single loop with zero train epochs.
if not flags_obj.train_epochs:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When is this true given we have a default value for train_epochs?

# If train_epoches is not set, perform a single loop with zero train epochs.
schedule, n_loops = [0], 1
else:
# Compute the number of times to loop while training. All but the last
Expand All @@ -546,22 +546,23 @@ def input_fn_eval():
classifier.train(input_fn=lambda: input_fn_train(num_train_epochs),
hooks=train_hooks, max_steps=flags_obj.max_train_steps)

tf.logging.info('Starting to evaluate.')
if not flags_obj.skip_eval:
tf.logging.info('Starting to evaluate.')

# flags_obj.max_train_steps is generally associated with testing and
# profiling. As a result it is frequently called with synthetic data, which
# will iterate forever. Passing steps=flags_obj.max_train_steps allows the
# eval (which is generally unimportant in those circumstances) to terminate.
# Note that eval will run for max_train_steps each loop, regardless of the
# global_step count.
eval_results = classifier.evaluate(input_fn=input_fn_eval,
steps=flags_obj.max_train_steps)
# flags_obj.max_train_steps is generally associated with testing and
# profiling. As a result it is frequently called with synthetic data,
# which will iterate forever. Passing steps=flags_obj.max_train_steps
# allows the eval (which is generally unimportant in those circumstances)
# to terminate. Note that eval will run for max_train_steps each loop,
# regardless of the global_step count.
eval_results = classifier.evaluate(input_fn=input_fn_eval,
steps=flags_obj.max_train_steps)

benchmark_logger.log_evaluation_result(eval_results)
benchmark_logger.log_evaluation_result(eval_results)

if model_helpers.past_stop_threshold(
flags_obj.stop_threshold, eval_results['accuracy']):
break
if model_helpers.past_stop_threshold(
flags_obj.stop_threshold, eval_results['accuracy']):
break

if flags_obj.export_dir is not None:
# Exports a saved model for the given classifier.
Expand Down Expand Up @@ -594,9 +595,8 @@ def define_resnet_flags(resnet_size_choices=None):
'If not None initialize all the network except the final layer with '
'these values'))
flags.DEFINE_boolean(
name='eval_only', default=False,
help=flags_core.help_wrap('Skip training and only perform evaluation on '
'the latest checkpoint.'))
name='skip_eval', default=False,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not add 'skip_eval' instead of modifying an existing flag?

help=flags_core.help_wrap('Skip evaluation when this flag is true'))

choice_kwargs = dict(
name='resnet_size', short_name='rs', default='50',
Expand Down