model.finetune(...) does not show the loss of the model

[JessicaLopezEspejel]

Hello,

I just ran the following jupyter notebook: t5-trivia.ipynb and it works well, however, when I run model.finetune(...) does not show the loss of the model.

Any idea how I can solve it?

Besides, I add the output I got when I ran the fine-tuning:

INFO:root:system_path_file_exists:gs://t5-data/pretrained_models/base/operative_config.gin
ERROR:root:Path not found: gs://t5-data/pretrained_models/base/operative_config.gin
INFO:root:Skipping import of unknown module `t5.data.sentencepiece_vocabulary` (skip_unknown=True).
From /usr/local/lib/python3.7/dist-packages/tensorflow/python/training/training_util.py:397: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:absl:Using an uncached FunctionDataset for training is not recommended since it often results in insufficient shuffling on restarts, resulting in overfitting. It is highly recommended that you cache this task before training with it or use a data source that supports lower-level shuffling (e.g., FileDataSource).
INFO:absl:Load dataset info from gs://sql-alias/models/data/trivia_qa/unfiltered.nocontext/1.1.0
INFO:absl:Reusing dataset trivia_qa (gs://sql-alias/models/data/trivia_qa/unfiltered.nocontext/1.1.0)
INFO:absl:Constructing tf.data.Dataset trivia_qa for split train, from gs://sql-alias/models/data/trivia_qa/unfiltered.nocontext/1.1.0
From /usr/local/lib/python3.7/dist-packages/seqio/dataset_providers.py:1479: sample_from_datasets_v2 (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.sample_from_datasets(...)`.
SimdMeshImpl ignoring devices ['', '', '', '', '', '', '', '']
Using default tf glorot_uniform_initializer for variable encoder/block_000/layer_000/SelfAttention/relative_attention_bias  The initialzer will guess the input and output dimensions  based on dimension order.
Using default tf glorot_uniform_initializer for variable decoder/block_000/layer_000/SelfAttention/relative_attention_bias  The initialzer will guess the input and output dimensions  based on dimension order.
From /usr/local/lib/python3.7/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py:758: Variable.load (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Prefer Variable.assign which has equivalent behavior in 2.X.

Thank you

[antonio-mastropaolo]

Has anyone managed to solve this annoying problem?

[JessicaLopezEspejel]

Hello @antonio-mastropaolo,

Please add this line in your notebook: from tqdm.notebook import tqdm

For me it worked, let me know if it works for you.

[antonio-mastropaolo]

Hello @JessicaLopezEspejel

Thanks a lot for getting back to me.
Unfortunately I didn't solve it yet.

My code looks like this:

from tqdm.notebook import tqdm
TRAIN_STEPS =  100000#@param {type: "integer"}

with gin.unlock_config():
    gin.parse_config_file(LOCAL_GIN_PATH)
    model.finetune(mixture_or_task_name=MIXTURE_NAME,
               finetune_steps=TRAIN_STEPS,
               pretrained_model_dir=PRETRAIN_MODEL_DIR)

[adarob]

Can you provide more details? Is the model training? Are results showing up in the tensorboard in the previous cell?

[antonio-mastropaolo]

Hello @adarob

Thanks for chiming in

Yep, we are talking about the model training.
As for the tensorboard, I'm not using it

[coreyfournier]

I'm pretty sure it outputs it to graph.pbtxt and Tensorboard reads it and graphs the loss.