Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix metric calculation in examples and setup tests to run on multi-gpu for no_trainer scripts #17331

Merged
merged 10 commits into from
May 18, 2022
Merged

Fix metric calculation in examples and setup tests to run on multi-gpu for no_trainer scripts #17331

merged 10 commits into from
May 18, 2022

Conversation

muellerzr
Copy link
Contributor

What does this PR do?

This PR fixes all failing tests in a multi-gpu setting for all no_trainer example scripts. This includes an issue with when the logger was called before Accelerator() was created, adjusting when the conditional to add to the samples_seen, and adjusting samples_seen to use the length when the labels are just a list instead of a torch tensor.

Because the tests were rewritten to use accelerate launch and the new write_basic_config, these tests will automatically allocate themselves properly to test on multigpu, gpu, or cpu depending on what environment is available

Fixes # (issue):

Closes #17214, #17200

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@sgugger

@muellerzr muellerzr added Examples Which is related to examples in general bug labels May 18, 2022
@muellerzr muellerzr requested a review from sgugger May 18, 2022 17:45
@muellerzr muellerzr linked an issue May 18, 2022 that may be closed by this pull request
almost all codes that related to generation in examples/pytorch/**_no_trainer.py have bugs #17200
Closed
4 tasks
@muellerzr muellerzr marked this pull request as draft May 18, 2022 17:45
@muellerzr muellerzr marked this pull request as ready for review May 18, 2022 17:47
sgugger
sgugger approved these changes May 18, 2022
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM as long as the tests pass :-)
Thanks for fixing those examples!

@muellerzr muellerzr merged commit 1762ded into main May 18, 2022
@muellerzr muellerzr deleted the muellerzr-length-examples branch May 18, 2022 18:17
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented May 18, 2022 •
edited
Loading

The documentation is not available anymore as the PR was closed or merged.

ArthurZucker added a commit to ArthurZucker/transformers that referenced this pull request May 20, 2022
@ArthurZucker
Squashed commit of the following:
55b1309 
commit 5419205
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Thu May 19 23:46:26 2022 +0200

    [Test OPT] Add batch generation test opt (huggingface#17359)

    * up

    * up

commit 48c2269
Author: ddobokki <44228269+ddobokki@users.noreply.github.com>
Date:   Fri May 20 05:42:44 2022 +0900

    Fix bug in Wav2Vec2 pretrain example (huggingface#17326)

commit 5d6feec
Author: Nathan Dahlberg <58701810+nadahlberg@users.noreply.github.com>
Date:   Thu May 19 16:21:19 2022 -0400

    fix for 17292 (huggingface#17293)

commit 518bd02
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Thu May 19 22:17:02 2022 +0200

    [Generation] Fix Transition probs (huggingface#17311)

    * [Draft] fix transition probs

    * up

    * up

    * up

    * make it work

    * fix

    * finish

    * update

commit e8714c0
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Thu May 19 22:15:36 2022 +0200

    [OPT] Run test in lower precision on GPU (huggingface#17353)

    * [OPT] Run test only in half precision

    * up

    * up

    * up

    * up

    * finish

    * fix on GPU

    * Update tests/models/opt/test_modeling_opt.py

commit 2b28229
Author: Nicolas Patry <patry.nicolas@protonmail.com>
Date:   Thu May 19 20:28:12 2022 +0200

    Adding `batch_size` test to QA pipeline. (huggingface#17330)

commit a4386d7
Author: Nicolas Patry <patry.nicolas@protonmail.com>
Date:   Thu May 19 10:29:16 2022 +0200

    [BC] Fixing usage of text pairs (huggingface#17324)

    * [BC] Fixing usage of text pairs

    The BC is actually preventing users from misusing the pipeline since
    users could have been willing to send text pairs and the pipeline would
    instead understand the thing as a batch returning bogus results.

    The correct usage of text pairs is preserved in this PR even when that
    makes the code clunky.

    Adds support for {"text":..,, "text_pair": ...} inputs for both dataset
    iteration and more explicit usage to pairs.

    * Updating the doc.

    * Update src/transformers/pipelines/text_classification.py

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

    * Update src/transformers/pipelines/text_classification.py

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

    * Update tests/pipelines/test_pipelines_text_classification.py

    Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

    * quality.

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
    Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

commit 3601aa8
Author: Stas Bekman <stas00@users.noreply.github.com>
Date:   Wed May 18 16:00:47 2022 -0700

    [tests] fix copy-n-paste error (huggingface#17312)

    * [tests] fix copy-n-paste error

    * fix

commit 1b20c97
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Wed May 18 21:49:08 2022 +0200

    Fix ci_url might be None (huggingface#17332)

    * fix

    * Update utils/notification_service.py

    Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
    Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>

commit 6aad387
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Wed May 18 21:26:44 2022 +0200

    fix (huggingface#17337)

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

commit 1762ded
Author: Zachary Mueller <muellerzr@gmail.com>
Date:   Wed May 18 14:17:40 2022 -0400

    Fix metric calculation in examples and setup tests to run on multi-gpu for no_trainer scripts (huggingface#17331)

    * Fix length in no_trainer examples

    * Add setup and teardown

    * Use new accelerator config generator to automatically make tests able to run based on environment

commit 6e195eb
Author: Jader Martins <jadermcs@users.noreply.github.com>
Date:   Wed May 18 14:18:43 2022 -0300

    docs for typical decoding (huggingface#17186)

    Co-authored-by: Jader Martins <jadermcs94@gmail.com>

commit 060fe61
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Wed May 18 19:07:48 2022 +0200

    Not send successful report (huggingface#17329)

    * send report only if there is any failure

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

commit b3b9f99
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Wed May 18 17:57:23 2022 +0200

    Fix test_t5_decoder_model_past_large_inputs (huggingface#17320)

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

commit 6da76b9
Author: Jingya HUANG <44135271+JingyaHuang@users.noreply.github.com>
Date:   Wed May 18 17:52:13 2022 +0200

    Add onnx export cuda support (huggingface#17183)

    Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

    Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>

commit adc0ff2
Author: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Date:   Wed May 18 17:47:18 2022 +0200

    Add CvT (huggingface#17299)

    * Adding cvt files

    * Adding cvt files

    * changes in init file

    * Adding cvt files

    * changes in init file

    * Style fixes

    * Address comments from code review

    * Apply suggestions from code review

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

    * Format lists in docstring

    * Fix copies

    * Apply suggestion from code review

    Co-authored-by: AnugunjNaman <anugunjjha@gmail.com>
    Co-authored-by: Ayushman Singh <singhayushman13@protonmail.com>
    Co-authored-by: Niels Rogge <nielsrogge@Nielss-MacBook-Pro.local>
    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

commit 4710702
Author: Sylvain Gugger <Sylvain.gugger@gmail.com>
Date:   Wed May 18 10:46:40 2022 -0400

    Fix style

commit 5fdb54e
Author: mraunak <83710963+mraunak@users.noreply.github.com>
Date:   Wed May 18 10:39:02 2022 -0400

    Add Information Gain Filtration algorithm (huggingface#16953)

    * Add information gain filtration algorithm

    * Complying with black requirements

    * Added author

    * Fixed import order

    * flake8 corrections

    Co-authored-by: Javier Turek <javier.turek@intel.com>

commit 91ede48
Author: Kamal Raj <kamalraj97@gmail.com>
Date:   Wed May 18 19:59:53 2022 +0530

    Fix typo (huggingface#17328)

commit fe28eb9
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Wed May 18 16:06:41 2022 +0200

    remove (huggingface#17325)

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

commit 2cb2ea3
Author: Nicolas Patry <patry.nicolas@protonmail.com>
Date:   Wed May 18 16:06:24 2022 +0200

    Accepting real pytorch device as arguments. (huggingface#17318)

    * Accepting real pytorch device as arguments.

    * is_torch_available.

commit 1c9d1f4
Author: Nicolas Patry <patry.nicolas@protonmail.com>
Date:   Wed May 18 15:46:12 2022 +0200

    Updating the docs for `max_seq_len` in QA pipeline (huggingface#17316)

commit 60ad734
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Wed May 18 15:08:56 2022 +0200

    [T5] Fix init in TF and Flax for pretraining (huggingface#17294)

    * fix init

    * Apply suggestions from code review

    * fix

    * finish

    * Update src/transformers/modeling_tf_utils.py

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

commit 7ba1d4e
Author: Joaq <55513213+jQuinRivero@users.noreply.github.com>
Date:   Wed May 18 09:23:47 2022 -0300

    Add type hints for ProphetNet (Pytorch) (huggingface#17223)

    * added type hints to prophetnet

    * reformatted with black

    * fix bc black misformatted some parts

    * fix imports

    * fix imports

    * Update src/transformers/models/prophetnet/configuration_prophetnet.py

    Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>

    * update OPTIONAL type hint and docstring

    Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>

commit d6b8e9c
Author: Carl <carl.cochet@gmail.com>
Date:   Wed May 18 01:07:43 2022 +0200

    Add trajectory transformer (huggingface#17141)

    * Add trajectory transformer

    Fix model init

    Fix end of lines for .mdx files

    Add trajectory transformer model to toctree

    Add forward input docs

    Fix docs, remove prints, simplify prediction test

    Apply suggestions from code review

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
    Apply suggestions from code review

    Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
    Update docs, more descriptive comments

    Apply suggestions from code review

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
    Update readme

    Small comment update and add conversion script

    Rebase and reformat

    Fix copies

    Fix rebase, remove duplicates

    Fix rebase, remove duplicates

    * Remove tapex

    * Remove tapex

    * Remove tapex

commit c352640
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Wed May 18 00:34:31 2022 +0200

    fix (huggingface#17310)

commit d9050dc
Author: Cesare Campagnano <cesare.campagnano@gmail.com>
Date:   Tue May 17 23:44:37 2022 +0200

    [LED] fix global_attention_mask not being passed for generation and docs clarification about grad checkpointing (huggingface#17112)

    * [LED] fixed global_attention_mask not passed for generation + docs clarification for gradient checkpointing

    * LED docs clarification

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    * [LED] gradient_checkpointing=True should be passed to TrainingArguments

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    * [LED] docs: remove wrong word

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    * [LED] docs fix typo

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

commit bad3583
Author: Jean Vancoppenolle <jean.vcop@gmail.com>
Date:   Tue May 17 23:42:14 2022 +0200

    Add support for pretraining recurring span selection to Splinter (huggingface#17247)

    * Add SplinterForSpanSelection for pre-training recurring span selection.

    * Formatting.

    * Rename SplinterForSpanSelection to SplinterForPreTraining.

    * Ensure repo consistency

    * Fixup changes

    * Address SplinterForPreTraining PR comments

    * Incorporate feedback and derive multiple question tokens per example.

    * Update src/transformers/models/splinter/modeling_splinter.py

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    * Update src/transformers/models/splinter/modeling_splinter.py

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    Co-authored-by: Jean Vancoppenole <jean.vancoppenolle@retresco.de>
    Co-authored-by: Tobias Günther <tobias.guenther@retresco.de>
    Co-authored-by: Tobias Günther <github@tobigue.de>
    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

commit 0511305
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Tue May 17 18:56:58 2022 +0200

    Add PR author in CI report + merged by info (huggingface#17298)

    * Add author info to CI report

    * Add merged by info

    * update

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

commit 032d63b
Author: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Date:   Tue May 17 12:56:24 2022 -0400

    Fix dummy creation script (huggingface#17304)

commit 986dd5c
Author: Sylvain Gugger <Sylvain.gugger@gmail.com>
Date:   Tue May 17 12:50:14 2022 -0400

    Fix style

commit 38ddab1
Author: Karim Foda <35491698+KMFODA@users.noreply.github.com>
Date:   Tue May 17 09:32:12 2022 -0700

    Doctest longformer (huggingface#16441)

    * Add initial doctring changes

    * make fixup

    * Add TF doc changes

    * fix seq classifier output

    * fix quality errors

    * t

    * swithc head to random init

    * Fix expected outputs

    * Update src/transformers/models/longformer/modeling_longformer.py

    Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>

    Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>

commit 10704e1
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Tue May 17 18:20:36 2022 +0200

    [Test] Fix W2V-Conformer integration test (huggingface#17303)

    * [Test] Fix W2V-Conformer integration test

    * correct w2v2

    * up

commit 28a0811
Author: regisss <15324346+regisss@users.noreply.github.com>
Date:   Tue May 17 17:58:14 2022 +0200

    Improve mismatched sizes management when loading a pretrained model (huggingface#17257)

    - Add --ignore_mismatched_sizes argument to classification examples

    - Expand the error message when loading a model whose head dimensions are different from expected dimensions

commit 1f13ba8
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Tue May 17 15:48:23 2022 +0200

    correct opt (huggingface#17301)

commit 349f1c8
Author: Matt <Rocketknight1@users.noreply.github.com>
Date:   Tue May 17 14:36:23 2022 +0100

    Rewrite TensorFlow train_step and test_step (huggingface#17057)

    * Initial commit

    * Better label renaming

    * Remove breakpoint before pushing (this is your job)

    * Test a lot more in the Keras fit() test

    * make fixup

    * Clarify the case where we flatten y dicts into tensors

    * Clarify the case where we flatten y dicts into tensors

    * Extract label name remapping to a method

commit 651e48e
Author: Matt <Rocketknight1@users.noreply.github.com>
Date:   Tue May 17 14:14:17 2022 +0100

    Fix tests of mixed precision now that experimental is deprecated (huggingface#17300)

    * Fix tests of mixed precision now that experimental is deprecated

    * Fix mixed precision in training_args_tf.py too

commit 6d21142
Author: SaulLu <55560583+SaulLu@users.noreply.github.com>
Date:   Tue May 17 14:33:13 2022 +0200

    fix retribert's `test_torch_encode_plus_sent_to_model` (huggingface#17231)
elusenji pushed a commit to elusenji/transformers that referenced this pull request Jun 12, 2022
@muellerzr @elusenji
Fix metric calculation in examples and setup tests to run on multi-gp…
d7299e2 
…u for no_trainer scripts (huggingface#17331)

* Fix length in no_trainer examples

* Add setup and teardown

* Use new accelerator config generator to automatically make tests able to run based on environment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sgugger sgugger sgugger approved these changes

Assignees
No one assigned
Labels
bug Examples Which is related to examples in general
Projects
None yet
Milestone
No milestone
3 participants
@muellerzr @HuggingFaceDocBuilderDev @sgugger