Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add musicgen melody #2

Merged
merged 188 commits into from
Mar 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
188 commits
Select commit Hold shift + click to select a range
5f96855
Add npu device for pipeline (#28885)
statelesshz Feb 7, 2024
328ade8
[Docs] Fix placement of tilde character (#28913)
khipp Feb 8, 2024
33df036
[Docs] Revert translation of '@slow' decorator (#28912)
khipp Feb 8, 2024
4b236ae
Fix utf-8 yaml load for marian conversion to pytorch in Windows (#28618)
SystemPanic Feb 8, 2024
115ac94
[`Core generation`] Adds support for static KV cache (#27931)
ArthurZucker Feb 8, 2024
693667b
Remove dead TF loading code (#28926)
Rocketknight1 Feb 8, 2024
0b693e9
fix: torch.int32 instead of torch.torch.int32 (#28883)
vodkaslime Feb 8, 2024
cc309fd
pass kwargs in stopping criteria list (#28927)
zucchini-nlp Feb 8, 2024
d628664
Support batched input for decoder start ids (#28887)
zucchini-nlp Feb 8, 2024
2749e47
[Docs] Fix broken links and syntax issues (#28918)
khipp Feb 8, 2024
de11e65
Fix max_position_embeddings default value for llama2 to 4096 #28241 (…
karl-hajjar Feb 9, 2024
ebf3ea2
Fix a wrong link to CONTRIBUTING.md section in PR template (#28941)
B-Step62 Feb 9, 2024
d123e66
Fix type annotations on neftune_noise_alpha and fsdp_config TrainingA…
peblair Feb 9, 2024
58e3d23
[i18n-de] Translate README.md to German (#28933)
khipp Feb 9, 2024
f278ef2
[Nougat] Fix pipeline (#28242)
NielsRogge Feb 12, 2024
ef5ab72
[Docs] Update README and default pipelines (#28864)
NielsRogge Feb 12, 2024
cf4c20b
Convert `torch_dtype` as `str` to actual torch data type (i.e. "float…
KossaiSbai Feb 12, 2024
1709886
[`pipelines`] updated docstring with vqa alias (#28951)
cmahmut Feb 12, 2024
e30bbb2
Tests: tag `test_save_load_fast_init_from_base` as flaky (#28930)
gante Feb 12, 2024
792819f
Updated requirements for image-classification samples: datasets>=2.14…
alekseyfa Feb 12, 2024
136cd89
Always initialize tied output_embeddings if it has a bias term (#28947)
hackyon Feb 12, 2024
c617f98
Clean up staging tmp checkpoint directory (#28848)
woshiyyya Feb 12, 2024
fe3df9d
[Docs] Add language identifiers to fenced code blocks (#28955)
khipp Feb 12, 2024
78ba9f4
[Docs] Add video section (#28958)
NielsRogge Feb 12, 2024
d90acc1
[i18n-de] Translate CONTRIBUTING.md to German (#28954)
khipp Feb 12, 2024
b445675
[`NllbTokenizer`] refactor with added tokens decoder (#27717)
ArthurZucker Feb 13, 2024
da20209
Add sudachi_projection option to BertJapaneseTokenizer (#28503)
hiroshi-matsuda-rit Feb 13, 2024
3e70a20
Static Cache: load models with MQA or GQA (#28975)
gante Feb 13, 2024
3de6a6b
Update configuration_llama.py: fixed broken link (#28946)
AdityaKane2001 Feb 13, 2024
bd4b83e
[`DETR`] Update the processing to adapt masks & bboxes to reflect pad…
amyeroberts Feb 13, 2024
1d12b8b
ENH: Do not pass warning message in case `quantization_config` is in …
younesbelkada Feb 14, 2024
164bdef
ENH [`AutoQuantizer`]: enhance trainer + not supported quant methods …
younesbelkada Feb 14, 2024
de6029a
Add `StableLM` (#28810)
jon-tow Feb 14, 2024
63ffd56
Add SiglipForImageClassification and CLIPForImageClassification (#28952)
NielsRogge Feb 14, 2024
1ecf5f7
AQLM quantizer support (#28928)
Feb 14, 2024
7252e8d
[`Doc`] Fix docbuilder - make `BackboneMixin` and `BackboneConfigMixi…
amyeroberts Feb 14, 2024
69ca640
Set the dataset format used by `test_trainer` to float32 (#28920)
statelesshz Feb 14, 2024
0507e69
Introduce AcceleratorConfig dataclass (#28664)
muellerzr Feb 14, 2024
354775b
Fix flaky test vision encoder-decoder generate (#28923)
zucchini-nlp Feb 14, 2024
3f4e79d
Mask Generation Task Guide (#28897)
merveenoyan Feb 14, 2024
725f4ad
Add tie_weights() to LM heads and set bias in set_output_embeddings()…
hackyon Feb 14, 2024
0199a48
Backbone kwargs in config (#28784)
amyeroberts Feb 14, 2024
5f06053
[TPU] Support PyTorch/XLA FSDP via SPMD (#28949)
alanwaketan Feb 14, 2024
7a0fccc
FIX [`Trainer` / tags]: Fix trainer + tags when users do not pass `"t…
younesbelkada Feb 14, 2024
609a176
[`CLeanup`] Revert SDPA attention changes that got in the static kv c…
ArthurZucker Feb 14, 2024
f3788b0
Fix static generation when compiling! (#28937)
ArthurZucker Feb 15, 2024
83e96dc
Add cuda_custom_kernel in DETA (#28989)
SangbumChoi Feb 15, 2024
5b6fa23
DeformableDetrModel support fp16 (#29013)
DonggeunYu Feb 15, 2024
8a0ed0a
Fix copies between DETR and DETA (#29037)
amyeroberts Feb 15, 2024
6d1f545
FIX: Fix error with `logger.warning` + inline with recent refactor (#…
younesbelkada Feb 15, 2024
4156f51
Patch to skip failing `test_save_load_low_cpu_mem_usage` tests (#29043)
amyeroberts Feb 15, 2024
b0a7f44
Removed obsolete attribute setting for AQLM quantization. (#29034)
Feb 15, 2024
f3aa7db
Fix a tiny typo in `generation/utils.py::GenerateEncoderDecoderOutput…
sadra-barikbin Feb 15, 2024
1e402b9
add test marker to run all tests with @require_bitsandbytes (#28278)
Titus-von-Koeller Feb 16, 2024
f497f56
Update all references to canonical models (#29001)
LysandreJik Feb 16, 2024
8876ce8
Update important model list (#29019)
LysandreJik Feb 16, 2024
aee11fe
Fix max_length criteria when using inputs_embeds (#28994)
zucchini-nlp Feb 16, 2024
0eb4085
Support : Leverage Accelerate for object detection/segmentation model…
Tanmaypatil123 Feb 16, 2024
258da40
fix num_assistant_tokens with heuristic schedule (#28759)
jmamou Feb 16, 2024
b262808
fix failing trainer ds tests (#29057)
pacman100 Feb 16, 2024
4c18ddb
`auto_find_batch_size` isn't yet supported with DeepSpeed/FSDP. Raise…
pacman100 Feb 16, 2024
be42c24
Honor trust_remote_code for custom tokenizers (#28854)
rl337 Feb 16, 2024
161fe42
Feature: Option to set the tracking URI for MLflowCallback. (#29032)
seanswyi Feb 16, 2024
636b032
Fix trainer test wrt DeepSpeed + auto_find_bs (#29061)
muellerzr Feb 16, 2024
2f1003b
Add chat support to text generation pipeline (#28945)
Rocketknight1 Feb 16, 2024
ce4fff0
[Docs] Spanish translation of task_summary.md (#28844)
aaronjimv Feb 16, 2024
864c8e6
[`Awq`] Add peft support for AWQ (#28987)
younesbelkada Feb 19, 2024
a75a6c9
FIX [`bnb` / `tests`]: Fix currently failing bnb tests (#29092)
younesbelkada Feb 19, 2024
593230f
fix the post-processing link (#29091)
davies-w Feb 19, 2024
9830858
Fix the `bert-base-cased` tokenizer configuration test (#29105)
LysandreJik Feb 19, 2024
79132d4
Fix a typo in `examples/pytorch/text-classification/run_classificatio…
Ja1Zhou Feb 19, 2024
b2724d7
change version (#29097)
ArthurZucker Feb 19, 2024
07e3454
[Docs] Add resources (#28705)
NielsRogge Feb 19, 2024
08cd694
ENH: added new output_logits option to generate function (#28667)
mbaak Feb 19, 2024
5ce90f3
Bnb test fix for different hardwares (#29066)
Titus-von-Koeller Feb 19, 2024
a4851d9
Fix two tiny typos in `pipelines/base.py::Pipeline::_sanitize_paramet…
sadra-barikbin Feb 19, 2024
4f09d0f
storing & logging gradient norm in trainer (#27326)
shijie-wu Feb 19, 2024
49c0b29
Fixed nll with label_smoothing to just nll (#28708)
nileshkokane01 Feb 20, 2024
9094abe
[`gradient_checkpointing`] default to use it for torch 2.3 (#28538)
ArthurZucker Feb 20, 2024
a7ff2f2
Move misplaced line (#29117)
kno10 Feb 20, 2024
f7ef7ce
FEAT [`Trainer` / `bnb`]: Add RMSProp from `bitsandbytes` to HF `Trai…
younesbelkada Feb 20, 2024
1c9134f
Abstract image processor arg checks. (#28843)
molbap Feb 20, 2024
ff76e7c
FIX [`bnb` / `tests`] Propagate the changes from #29092 to 4-bit test…
younesbelkada Feb 20, 2024
7d312ad
Llama: fix batched generation (#29109)
gante Feb 20, 2024
a7755d2
Generate: unset GenerationConfig parameters do not raise warning (#29…
gante Feb 20, 2024
5e95dca
[`cuda kernels`] only compile them when initializing (#29133)
ArthurZucker Feb 20, 2024
efdd436
FIX [`PEFT` / `Trainer` ] Handle better peft + quantized compiled mod…
younesbelkada Feb 20, 2024
15cfe38
[`Core tokenization`] `add_dummy_prefix_space` option to help with l…
ArthurZucker Feb 20, 2024
0996a10
Revert low cpu mem tie weights (#29135)
amyeroberts Feb 20, 2024
ee3af60
Add support for fine-tuning CLIP-like models using contrastive-image-…
tjs-intel Feb 20, 2024
7688d8d
Save (circleci) cache at the end of a job (#29141)
ydshieh Feb 20, 2024
b8b1647
[Phi] Add support for sdpa (#29108)
hackyon Feb 20, 2024
1c81132
Raise unused kwargs image processor (#29063)
molbap Feb 20, 2024
857fd8e
Generate: missing generation config eos token setting in encoder-deco…
gante Feb 20, 2024
3c00b88
Added image_captioning version in es and included in toctree file (#2…
gisturiz Feb 20, 2024
c47576c
Fix drop path being ignored in DINOv2 (#29147)
fepegar Feb 20, 2024
e770f03
[`pipeline`] Add pool option to image feature extraction pipeline (#2…
amyeroberts Feb 20, 2024
1a77f07
v4.39.dev.0
ArthurZucker Feb 21, 2024
3994fa5
🚨 Llama: update rope scaling to match static cache changes (#29143)
gante Feb 21, 2024
cc4a664
`torch.compile` compatibility with `generate` + static cache (#29114)
fxmarty Feb 21, 2024
1d0ea7a
support SDPA Attention in stablelm (#29106)
eaidova Feb 21, 2024
58245ba
[`Maskformer`] safely get backbone config (#29166)
amyeroberts Feb 21, 2024
594c127
[ `gemma`] Adds support for Gemma 💎 (#29167)
ArthurZucker Feb 21, 2024
ae49b21
FIX [`Gemma`] Fix bad rebase with transformers main (#29170)
younesbelkada Feb 21, 2024
fc37f38
Add training version check for AQLM quantizer. (#29142)
BlackSamorez Feb 21, 2024
2a9b1f8
[Gemma] Fix eager attention (#29187)
sanchit-gandhi Feb 22, 2024
dabe855
[Mistral, Mixtral] Improve docs (#29084)
NielsRogge Feb 22, 2024
2cc8cf6
Fix `torch.compile` with `fullgraph=True` when `attention_mask` input…
fxmarty Feb 22, 2024
4524494
fix(mlflow): check mlflow version to use the synchronous flag (#29195)
cchen-dialpad Feb 23, 2024
75ed76e
Fix missing translation in README_ru (#29054)
strikoder Feb 23, 2024
3f60d11
Improve _update_causal_mask performance (#29210)
alessandropalla Feb 23, 2024
89c6481
[`Doc`] update model doc qwen2 (#29238)
ArthurZucker Feb 23, 2024
371b572
Allow remote code repo names to contain "." (#29175)
Rocketknight1 Feb 23, 2024
c8d9840
Use torch 2.2 for daily CI (model tests) (#29208)
ydshieh Feb 23, 2024
9fe3608
Cache `is_vision_available` result (#29280)
bmuskalla Feb 26, 2024
93f8617
Use `DS_DISABLE_NINJA=1` (#29290)
ydshieh Feb 26, 2024
2a7746c
Add `non_device_test` pytest mark to filter out non-device tests (#29…
fxmarty Feb 26, 2024
7c4995f
Add feature extraction mapping for automatic metadata update (#28944)
merveenoyan Feb 26, 2024
755960a
Merge branch 'main' into add-musicgen-melody
ylacombe Feb 26, 2024
24d59c7
Use `torch.bool` instead of `torch.int64` for non-persistant causal m…
fxmarty Feb 26, 2024
ece1b62
Generate: v4.38 removals and related updates (#29171)
gante Feb 26, 2024
8f2f0f0
Track each row separately for stopping criteria (#29116)
zucchini-nlp Feb 26, 2024
9f7535b
[docs] Spanish translation of tasks_explained.md (#29224)
aaronjimv Feb 26, 2024
b433404
[i18n-zh] Translated torchscript.md into Chinese (#29234)
windsonsea Feb 26, 2024
734eb25
🌐 [i18n-ZH] Translate chat_templating.md into Chinese (#28790)
shibing624 Feb 26, 2024
c291350
[i18n-vi] Translate README.md to Vietnamese (#29229)
hoangsvit Feb 26, 2024
a44d2dc
[i18n-zh] Translated task/asr.md into Chinese (#29233)
windsonsea Feb 26, 2024
3b8c053
Fixed Deformable Detr typo when loading cuda kernels for MSDA (#29294)
EduardoPach Feb 26, 2024
3fcfbe7
Adding SegGPT (#27735)
EduardoPach Feb 26, 2024
871ba71
GenerationConfig validate both constraints and force_words_ids (#29163)
FredericOdermatt Feb 27, 2024
a3f9221
Add generate kwargs to VQA pipeline (#29134)
regisss Feb 27, 2024
e3fc90a
Cleaner Cache `dtype` and `device` extraction for CUDA graph generati…
BlackSamorez Feb 27, 2024
83e366b
Image Feature Extraction docs (#28973)
merveenoyan Feb 27, 2024
6d3b643
Fix `attn_implementation` documentation (#29295)
fxmarty Feb 27, 2024
63a0c8f
[tests] enable benchmark unit tests on XPU (#29284)
faaany Feb 27, 2024
5c341d4
Use torch 2.2 for deepspeed CI (#29246)
ydshieh Feb 27, 2024
8a1faf2
Add compatibility with skip_memory_metrics for mps device (#29264)
SunMarc Feb 27, 2024
ddf7ac4
Token level timestamps for long-form generation in Whisper (#29148)
zucchini-nlp Feb 27, 2024
227cd54
Fix a few typos in `GenerationMixin`'s docstring (#29277)
sadra-barikbin Feb 27, 2024
83ab011
[i18n-zh] Translate fsdp.md into Chinese (#29305)
windsonsea Feb 27, 2024
63caa37
Starcoder2 model - bis (#29215)
RaymondLi0 Feb 28, 2024
bd5b986
simplify get_class_in_module and fix for paths containing a dot (#29262)
cebtenzzre Feb 28, 2024
ad00c48
FIX [`Gemma` / `CI`] Make sure our runners have access to the model (…
younesbelkada Feb 28, 2024
e715c78
Remove numpy usage from owlvit (#29326)
fxmarty Feb 28, 2024
a528885
[`require_read_token`] fix typo (#29345)
ArthurZucker Feb 28, 2024
7c87f35
[`T5 and Llama Tokenizer`] remove warning (#29346)
ArthurZucker Feb 28, 2024
8a8a0a4
[`Llama ROPE`] Fix torch export but also slow downs in forward (#29198)
ArthurZucker Feb 28, 2024
2ce56d3
Disable Mixtral `output_router_logits` during inference (#29249)
LeonardoEmili Feb 28, 2024
7628b3a
Idefics: generate fix (#29320)
gante Feb 28, 2024
d3a4b47
RoPE loses precision for Llama / Gemma + Gemma logits.float() (#29285)
danielhanchen Feb 28, 2024
554e7ad
check if position_ids exists before using it (#29306)
jiqing-feng Feb 28, 2024
f54d82c
[CI] Quantization workflow (#29046)
SunMarc Feb 28, 2024
49204c1
Better SDPA unmasking implementation (#29318)
fxmarty Feb 28, 2024
2209b7a
[i18n-zh] Sync source/zh/index.md (#29331)
windsonsea Feb 28, 2024
1aee9af
FIX [`CI` / `starcoder2`] Change starcoder2 path to correct one for s…
younesbelkada Feb 29, 2024
8d8ac9c
FIX [`CI`]: Fix failing tests for peft integration (#29330)
younesbelkada Feb 29, 2024
b647acd
FIX [`CI`] `require_read_token` in the llama FA2 test (#29361)
younesbelkada Feb 29, 2024
44fe1a1
Avoid using uncessary `get_values(MODEL_MAPPING)` (#29362)
ydshieh Feb 29, 2024
bb4f816
Patch YOLOS and others (#29353)
NielsRogge Feb 29, 2024
0ad770c
Fix @require_read_token in tests (#29367)
Wauplin Feb 29, 2024
5ee0868
Expose `offload_buffers` parameter of `accelerate` to `PreTrainedMode…
notsyncing Mar 1, 2024
2858d6c
Fix Base Model Name of LlamaForQuestionAnswering (#29258)
lenglaender Mar 1, 2024
50db7ca
FIX [`quantization` / `ESM`] Fix ESM 8bit / 4bit with bitsandbytes (#…
younesbelkada Mar 1, 2024
e7b9837
[`Llama + AWQ`] fix `prepare_inputs_for_generation` 🫠 (#29381)
ArthurZucker Mar 1, 2024
0a0a279
🚨🚨[Whisper Tok] Update integration test (#29368)
sanchit-gandhi Mar 1, 2024
f1b1379
[`YOLOS`] Fix - return padded annotations (#29300)
amyeroberts Mar 1, 2024
15f8296
Support subfolder with `AutoProcessor` (#29169)
JingyaHuang Mar 1, 2024
cec7733
Fix llama + gemma accelete tests (#29380)
SunMarc Mar 1, 2024
1a7c117
Fix deprecated arg issue (#29372)
muellerzr Mar 1, 2024
831bc25
Correct zero division error in inverse sqrt scheduler (#28982)
DavidAfonsoValente Mar 1, 2024
aade711
[tests] enable automatic speech recognition pipeline tests on XPU (#…
faaany Mar 4, 2024
38953a7
update path to hub files in the error message (#29369)
poedator Mar 4, 2024
39ef3fb
[Mixtral] Fixes attention masking in the loss (#29363)
DesmonDay Mar 4, 2024
704b3f7
Add mlx support to BatchEncoding.convert_to_tensors (#29406)
tidely Mar 4, 2024
c38a122
Workaround for #27758 to avoid ZeroDivisionError (#28756)
tleyden Mar 4, 2024
5e4b69d
Convert SlimSAM checkpoints (#28379)
NielsRogge Mar 4, 2024
81220cb
Fix: Fixed the previous tracking URI setting logic to prevent clashes…
seanswyi Mar 4, 2024
8ef9862
Fix OneFormer `post_process_instance_segmentation` for panoptic tasks…
nickthegroot Mar 4, 2024
1681a6d
🚨 Fully revert atomic checkpointing 🚨 (#29370)
muellerzr Mar 4, 2024
7941769
Fix grad_norm unserializable tensor log failure (#29212)
svenschultze Mar 4, 2024
bcd23a5
Avoid edge case in audio utils (#28836)
ylacombe Mar 4, 2024
8b9177f
Merge branch 'main' into add-musicgen-melody
ylacombe Mar 4, 2024
ad26dc9
replace musicgen melody checkpoints with official orga
ylacombe Mar 4, 2024
7595256
rename ylacombe->facebook in checkpoints
ylacombe Mar 4, 2024
2576806
fix copies
ylacombe Mar 4, 2024
379d70b
remove unecessary warning
ylacombe Mar 4, 2024
9795c6f
add shape in code docstrings
ylacombe Mar 4, 2024
b03b36d
add files to slow doc tests
ylacombe Mar 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
37 changes: 20 additions & 17 deletions .circleci/create_circleci_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ class CircleCIJob:
name: str
additional_env: Dict[str, Any] = None
cache_name: str = None
cache_version: str = "0.8"
cache_version: str = "0.8.2"
docker_image: List[Dict[str, str]] = None
install_steps: List[str] = None
marker: Optional[str] = None
Expand Down Expand Up @@ -128,22 +128,6 @@ def to_dict(self):
steps.extend([{"run": l} for l in self.install_steps])
steps.extend([{"run": 'pip install "fsspec>=2023.5.0,<2023.10.0"'}])
steps.extend([{"run": "pip install pytest-subtests"}])
steps.append(
{
"save_cache": {
"key": f"v{self.cache_version}-{self.cache_name}-{cache_branch_prefix}-pip-" + '{{ checksum "setup.py" }}',
"paths": ["~/.cache/pip"],
}
}
)
steps.append(
{
"save_cache": {
"key": f"v{self.cache_version}-{self.cache_name}-{cache_branch_prefix}-site-packages-" + '{{ checksum "setup.py" }}',
"paths": ["~/.pyenv/versions/"],
}
}
)
steps.append({"run": {"name": "Show installed libraries and their versions", "command": "pip freeze | tee installed.txt"}})
steps.append({"store_artifacts": {"path": "~/transformers/installed.txt"}})

Expand Down Expand Up @@ -264,6 +248,25 @@ def to_dict(self):

steps.append({"store_artifacts": {"path": "~/transformers/tests_output.txt"}})
steps.append({"store_artifacts": {"path": "~/transformers/reports"}})

# save cache at the end: so pytest step runs before cache saving and we can see results earlier
steps.append(
{
"save_cache": {
"key": f"v{self.cache_version}-{self.cache_name}-{cache_branch_prefix}-pip-" + '{{ checksum "setup.py" }}',
"paths": ["~/.cache/pip"],
}
}
)
steps.append(
{
"save_cache": {
"key": f"v{self.cache_version}-{self.cache_name}-{cache_branch_prefix}-site-packages-" + '{{ checksum "setup.py" }}',
"paths": ["~/.pyenv/versions/"],
}
}
)

job["steps"] = steps
return job

Expand Down
2 changes: 1 addition & 1 deletion .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
to it if that's the case.
Expand Down
40 changes: 29 additions & 11 deletions .github/workflows/build-docker-images.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,18 +20,8 @@ concurrency:
jobs:
latest-docker:
name: "Latest PyTorch + TensorFlow [dev]"
runs-on: ubuntu-22.04
runs-on: [intel-cpu, 8-cpu, ci]
steps:
- name: Cleanup disk
run: |
sudo ls -l /usr/local/lib/
sudo ls -l /usr/share/
sudo du -sh /usr/local/lib/
sudo du -sh /usr/share/
sudo rm -rf /usr/local/lib/android
sudo rm -rf /usr/share/dotnet
sudo du -sh /usr/local/lib/
sudo du -sh /usr/share/
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
Expand Down Expand Up @@ -307,3 +297,31 @@ jobs:
# REF=main
# push: true
# tags: huggingface/transformers-pytorch-deepspeed-amd-gpu-push-ci

latest-quantization-torch-docker:
name: "Latest Pytorch + Quantization [dev]"
# Push CI doesn't need this image
if: inputs.image_postfix != '-push-ci'
runs-on: [intel-cpu, 8-cpu, ci]
steps:
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
-
name: Check out code
uses: actions/checkout@v3
-
name: Login to DockerHub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_PASSWORD }}
-
name: Build and push
uses: docker/build-push-action@v5
with:
context: ./docker/transformers-quantization-latest-gpu
build-args: |
REF=main
push: true
tags: huggingface/transformers-quantization-latest-gpu${{ inputs.image_postfix }}
56 changes: 54 additions & 2 deletions .github/workflows/self-scheduled.yml
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@ jobs:
working-directory: /workspace
run: |
python3 -m pip uninstall -y deepspeed
DS_BUILD_CPU_ADAM=1 DS_BUILD_FUSED_ADAM=1 python3 -m pip install deepspeed --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check
DS_DISABLE_NINJA=1 DS_BUILD_CPU_ADAM=1 DS_BUILD_FUSED_ADAM=1 python3 -m pip install deepspeed --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check

- name: NVIDIA-SMI
run: |
Expand Down Expand Up @@ -297,6 +297,56 @@ jobs:
name: ${{ matrix.machine_type }}_run_tests_torch_cuda_extensions_gpu_test_reports
path: /workspace/transformers/reports/${{ matrix.machine_type }}_tests_torch_cuda_extensions_gpu

run_tests_quantization_torch_gpu:
name: Quantization tests
strategy:
fail-fast: false
matrix:
machine_type: [single-gpu, multi-gpu]
runs-on: ['${{ matrix.machine_type }}', nvidia-gpu, t4, daily-ci]
container:
image: huggingface/transformers-quantization-latest-gpu
options: --gpus all --shm-size "16gb" --ipc host -v /mnt/cache/.cache/huggingface:/mnt/cache/
needs: setup
steps:
- name: Update clone
working-directory: /transformers
run: git fetch && git checkout ${{ github.sha }}

- name: Reinstall transformers in edit mode (remove the one installed during docker image build)
working-directory: /transformers
run: python3 -m pip uninstall -y transformers && python3 -m pip install -e .

- name: NVIDIA-SMI
run: |
nvidia-smi

- name: Environment
working-directory: /transformers
run: |
python3 utils/print_env.py

- name: Show installed libraries and their versions
working-directory: /transformers
run: pip freeze

- name: Run quantization tests on GPU
working-directory: /transformers
run: |
python3 -m pytest -v --make-reports=${{ matrix.machine_type }}_tests_quantization_torch_gpu tests/quantization

- name: Failure short reports
if: ${{ failure() }}
continue-on-error: true
run: cat /transformers/reports/${{ matrix.machine_type }}_tests_quantization_torch_gpu/failures_short.txt

- name: "Test suite reports artifacts: ${{ matrix.machine_type }}_run_tests_quantization_torch_gpu"
if: ${{ always() }}
uses: actions/upload-artifact@v3
with:
name: ${{ matrix.machine_type }}_run_tests_quantization_torch_gpu
path: /transformers/reports/${{ matrix.machine_type }}_tests_quantization_torch_gpu

run_extract_warnings:
name: Extract warnings in CI artifacts
runs-on: ubuntu-22.04
Expand All @@ -307,7 +357,8 @@ jobs:
run_examples_gpu,
run_pipelines_tf_gpu,
run_pipelines_torch_gpu,
run_all_tests_torch_cuda_extensions_gpu
run_all_tests_torch_cuda_extensions_gpu,
run_tests_quantization_torch_gpu,
]
steps:
- name: Checkout transformers
Expand Down Expand Up @@ -355,6 +406,7 @@ jobs:
run_pipelines_tf_gpu,
run_pipelines_torch_gpu,
run_all_tests_torch_cuda_extensions_gpu,
run_tests_quantization_torch_gpu,
run_extract_warnings
]
steps:
Expand Down
20 changes: 10 additions & 10 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ We have added [templates](https://github.com/huggingface/transformers/tree/main/

## Do you want to implement a new model?

New models are constantly released and if you want to implement a new model, please provide the following information
New models are constantly released and if you want to implement a new model, please provide the following information:

* A short description of the model and a link to the paper.
* Link to the implementation if it is open-sourced.
Expand All @@ -129,7 +129,7 @@ You will need basic `git` proficiency to contribute to
manual. Type `git --help` in a shell and enjoy! If you prefer books, [Pro
Git](https://git-scm.com/book/en/v2) is a very good reference.

You'll need **[Python 3.8]((https://github.com/huggingface/transformers/blob/main/setup.py#L426))** or above to contribute to 🤗 Transformers. Follow the steps below to start contributing:
You'll need **[Python 3.8](https://github.com/huggingface/transformers/blob/main/setup.py#L426)** or above to contribute to 🤗 Transformers. Follow the steps below to start contributing:

1. Fork the [repository](https://github.com/huggingface/transformers) by
clicking on the **[Fork](https://github.com/huggingface/transformers/fork)** button on the repository's page. This creates a copy of the code
Expand Down Expand Up @@ -305,7 +305,7 @@ the [tests](https://github.com/huggingface/transformers/tree/main/tests) folder
[examples](https://github.com/huggingface/transformers/tree/main/examples) folder.

We like `pytest` and `pytest-xdist` because it's faster. From the root of the
repository, specify a *path to a subfolder or a test file* to run the test.
repository, specify a *path to a subfolder or a test file* to run the test:

```bash
python -m pytest -n auto --dist=loadfile -s -v ./tests/models/my_new_model
Expand Down Expand Up @@ -377,7 +377,7 @@ One way to run the `make` command on Windows is with MSYS2:
3. Run in the shell: `pacman -Syu` and install `make` with `pacman -S make`.
4. Add `C:\msys64\usr\bin` to your PATH environment variable.

You can now use `make` from any terminal (Powershell, cmd.exe, etc.)! 🎉
You can now use `make` from any terminal (PowerShell, cmd.exe, etc.)! 🎉

### Sync a forked repository with upstream main (the Hugging Face repository)

Expand All @@ -386,9 +386,9 @@ When updating the main branch of a forked repository, please follow these steps
1. When possible, avoid syncing with the upstream using a branch and PR on the forked repository. Instead, merge directly into the forked main.
2. If a PR is absolutely necessary, use the following steps after checking out your branch:

```bash
git checkout -b your-branch-for-syncing
git pull --squash --no-commit upstream main
git commit -m '<your message without GitHub references>'
git push --set-upstream origin your-branch-for-syncing
```
```bash
git checkout -b your-branch-for-syncing
git pull --squash --no-commit upstream main
git commit -m '<your message without GitHub references>'
git push --set-upstream origin your-branch-for-syncing
```
Loading
Loading