Pr nncf jpqd with bert example only #2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…nd distillation via NNCF
#2) * use nncf official branch for install since JPQD is merged * copy ac scripts from transformer repo * init commit for wav2vec2 * add onnx_config argument in OVTrainer for onnx export with unsupported model * enable customized teancher kd * add readme * delete debugging lines
* test part 1 * clean "compute_distillation_loss' in OVTrainer * add test of OVTrainer for int8+kd / movement / movement+int8/ movement+int8+kd * add expectedFailuremark to test of OVModelForAudioClassification * revert unncessary codes about "OVModelForAudioClassification" * change to a shorter train for w2v2 in readme * revert compute_metrics change since it is not unnecessary * fix task_loss non-scalar bug for kd logging * make regex clearer in QA bert config
* MO IR pruning depends on scheduler stage * Readme tweaks for all example tasks * Minor tweak on tests * Align setup.py for openvino-dev and nncf versions needed for JPQD
* copy text-classification example from transformers * init draft for sst example * update sst2 accuracy & training time
… quantization config
…oint compression variations, misc. patches * use builtin onnx configs for wav2vec onnx export * move teacher model argument from OVTrainingArgs to model args * fix duplicate call of `epoch_step` * temporal workaround about compression metrics * test for all training cases * temporal workaround for eval only * cover train/eval tests * style fix
…training loss check (#6) * removing old tests in test_quantization since they are now in `test_training` * bugfix in checking compression metrics during training
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
yujiepan-work
force-pushed
the
pr-nncf-jpqd-with-bert-example-only
branch
from
fbb18af to
e2cbed4
Compare
yujiepan-work
added a commit
that referenced
this pull request
Feb 24, 2023
…uggingface#150) * Initial commit to enable OVTrainer with joint pruning, quantization and distillation via NNCF * Review OpenVINO Q&A readme and configs * Update README.md * Add post init value checker to OVTrainingArguments * Initial enabling of audio classification/wav2vec2 [tests not included] (#2) * use nncf official branch for install since JPQD is merged * copy ac scripts from transformer repo * init commit for wav2vec2 * add onnx_config argument in OVTrainer for onnx export with unsupported model * enable customized teancher kd * add readme * delete debugging lines * Update openvino-dev and nncf version in setup.py * refactor _enable_standard_onnx_export_option to _set_standard_onnx_export_option * add tests for (movement/quantization) with distillation (#3) * test part 1 * clean "compute_distillation_loss' in OVTrainer * add test of OVTrainer for int8+kd / movement / movement+int8/ movement+int8+kd * add expectedFailuremark to test of OVModelForAudioClassification * revert unncessary codes about "OVModelForAudioClassification" * change to a shorter train for w2v2 in readme * revert compute_metrics change since it is not unnecessary * fix task_loss non-scalar bug for kd logging * make regex clearer in QA bert config * Refactor compression-related logging * Refactor OpenVINO IR generation and patch tests * Miscellaneous refactoring * MO IR pruning depends on scheduler stage * Readme tweaks for all example tasks * Minor tweak on tests * Align setup.py for openvino-dev and nncf versions needed for JPQD * Fix lint with Black * Refactor OpenVINO IR generation using python api * Fix via isort * Handle IR generation error to avoid run termination * Update QA readme * Enable distillation on openvino's image classification example * Minor refactoring in openvino's audio classification example * Move openvino-dev dependency to be extra of NNCF * Configure IR model to accept dynamic-shaped input * Revert _enable_standard_onnx_export_option method in OVConfig * Update wav2vec2 configs for audio classification * Add BERT-base/glue-sst2 example with QAT / JPQD (#4) * copy text-classification example from transformers * init draft for sst example * update sst2 accuracy & training time * Revise wav2vec2 config and audio classification readme * Patch _enable_standard_onnx_export_option to only add the key pair to quantization config * Set logging level to INFO in openvino/trainer.py * Review readme of text and image classification * Revert IR generation with static input shape for joint compression * Add distillation and advance optimization section in optimization_ov.mdx * Patch tests * Revise formatting of optimization_ov.mdx * Limit #checkpoint saved for JPQD samples * Handle NNCF output to text log and only print errors to stdout * Replace hardcoded model.onnx filename with constant variable * Fix movement sparsity config in optimization_ov.mdx * Change _set_feature to _set_task to align with OVQuantizer * Revert onnx_config exposure in OVTrainer, expand test coverages for joint compression variations, misc. patches * use builtin onnx configs for wav2vec onnx export * move teacher model argument from OVTrainingArgs to model args * fix duplicate call of `epoch_step` * temporal workaround about compression metrics * test for all training cases * temporal workaround for eval only * cover train/eval tests * style fix * Move old ovtrainer tests to a new `test_training.py` file; bugfix in training loss check (#6) * removing old tests in test_quantization since they are now in `test_training` * bugfix in checking compression metrics during training * keep bert examples only and misc. fixes (#7) * temporarily keep bert examples only; remove w2v2 and swin * move nncf_compression_config out of OVTrainingArguments * type hint change for nncf_compression_config * documnet rename feature to task * revert existing QAT image classification example * delete useless codes in test quantization * revert existing test_ quantization * misc change in compute_metric * revert unnecessary changes * temporal workaround for logging distill & compression loss (not using dist. reduce) * revert set_task method * bugfix in compression metric in qa task * bugfix in importing tpu * simplify pruning ir codes * clean unncessary distillation weight attribute in trainer * Change nncf requirement to official 2.4 * Log nncf compression statistics at the beginning of each training epoch * Revise optimization_ov.mdx documentation * Consolidate during training optimization to QAT and JPQD * Add known limitation regarding OpenVINO IR with static input shape * fix data parallel crashes and add tests for DP/DDP (#8) * fix "not same device" error in data parallel * wrap teacher model with data parallel * add sst2 tests for dp/ddp with fixes * Add remark in optimization_ov.mdx on supported model architecture for structured pruning * Refactor JPQD IR generation where final IR is dynamic in input shape * Revise optimization_ov.mdx to remove static IR limitations * revert snippet for inference with Transformers pipeline * Remove commented codes in openvino/trainer.py * Add tests about new OV IR export - check dynamic graph and output equivalence to torch model (#9) * draft for new export with some todos * draft for tests * delete onnx export debugging when errors on saving * add back the debug info when ir export fails * bugfix in random setting zeros in movement masks * Add tests on OV IR reshape-ability * Remove unused imports in openvino/trainer.py * Refine inference pipeline with OVModel in optimization_ov.mdx * Revise openvino extras in setup.py --------- Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com> Co-authored-by: Yujie Pan <yujie.pan@intel.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes # (issue)
Before submitting