Simple Progress Bar Inclusion #2334

GiulioZizzo · 2023-11-21T16:11:35Z

Description

Currently, the core training routines for many ART estimators do not had progress bars to inform the user of how long models require for training. Although this was not a problem when ART was first developed as models were small and trained quickly, with ever larger models being evaluated for robustness, it Is very useful to have an indication of how long training will take.
For this PR we introduce a simple tqdm progress bar in the pytorch/tensorflow/huggingface estimators for the fit and fit_generate functions.

This feature branches off the current open PR #2300

Fixes (partially) #2288

Type of change

Please check all relevant options.

Improvement (non-breaking)
Bug fix (non-breaking)
New feature (non-breaking)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

Updated the current CI test test_fit_kwargs to include a check for using the progress bar kwarg

Test Configuration:

OS: Mac OS
Python version: 3.8
ART version or commit number: ART 1.16
TensorFlow / Keras / PyTorch / MXNet version: TensorFlow 2.10, pytorch: 1.13

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
My changes have been tested using both CPU and GPU devices

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

codecov-commenter · 2023-11-21T16:16:29Z

Codecov Report

Attention: 10 lines in your changes are missing coverage. Please review.

Comparison is base (a281f62) 76.18% compared to head (b817b9c) 85.47%.
Report is 6 commits behind head on dev_1.17.0.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@              Coverage Diff               @@
##           dev_1.17.0    #2334      +/-   ##
==============================================
+ Coverage       76.18%   85.47%   +9.29%     
==============================================
  Files             327      327              
  Lines           30850    30206     -644     
  Branches         5716     5591     -125     
==============================================
+ Hits            23503    25819    +2316     
+ Misses           5914     2948    -2966     
- Partials         1433     1439       +6

Files	Coverage Δ
art/attacks/extraction/knockoff_nets.py	`89.93% <ø> (ø)`
art/defences/trainer/adversarial_trainer.py	`92.24% <100.00%> (ø)`
art/defences/trainer/dp_instahide_trainer.py	`96.07% <100.00%> (ø)`
...rs/certification/derandomized_smoothing/pytorch.py	`58.36% <ø> (-0.38%)`	⬇️
...certification/derandomized_smoothing/tensorflow.py	`62.38% <ø> (-0.92%)`	⬇️
...ertification/randomized_smoothing/macer/pytorch.py	`98.38% <100.00%> (ø)`
...ification/randomized_smoothing/macer/tensorflow.py	`91.89% <100.00%> (+56.75%)`	⬆️
...tors/certification/randomized_smoothing/pytorch.py	`85.54% <100.00%> (ø)`
...ation/randomized_smoothing/randomized_smoothing.py	`97.33% <100.00%> (-0.04%)`	⬇️
...ication/randomized_smoothing/smooth_adv/pytorch.py	`98.36% <100.00%> (ø)`
... and 8 more

... and 59 files with indirect coverage changes

beat-buesser

Hi @GiulioZizzo Thank you very much for your pull request! I think it would be great to extend the PR to also cover Keras and TensorFlow for the new progress bar.

beat-buesser · 2023-12-13T15:22:07Z

art/estimators/classification/pytorch.py

@@ -389,12 +390,14 @@ def fit(  # pylint: disable=W0221
                          the batch size. If ``False`` and the size of dataset is not divisible by the batch size, then
                          the last batch will be smaller. (default: ``False``)
        :param scheduler: Learning rate scheduler to run at the start of every epoch.
-        :param kwargs: Dictionary of framework-specific arguments. This parameter is not currently supported for PyTorch
-                       and providing it takes no effect.
+        :param kwargs: Dictionary of framework-specific arguments. Currently supports "display_progress_bar" to


Let's change this new argument display_progress_bar to verbose to follow the ART pattern and add it as full argument outside kwargs.

beat-buesser · 2023-12-13T15:23:15Z

tests/estimators/classification/test_deeplearning_common.py

+        if framework in ["kerastf", "keras"]:
+            kwargs = {"callbacks": [LearningRateScheduler(get_lr)]}
+        else:
+            kwargs = {"callbacks": [LearningRateScheduler(get_lr)], "display_progress_bar": True}


Please add progress bars also for Keras and TensorFlow v1.

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

…andomized tools Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

…for kwargs checks Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

art/estimators/certification/derandomized_smoothing/pytorch.py

beat-buesser · 2023-12-19T15:47:20Z

art/estimators/classification/tensorflow.py

+        display_pb = self.process_verbose(verbose)
+


Suggested change

display_pb = self.process_verbose(verbose)

beat-buesser · 2023-12-19T15:47:38Z

art/estimators/classification/tensorflow.py

+            for epoch in tqdm(range(nb_epochs), disable=not display_pb, desc="Epochs"):
+                for i_batch, o_batch in tqdm(generator.iterator, disable=not display_pb, desc="Batches"):


Suggested change

for epoch in tqdm(range(nb_epochs), disable=not display_pb, desc="Epochs"):

for i_batch, o_batch in tqdm(generator.iterator, disable=not display_pb, desc="Batches"):

for epoch in tqdm(range(nb_epochs), disable=not verbose, desc="Epochs"):

for i_batch, o_batch in generator.iterator:

beat-buesser · 2023-12-19T15:51:26Z

tests/estimators/classification/test_deeplearning_common.py

-        kwargs = {"epochs": 1}
-        with pytest.raises(TypeError) as exception:
-            classifier.fit(x_train_mnist, y_train_mnist, batch_size=default_batch_size, nb_epochs=1, **kwargs)
+        kwargs = {"callbacks": [LearningRateScheduler(get_lr)], "verbose": True}


Suggested change

kwargs = {"callbacks": [LearningRateScheduler(get_lr)], "verbose": True}

kwargs = {"callbacks": [LearningRateScheduler(get_lr)]}

beat-buesser · 2023-12-19T15:52:08Z

tests/estimators/classification/test_deeplearning_common.py

-        with pytest.raises(TypeError) as exception:
-            classifier.fit(x_train_mnist, y_train_mnist, batch_size=default_batch_size, nb_epochs=1, **kwargs)
+        kwargs = {"callbacks": [LearningRateScheduler(get_lr)], "verbose": True}
+        classifier.fit(x_train_mnist, y_train_mnist, batch_size=default_batch_size, nb_epochs=1, **kwargs)


Suggested change

classifier.fit(x_train_mnist, y_train_mnist, batch_size=default_batch_size, nb_epochs=1, **kwargs)

classifier.fit(x_train_mnist, y_train_mnist, batch_size=default_batch_size, nb_epochs=1, verbose=True, **kwargs)

beat-buesser · 2023-12-19T15:53:14Z

tests/estimators/classification/test_deeplearning_common.py

+
+        # Test failure for invalid parameters: does not apply to many frameworks which allow arbitrary kwargs
+        if framework not in [
+            "tensorflow1",
+            "tensorflow2",
+            "tensorflow2v1",
+            "huggingface",
+            "pytorch",
+        ]:
+            kwargs = {"epochs": 1}
+            with pytest.raises(TypeError) as exception:
+                classifier.fit(x_train_mnist, y_train_mnist, batch_size=default_batch_size, nb_epochs=1, **kwargs)
+
+            assert "multiple values for keyword argument" in str(exception)


Suggested change

# Test failure for invalid parameters: does not apply to many frameworks which allow arbitrary kwargs

if framework not in [

"tensorflow1",

"tensorflow2",

"tensorflow2v1",

"huggingface",

"pytorch",

]:

kwargs = {"epochs": 1}

with pytest.raises(TypeError) as exception:

classifier.fit(x_train_mnist, y_train_mnist, batch_size=default_batch_size, nb_epochs=1, **kwargs)

assert "multiple values for keyword argument" in str(exception)

art/estimators/classification/tensorflow.py

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

GiulioZizzo added 5 commits November 8, 2023 20:42

progress bar development

d8bab78

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

revert check on dim for fit-generator and move to a separate PR

7bed09d

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

update kwarg test to run with pb display

b3dec0f

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

run on CI pipeline

a32d798

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

remove CI to run on feature branch

043752d

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

GiulioZizzo mentioned this pull request Nov 28, 2023

Hugging Face Notebook Improvements #2338

Merged

13 tasks

beat-buesser self-requested a review November 30, 2023 15:36

beat-buesser self-assigned this Nov 30, 2023

beat-buesser added the improvement Improve implementation label Nov 30, 2023

beat-buesser added this to the ART 1.17.0 milestone Nov 30, 2023

beat-buesser linked an issue Nov 30, 2023 that may be closed by this pull request

Implementation of Progress Bars for ART Estimators #2288

Closed

beat-buesser added 2 commits November 30, 2023 18:17

Merge branch 'dev_1.17.0' into simple_pb_inclusion

b71810a

Merge branch 'dev_1.17.0' into simple_pb_inclusion

6c1bc43

beat-buesser requested changes Dec 13, 2023

View reviewed changes

GiulioZizzo added 7 commits December 14, 2023 13:03

change to verbose, and add support for tf1

39ab9cd

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

standardise verbose use across tools

5fb24d5

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

mypy fixes

2a3290a

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

split verbosity processing into separate method.

3c6458d

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

split verbosity processing into separate method.

cc441c9

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

general inclusion of pylint: disable=W0221. Updates to randomized/der…

9c3f572

…andomized tools Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

Better handling of int values for verbosity. Adding fit_gen to tests …

804af32

…for kwargs checks Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

beat-buesser requested changes Dec 19, 2023

View reviewed changes

beat-buesser added 3 commits December 19, 2023 16:55

Update art/estimators/certification/derandomized_smoothing/pytorch.py

94cf59f

Apply suggestions from code review

1f3026b

Merge branch 'dev_1.17.0' into simple_pb_inclusion

a405f49

github-advanced-security bot found potential problems Dec 19, 2023

View reviewed changes

art/estimators/classification/tensorflow.py Fixed Show fixed Hide fixed

beat-buesser reviewed Dec 20, 2023

View reviewed changes

art/estimators/classification/tensorflow.py Outdated Show resolved Hide resolved

beat-buesser and others added 2 commits December 20, 2023 10:45

Apply suggestions from code review

aa6a0ee

unifying art tools in verbose interface

56f7f4a

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

GiulioZizzo added 2 commits December 20, 2023 15:23

mypy fixes

63916b2

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

mypy fixes

b817b9c

Signed-off-by: GiulioZizzo <giulio.zizzo@yahoo.co.uk>

beat-buesser merged commit b817b9c into Trusted-AI:dev_1.17.0 Dec 22, 2023
35 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simple Progress Bar Inclusion #2334

Simple Progress Bar Inclusion #2334

GiulioZizzo commented Nov 21, 2023

codecov-commenter commented Nov 21, 2023 •

edited

Loading

beat-buesser left a comment

beat-buesser Dec 13, 2023

beat-buesser Dec 13, 2023

beat-buesser Dec 19, 2023

beat-buesser Dec 19, 2023

beat-buesser Dec 19, 2023

beat-buesser Dec 19, 2023

beat-buesser Dec 19, 2023

		for epoch in tqdm(range(nb_epochs), disable=not display_pb, desc="Epochs"):
		for i_batch, o_batch in tqdm(generator.iterator, disable=not display_pb, desc="Batches"):

	kwargs = {"callbacks": [LearningRateScheduler(get_lr)], "verbose": True}
	kwargs = {"callbacks": [LearningRateScheduler(get_lr)]}

	classifier.fit(x_train_mnist, y_train_mnist, batch_size=default_batch_size, nb_epochs=1, **kwargs)
	classifier.fit(x_train_mnist, y_train_mnist, batch_size=default_batch_size, nb_epochs=1, verbose=True, **kwargs)

Simple Progress Bar Inclusion #2334

Simple Progress Bar Inclusion #2334

Conversation

GiulioZizzo commented Nov 21, 2023

Description

Type of change

Testing

Checklist

codecov-commenter commented Nov 21, 2023 • edited Loading

Codecov Report

beat-buesser left a comment

Choose a reason for hiding this comment

beat-buesser Dec 13, 2023

Choose a reason for hiding this comment

beat-buesser Dec 13, 2023

Choose a reason for hiding this comment

beat-buesser Dec 19, 2023

Choose a reason for hiding this comment

beat-buesser Dec 19, 2023

Choose a reason for hiding this comment

beat-buesser Dec 19, 2023

Choose a reason for hiding this comment

beat-buesser Dec 19, 2023

Choose a reason for hiding this comment

beat-buesser Dec 19, 2023

Choose a reason for hiding this comment

codecov-commenter commented Nov 21, 2023 •

edited

Loading