-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Progress Reporting for long running operations #3243
Merged
Merged
Changes from 201 commits
Commits
Show all changes
202 commits
Select commit
Hold shift + click to select a range
b2fbfb7
minor changes
6fa7bc6
progress classes and added reportign
68b8f28
Merge branch 'master' of https://github.com/azure/azure-cli into prog…
2a9e669
repackage
8445e2f
check for value before getting it in parsed args
c71ebc6
Merge branch 'master' of https://github.com/azure/azure-cli into prog…
ac350ea
undoing check
d88ad72
minor
7491ec9
mvc for progress outline
3ca3bd7
Merge branch 'master' of https://github.com/azure/azure-cli into prog…
b08e133
ask user for feedback
e0d39fc
shell additions
5d1c333
repackaging + shell implementation start
8c7b18c
remove vscode
adea8f1
heart beat
7a9e789
progress bar
64c5a7a
minor changes
72a77ee
CLI spinning wheel
72b954d
upload download storage
fdfb13e
humanfriendly in dependencies
ae24830
clean up controller
b50e085
minor changes
bd6892d
split deterministic and nondeterministic
29ebff5
separate the reporter
de8b0c5
design simplification
16f94d5
minor
3d53b00
Merge branch 'master' of https://github.com/azure/azure-cli into prog…
a36962b
errors
b728bac
minor
2532835
clear changing
55c6ad9
update readme
8bc52a0
Revert "clear changing"
2c5ff53
Revert "update readme"
ace1cc7
minor
086f2a6
mute stderr
c1f55c6
varying number of parameters capability
93bffb3
hooking up shell with progress
32b878f
hooking up progress bar
771a0ee
message in determiniate progress
c75bd4d
add testing
217147a
cleaning code + writing tests
b386600
tests
6d9debf
removed files
5fa1228
shell + other modifications
2c1ff74
minor changes
b92392b
history
ae11bcc
pylint + formatting
454f42c
flake8
94cb45a
flake8
6a4239d
pylint
42ab25c
cleaning + comments
36cd5ca
tests
68941db
reviews
ab4d0dc
kwargs cleaning
dbb754f
clean
942d9a1
flake8
30533d8
tests + cleaning formatting
9f670e3
travis failing
8ddba04
removed namespace in nspkg
a304486
shell correction
7ac9df1
cleaning + formatting
66243c8
char hack
0618566
redir stderr
b49fb99
tests + patch
e21a6d6
change toolbar for 20 seconds
b4812be
feedback in toolbar
e375839
Merge branch 'master' into feedback
a97d154
tests
68d8854
flake8
1b56a53
test
6a6215b
tests
352a695
tests
0c6ed18
tests
73a199d
tests
47c295a
tests
243971c
tests
c0e9f0e
tests
1066afd
tests
affb745
tests
1b1cbfc
tests
a7acfcc
tests
5cc8bac
remove dead code
c5c8481
flake8
2145b54
Merge branch 'feedback' of https://github.com/oakeyc/azure-cli into f…
74d8e64
flake8
461f2ff
Merge branch 'master' of https://github.com/azure/azure-cli into feed…
517cb60
small changes
76fcb63
commenting out test
2a3768e
removed unnecessary subclass
ecb676e
mock up the progress for tests
dc798f0
wording
4c13385
pylint
4d34129
typo
11753a6
some style + renaming
6e86c34
keyword
d540b83
kwargs
efc725b
calling function
1d7c47f
calling function
f87c0a8
adding tests back
7e6f092
changed outstream
674d8c3
comment out
f76fa9d
flake8
569eb43
stringio
4d61e10
trying fdopen
76f9e33
reviewed changes
6fe564f
change to not close
056ea14
delete file
17d3501
tests
31583f4
remove completion tests
ea6d575
try fdopen
90254eb
change which tests dont run
63735c2
remove from longrunning
75fbe03
cleaning
16fe6d2
does this work?
0a1d367
patch storage
1537d47
write
2f519e2
write
83695c3
tests
bfb5751
tests
c57d889
formatting
847afa4
mock outstream
0b273c4
undo
9590bd3
making tests pass
561091a
Merge branch 'master' of https://github.com/oakeyc/azure-cli into pro…
7abb962
remodel to decouple the view
53e2fba
reorg
63920ab
patch tests
cf4a9eb
style
4fddd4b
remocked
0dc79d5
Merge branch 'master' of https://github.com/azure/azure-cli into prog…
19955f4
merge
d3d6c31
Merge branch 'master' into progress
5dc5472
pylint
3cebef4
Merge branch 'progress' of https://github.com/oakeyc/azure-cli into p…
570c066
reformat
f42233b
test
9f77800
typo
ce45cd6
outstream for indetstout
85dd6d1
flake8
b2a5406
vcr
9d46cba
mock view
cb21770
pylint
ce5d11b
minor
9ebacb0
remock
6f84dd9
remock
497c436
reorg
4344a57
test
d2464f0
change to cli config
d3d73e5
typos
d438c49
feedback as command
72a996d
utc
e3f7506
Merge branch 'master' into feedback
a38c823
remove from shell config
2a528f5
Merge branch 'feedback' of https://github.com/oakeyc/azure-cli into f…
bb78aba
flake8
7ca19c4
files
cc8a8c5
fix shell progress
ab1479d
remove extra windows characters
9a4cae2
write file
81281d3
fix test
7dfefdd
'cursor'
594f00e
small changes
1ada7aa
removed all the char windows
71eb091
change error for p2
1c1adf8
muting parsing
ff4dfe7
fix conflicts
5f62753
added progress back in
f2bb5ed
Merge branch 'feedback' of https://github.com/oakeyc/azure-cli into f…
e986cb6
Revert "Merge branch 'feedback' of https://github.com/oakeyc/azure-cl…
c755c5f
Revert "added progress back in"
f283591
revert changes
d489d53
remove with interupt
61e0e54
remove feedback
a7b32a0
shell progress tests
af4d40e
shell progress view test
af2f981
flake8
189adf8
patch back error code when you neeed it
8add8e5
small changes
ad3dc16
add dump for ending for views + force implement flush
5e8c1c7
flake8
4c4d057
mock test
0ab8de0
Merge branch 'master' of https://github.com/azure/azure-cli into prog…
1c027c5
Merge branch 'master' of https://github.com/azure/azure-cli into prog…
8a028ff
update dependencies
5f2d3fe
history
f2a912d
remove feedback stuff
347518e
remove excess
89711bb
progress with storage all good
705fb64
minor
437957e
flake8
7910d19
pylint
c157edc
minor
11b5740
flake8
a364f2f
Merge branch 'master' into progress
1b57307
import error
ae26f04
progress things
2726472
fixing tests
21d3cb0
flake8
b0c831e
tests
ebf183c
tests;
8014e06
interactive mode progress multi-process fix
d589c8d
updates
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,169 @@ | ||
# -------------------------------------------------------------------------------------------- | ||
# Copyright (c) Microsoft Corporation. All rights reserved. | ||
# Licensed under the MIT License. See License.txt in the project root for license information. | ||
# -------------------------------------------------------------------------------------------- | ||
from __future__ import division | ||
import sys | ||
import platform | ||
|
||
import humanfriendly | ||
BAR_LEN = 70 | ||
|
||
if platform.system() == 'Windows': | ||
humanfriendly.erase_line_code = '' | ||
|
||
|
||
class ProgressViewBase(object): | ||
""" a view base for progress reporting """ | ||
def __init__(self, out): | ||
self.out = out | ||
|
||
def write(self, args): | ||
""" writes the progress """ | ||
raise NotImplementedError | ||
|
||
def flush(self): | ||
""" flushes the message out the pipeline""" | ||
raise NotImplementedError | ||
|
||
def clear(self): | ||
""" resets the view to neutral """ | ||
pass | ||
|
||
|
||
class ProgressReporter(object): | ||
""" generic progress reporter """ | ||
def __init__(self, message='', value=None, total_value=None): | ||
self.message = message | ||
self.value = value | ||
self.total_val = total_value | ||
self.closed = False | ||
|
||
def add(self, **kwargs): | ||
""" | ||
adds a progress report | ||
:param kwargs: dictionary containing 'message', 'total_val', 'value' | ||
""" | ||
message = kwargs.get('message', self.message) | ||
total_val = kwargs.get('total_val', self.total_val) | ||
value = kwargs.get('value', self.value) | ||
if value and total_val: | ||
assert value >= 0 and value <= total_val and total_val >= 0 | ||
self.closed = value == total_val | ||
self.total_val = total_val | ||
self.value = value | ||
self.message = message | ||
|
||
def report(self): | ||
""" report the progress """ | ||
percent = self.value / self.total_val if self.value is not None and self.total_val else None | ||
return {'message': self.message, 'percent': percent} | ||
|
||
|
||
class ProgressHook(object): | ||
""" sends the progress to the view """ | ||
def __init__(self): | ||
self.reporter = ProgressReporter() | ||
self.active_progress = None | ||
|
||
def init_progress(self, progress_view): | ||
""" activate a view """ | ||
self.active_progress = progress_view | ||
|
||
def add(self, **kwargs): | ||
""" adds a progress report """ | ||
self.reporter.add(**kwargs) | ||
self.update() | ||
|
||
def update(self): | ||
""" updates the view with the progress """ | ||
# TODO APPROPRIATE PROGRESS SENT | ||
self.active_progress.write(self.reporter.report()) | ||
self.active_progress.flush() | ||
|
||
def stop(self): | ||
""" if there is an abupt stop before ending """ | ||
self.reporter.closed = True | ||
self.add(message='Interrupted') | ||
self.active_progress.clear() | ||
|
||
def begin(self, **kwargs): | ||
""" start reporting progress """ | ||
kwargs['message'] = kwargs.get('message', 'Starting') | ||
self.add(**kwargs) | ||
self.reporter.closed = False | ||
|
||
def end(self, **kwargs): | ||
""" ending reporting of progress """ | ||
kwargs['message'] = kwargs.get('message', 'Finished') | ||
self.reporter.closed = True | ||
self.add(**kwargs) | ||
self.active_progress.clear() | ||
|
||
def is_running(self): | ||
""" whether progress is continuing """ | ||
return not self.reporter.closed | ||
|
||
|
||
class IndeterminateStandardOut(ProgressViewBase): | ||
""" custom output for progress reporting """ | ||
def __init__(self, out=None): | ||
super(IndeterminateStandardOut, self).__init__( | ||
out if out else sys.stderr) | ||
self.spinner = None | ||
|
||
def write(self, args): | ||
""" | ||
writes the progress | ||
:param args: dictionary containing key 'message' | ||
""" | ||
if self.spinner is None: | ||
self.spinner = humanfriendly.Spinner( | ||
label='In Progress', stream=self.out, hide_cursor=False) | ||
msg = args.get('message', 'In Progress') | ||
self.spinner.step(label=msg) | ||
|
||
def flush(self): | ||
self.out.flush() | ||
|
||
|
||
def _format_value(msg, percent): | ||
bar_len = BAR_LEN - len(msg) - 1 | ||
completed = int(bar_len * percent) | ||
|
||
message = '\r{}['.format(msg) | ||
message += ('#' * completed).ljust(bar_len) | ||
message += '] {:.4%}'.format(percent) | ||
return message | ||
|
||
|
||
class DeterminateStandardOut(ProgressViewBase): | ||
""" custom output for progress reporting """ | ||
def __init__(self, out=None): | ||
super(DeterminateStandardOut, self).__init__(out if out else sys.stderr) | ||
|
||
def write(self, args): | ||
""" | ||
writes the progress | ||
:param args: args is a dictionary containing 'percent', 'message' | ||
""" | ||
percent = args.get('percent', 0) | ||
message = args.get('message', '') | ||
|
||
if percent: | ||
progress = _format_value(message, percent) | ||
self.out.write(progress) | ||
|
||
def clear(self): | ||
self.out.write('\n') | ||
|
||
def flush(self): | ||
self.out.flush() | ||
|
||
|
||
def get_progress_view(determinant=False, outstream=sys.stderr): | ||
""" gets your view """ | ||
if determinant: | ||
return DeterminateStandardOut(out=outstream) | ||
else: | ||
return IndeterminateStandardOut(out=outstream) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,100 @@ | ||
# -------------------------------------------------------------------------------------------- | ||
# Copyright (c) Microsoft Corporation. All rights reserved. | ||
# Licensed under the MIT License. See License.txt in the project root for license information. | ||
# -------------------------------------------------------------------------------------------- | ||
import unittest | ||
|
||
import azure.cli.core.commands.progress as progress | ||
|
||
|
||
class MockOutstream(progress.ProgressViewBase): | ||
""" mock outstream for testing """ | ||
def __init__(self): | ||
self.string = '' | ||
|
||
def write(self, message): | ||
self.string = message | ||
|
||
def flush(self): | ||
pass | ||
|
||
|
||
class TestProgress(unittest.TestCase): # pylint: disable=too-many-public-methods | ||
""" test the progress reporting """ | ||
|
||
def test_progress_indicator_det_model(self): | ||
""" test the progress reporter """ | ||
reporter = progress.ProgressReporter() | ||
args = reporter.report() | ||
self.assertEqual(args['message'], '') | ||
self.assertEqual(args['percent'], None) | ||
|
||
reporter.add(message='Progress', total_val=10, value=0) | ||
self.assertEqual(reporter.message, 'Progress') | ||
self.assertEqual(reporter.value, 0) | ||
self.assertEqual(reporter.total_val, 10) | ||
args = reporter.report() | ||
self.assertEqual(args['message'], 'Progress') | ||
self.assertEqual(args['percent'], 0) | ||
|
||
with self.assertRaises(AssertionError): | ||
reporter.add(message='In words', total_val=-1, value=10) | ||
with self.assertRaises(AssertionError): | ||
reporter.add(message='In words', total_val=1, value=-10) | ||
with self.assertRaises(AssertionError): | ||
reporter.add(message='In words', total_val=30, value=12340) | ||
|
||
reporter = progress.ProgressReporter() | ||
message = reporter.report() | ||
self.assertEqual(message['message'], '') | ||
|
||
reporter.add(message='Progress') | ||
self.assertEqual(reporter.message, 'Progress') | ||
|
||
message = reporter.report() | ||
self.assertEqual(message['message'], 'Progress') | ||
|
||
def test_progress_indicator_indet_stdview(self): | ||
""" tests the indeterminate progress standardout view """ | ||
outstream = MockOutstream() | ||
view = progress.IndeterminateStandardOut(out=outstream) | ||
view.write({}) | ||
self.assertEqual(view.spinner.label, 'In Progress') | ||
before = view.spinner.total | ||
view.write({}) | ||
after = view.spinner.total | ||
self.assertTrue(after >= before) | ||
view.write({'message': 'TESTING'}) | ||
|
||
def test_progress_indicator_det_stdview(self): | ||
""" test the determinate progress standardout view """ | ||
outstream = MockOutstream() | ||
view = progress.DeterminateStandardOut(out=outstream) | ||
view.write({'message': 'hihi', 'percent': .5}) | ||
# 95 length, 48 complete, 4 dec percent | ||
bar_str = ('#' * int(.5 * 65)).ljust(65) | ||
self.assertEqual(outstream.string, '\rhihi[{}] {:.4%}'.format(bar_str, .5)) | ||
|
||
view.write({'message': '', 'percent': .9}) | ||
# 99 length, 90 complete, 4 dec percent | ||
bar_str = ('#' * int(.9 * 69)).ljust(69) | ||
self.assertEqual(outstream.string, '\r[{}] {:.4%}'.format(bar_str, .9)) | ||
|
||
def test_progress_indicator_controller(self): | ||
""" tests the controller for progress reporting """ | ||
controller = progress.ProgressHook() | ||
view = MockOutstream() | ||
|
||
controller.init_progress(view) | ||
self.assertTrue(view == controller.active_progress) | ||
|
||
controller.begin() | ||
|
||
self.assertEqual(controller.active_progress.string['message'], 'Starting') | ||
|
||
controller.end() | ||
self.assertEqual(controller.active_progress.string['message'], 'Finished') | ||
|
||
|
||
if __name__ == '__main__': | ||
unittest.main() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove todo?