Documentation- Basic Profiling for Ray Users #2326

crystalzyan · 2018-06-29T10:38:17Z

What do these changes do?

Added a documentation tutorial (with graphics) on profiling the performance of a basic Ray example using Python time, Python cProfile, line_profiler (a third-party profiler), and the Ray timeline web UI. There is an existing documentation page profiling.rst for Ray developers, so I put mine's as a separate page user-profiling.rst under the same Help section.

Related issue number

No issue number. Requested by @pcmoritz

…posed to current Profiling section for Ray developers. Completed three sections 'A Basic Profiling Example', 'Timing Performance Using Python's Timestamps', and 'Profiling Using An External Profiler (Line_Profiler).' Left to-do two sections on CProfile and Ray Timeline Visualization.'

…ization section, including graphical images

…wording edits

AmplabJenkins · 2018-06-29T11:48:17Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6395/
Test PASSed.

pcmoritz · 2018-07-08T20:41:59Z

doc/source/user-profiling.rst

+This document is intended for users of Ray who want to know how to evaluate 
+the performance of their code while running on Ray. Profiling the 
+performance of your code can be very helpful to determine performance 
+bottlenecks or where your code may not be parallelizing properly. If you 


"or to find out where your code may not be parallelized properly"

pcmoritz · 2018-07-08T20:42:38Z

doc/source/user-profiling.rst

+performance of your code can be very helpful to determine performance 
+bottlenecks or where your code may not be parallelizing properly. If you 
+are interested in pinpointing why your Ray application may not be 
+achieving speedups as expected, then do read on!


"the expected speedup, read on!"

robertnishihara

Some comments:

Let's remove the images. They're really nice, but I'm a bit concerned about their size. As the codebase gets larger, it becomes more difficult to download and work with.
Maybe show people how to use cProfile inside of an actor.

pcmoritz · 2018-07-08T20:54:36Z

doc/source/user-profiling.rst

+A Basic Example to Profile
+--------------------------
+
+Let's try to profile a simple example, and compare how different looping 


"and compare how different ways to write a simple loop can affect performance"

pcmoritz · 2018-07-08T20:55:10Z

doc/source/user-profiling.rst

+Let's try to profile a simple example, and compare how different looping 
+call structures of the same remote function affects performance.
+
+As a stand-in for a computationally intensive and possibly slower function,


"As a proxy"? stand-in sounds a little strange to me :)

pcmoritz · 2018-07-08T20:58:35Z

doc/source/user-profiling.rst

+      list2.append(func.remote())
+    ray.get(list2)
+
+Finally, as a demonstration of Ray's parallelism abilities, let's create a 


Oh, this example should be a bit different. The idea is to call another function (which is not remote) as other_func; this is going to slow down the loop since that part is not parallelized (so just remove the @ray.remote from other_func and also the .remote). This shows that if there is still a large serial component to the loop, parallelism speedup might not be as large as expected (this is called https://en.wikipedia.org/wiki/Amdahl%27s_law). Does it make sense? Please modify the description accordingly.

pcmoritz · 2018-07-08T20:59:10Z

doc/source/user-profiling.rst

+time how long it takes to complete each loop version. We can do this using 
+python's built-in ``time`` `module`_.
+
+.. _`module`: https://docs.python.org/2/library/time.html


Let's link to the Python 3 version here.

pcmoritz · 2018-07-08T21:10:43Z

doc/source/user-profiling.rst

+  if __name__ == "__main__":
+    main()
+
+Alternatively, to print out the timer on **selective** calls to ``ex1()``,


Why don't we get rid of this paragraph and the code after? It doesn't really add much I think and is not needed for profiling.

pcmoritz · 2018-07-08T21:11:26Z

doc/source/user-profiling.rst

+
+Let's interpret these results. 
+
+Most pertinently, ``ex1()`` took substantially more time than ``ex2()``, 


Here, ex1() took

pcmoritz · 2018-07-08T21:12:00Z

doc/source/user-profiling.rst

+Let's interpret these results. 
+
+Most pertinently, ``ex1()`` took substantially more time than ``ex2()``, 
+despite their only difference being that ``ex1()`` calls ``ray.get`` on the 


get rid of "despite" and just way what the difference is.

pcmoritz · 2018-07-08T21:12:48Z

doc/source/user-profiling.rst

+
+By calling ``ray.get`` after each call to the remote function, ``ex1()`` 
+removes all ability to parallelize work, by forcing the driver to wait for 
+each ``func()``'s result in succession. We are completely sabotaging any 


Instead of sabotaging, let's say "we are not taking advantage of Ray parallelization here" :)

pcmoritz · 2018-07-08T21:13:15Z

doc/source/user-profiling.rst

+Ray and multiple CPUs, this loop would take at least 3.5 seconds to finish.
+
+
+Profiling Using An External Profiler (Line_Profiler)


in this caption, let's remove the _ from Line_Profiler

pcmoritz · 2018-07-08T21:21:30Z

Let's try to keep the images first and compress them with JPEG. If we can get them under 50KB, let's keep them!

pcmoritz · 2018-07-08T21:24:06Z

doc/source/user-profiling.rst

+Profiling the performance of your Ray application doesn't need to be 
+an eye-straining endeavor of interpreting numbers among hundreds of 
+lines of text. Ray comes with its own visual web UI to visualize the 
+parallelization (or lack thereof) of user tasks submitted to Ray!


Point out the limitation somewhere, i.e. this only shows timing info about ray tasks, but doesn't break timing down to normal python functions. This can be a problem especially on a driver, that's why we explain the other profiling techniques above.

pcmoritz · 2018-07-08T21:25:28Z

doc/source/user-profiling.rst

+parallelization (or lack thereof) of user tasks submitted to Ray!
+
+Currently, whenever initializing Ray, a URL is automatically generated and
+printed to terminal on where to view Ray's web UI as a Jupyter notebook:


generated and printed in the terminal. This URL can be used to view...

pcmoritz · 2018-07-08T21:26:05Z

doc/source/user-profiling.rst

+example, it has opened on port 8897.
+
+Because this web UI is only available as long as your Ray application 
+is currently running, you may need to add a user prompt to stall 


to prevent your Ray application

pcmoritz · 2018-07-08T21:27:00Z

doc/source/user-profiling.rst

+    ex3()
+
+    # Require user input confirmation before exiting
+    hang = int(input('Examples finished executing. Enter any integer to exit:'))


hang = input('Examples finished executing. Press enter to exit:')

pcmoritz · 2018-07-08T21:28:21Z

doc/source/user-profiling.rst

+the first batch of ``func()`` calls is finished.
+
+**For more on Ray's Web UI,** such as how to access the UI on a remote
+node over ssh, or for troubleshooting installation, **please see our** 


remove bolding from "please see our"

pcmoritz

This is really cool! There is a little bit of reworking needed before we can merge it, please see the comments.

…ed to address (1) compressing the image files, (2) correcting ex 3 to not be remote, and (3) using cProfile on an actor

… a semi-parallelized example. Compressed timeline example image to be under 50 KB, removed view timeline GUI image. Updated timeline example image to reflect revised example 3. cProfile actor example left

…ing actors in the cProfile section

AmplabJenkins · 2018-07-12T02:55:39Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6614/
Test PASSed.

…the Ray actor cProfile example

crystalzyan · 2018-07-12T05:15:50Z

I updated the following:

Created new example ex4 of profiling Ray actors under the cProfile section. I decided to create an example of another possible user mistake, where they use one actor instead of several to parallelize code. Code will not speed up even if ray.wait() is called at the very end (the fix that would work on ex1).
Corrected ex3 to alternate to a local function instead of to a remote function. Consequently updated profiling interpretation/explanation of ex3, ex3 profiling output results for all profiling technique sections, and the Ray timeline image.
Removed one of the images and compressed the other image to be <50 KB. .GIF is actually a better compression method than .JPG in this case, because .JPG compression is optimal for low-contrast images, not sharp-contrast typeface and GUIs. Additionally, further compressed the image by lowering the dimensions/resolution and reducing the color palette to 128 colors.
I'll point out though that there are multiple other existing images in the documentation folder as well, some >500 KB in size (such as ray-tune-parcoords.png), which is why I did not realize that image files might be considered too big to keep in the codebase. Additionally, the current documentation folder is a little messy as the image files are all mixed in with the documentation .rst files, and there is actually also an "images" directory that's not being completely used within the documentation folder. This may need to be cleaned up?
Minor wording changes

AmplabJenkins · 2018-07-12T05:48:30Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6615/
Test FAILed.

AmplabJenkins · 2018-07-12T20:34:06Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6620/
Test FAILed.

AmplabJenkins · 2018-07-12T20:37:54Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6619/
Test PASSed.

AmplabJenkins · 2018-07-12T20:43:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6622/
Test FAILed.

AmplabJenkins · 2018-07-12T20:52:15Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6625/
Test FAILed.

AmplabJenkins · 2018-07-12T21:06:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6624/
Test FAILed.

AmplabJenkins · 2018-07-12T21:23:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6626/
Test FAILed.

changes have been incorporated

crystalzyan added 7 commits June 28, 2018 02:51

Ray documentation - Fixed rst codeblock linebreaks in 'User Profiling'

87a5855

Ray documentation - For User Profiling, added section on cProfile

37e0a3f

Ray documentation - For User Profiling, completed Ray Timeline Visual…

0c54393

…ization section, including graphical images

Ray documentation - made User Profiling timeline image larger, minor …

62e1e83

…wording edits

Ray documentation - minor wording edits to User Profiling

98f2240

Ray documentation - User Profiling- fixed broken link

2bd4889

pcmoritz reviewed Jul 8, 2018

View reviewed changes

robertnishihara previously requested changes Jul 8, 2018

View reviewed changes

pcmoritz reviewed Jul 8, 2018

View reviewed changes

crystalzyan added 3 commits July 10, 2018 22:25

Minor wording changes requested by Philipp Moritz addressed. Still ne…

3ee1616

…ed to address (1) compressing the image files, (2) correcting ex 3 to not be remote, and (3) using cProfile on an actor

Ray documentation - For user-profiling.rst, revised example 3 to show…

483c604

… a semi-parallelized example. Compressed timeline example image to be under 50 KB, removed view timeline GUI image. Updated timeline example image to reflect revised example 3. cProfile actor example left

Ray documentation - in user-profiling.rst, added a new example includ…

23f39d4

…ing actors in the cProfile section

Ray documentation - For user-profiling.rst, added section header for …

063bda7

…the Ray actor cProfile example

pcmoritz added 7 commits July 12, 2018 12:13

Update user-profiling.rst

9e75f78

Update user-profiling.rst

86cdf7b

4 space indentation

526432d

Update user-profiling.rst

082167b

Update user-profiling.rst

83b09f1

Update user-profiling.rst

0a3e1a5

corrections

ba8fc9f

pcmoritz approved these changes Jul 12, 2018

View reviewed changes

pcmoritz merged commit ebf4070 into ray-project:master Jul 12, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Documentation- Basic Profiling for Ray Users #2326

Documentation- Basic Profiling for Ray Users #2326

crystalzyan commented Jun 29, 2018

AmplabJenkins commented Jun 29, 2018

pcmoritz Jul 8, 2018

pcmoritz Jul 8, 2018

robertnishihara left a comment

pcmoritz Jul 8, 2018

pcmoritz Jul 8, 2018

pcmoritz Jul 8, 2018

pcmoritz Jul 8, 2018

pcmoritz Jul 8, 2018 •

edited

Loading

pcmoritz Jul 8, 2018

pcmoritz Jul 8, 2018

pcmoritz Jul 8, 2018

pcmoritz Jul 8, 2018

pcmoritz commented Jul 8, 2018

pcmoritz Jul 8, 2018

pcmoritz Jul 8, 2018

pcmoritz Jul 8, 2018

pcmoritz Jul 8, 2018

pcmoritz Jul 8, 2018

pcmoritz left a comment

AmplabJenkins commented Jul 12, 2018

crystalzyan commented Jul 12, 2018

AmplabJenkins commented Jul 12, 2018

AmplabJenkins commented Jul 12, 2018

AmplabJenkins commented Jul 12, 2018

AmplabJenkins commented Jul 12, 2018

AmplabJenkins commented Jul 12, 2018

AmplabJenkins commented Jul 12, 2018

AmplabJenkins commented Jul 12, 2018


		Let's interpret these results.

		Most pertinently, ``ex1()`` took substantially more time than ``ex2()``,

		Ray and multiple CPUs, this loop would take at least 3.5 seconds to finish.


		Profiling Using An External Profiler (Line_Profiler)

Documentation- Basic Profiling for Ray Users #2326

Documentation- Basic Profiling for Ray Users #2326

Conversation

crystalzyan commented Jun 29, 2018

What do these changes do?

Related issue number

AmplabJenkins commented Jun 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robertnishihara left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcmoritz Jul 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcmoritz commented Jul 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcmoritz left a comment

Choose a reason for hiding this comment

AmplabJenkins commented Jul 12, 2018

crystalzyan commented Jul 12, 2018

AmplabJenkins commented Jul 12, 2018

AmplabJenkins commented Jul 12, 2018

AmplabJenkins commented Jul 12, 2018

AmplabJenkins commented Jul 12, 2018

AmplabJenkins commented Jul 12, 2018

AmplabJenkins commented Jul 12, 2018

AmplabJenkins commented Jul 12, 2018

pcmoritz Jul 8, 2018 •

edited

Loading