Add callgraph examples to README.md (#528)

Add callgraph examples to README.md

Co-authored-by: Maggie Nolan <nolanmar@google.com>
Co-authored-by: Alexey Alexandrov <aalexand@users.noreply.github.com>

doc/README.md

@@ -24,7 +24,7 @@ pprof operates on data in the profile.proto format. Each profile is a collection
 of samples, where each sample is associated to a point in a location hierarchy,
 one or more numeric values, and a set of labels. Often these profiles represents
 data collected through statistical sampling of a program, so each sample
-describes a program call stack and a number or weight of samples collected at a
+describes a program call stack and a number or value of samples collected at a
 location. pprof is agnostic to the profile semantics, so other uses are
 possible. The interpretation of the reports generated by pprof depends on the
 semantics defined by the source of the profile.
@@ -66,10 +66,13 @@ in a browser to see the interface.
 
 The objective of pprof is to generate a report for a profile. The report is
 generated from a location hierarchy, which is reconstructed from the profile
-samples. Each location contains two values: *flat* is the value of the location
-itself, while *cum* is the value of the location plus all its
-descendants. Samples that include a location multiple times (eg for recursive
-functions) are counted only once per location.
+samples. Each location contains two values: 
+
+* *flat*: the value of the location itself.
+* *cum*: the value of the location plus all its descendants. 
+
+Samples that include a location multiple times (e.g. for recursive functions) 
+are counted only once per location.
 
 ## Options
 
@@ -81,19 +84,20 @@ other.
 Some common pprof options are:
 
 * **-flat** [default], **-cum**: Sort entries based on their flat or cumulative
-  weight respectively, on text reports.
+  value respectively, on text reports.
 * **-functions** [default], **-filefunctions**, **-files**, **-lines**,
   **-addresses**: Generate the report using the specified granularity.
 * **-noinlines**: Attribute inlined functions to their first out-of-line caller.
   For example, a command like `pprof -list foo -noinlines profile.pb.gz` can be
   used to produce the annotated source listing attributing the metrics in the
   inlined functions to the out-of-line calling line.
-* **-nodecount= _int_:** Maximum number of entries in the report. pprof will only print
-  this many entries and will use heuristics to select which entries to trim.
+* **-nodecount= _int_:** Maximum number of entries in the report. pprof will
+  only print this many entries and will use heuristics to select which entries
+  to trim.
 * **-focus= _regex_:** Only include samples that include a report entry matching
   *regex*.
-* **-ignore= _regex_:** Do not include samples that include a report entry matching
-  *regex*.
+* **-ignore= _regex_:** Do not include samples that include a report entry
+  matching *regex*.
 * **-show\_from= _regex_:** Do not show entries above the first one that
   matches *regex*.
 * **-show= _regex_:** Only show entries that match *regex*.
@@ -161,8 +165,8 @@ range) match a given sample, then the sample will be discarded.
 
 pprof text reports show the location hierarchy in text format.
 
-* **-text:** Prints the location entries, one per line, including the flat and cum
-  values.
+* **-text:** Prints the location entries, one per line, including the flat and
+  cum values.
 * **-tree:** Prints each location entry with its predecessors and successors.
 * **-peek= _regex_:** Print the location entry with all its predecessors and
   successors, without trimming any entries.
@@ -174,23 +178,68 @@ pprof can generate graphical reports on the DOT format, and convert them to
 multiple formats using the graphviz package.
 
 These reports represent the location hierarchy as a graph, with a report entry
-represented as a node. Solid edges represent a direct connection between
-entries, while dotted edges represent a connection where some intermediate nodes
-have been removed. Nodes are removed using heuristics to limit the size of
+represented as a node. Nodes are removed using heuristics to limit the size of
 the graph, controlled by the *nodecount* option.
 
-The size of each node represents the flat weight of the node, and the width of
-each edge represents the cumulative weight of all samples going through
-it. Nodes are colored according to their cumulative weight, highlighting the
-paths with the highest cum weight.
-
-* **-dot:** Generates a report in .dot format. All other formats are generated from
-  this one.
+* **-dot:** Generates a report in .dot format. All other formats are generated
+  from this one.
 * **-svg:** Generates a report in SVG format.
 * **-web:** Generates a report in SVG format on a temp file, and starts a web
   browser to view it.
-* **-png, -jpg, -gif, -pdf:** Generates a report in these formats,
-
+* **-png, -jpg, -gif, -pdf:** Generates a report in these formats.
+
+### Interpreting the Callgraph
+
+* **Flat Value**: the value of a location itself. This is indicated by the font 
+  size.
+    *   Nodes with larger font sizes have larger flat values.
+    *   Nodes with smaller font sizes have smaller flat values.
+
+* **Cum Value**: the value of a location plus all of its descendants. This is
+  indicated by the node's color.
+    *   Redder nodes have greater cum values.
+    *   Greyer nodes have smaller cum values.
+
+* **Dashed Edges**: some locations between the two connected locations were
+  removed.
+
+* **Solid Edges**: one location directly calls the other.
+
+* **Thicker & Redder Edges**: more resources (i.e. larger value) were used
+  along that path.
+
+* **Thinner & Greyer Edges**: fewer resources (i.e. smaller value) were used
+  along that path.
+
+* **"(inline)" Edge Marker**: the call has been inlined into the caller.
+
+Let's consider the following example graph:
+
+![callgraph](images/callgraph.png)
+
+* For nodes:
+  * `(*Rand).Read` has a small flat value and a small cum value because the
+    the font is small and the node is grey.
+  * `(*compressor).deflate` has a large flat value and a large cum value because the font 
+    is large and the node is red.
+  * `(*Writer).Flush` has a small flat value and a large cum value because the font is
+    small and the node is red.
+
+* For edges:
+  * the edge between `(*Writer).Write` and `(*compressor).write`: 
+    * Since it is a dashed edge, some nodes were removed between those two.
+    * Since it is thick and red, more resources were used in call stacks between
+    those two nodes.
+  * the edge between `(*Rand).Read` and `read`:
+    * Since it is a dashed edge, some nodes were removed between those two.
+    * Since it is thin and grey, fewer resources were used in call stacks 
+    between those two nodes.
+  * the edge between `read` and `(*rngSource).Int63`:
+    * Since it is a solid edge, there are no nodes between those two (i.e. it
+      was a direct call).
+    * Since it is thin and grey, fewer resrouces were used in call stacks
+      between those two nodes.
+
 ## Annotated code
 
 pprof can also generate reports of annotated source with samples associated to
@@ -210,7 +259,7 @@ search for them in a directory pointed to by the environment variable
 `$PPROF_TOOLS`.
 
 * **-list= _regex_:** Generates an annotated source listing for functions
-  matching *regex*, with flat/cum weights for each source line.
+  matching *regex*, with flat/cum values for each source line.
 * **-disasm= _regex_:** Generates an annotated disassembly listing for
   functions matching *regex*.
 * **-weblist= _regex_:** Generates a source/assembly combined annotated listing
@@ -264,12 +313,12 @@ also accept some legacy formats generated by
 When fetching from a URL handler, pprof accepts options to indicate how much to
 wait for the profile.
 
-* **-seconds= _int_:** Makes pprof request for a profile with the specified duration
-  in seconds. Only makes sense for profiles based on elapsed time, such as CPU
-  profiles.
-* **-timeout= _int_:** Makes pprof wait for the specified timeout when retrieving a
-  profile over http. If not specified, pprof will use heuristics to determine a
-  reasonable timeout.
+* **-seconds= _int_:** Makes pprof request for a profile with the specified
+  duration in seconds. Only makes sense for profiles based on elapsed time, such
+  as CPU profiles.
+* **-timeout= _int_:** Makes pprof wait for the specified timeout when
+  retrieving a profile over http. If not specified, pprof will use heuristics to
+  determine a reasonable timeout.
 
 pprof also accepts options which allow a user to specify TLS certificates to
 use when fetching or symbolizing a profile from a protected endpoint. For more