visitor functions

[sbromberger]

This is the BFS algorithm from Experimental.ShortestPaths:

julia> @benchmark shortest_paths(g, 1, BFS())                                                                                                
BenchmarkTools.Trial:                                                                                                                        
  memory estimate:  30.64 MiB                                                                                                                
  allocs estimate:  14                                                                                                                       
  --------------                                                                                                                             
  minimum time:     421.353 ms (0.00% GC)                                                                                                    
  median time:      425.748 ms (0.00% GC)                                                                                                    
  mean time:        428.300 ms (0.06% GC)                                                                                                    
  maximum time:     463.228 ms (0.24% GC)                                                                                                    
  --------------                                                                                                                             
  samples:          12                                                                                                                       
  evals/sample:     1                                                                                                                        

Here's bfs_visit with inlined NOP functions (see code):

julia> @benchmark bfs_visit(g, [1])
BenchmarkTools.Trial: 
  memory estimate:  30.64 MiB
  allocs estimate:  13
  --------------
  minimum time:     419.395 ms (0.00% GC)
  median time:      421.325 ms (0.00% GC)
  mean time:        421.280 ms (0.06% GC)
  maximum time:     425.055 ms (0.28% GC)
  --------------
  samples:          12
  evals/sample:     1

Ref: #1272

[sbromberger]

The idea here would be to make a traverse_graph function that takes a graph, a set of sources, an algorithm (BFS or DFS), a state object, and visitor functions, and would return the state after traversal.

This first cut is to demonstrate that inlined NOP visitor functions do not impact performance.

[codecov]

Codecov Report

Merging #1278 into master will increase coverage by 0.52%.
The diff coverage is 99.18%.

@@            Coverage Diff             @@
##           master    #1278      +/-   ##
==========================================
+ Coverage   99.14%   99.66%   +0.52%     
==========================================
  Files         106      107       +1     
  Lines        5053     5139      +86     
==========================================
+ Hits         5010     5122     +112     
+ Misses         43       17      -26

[sbromberger]

BFS shortest paths vs visited vertices:

BFS:

julia> @benchmark shortest_paths(g, 1, BFS())                                                                                                
BenchmarkTools.Trial:                                                                                                                        
  memory estimate:  30.64 MiB                                                                                                                
  allocs estimate:  14                                                                                                                       
  --------------                                                                                                                             
  minimum time:     410.802 ms (0.00% GC)                                                                                                    
  median time:      413.111 ms (0.00% GC)
  mean time:        415.450 ms (0.05% GC)
  maximum time:     435.784 ms (0.00% GC)
  --------------
  samples:          13
  evals/sample:     1

Visited vertices:

julia> @benchmark visited_vertices(g, [1], BFS())
BenchmarkTools.Trial:              
  memory estimate:  38.27 MiB
  allocs estimate:  15
  --------------
  minimum time:     437.776 ms (0.00% GC)
  median time:      442.688 ms (0.00% GC)
  mean time:        448.781 ms (0.02% GC)
  maximum time:     485.264 ms (0.00% GC)
  --------------
  samples:          12
  evals/sample:     1

(g = (1m, 5m) undirected)

[sbromberger]

Further optimized:

BFS shortest paths:

julia> @benchmark shortest_paths(g, 1, BFS())                                                                                         [5/843]
BenchmarkTools.Trial:                                                                                                                        
  memory estimate:  61.27 MiB                                                                                                                
  allocs estimate:  14                                                                                                                       
  --------------                                                                                                                             
  minimum time:     1.444 s (0.00% GC)                                                                                                       
  median time:      1.451 s (0.00% GC)
  mean time:        1.450 s (0.00% GC)
  maximum time:     1.454 s (0.00% GC)
  --------------
  samples:          4
  evals/sample:     1

Visited vertices:

julia> @benchmark visited_vertices(g, [1], BFS())
BenchmarkTools.Trial: 
  memory estimate:  46.02 MiB
  allocs estimate:  11
  --------------
  minimum time:     1.341 s (0.00% GC)
  median time:      1.362 s (0.00% GC)
  mean time:        1.367 s (0.00% GC)
  maximum time:     1.404 s (0.00% GC)
  --------------
  samples:          4
  evals/sample:     1

(This is a 2m, 20m graph)

[sbromberger]

Wow, look at this:

julia> g = Graph(2_000_000, 20_000_000)                                                                                                      
{2000000, 20000000} undirected simple Int64 graph    

BFS shortest paths:

julia> @benchmark shortest_paths(g, 1, BFS())                                                                                                
BenchmarkTools.Trial:                                                                                                                        
  memory estimate:  61.27 MiB                                                                                                                
  allocs estimate:  14                                                                                                                       
  --------------                                                                                                                             
  minimum time:     1.429 s (0.00% GC)                                                                                                       
  median time:      1.435 s (0.00% GC)
  mean time:        1.434 s (0.00% GC)
  maximum time:     1.438 s (0.00% GC)
  --------------
  samples:          4
  evals/sample:     1

bfssp (bfs shortest paths using traverse_graph):

julia> @benchmark bfssp(g, [1], BFS())
BenchmarkTools.Trial: 
  memory estimate:  61.27 MiB      
  allocs estimate:  14
  --------------
  minimum time:     1.425 s (0.00% GC)
  median time:      1.427 s (0.00% GC)
  mean time:        1.428 s (0.00% GC)
  maximum time:     1.435 s (0.00% GC)
  --------------
  samples:          4
  evals/sample:     1

[sbromberger]

I plan on creating documentation and tests for this and then merging since it's Experimental. The larger plan is to then deprecate the current shortest paths algorithms for the ones in Experimental, tag a new minor version in the next week or two, then move Experimental.ShortestPaths into the main codebase, and tag a new major version in the next couple of months.

cc @matbesancon @jpfairbanks @simonschoelly @somil55 for feedback on this plan.

[matbesancon]

sounds good to me. Before releasing a major release, we should check what needs breaking to do it once.

[sbromberger]

New topological sort (getting ready to push) is a bit slower but perhaps still acceptable (and there might be optimizations we can do):

julia> @benchmark topological_sort_by_dfs(dg)                                                                                      [118/1924]
BenchmarkTools.Trial:                                                                                                                        
  memory estimate:  69.00 MiB                                                                                                                
  allocs estimate:  32                                                                                                                       
  --------------                                                                                                                             
  minimum time:     138.882 ms (0.00% GC)                                                                                                    
  median time:      140.795 ms (0.00% GC)                                                                                                    
  mean time:        142.838 ms (1.15% GC)                                                                                                    
  maximum time:     161.023 ms (10.07% GC)                                                                                                   
  --------------                                                                                                                             
  samples:          35                                                                                                                       
  evals/sample:     1                                                                                                                        
                                                                                                                                             
julia> @benchmark topological_sort(dg)                                                                                                       
BenchmarkTools.Trial:                                                                                                                        
  memory estimate:  69.50 MiB                                                                                                                
  allocs estimate:  36                                                                                                                       
  --------------                                                                                                                             
  minimum time:     154.339 ms (0.00% GC)                                                                                                    
  median time:      156.935 ms (0.00% GC)                                                                                                    
  mean time:        159.195 ms (1.05% GC)                                                                                                    
  maximum time:     174.461 ms (8.74% GC)                                                                                                    
  --------------                                              
  samples:          32             
  evals/sample:     1

[simonschoelly]

New topological sort (getting ready to push) is a bit slower but perhaps still acceptable (and there might be optimizations we can do):

From a quick glance it looks as if the old topological sort was not implemented verify efficiently.

[sbromberger]

From a quick glance it looks as if the old topological sort was not implemented verify efficiently.

If you have any suggestions for improvement, please suggest them in the new topological sort, which is essentially the same as the old one except we're using the new traverse_graph! function.

[sbromberger]

I am going to merge this since it's in Experimental. I will be tagging a new release (1.3.1) in the next week.