Skip to content

Commit

Permalink
Added sort performance tests
Browse files Browse the repository at this point in the history
  • Loading branch information
kmsquire committed Jul 22, 2013
1 parent c9a7206 commit 453b4ef
Show file tree
Hide file tree
Showing 3 changed files with 95 additions and 4 deletions.
11 changes: 7 additions & 4 deletions test/perf/Makefile
Original file line number Diff line number Diff line change
@@ -1,15 +1,18 @@
JULIAHOME = $(abspath ../..)
include ../../Make.inc

all: micro kernel cat shootout
all: micro kernel cat shootout sort

micro kernel cat shootout:
@$(MAKE) $(QUIET_MAKE) -C shootout
micro kernel cat sort:
@$(call spawn,$(JULIA_EXECUTABLE)) $@/perf.jl | perl -nle '@_=split/,/; printf "%-18s %8.3f %8.3f %8.3f %8.3f\n", $$_[1], $$_[2], $$_[3], $$_[4], $$_[5]'

shootout:
@$(MAKE) $(QUIET_MAKE) -C $@
@$(call spawn,$(JULIA_EXECUTABLE)) $@/perf.jl | perl -nle '@_=split/,/; printf "%-18s %8.3f %8.3f %8.3f %8.3f\n", $$_[1], $$_[2], $$_[3], $$_[4], $$_[5]'

clean:
rm -f *~
$(MAKE) -C micro $@
$(MAKE) -C shootout $@

.PHONY: micro kernel cat shootout clean
.PHONY: micro kernel cat shootout clean sort
19 changes: 19 additions & 0 deletions test/perf/perfutil.jl
Original file line number Diff line number Diff line change
Expand Up @@ -31,5 +31,24 @@ macro timeit1(ex,name)
end
end

macro timeit_init(ex,init,name)
quote
t = zeros(ntrials)
for i=0:ntrials
$(esc(init))
e = 1000*(@elapsed $(esc(ex)))
if i > 0
# warm up on first iteration
t[i] = e
end
end
if print_output
@printf "julia,%s,%f,%f,%f,%f\n" $name min(t) max(t) mean(t) std(t)
end
gc()
end
end


# seed rng for more consistent timings
srand(1776)
69 changes: 69 additions & 0 deletions test/perf/sort/perf.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
import Base.Sort: QuickSort, MergeSort, TimSort, InsertionSort

include("../perfutil.jl")

sorts = [QuickSort, MergeSort, TimSort, InsertionSort]

randstr_fn!(str_len::Int) = d -> (for i = 1:length(d); d[i] = randstring(str_len); end; d)
randint_fn!(m::Int) = d -> rand!(1:m,d)

for (T, typename, randfn!) in Any[(Int, string(Int), randint_fn!(10)),
(Float64, string(Float64), rand!),
(String, "String_05", randstr_fn!(5)),
(String, "String_10", randstr_fn!(10))]
for logsize = 6:2:18
size = 2^logsize
for s in sorts
if s == InsertionSort && logsize >=14; continue; end
data = Array(T, size)
gc()

## Random
name = "$(typename)_$(logsize)_$(string(s)[1:end-5])_random"
@timeit_init(sort!(data, alg=s), randfn!(data), name)

## Sorted
name = "$(typename)_$(logsize)_$(string(s)[1:end-5])_sorted"
@timeit(sort!(data, alg=s), name)

## Reverse sorted
name = "$(typename)_$(logsize)_$(string(s)[1:end-5])_reversed"
@timeit_init(sort!(data, alg=s), reverse!(data), name)

## Sorted with 3 exchanges
name = "$(typename)_$(logsize)_$(string(s)[1:end-5])_3exchanges"
@timeit_init(sort!(data, alg=s),
begin
for i = 1:3
n1 = rand(1:size)
n2 = rand(1:size)
data[n1], data[n2] = data[n2], data[n1]
end
end,
name)

## Sorted with 10 unsorted values appended
name = "$(typename)_$(logsize)_$(string(s)[1:end-5])_append"
@timeit_init(sort!(data, alg=s), begin data[end-9:end]=randfn!(Array(T,10)) end, name)

## Random data with 4 unique values
name = "$(typename)_$(logsize)_$(string(s)[1:end-5])_4unique"
@timeit_init(sort!(data4, alg=s), begin data4=data[rand(1:4,size)] end, name)

## All values equal
name = "$(typename)_$(logsize)_$(string(s)[1:end-5])_allequal"
data1 = data[ones(Int, size)]
@timeit(sort!(data1, alg=s), name)

## QuickSort median killer
if s == QuickSort && logsize > 16; continue; end # too slow!

name = "$(typename)_$(logsize)_$(string(s)[1:end-5])_qsortkiller"
data = data[1:size>>1]
data = sort!(data, alg=s)
data = vcat(reverse(data), data)
@timeit_init(sort!(qdata, alg=s), begin qdata=copy(data) end, name)
end
end
end

7 comments on commit 453b4ef

@staticfloat
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I've tried this out, this is..... a LOT of tests. I'm not sure we want to throw these into codespeed, as even visualizing ~800 tests is pretty overwhelming. I could special-case a few into codespeed (I have if codespeed checks in perfutil.jl, so the same can happen here), say about a dozen..... which ones do you think would be most informative/interesting to track?

@ViralBShah
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think everything in the perfsuite should have something that is comprehensive, and a subset that is run repeatedly for reporting - like a handful of tests. I suspected we would need this a while back, but it has come up much sooner than I expected. :-)

@kmsquire
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, for codespeed, it would proabably be better just to do a few tests here. Maybe something like:

  • 2 sizes (2^6 and 2^18) (only use 2^6 for InsertionSort)
  • random and sorted with 10 random appended
  • String and Int

That brings it down to 28 tests. Reasonable, or still too many?

As soon as #3809 is resolved, I'll submit https://github.com/kmsquire/SortPerf.jl to METADATA.jl, if anyone wants to run something more comprehensive or test their own data types.

@Keno
Copy link
Member

@Keno Keno commented on 453b4ef Jul 24, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't you just run a small subset of them individually for reporting and then run another one that just adds the results of the entire sort perf suite?

@ViralBShah
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - I suggest keeping all the tests in perf and not doing a SortPerf.jl package. In any case we should have a way to do quick performance tests, and comprehensive performance tests.

@staticfloat
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keno and I are on the same wavelength; I can special-case some tests into codespeed reporting, so that make codespeed runs the "dozen" tests, and make sort (or just make) runs them all.

I'll do those few tests you suggested, Kevin, we'll see what it looks like. It's certainly better than ~800!

@kmsquire
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ViralBShah, one of the things I've done in SortPerf.jl is to make it easy for users to test sorting of data types other than Ints, Floats, or Strings using different algorithms. Obviously, deep down, pretty much everything made up of those types, but there may be characteristics of certain datasets that suggest one algorithm over another, and this would let users test that. I don't think that really belongs in perf. What do you think?

Please sign in to comment.