countlines slower on master than v0.4

[zhmz90]

Compare countlines on master and v0.4

On master:

julia> @time countlines("../data/ref/ucsc.hg19/ucsc.hg19.fasta")
  3.453905 seconds (167 allocations: 18.294 KB)
62743362

julia> @time countlines("../data/ref/ucsc.hg19/ucsc.hg19.fasta")
  3.466343 seconds (20 allocations: 9.328 KB)
62743362

julia> @time countlines("../data/ref/ucsc.hg19/ucsc.hg19.fasta")
  3.455346 seconds (20 allocations: 9.328 KB)
62743362

julia> 

On Jula v0.4:

guo@x02:~/haplox/private/Zhongshan/sec_src$ ~/julia_v4/julia 
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.4.6-pre+37 (2016-05-27 22:56 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit 430601c (11 days old release-0.4)
|__/                   |  x86_64-linux-gnu

julia> @time countlines("../data/ref/ucsc.hg19/ucsc.hg19.fasta")
  1.928987 seconds (806.67 k allocations: 48.869 MB, 1.99% gc time)
62743362

julia> @time countlines("../data/ref/ucsc.hg19/ucsc.hg19.fasta")
  1.885332 seconds (781.25 k allocations: 47.691 MB, 0.61% gc time)
62743362

But the speed of readall/readstring seems close between master and v0.4.

compare the speed of readall/readstring on master and v0.4

guo@x02:~/haplox/private/Zhongshan/sec_src$ julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.0-dev+4582 (2016-06-07 22:00 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit d98f2c0* (0 days old master)
|__/                   |  x86_64-linux-gnu

julia> @time readstring("../data/ref/ucsc.hg19/ucsc.hg19.fasta");
  2.201073 seconds (185 allocations: 5.960 GB, 4.11% gc time)

julia> 
guo@x02:~/haplox/private/Zhongshan/sec_src$ ~/julia_v4/julia 
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.4.6-pre+37 (2016-05-27 22:56 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit 430601c (11 days old release-0.4)
|__/                   |  x86_64-linux-gnu

julia> @time readall("../data/ref/ucsc.hg19/ucsc.hg19.fasta");
  2.175163 seconds (17.95 k allocations: 2.981 GB, 0.53% gc time)

version info

My Julia master is the latest now:

julia> versioninfo()
Julia Version 0.5.0-dev+4582
Commit d98f2c0* (2016-06-07 22:00 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, haswell)

Julia v0.4 version is:

  _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.4.6-pre+37 (2016-05-27 22:56 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit 430601c (11 days old release-0.4)
|__/                   |  x86_64-linux-gnu

julia> versioninfo()
Julia Version 0.4.6-pre+37
Commit 430601c (2016-05-27 22:56 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

[StefanKarpinski]

I don't see a slowdown on my system, but I also don't have access to that data file, so it's hard to tell. Can you try this redefinition of Base.countlines:

function Base.countlines(io::IO, eol::Char='\n')
    isascii(eol) || throw(ArgumentError("only ASCII line terminators are supported"))
    aeol = UInt8(eol)
    a = Array{UInt8}(8192)
    nl = 0
    while !eof(io)
        nb = readbytes!(io, a)
        @simd for i=1:nb
            @inbounds nl += a[i] == aeol
        end
    end
    nl
end
Base.countlines(f::AbstractString, eol::Char='\n') = open(io->countlines(io,eol), f)::Int

[zhmz90]

julia> @time countlines("../data/ref/ucsc.hg19/ucsc.hg19.fasta")
  1.321105 seconds (14.45 k allocations: 662.756 KB)
62743362

julia> @time countlines("../data/ref/ucsc.hg19/ucsc.hg19.fasta")
  1.315232 seconds (20 allocations: 9.328 KB)
62743362

Now the speed of countlines on the master is faster than v0.4 with your redefinition of Base.countlines.

It seems the slowdown on master before is caused by UInt8(eol) .

Thanks. @StefanKarpinski

is this similar pattern can be optimized by the compiler since eol in UInt8(eol) doesn't change in the loop?

[StefanKarpinski]

I had originally written the function that way but @nalimilan convinced me that I didn't need to manually do this transformation, but that seems to be inconsistent across different systems, which is concerning. Does anyone have any ideas as to why that optimization would kick in on some systems and not others?