panic: runtime error: index out of range

[despens]

Version 1.0.0 crashes when analysing a large file set. If required, I can find out the exact point or file this happens with.

panic: runtime error: index out of range

goroutine 526 [running]:
github.com/richardlehane/mscfb.(*Reader).fileOffset(0xc208505e30, 0x1ffffffff, 0xffffffff)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/mscfb/mscfb.go:97 +0x7d
github.com/richardlehane/mscfb.(*Reader).stream(0xc208505e30, 0x140000c2ffffffff, 0x58, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/mscfb/streams.go:90 +0x3d5
github.com/richardlehane/mscfb.(*File).Read(0xc20a8c0360, 0xc20a8c2000, 0x1000, 0x2000, 0x7f9cf3540000, 0x0, 0x0)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/mscfb/directory.go:144 +0xaa
github.com/richardlehane/siegfried/pkg/core/siegreader.(*stream).fill(0xc20a8c04e0, 0xc20a89b600, 0x0, 0x0)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/pkg/core/siegreader/stream.go:108 +0x23d
github.com/richardlehane/siegfried/pkg/core/siegreader.(*stream).Slice(0xc20a8c04e0, 0x0, 0x1000, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/pkg/core/siegreader/stream.go:132 +0xdb
github.com/richardlehane/siegfried/pkg/core/siegreader.(*Reader).setBuf(0xc20a8a32c0, 0x0, 0x0, 0x0)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/pkg/core/siegreader/reader.go:46 +0x88
github.com/richardlehane/siegfried/pkg/core/siegreader.(*LimitReader).ReadByte(0xc20a8afa40, 0xc20a89b600, 0x0, 0x0)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/pkg/core/siegreader/reader.go:227 +0x1b5
github.com/richardlehane/match/wac.(*Wac).match(0xc20a8bb320, 0x7f9cf1d34000, 0xc20a8afa40, 0xc20a8c0840)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/match/wac/wac.go:280 +0xb5
created by github.com/richardlehane/match/wac.(*Wac).Index
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/match/wac/wac.go:263 +0x6e

goroutine 1 [chan receive]:
github.com/richardlehane/siegfried.(*Siegfried).Identify(0xc208010af0, 0xc20a8bb140, 0x12, 0x7f9cf3551c88, 0xc20833ef70, 0x0, 0x0, 0x0)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/siegfried.go:371 +0x5b8
main.identifyFile(0x7f9cf355d2b0, 0xc20852fcb0, 0xc208010af0, 0xc20a8bb140, 0x12, 0x8200)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:132 +0x2c2
main.func·004(0xc20a8bb140, 0x12, 0x7f9cf35520a0, 0xc20a8be140, 0x0, 0x0, 0x0, 0x0)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:119 +0x158
path/filepath.walk(0xc20a8bb140, 0x12, 0x7f9cf35520a0, 0xc20a8be140, 0xc20878fd60, 0x0, 0x0)
    /usr/local/go/src/path/filepath/path.go:347 +0x91
path/filepath.walk(0xc20a895ef8, 0x8, 0x7f9cf35520a0, 0xc20a8736d0, 0xc20878fd60, 0x0, 0x0)
    /usr/local/go/src/path/filepath/path.go:372 +0x51d
path/filepath.walk(0x7ffdedead2b8, 0x1, 0x7f9cf35520a0, 0xc20850cd20, 0xc20878fd60, 0x0, 0x0)
    /usr/local/go/src/path/filepath/path.go:372 +0x51d
path/filepath.Walk(0x7ffdedead2b8, 0x1, 0xc20878fd60, 0x0, 0x0)
    /usr/local/go/src/path/filepath/path.go:394 +0xf2
main.multiIdentifyS(0x7f9cf355d2b0, 0xc20852fcb0, 0xc208010af0, 0x7ffdedead2b8, 0x1, 0x0, 0x0, 0x0)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:122 +0x94
main.main()
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:224 +0xd83

goroutine 525 [chan receive]:
github.com/richardlehane/siegfried/pkg/core/bytematcher.(*scorer).score(0xc208505f10)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/pkg/core/bytematcher/scorer.go:73 +0x5d
created by github.com/richardlehane/siegfried/pkg/core/bytematcher.(*Matcher).newScorer
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/pkg/core/bytematcher/scorer.go:68 +0x5a2

goroutine 522 [chan receive]:
github.com/richardlehane/siegfried/pkg/core/containermatcher.(*CTest).identify(0xc2084f9040, 0xc2084ed880, 0xc20a8be280, 0x7f9cf1d3c438, 0xc20a8af9f0, 0xc20a8af9e0, 0x7, 0x0, 0x0, 0x0)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/pkg/core/containermatcher/identify.go:120 +0x3d2
github.com/richardlehane/siegfried/pkg/core/containermatcher.(*ContainerMatcher).identify(0xc2084ed880, 0x7f9cf1d3c438, 0xc20a8af9f0, 0xc20a8c0180)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/pkg/core/containermatcher/identify.go:101 +0x1f4
created by github.com/richardlehane/siegfried/pkg/core/containermatcher.Matcher.Identify
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/pkg/core/containermatcher/identify.go:48 +0x3c0

goroutine 523 [chan receive]:
github.com/richardlehane/siegfried/pkg/core/bytematcher.(*Matcher).identify(0xc2084f2c60, 0x7f9cf1d3c4a0, 0xc20a8c04e0, 0xc20a8c05a0, 0xc20a8c0600)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/pkg/core/bytematcher/identify.go:63 +0x1a2
created by github.com/richardlehane/siegfried/pkg/core/bytematcher.(*Matcher).Identify
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/pkg/core/bytematcher/bytematcher.go:164 +0xad

goroutine 524 [select]:
github.com/richardlehane/siegfried/pkg/core/bytematcher.(*scorer).filterHits(0xc208505f10)
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/pkg/core/bytematcher/scorer.go:530 +0x2ab
created by github.com/richardlehane/siegfried/pkg/core/bytematcher.(*Matcher).newScorer
    /home/vagrant/packaging/siegfried-1.0.0/obj-x86_64-linux-gnu/src/github.com/richardlehane/siegfried/pkg/core/bytematcher/scorer.go:67 +0x588

[despens]

I have identified the directory that causes the panic.
@richardlehane, please let me know if I should send it to you as a sample.

[richardlehane]

I'm sorry you keep running into issues, but I really appreciate the reports!
This one looks to be a bug with https://github.com/richardlehane/mscfb: the library that siegfried uses to parse OLE files. The problem should be with a single OLE file, likely a doc, xls, or thumbs.db or something like that. I think I can spot the line that is causing this, but a sample would really help. Can use richard@itforarchivists.com or I can share a dropbox folder with you if too large for email.

[despens]

I identified the file and sent it to you.