Parse failure when crossing chunk boundaries

[pjones]

I believe I've found a bug in attoparsec and I need some help writing a failing test. Some background:

I have a tool that processes CSV files using Cassava. After upgrading attoparsec I encountered a regression in my tool while processing production files. The problem only happens in some of the CSV files, and only when the files are big enough to be fed into attoparsec in chunks. Feeding only whole lines into attoparsec works fine.

After some experimenting and some help from git bisect, I've tracked the issue down to commit 791e046c526710dce7b87e308ee48f2fb6811d7b. Before that commit all my regression tests pass. After that commit they fail every time.

I haven't yet been able to shrink this down to a small, reproducible test case. The tool reads chunks of ByteString.Char8 values using io-streams and feeds them into Cassava, which feeds them into attoparsec. The smallest CSV file that produces the problem is about 2MB.

Due to time constraints I don't have the spare cycles to track this down further right now. Since I can lock cabal down to the last working version of attoparsec it's not a show stopper for me. That said, if someone else wants to chase this down I'd be happy to try out patches or provide more detail.

[basvandijk]

Hi @pjones, thanks for reporting this!

Is it possible you can share your tool's source code with us? If not, because of secrecy reasons for example, could you strip out the sensitive parts?

It will also be helpfull if you can share the CSV file and the error message you are getting.

Thanks!

[pjones]

Here's a small program and CSV file which reproduces the bug:

http://www.pmade.com/static/tmp/boundary-bug.tar.bz2

I'll delete that tarball after a couple of days.

The error message is: "parse error (endOfInput)" which I suspect is partially constructed by the cassava Parser and attoparsec.

[bos]

Thanks for the repro, @pjones. The error is being reported from Data.Csv.Incremental.decodeWithP. In fact, merely finding that location took me a while, as there were several packages that could have been reporting that exact error message.

I've been able to cut your CSV file down to 29% of its original size while preserving the behaviour. No idea yet what's actually going on, but a smaller test case has to be strictly better than a larger one ... right?

(Oh, and cc @tibbe for good measure.)

[bos]

I had a wild notion that the problem must have something to do with the sizes of the chunks of data being fed to the parser, and amazingly this turned out to be a helpful direction to pursue.

I've now got the failing input trimmed down to just two lines: the header and a row of actual data. This gives a fighting chance at figuring out what's wrong.

[pjones]

Awesome work @bos, thanks!

[bos]

I should mention that I released a new version of attoparsec already that has this fixed. Sorry for the inconvenience!