ERROR ArcRecordUtils - Read 1224 bytes but expected 1300 bytes

[ruebot]

This is coming up a lot in the build.

Example:

2017-11-30 09:19:54,601 [Executor task launch worker for task 0] ERROR ArcRecordUtils - Read 1224 bytes but expected 1300 bytes. Continuing...
WARNING Record STARTING at 0 has 1761 trailing byte(s): file:/home/nruest/git/aut/target/test-classes/arc/badexample.arc.gz: {subject-uri=filedesc://IAH-20080430204825-00000-blackbook.arc, ip-address=0.0.0.0, origin=InternetArchive, length=1300, absolute-offset=0, creation-date=20080430204825, content-type=text/plain, version=1.1}

[ianmilligan1]

We had this back in Warcbase - see issue here lintool/warcbase#199.

According to this it should have been fixed in webarchive-commons (1.1.7). @jrwiebe did quite a bit of digging on this problem.

[ruebot]

@anjackson any chance this resurfaced in webarchive-commons 1.1.8?

[anjackson]

@ruebot Possibly, although the changeset doesn't seem to affect the original fix. Any chance you've got an older version on the classpath somewhere?

[ruebot]

+- org.netpreserve.commons:webarchive-commons:jar:1.1.8:compile is the only webarchive-commons package that comes up with mvn dependency:tree.

[anjackson]

Hmmm, I'm confused. Your version of ArcRecordUtils doesn't have the additional code @jrwiebe added to cope with this situation.

Or am I misunderstanding the issue?

[ruebot]

Oh, that weird.

[ruebot]

@lintool https://github.com/lintool/warcbase/blob/7bdc8b55fdbf96a6fd5a246de761fb344563ce1f/src/main/java/org/warcbase/data/ArcRecordUtils.java#L61-L66 vs https://github.com/lintool/warcbase/commits/master/warcbase-core/src/main/java/org/warcbase/data/ArcRecordUtils.java

Do you know what happen to the git history there? When you split it up to warcbase-core, do you remember if you used git mv or just mv? And, did @jrwiebe's work get refactored in what we have now, or removed?

[ianmilligan1]

I think that predates the separation btw warcbase-core and other.

But aha, the branch @jrwiebe was working on was never merged. Here's the differences there: https://github.com/lintool/warcbase/compare/arc-tobytes

[ianmilligan1]

Shoot, I wish we'd merged this.

After @jrwiebe did this branch, we then had the split between warcbase-core and warcbase-hbase, and then subsequent i/o handling changes.

I don't think there's a straightforward way to port this stuff over (without knowing Java, which I don't).

Another option is to suppress some of the error handling by default on expecting X bytes but getting Y bytes.

[jrwiebe]

Since we know why that error is occurring, you could fairly safely suppress the error for now -- perhaps only for cases where the difference between bytes-read and bytes-expected is 76 (see (lintool/warcbase#199 (comment)) -- until someone has time to merge my branch.

[ruebot]

Resolved with 3eb093a