htable.save problem (table size dependent?)

[znruss]

normalize-by-median.py and load-into-counting.py seem to be working on my system, but the htable.save command seems to fail silently. The memory allocates properly when initializing the table, and normalize-by-median.py runs and generates an appropriate .keep file, but the saved .kh is only 16 bytes. The problem seems to be sensitive to the minimum table size but not number of tables or kmer size or the bigcount flag. x=4e9 causes this problem, 2e9 and 1e9 do not.
Version:

khmer version 1.0-11-g059a8e3
screed version 0.7

Here are some of my tests:

hexdump -C k19n4x4e9.kh
00000000  04 01 01 13 00 00 00 04  07 28 6b ee 00 00 00 00  |.........(k.....|
00000010
hexdump -C k20n4x4e9.kh
00000000  04 01 01 14 00 00 00 04  07 28 6b ee 00 00 00 00  |.........(k.....|
00000010
hexdump -C k19n1x4e9.kh
00000000  04 01 01 13 00 00 00 01  07 28 6b ee 00 00 00 00  |.........(k.....|
00000010
hexdump -C k19n4x2e9.kh
00000000  04 01 01 13 00 00 00 04  0b 94 35 77 00 00 00 00  |..........5w....|
00000010  ff 13 0b 06 02 03 0f 00  08 04 03 01 07 00 00 01  |................|
00000020  02 00 00 00 00 02 00 02  0d 00 02 00 00 00 01 00  |................|
00000030  05 01 01 03 00 02 01 01  00 03 02 00 00 00 01 00  |................|
... (this seems like a good kh)

I also ran ./setup.py nosetests (295 tests, 8 seconds, no fails). Is there anything else I should test? Thanks!

[ctb]

This is very puzzling BUT we do know that there are problems with loading/saving so I am eager to track it down. @znruss, two questions -- first, what platform are you running on? and second, do you have a simple script that reproduces the problem on your platform? thanks!

[znruss]

Here's my test script. test.fasta contains 25k sequences, but it seems to be size independent (though obviously less sequences lets me get to the problem faster)

import sys
import threading
import textwrap
import khmer
from khmer.khmer_args import build_counting_args, report_on_config, info
from khmer.threading_args import add_threading_args
from khmer.file import check_file_status, check_space
from khmer.file import check_space_for_hashtable
k = 19
n = 4
x = float(4*(10**9))
t = 1

info('load-into-counting.py', ['counting'])
check_space_for_hashtable(n * x)
htable = khmer.new_counting_hash(k, x,n, t)
htable.set_use_bigcount(False)
config = khmer.get_config()
config.set_reads_input_buffer_size(t * 64 * 1024)
rparser = khmer.ReadParser('test.fasta', t)
threads = []
for _ in xrange(1):
    cur_thrd = threading.Thread(target=htable.consume_fasta_with_reads_parser,args=(rparser, ))
    threads.append(cur_thrd)
    cur_thrd.start()

for _ in threads:
    _.join()

htable.save('test.kh')

My platform is some kind of redhat.
Linux version 2.6.32-042stab055.10 (root@rh6-build-x64) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Thu May 10 15:38:32 MSD 2012
gcc is gcc (GCC) 4.4.6 20110731 (Red Hat 4.4.6-3)

I'm currently recompiling a new version of gcc to see if that solves the problem (since the saving seems to be handled by https://github.com/ged-lab/khmer/blob/master/khmer/_khmermodule.cc#L3091

[znruss]

this is run in python interactive mode btw, but it doesn't seem to matter

[ctb]

OK, that's disturbing. Thanks, we'll look into it!

[ctb]

Hi @znruss, I cannot replicate the problem on athyra (our local development server). So that's good to know :). Note to ged-lab folk: see ~t/dev/khmer/tst.py on athyra.

[znruss]

I did a little more testing and found the switch between work and fail happens - it's between 2.1e9 2.2e9 which conveniently happens to be 2^31 or int32. Still messing with gcc to see if it was a compile problem.

[znruss]

I'm pretty sure the problem occurs with this command:
https://github.com/ged-lab/khmer/blob/master/lib/counting.cc#L695
but I don't know why it fails when save_tablesize is greater than the maximum value of int32.

With new recompiled gcc 4.8.2, problem persists.
I tried flags like -m64, -D_LARGEFILE_SOURCE, etc. with no apparent effect.
Finally, checking the size of things like streamoff and streamsize both gave 8 bytes or int64, so they should be fine with save_tablesize.

Additional data: distro is CentOS release 6.2 (Final).

The problem isn't in khmer at all, it's some issue with the underlying system. Here's a c++ program that fails to write on my machine when the buffer size is larger than 2e9, just like the hashtable problem. It should work fine on your system.

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <fstream>
#include <iostream>
#include <limits>

int main()
{
      using namespace std;
int i;
int64_t buffsize=4000000000;
char * writebuffer;
writebuffer=(char*) malloc (buffsize+1);
    cout << numeric_limits<streamoff>::max() << endl;
    ofstream outfile("test.txt", ios::binary);
    cout << outfile.is_open();
  cout << outfile.fail();
 outfile.write((const char *) writebuffer, buffsize);
    cout << outfile.is_open();
  cout << outfile.fail();
perror("A");
outfile.close();
}

The error it gives is "Bad address"

[mr-c]

@znruss Feel free to update this issue later; but for now I'm going to close it.

[znruss]

Certainly. I just have a couple more details about the problem:
The system was actually virtualized and the filesystem was NFS based. I managed to confirm that NFS was responsible for failing to complete writes >2GB. However, when using a local ext4 disk, the failbit was still being set even though the write was successful; the set failbit was causing all subsequent writes to fail. Since khmer writes its table in multiple steps if N>1 this is an issue, but I would chalk it up to a very particular issue with our virtualized setup.

I have a fix that breaks the writes up so that the table is written 2GB at a time, and I can contribute that if desired. Thanks!

[mr-c]

Interesting!

Please send a pull request, thanks :-)

On Fri, May 9, 2014 at 5:42 PM, znruss notifications@github.com wrote:

Certainly. I just have a couple more details about the problem:
The system was actually virtualized and the filesystem was NFS based. I
managed to confirm that NFS was responsible for failing to complete writes

2GB. However, when using a local ext4 disk, the failbit was still being
set even though the write was successful; the set failbit was causing all
subsequent writes to fail. Since khmer writes its table in multiple steps
if N>1 this is an issue, but I would chalk it up to a very particular issue
with our virtualized setup.

I have a fix that breaks the writes up so that the table is written 2GB at
a time, and I can contribute that if desired. Thanks!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/408#issuecomment-42717146
.