Mmap support for reference sequences.

[jkbonfield]

The mmap support is useful for reducing memory usage and also minimising disk I/O on querying small regions.

The second commit is rather orthogonal to this, but still desireable for portability as it means I can detect whether mmap is available.

[jmarshall]

I've merged 3af49b4 and added AC_FUNC_MMAP to a config.h-based #define configury (see 5f5aa02).

[ekg]

@jkbonfield Do you have examples of tests showing that mmap is better than seeking? I experimented quite a bit with this in fastahack and didn't notice any difference in performance. Similar results in rocksdb, which allows mmapping the db files although notes that this tends to decrease performance.

In my experience, it seems mostly to make coding easier by allowing files to look like large contiguous regions of memory. In principle it seems great, so I'm curious if there might be a particular way to use it that yields better performance.

[jkbonfield]

On Wed, Apr 01, 2015 at 03:30:31AM -0700, Erik Garrison wrote:

@jkbonfield Do you have examples of tests showing that mmap is
better than seeking? I experimented quite a bit with this in fastahack
and didn't notice any difference in performance. Similar results in
rocksdb, which allows mmapping the db files although notes that this
tends to decrease performance.

No I don't, but if doing lots of small fetches from within the same
disk block it'll typically be less overhead to memcpy from an mmapped
page than to go into the kernel and back out again to fetch from the
file system caching layer. I have nothing to back up that assertion
though; it's just my gut instinct.

For seeking to larger offsets, fundamentally the OS has to do a
physical seek and read somewhere behind the scenes so it wouldn't make
much difference. Where mmap can be poorer though is that the mmap
page size may be much smaller than the file systems preferred buffer
size, causing more I/O overheads. This can be seen in things like
Lustre where the cost of any I/O call is high (by default I think the
file system block size is somewhere around 1Mb there).

In my experience, it seems mostly to make coding easier by allowing
files to look like large contiguous regions of memory. In principle
it seems great, so I'm curious if there might be a particular way to
use it that yields better performance.

The primary goal for adding mmap was robustness when dealing with
multiple programs running on the same host.

We had a corner case of running lots of single threaded jobs on small
CRAM files where each container/slice spanned numerous chromosomes
(due to the exceptionally shallow coverage). This made each job load
up multiple reference sequences into memory as they decode slice by
slice. Multiply this by many processes and it hit a memory limit.

The clear implication here is that the memory to hold the reference
sequences should have been shared between processes. Mmap is the
simplest way to achieve this.

James

James Bonfield (jkb@sanger.ac.uk) | Hora aderat briligi. Nunc et Slythia Tova
| Plurima gyrabant gymbolitare vabo;
A Staden Package developer: | Et Borogovorum mimzebant undique formae,
https://sf.net/projects/staden/ | Momiferique omnes exgrabure Rathi.

The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

[ekg]

Thanks James, this really helps.
On Apr 1, 2015 11:57 AM, "James Bonfield" notifications@github.com wrote:

On Wed, Apr 01, 2015 at 03:30:31AM -0700, Erik Garrison wrote:

@jkbonfield Do you have examples of tests showing that mmap is
better than seeking? I experimented quite a bit with this in fastahack
and didn't notice any difference in performance. Similar results in
rocksdb, which allows mmapping the db files although notes that this
tends to decrease performance.

No I don't, but if doing lots of small fetches from within the same
disk block it'll typically be less overhead to memcpy from an mmapped
page than to go into the kernel and back out again to fetch from the
file system caching layer. I have nothing to back up that assertion
though; it's just my gut instinct.

For seeking to larger offsets, fundamentally the OS has to do a
physical seek and read somewhere behind the scenes so it wouldn't make
much difference. Where mmap can be poorer though is that the mmap
page size may be much smaller than the file systems preferred buffer
size, causing more I/O overheads. This can be seen in things like
Lustre where the cost of any I/O call is high (by default I think the
file system block size is somewhere around 1Mb there).

In my experience, it seems mostly to make coding easier by allowing
files to look like large contiguous regions of memory. In principle
it seems great, so I'm curious if there might be a particular way to
use it that yields better performance.

The primary goal for adding mmap was robustness when dealing with
multiple programs running on the same host.

We had a corner case of running lots of single threaded jobs on small
CRAM files where each container/slice spanned numerous chromosomes
(due to the exceptionally shallow coverage). This made each job load
up multiple reference sequences into memory as they decode slice by
slice. Multiply this by many processes and it hit a memory limit.

The clear implication here is that the memory to hold the reference
sequences should have been shared between processes. Mmap is the
simplest way to achieve this.

James

James Bonfield (jkb@sanger.ac.uk) | Hora aderat briligi. Nunc et Slythia
Tova
| Plurima gyrabant gymbolitare vabo;
A Staden Package developer: | Et Borogovorum mimzebant undique formae,
https://sf.net/projects/staden/ | Momiferique omnes exgrabure Rathi.

The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

—
Reply to this email directly or view it on GitHub
#187 (comment).