`memory_pool_collection` carves too little memory from block if there are many buckets

[psalz]

Basically, this:

const size_t max_node_size = 512 * 1024 * 1024;
const size_t block_size = 1024 * 1024 * 1024;
memory::memory_pool_collection<memory::node_pool, memory::log2_buckets> my_pool(max_node_size, block_size);
my_pool.allocate_node(256 * 1024 * 1024);

This fails FOONATHAN_MEMORY_ASSERT(no_nodes > 0); in free_list.cpp because memory_pool_collection::def_capacity() returns arena._next_block_size() / pools_.size(), i.e., 2147483632 / 27 = 79536430, which is less than the requested 268435456.

[foonathan]

I mean, yes, it fails. This is because the initial block size is simply too small for the block size. You asked for 27 buckets with huge node sizes, and give it a block size that's too small to contain their initial memory. I'm not sure what its supposed to do in that case?

As an aside, as soon as you're talking about nodes in the size of a couple of KiB, the default new/malloc implementation is probably better suited for allocation than the use of a free list.

[psalz]

Being new to this library and from looking at the documentation, it's not obvious to me that (1) all buckets are allocated eagerly instead of on-demand, and in particular (2) that they need to fit into a single block.

Given that I triggered some assertion five levels deep into the library, I assumed this to be a bug. Not to be overbearing in requesting features from an open source project -- but since you asked: If this is intended behavior, I would prefer it to tell me during construction of the collection.

As an aside, as soon as you're talking about nodes in the size of a couple of KiB, the default new/malloc implementation is probably better suited for allocation than the use of a free list.

Good to know, thanks. In my case I'm using pinned memory, which is extremely slow to allocate, so I'm actually gaining performance even for large (~10 MiB) nodes. I ended up using a simple pool though, instead of the collection.

[foonathan]

Being new to this library and from looking at the documentation, it's not obvious to me that (1) all buckets are allocated eagerly instead of on-demand, and in particular (2) that they need to fit into a single block.

This isn't the case. By default, the initial block size is partitioned into equally sized chunks for all free lists, which is lazily committed.

If this is intended behavior, I would prefer it to tell me during construction of the collection.

Good point. I have added a check in the constructor in 9d3bda1.

[psalz]

Nice - thanks!