Summary
VariableLengthArray::reserve() and shrink_to_fit() call the allocator's reallocate while the buffer holds live elements. cetl::pf17::pmr::memory_resource::reallocate follows std::realloc semantics: when it can't resize in place it moves the block with a byte copy. For any T that isn't trivially copyable, that relocates objects without running their move constructors or destructors. This is undefined behaviour, and in practice it causes dangling self-pointers, heap corruption and crashes.
The VLA picks up reallocate automatically for cetl::pf17::pmr::polymorphic_allocator. CETL's default pf17 new_delete_resource() (deviant::MaxAlignNewDeleteResource) implements it with std::realloc. So a plain VariableLengthArray<std::string, cetl::pf17::pmr::polymorphic_allocator<std::string>> on the default resource is affected, with no custom resource involved.
Affected code
VariableLengthArray<bool> stores unsigned char and isn't affected. Builds that use std::pmr (C++17 and later) aren't affected through std::pmr::polymorphic_allocator, which has no reallocate. Custom allocators that provide reallocate are affected in any standard.
Reproducer
The reproducer below covers four cases:
- A self-referential element after
reserve() with a conforming resource whose reallocate always moves.
- The same element after
shrink_to_fit().
- The same element after
reserve() on CETL's default new_delete_resource().
std::string after reserve().
Build and run it from the repository root:
g++ -std=c++14 -Wall -Wextra -pedantic-errors -I include realloc_repro.cpp -o realloc-repro && ./realloc-repro
gcc 13.3 / libstdc++, arm64 (toolshed ts24.4.3 dev container):
reserve, moving resource: 2 of 2 elements broken
shrink_to_fit, moving resource: 2 of 2 elements broken
reserve, new_delete_resource(): 1 of 1 elements broken (block moved)
reserve, std::string (SSO): data() points outside the object (dangling SSO pointer)
munmap_chunk(): invalid pointer
Aborted
With -fsanitize=address:
ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed
#0 operator delete(void*)
#1 VariableLengthArrayBase<std::string, polymorphic_allocator<std::string>>::fast_destroy variable_length_array.hpp:227
...
#3 VariableLengthArray<std::string, ...>::~VariableLengthArray()
freed by thread T0 here:
#1 AlwaysMovingResource::do_reallocate
...
#7 VariableLengthArrayBase<std::string, ...>::reserve variable_length_array.hpp:634
Homebrew clang 22.1.8 / libc++, macOS arm64: cases 1–3 show the same broken elements. libc++'s std::string doesn't point into itself, so case 4 happens to survive there.
realloc_repro.cpp
// VLA reserve()/shrink_to_fit() relocate live non-trivially-copyable elements with a byte copy.
#include "cetl/pf17/cetlpf.hpp"
#include "cetl/pf17/sys/memory_resource.hpp"
#include "cetl/variable_length_array.hpp"
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <string>
// An element whose invariant depends on its own address, the way libstdc++'s SSO std::string does.
struct SelfRef
{
int value;
int* self;
explicit SelfRef(int v) noexcept
: value{v}
, self{&value}
{
}
SelfRef(const SelfRef& rhs) noexcept
: value{rhs.value}
, self{&value}
{
}
SelfRef(SelfRef&& rhs) noexcept
: value{rhs.value}
, self{&value}
{
}
SelfRef& operator=(const SelfRef& rhs) noexcept
{
value = rhs.value;
return *this;
}
~SelfRef() = default;
bool intact() const noexcept
{
return self == &value;
}
};
// A conforming memory_resource whose reallocate always moves the block, as std::realloc may.
class AlwaysMovingResource final : public cetl::pf17::pmr::memory_resource
{
void* do_allocate(std::size_t size_bytes, std::size_t) override
{
return std::malloc(size_bytes);
}
void do_deallocate(void* p, std::size_t, std::size_t) override
{
std::free(p);
}
void* do_reallocate(void* p, std::size_t old_size_bytes, std::size_t new_size_bytes, std::size_t) override
{
void* moved = std::malloc(new_size_bytes);
std::memcpy(moved, p, (old_size_bytes < new_size_bytes) ? old_size_bytes : new_size_bytes);
std::free(p);
return moved;
}
bool do_is_equal(const memory_resource& rhs) const noexcept override
{
return &rhs == this;
}
};
template <typename T>
using Vla = cetl::VariableLengthArray<T, cetl::pf17::pmr::polymorphic_allocator<T>>;
static int broken(const Vla<SelfRef>& vla)
{
int n = 0;
for (const auto& e : vla)
{
n += e.intact() ? 0 : 1;
}
return n;
}
int main()
{
int failures = 0;
// 1. reserve() growth with a resource that moves on reallocate.
{
AlwaysMovingResource resource;
Vla<SelfRef> vla{cetl::pf17::pmr::polymorphic_allocator<SelfRef>{&resource}};
vla.reserve(2);
vla.emplace_back(1);
vla.emplace_back(2);
vla.reserve(64);
const int b = broken(vla);
std::printf("reserve, moving resource: %d of %zu elements broken\n", b, vla.size());
failures += b;
}
// 2. shrink_to_fit() with a resource that moves on reallocate.
{
AlwaysMovingResource resource;
Vla<SelfRef> vla{cetl::pf17::pmr::polymorphic_allocator<SelfRef>{&resource}};
vla.reserve(64);
vla.emplace_back(1);
vla.emplace_back(2);
vla.shrink_to_fit();
const int b = broken(vla);
std::printf("shrink_to_fit, moving resource: %d of %zu elements broken\n", b, vla.size());
failures += b;
}
// 3. CETL's default pf17 new_delete_resource(), which reallocates with std::realloc. Whether realloc moves
// the block is up to the C library; growing well past the original size usually forces it.
{
Vla<SelfRef> vla{cetl::pf17::pmr::polymorphic_allocator<SelfRef>{cetl::pf17::pmr::new_delete_resource()}};
vla.reserve(1);
vla.emplace_back(1);
const void* before = vla.data();
vla.reserve(1U << 20U);
const int b = broken(vla);
std::printf("reserve, new_delete_resource(): %d of %zu elements broken (block %s)\n",
b,
vla.size(),
(before == vla.data()) ? "extended in place" : "moved");
failures += b;
}
// 4. A real standard type. libstdc++'s std::string points into its own SSO buffer; libc++'s does not.
{
AlwaysMovingResource resource;
Vla<std::string> vla{cetl::pf17::pmr::polymorphic_allocator<std::string>{&resource}};
vla.reserve(1);
vla.emplace_back("short");
vla.reserve(64);
// With libstdc++ the string's data pointer now refers to the freed old block: reading it is a use after free.
const bool dangling = (static_cast<const void*>(vla[0].data()) != static_cast<const void*>(&vla[0]) &&
vla[0].size() < 15U &&
(reinterpret_cast<const char*>(vla[0].data()) < reinterpret_cast<const char*>(&vla[0]) ||
reinterpret_cast<const char*>(vla[0].data()) >=
reinterpret_cast<const char*>(&vla[0] + 1)));
std::printf("reserve, std::string (SSO): %s\n",
dangling ? "data() points outside the object (dangling SSO pointer)" : "not affected on this library");
failures += dangling ? 1 : 0;
}
return (failures == 0) ? 0 : 1;
}
Suggested fix
Only use reallocate on a buffer with live elements when relocating them with a byte copy is valid, which means std::is_trivially_copyable<T>. Otherwise always take the allocate, move-construct, destroy, deallocate path. The detection could be gated in the reallocate helper, or reserve()/shrink_to_fit() could branch on the trait. Correct the comment in reserve() at the same time.
Longer term, an "extend in place only, never move" allocator extension would let non-trivial types benefit too. That's the same open question as #212.
Tests: add a non-trivially-copyable, address-sensitive element type to the VLA allocation tests, run it against a resource whose reallocate always moves, and cover both reserve() growth and shrink_to_fit().
Related
Summary
VariableLengthArray::reserve()andshrink_to_fit()call the allocator'sreallocatewhile the buffer holds live elements.cetl::pf17::pmr::memory_resource::reallocatefollowsstd::reallocsemantics: when it can't resize in place it moves the block with a byte copy. For anyTthat isn't trivially copyable, that relocates objects without running their move constructors or destructors. This is undefined behaviour, and in practice it causes dangling self-pointers, heap corruption and crashes.The VLA picks up
reallocateautomatically forcetl::pf17::pmr::polymorphic_allocator. CETL's default pf17new_delete_resource()(deviant::MaxAlignNewDeleteResource) implements it withstd::realloc. So a plainVariableLengthArray<std::string, cetl::pf17::pmr::polymorphic_allocator<std::string>>on the default resource is affected, with no custom resource involved.Affected code
reserve(): whenreallocatesucceeds the returned buffer is adopted as-is. The comment on the fallback branch, "unable to extend the reserved area for the same memory pointer", assumes reallocate only ever extends in place, but thememory_resource::reallocatecontract allows the block to move.shrink_to_fit(): same pattern.reallocate: applies to any allocator with the member, regardless ofT.VariableLengthArray<bool>storesunsigned charand isn't affected. Builds that usestd::pmr(C++17 and later) aren't affected throughstd::pmr::polymorphic_allocator, which has noreallocate. Custom allocators that providereallocateare affected in any standard.Reproducer
The reproducer below covers four cases:
reserve()with a conforming resource whosereallocatealways moves.shrink_to_fit().reserve()on CETL's defaultnew_delete_resource().std::stringafterreserve().Build and run it from the repository root:
g++ -std=c++14 -Wall -Wextra -pedantic-errors -I include realloc_repro.cpp -o realloc-repro && ./realloc-reprogcc 13.3 / libstdc++, arm64 (toolshed
ts24.4.3dev container):With
-fsanitize=address:Homebrew clang 22.1.8 / libc++, macOS arm64: cases 1–3 show the same broken elements. libc++'s
std::stringdoesn't point into itself, so case 4 happens to survive there.realloc_repro.cpp
Suggested fix
Only use
reallocateon a buffer with live elements when relocating them with a byte copy is valid, which meansstd::is_trivially_copyable<T>. Otherwise always take the allocate, move-construct, destroy, deallocate path. The detection could be gated in thereallocatehelper, orreserve()/shrink_to_fit()could branch on the trait. Correct the comment inreserve()at the same time.Longer term, an "extend in place only, never move" allocator extension would let non-trivial types benefit too. That's the same open question as #212.
Tests: add a non-trivially-copyable, address-sensitive element type to the VLA allocation tests, run it against a resource whose
reallocatealways moves, and cover bothreserve()growth andshrink_to_fit().Related
reallocatein copy assignment (raises the same in-place vs. move question)