Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add default pinned pool that falls back to new pinned allocations #15665

Merged
merged 44 commits into from
May 20, 2024
Merged
Changes from 1 commit
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
163ad97
pool with fallback
vuule May 6, 2024
1e850d6
don't use default pool
vuule May 6, 2024
3be42ba
fix allocator copy assignment
vuule May 6, 2024
395dcf1
fix ver2
vuule May 6, 2024
70ae74e
Merge branch 'branch-24.06' into bug-allocator-copy-wrong-stream
vuule May 6, 2024
503d170
Merge branch 'bug-allocator-copy-wrong-stream' into perf-defaul-piine…
vuule May 6, 2024
0873b1f
copyright
vuule May 6, 2024
ff18a21
fix operator==
vuule May 7, 2024
f5a735c
Merge branch 'branch-24.06' of https://github.com/rapidsai/cudf into …
vuule May 7, 2024
5766805
Merge branch 'bug-allocator-copy-wrong-stream' of https://github.com/…
vuule May 7, 2024
0bd92bf
Merge branch 'bug-allocator-copy-wrong-stream' into perf-defaul-piine…
vuule May 7, 2024
854c0ab
simplify pool creation
vuule May 7, 2024
5bf0ce4
namespace; comments
vuule May 7, 2024
284654d
Merge branch 'branch-24.06' into perf-defaul-piined-pool
vuule May 7, 2024
ff4d7f6
clean up
vuule May 7, 2024
f5b2c84
Merge branch 'branch-24.06' of https://github.com/rapidsai/cudf into …
vuule May 7, 2024
cf3f8a3
Merge branch 'perf-defaul-piined-pool' of https://github.com/vuule/cu…
vuule May 7, 2024
80b5963
mild polish
vuule May 7, 2024
d23684d
Merge branch 'branch-24.06' of https://github.com/rapidsai/cudf into …
vuule May 8, 2024
1828e05
Merge branch 'branch-24.06' of https://github.com/rapidsai/cudf into …
vuule May 9, 2024
0122038
remove inline
vuule May 9, 2024
60030da
scoped_lock
vuule May 9, 2024
a62377e
try
vuule May 9, 2024
6733c45
clean up
vuule May 9, 2024
abf40a8
clarify try-catch fallback
vuule May 9, 2024
fa7dce7
remove export
vuule May 9, 2024
a244d7c
non-indexed stream
vuule May 9, 2024
7076e73
Update cpp/src/io/utilities/config_utils.cpp
ttnghia May 9, 2024
0b8aa44
Merge branch 'branch-24.06' into perf-defaul-piined-pool
vuule May 9, 2024
3db44a3
Merge branch 'branch-24.06' of https://github.com/rapidsai/cudf into …
vuule May 13, 2024
27d30c8
make default_pinned_mr cheap to call multiple times
vuule May 13, 2024
224e68f
static_assert fixed_pinned_pool_memory_resource
vuule May 13, 2024
382e7b3
Merge branch 'perf-defaul-piined-pool' of https://github.com/vuule/cu…
vuule May 13, 2024
b2fd734
fix host_mr
vuule May 13, 2024
0eccf9a
add config function
vuule May 13, 2024
ecd6481
align config size; add missing header
vuule May 13, 2024
709123f
Merge branch 'branch-24.06' of https://github.com/rapidsai/cudf into …
vuule May 13, 2024
01b1bdb
Merge branch 'branch-24.06' of https://github.com/rapidsai/cudf into …
vuule May 14, 2024
ecb5f5a
CUDF_EXPORT
vuule May 14, 2024
f0d0bf0
fail config if resource is already created
vuule May 14, 2024
f989a56
fix config check
vuule May 15, 2024
d0e6dd7
Merge branch 'branch-24.06' of https://github.com/rapidsai/cudf into …
vuule May 15, 2024
2b4952a
docs
vuule May 15, 2024
fdcfad3
Merge branch 'branch-24.06' into perf-defaul-piined-pool
vuule May 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
simplify pool creation
  • Loading branch information
vuule committed May 7, 2024
commit 854c0abfd842a8d34bbbb2d8ee2bd4a984b07142
35 changes: 16 additions & 19 deletions cpp/src/io/utilities/config_utils.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -93,47 +93,50 @@ bool is_stable_enabled() { return is_all_enabled() or get_env_policy() == usage_
using rmm_pinned_pool_t = rmm::mr::pool_memory_resource<rmm::mr::pinned_host_memory_resource>;

class fixed_pinned_pool_memory_resource {
using upstream_mr = rmm::mr::pinned_host_memory_resource;
using host_pooled_mr = rmm::mr::pool_memory_resource<upstream_mr>;

private:
rmm_pinned_pool_t* _pool;
upstream_mr upstream_mr_{};
size_t pool_size_{};
host_pooled_mr* pool_;
void* pool_begin_{};
void* pool_end_{};
size_t pool_size_{};
cuda::stream_ref stream_{cudf::detail::global_cuda_stream_pool().get_stream(0).value()};
vuule marked this conversation as resolved.
Show resolved Hide resolved
rmm::mr::pinned_host_memory_resource fallback_mr_{};

public:
fixed_pinned_pool_memory_resource(rmm_pinned_pool_t* pool)
: _pool(pool), pool_size_{pool->pool_size()}
fixed_pinned_pool_memory_resource(size_t size)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to check and throw if size == 0?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so; the resource works fine with a zero-capacity pool. I used this when benchmarking. Basically to verify that the performance is the same as the non-pooled resource. So zero is a valid value for the size IMO.

: pool_size_{size}, pool_{new host_pooled_mr(upstream_mr_, size, size)}
{
// allocate from the pinned pool the full size to figure out
// our beginning and end address.
if (pool_size_ != 0) {
pool_begin_ = pool->allocate_async(pool_size_, stream_);
pool_begin_ = pool_->allocate_async(pool_size_, stream_);
pool_end_ = static_cast<void*>(static_cast<uint8_t*>(pool_begin_) + pool_size_);
pool->deallocate_async(pool_begin_, pool_size_, stream_);
pool_->deallocate_async(pool_begin_, pool_size_, stream_);
}
}

void* do_allocate_async(std::size_t bytes, std::size_t alignment, cuda::stream_ref stream)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm late, but these do_ versions should probably be protected/private?

{
if (bytes <= pool_size_) {
try {
return _pool->allocate_async(bytes, alignment, stream);
return pool_->allocate_async(bytes, alignment, stream);
} catch (const std::exception& unused) {
}
}

return fallback_mr_.allocate_async(bytes, alignment, stream);
return upstream_mr_.allocate_async(bytes, alignment, stream);
}
void do_deallocate_async(void* ptr,
ttnghia marked this conversation as resolved.
Show resolved Hide resolved
std::size_t bytes,
std::size_t alignment,
cuda::stream_ref stream) noexcept
{
if (bytes <= pool_size_ && ptr >= pool_begin_ && ptr <= pool_end_) {
_pool->deallocate_async(ptr, bytes, alignment, stream);
pool_->deallocate_async(ptr, bytes, alignment, stream);
} else {
fallback_mr_.deallocate_async(ptr, bytes, alignment, stream);
upstream_mr_.deallocate_async(ptr, bytes, alignment, stream);
}
}

Expand Down Expand Up @@ -193,8 +196,6 @@ class fixed_pinned_pool_memory_resource {

inline rmm::host_async_resource_ref default_pinned_mr()
vuule marked this conversation as resolved.
Show resolved Hide resolved
{
using host_pooled_mr = rmm::mr::pool_memory_resource<rmm::mr::pinned_host_memory_resource>;

auto const size = []() -> size_t {
if (auto const env_val = getenv("LIBCUDF_PINNED_POOL_SIZE")) { return std::atol(env_val); }
vuule marked this conversation as resolved.
Show resolved Hide resolved

Expand All @@ -207,13 +208,9 @@ inline rmm::host_async_resource_ref default_pinned_mr()
CUDF_LOG_INFO("Pinned pool size = {}", size);

// make the pool with max size equal to the initial size
static std::shared_ptr<host_pooled_mr> pool_mr = std::make_shared<host_pooled_mr>(
std::make_shared<rmm::mr::pinned_host_memory_resource>().get(), size, size);

static std::shared_ptr<fixed_pinned_pool_memory_resource> mr =
std::make_shared<fixed_pinned_pool_memory_resource>(pool_mr.get());
static fixed_pinned_pool_memory_resource mr{size};

return *mr;
return mr;
}

inline std::mutex& host_mr_lock()
Expand Down