Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify Copy-On-Write and Spilling #15436

Merged
merged 30 commits into from
Apr 20, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
f2bf171
clean up
madsbk Apr 2, 2024
276dd80
remove redundant methods
madsbk Apr 2, 2024
2413e99
impl. and use BufferOwner.__init__
madsbk Apr 2, 2024
9f4b0e1
SpillableBufferOwner: use __init__
madsbk Apr 2, 2024
834f6d5
remove SpillableBuffer.mark_exposed()
madsbk Apr 2, 2024
68524e4
remove SpillableBuffer.exposed
madsbk Apr 2, 2024
2e43f85
SpillableBuffer(ExposureTrackedBuffer)
madsbk Apr 2, 2024
659e832
mark cow tests as "spilling"
madsbk Apr 3, 2024
c5355a5
impl. SpillableBuffer.copy
madsbk Apr 3, 2024
3581579
option: allow cow and spilling at the same time
madsbk Apr 3, 2024
58f7e8e
test: skip zero-copy and no-copy-on-write test whe spilling is enabled
madsbk Apr 3, 2024
6018949
cleanup
madsbk Apr 3, 2024
f48c529
impl. set_spill_on_demand_globally
madsbk Apr 3, 2024
dcff299
impl. spill_on_demand_globally
madsbk Apr 3, 2024
4a8b43e
cleanup
madsbk Apr 3, 2024
7344f9a
impl. test_spilling_and_copy_on_write
madsbk Apr 3, 2024
1e9d061
don't copy spilled data
madsbk Apr 3, 2024
dc715e1
test_get_rmm_memory_resource_stack is safe again
madsbk Apr 3, 2024
3eaf5e3
Apply suggestions from code review
madsbk Apr 4, 2024
a4a3ff4
BufferOwner: forcing keyword-only
madsbk Apr 4, 2024
55027fc
Merge branch 'branch-24.06' of github.com:rapidsai/cudf into cow_and_…
madsbk Apr 4, 2024
504ae9a
typo
madsbk Apr 4, 2024
53106ee
Merge branch 'branch-24.06' of github.com:rapidsai/cudf into cow_and_…
madsbk Apr 11, 2024
ac6831a
doc
madsbk Apr 11, 2024
49625fa
rename _from_*_memory => from_*_memory
madsbk Apr 11, 2024
f4458b8
enable must of test_series_zero_copy_cow_off again
madsbk Apr 11, 2024
30cd8e6
doc
madsbk Apr 11, 2024
580dead
Update python/cudf/cudf/tests/test_spilling.py
madsbk Apr 11, 2024
6fe2d58
more tests
madsbk Apr 11, 2024
51b4b82
Merge branch 'branch-24.06' of github.com:rapidsai/cudf into cow_and_…
madsbk Apr 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
remove SpillableBuffer.mark_exposed()
  • Loading branch information
madsbk committed Apr 2, 2024
commit 834f6d54383b0d4c836cba8c2ad553bc6c7260d4
3 changes: 0 additions & 3 deletions python/cudf/cudf/core/buffer/spillable_buffer.py
Original file line number Diff line number Diff line change
Expand Up @@ -408,9 +408,6 @@ def memory_info(self) -> Tuple[int, int, str]:
(ptr, _, device_type) = self._owner.memory_info()
return (ptr + self._offset, self.nbytes, device_type)

def mark_exposed(self) -> None:
self._owner.mark_exposed()

def serialize(self) -> Tuple[dict, list]:
"""Serialize the Buffer

Expand Down
10 changes: 5 additions & 5 deletions python/cudf/cudf/tests/test_spilling.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ def test_spillable_buffer(manager: SpillManager):
buf = as_buffer(data=rmm.DeviceBuffer(size=10), exposed=False)
assert isinstance(buf, SpillableBuffer)
assert buf.spillable
buf.mark_exposed()
buf.owner.mark_exposed()
assert buf.exposed
assert not buf.spillable
buf = as_buffer(data=rmm.DeviceBuffer(size=10), exposed=False)
Expand Down Expand Up @@ -210,7 +210,7 @@ def test_spilling_buffer(manager: SpillManager):
buf = as_buffer(rmm.DeviceBuffer(size=10), exposed=False)
buf.spill(target="cpu")
assert buf.is_spilled
buf.mark_exposed() # Expose pointer and trigger unspill
buf.owner.mark_exposed() # Expose pointer and trigger unspill
assert not buf.is_spilled
with pytest.raises(ValueError, match="unspillable buffer"):
buf.spill(target="cpu")
Expand Down Expand Up @@ -653,7 +653,7 @@ def test_statistics_expose(manager: SpillManager):
]

# Expose the first buffer
buffers[0].mark_exposed()
buffers[0].owner.mark_exposed()
assert len(manager.statistics.exposes) == 1
stat = list(manager.statistics.exposes.values())[0]
assert stat.count == 1
Expand All @@ -662,7 +662,7 @@ def test_statistics_expose(manager: SpillManager):

# Expose all 10 buffers
for i in range(10):
buffers[i].mark_exposed()
buffers[i].owner.mark_exposed()

# The rest of the ptr accesses should accumulate to a single stat
# because they resolve to the same traceback.
Expand All @@ -682,7 +682,7 @@ def test_statistics_expose(manager: SpillManager):

# Expose the new buffers and check that they are counted as spilled
for i in range(10):
buffers[i].mark_exposed()
buffers[i].owner.mark_exposed()
assert len(manager.statistics.exposes) == 3
stat = list(manager.statistics.exposes.values())[2]
assert stat.count == 10
Expand Down