Skip to content

POC: ArrayManager -- array-based data manager for columnar store #36010

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 42 commits into from
Jan 13, 2021
Merged
Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
a51835b
POC: ArrayManager -- array-based data manager for columnar store
jorisvandenbossche Jun 1, 2020
591579b
Update with latest master + some fixes
jorisvandenbossche Aug 27, 2020
896080a
add pd.options.mode.data_manager to switch
jorisvandenbossche Sep 4, 2020
f9c4dda
Merge remote-tracking branch 'upstream/master' into array-manager
jorisvandenbossche Sep 5, 2020
d18082a
add apply_with_block workaround
jorisvandenbossche Sep 5, 2020
cf3c07a
fix alignment in apply
jorisvandenbossche Sep 5, 2020
b252c6d
reorder methods to match BlockManager
jorisvandenbossche Sep 5, 2020
0fb645e
skip json tests for now
jorisvandenbossche Sep 5, 2020
eb55fef
skip more json tests + to_csv with to_native_types
jorisvandenbossche Sep 5, 2020
d241f31
Merge remote-tracking branch 'upstream/master' into array-manager
jorisvandenbossche Sep 6, 2020
47c3ee3
support both ndarrays and ExtensionArrays
jorisvandenbossche Sep 17, 2020
75f7de2
Merge remote-tracking branch 'upstream/master' into array-manager
jorisvandenbossche Sep 17, 2020
f36e395
add unstack
jorisvandenbossche Sep 17, 2020
be20816
fix native types, skip quantile, hdf, stata tests
jorisvandenbossche Sep 17, 2020
8b7cc81
remove skip in the benchmarks
jorisvandenbossche Sep 17, 2020
a239f50
Merge remote-tracking branch 'upstream/master' into array-manager
jorisvandenbossche Sep 17, 2020
a0ccf9a
Merge remote-tracking branch 'upstream/master' into array-manager
jorisvandenbossche Sep 22, 2020
dc1b190
Merge remote-tracking branch 'upstream/master' into array-manager
jorisvandenbossche Oct 16, 2020
55d38be
remove manager keyword from DataFrame constructor, add _as_manager in…
jorisvandenbossche Oct 16, 2020
3dea0d7
move new ArrayManager code to separate file
jorisvandenbossche Oct 16, 2020
1a61333
Merge branch 'master' of https://github.com/pandas-dev/pandas into ar…
jbrockmendel Nov 10, 2020
9751d33
de-privatize
jbrockmendel Nov 10, 2020
e45b645
Merge remote-tracking branch 'upstream/master' into array-manager
jorisvandenbossche Dec 11, 2020
3749c7d
try fix up typing
jorisvandenbossche Dec 11, 2020
af53040
add pytest option + add one github actions build to run them
jorisvandenbossche Dec 11, 2020
cc45673
fix pytest marks for skipping when using array-manager
jorisvandenbossche Dec 12, 2020
27cf215
several fixes - get tests/frame/methods tests passing
jorisvandenbossche Dec 12, 2020
f6a97df
ci - only run the tests/frame/methods tests
jorisvandenbossche Dec 12, 2020
67c4c2b
Merge remote-tracking branch 'upstream/master' into array-manager
jorisvandenbossche Dec 12, 2020
670ed76
mypy fix
jorisvandenbossche Dec 12, 2020
5128ad1
Merge remote-tracking branch 'upstream/master' into array-manager
jorisvandenbossche Dec 18, 2020
5c73688
Merge remote-tracking branch 'upstream/master' into array-manager
jorisvandenbossche Jan 8, 2021
a9a8c2d
move to internals/construction.py
jorisvandenbossche Jan 8, 2021
c7898fb
update for latest changes - fix tests/mypy
jorisvandenbossche Jan 8, 2021
3430307
fix todo
jorisvandenbossche Jan 8, 2021
1a30013
fix import in tests
jorisvandenbossche Jan 8, 2021
ef86b1e
Merge remote-tracking branch 'upstream/master' into array-manager
jorisvandenbossche Jan 10, 2021
c5548d9
add union alias to typing
jorisvandenbossche Jan 10, 2021
afe8f80
updates based on review
jorisvandenbossche Jan 10, 2021
b88c757
skip json tests to avoid segfaults
jorisvandenbossche Jan 10, 2021
ddc51d0
Merge remote-tracking branch 'upstream/master' into array-manager
jorisvandenbossche Jan 12, 2021
9dc5600
fix for Label -> Hashable change in master
jorisvandenbossche Jan 12, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix import in tests
  • Loading branch information
jorisvandenbossche committed Jan 8, 2021
commit 1a3001364cade395c7939e1c93e259adc7dabf41
4 changes: 3 additions & 1 deletion pandas/tests/internals/test_managers.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
"""
Testing interaction between the different managers (BlockManager, ArrayManager)
"""
from pandas.core.dtypes.missing import array_equivalent

import pandas as pd
import pandas._testing as tm
from pandas.core.internals import ArrayManager, BlockManager
Expand All @@ -26,7 +28,7 @@ def test_dataframe_creation():
assert isinstance(result._mgr, ArrayManager)
tm.assert_frame_equal(result, df_block)
assert all(
tm.array_equivalent(left, right)
array_equivalent(left, right)
for left, right in zip(result._mgr.arrays, df_array._mgr.arrays)
)

Expand Down