Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARROW-759: [Python] Serializing large class of Python objects in Apache Arrow #965

Closed
wants to merge 55 commits into from
Closed

ARROW-759: [Python] Serializing large class of Python objects in Apache Arrow #965

wants to merge 55 commits into from

Commits on Aug 15, 2017

  1. python to arrow serialization

    @pcmoritz
    pcmoritz committed Aug 15, 2017
    Configuration menu
    Copy the full SHA
    5766b8c View commit details
    Browse the repository at this point in the history

  2. rename serialization entry point

    @pcmoritz
    pcmoritz committed Aug 15, 2017
    Configuration menu
    Copy the full SHA
    deb3b46 View commit details
    Browse the repository at this point in the history

  3. deserialization path (need to figure out if base object and refcounti… 

    …ng is handled correctly)
    @pcmoritz
    pcmoritz committed Aug 15, 2017
    Configuration menu
    Copy the full SHA
    3af1c67 View commit details
    Browse the repository at this point in the history

  4. work in progress

    @pcmoritz
    pcmoritz committed Aug 15, 2017
    Configuration menu
    Copy the full SHA
    44fb98b View commit details
    Browse the repository at this point in the history

  5. roundtrip working for the first time

    @pcmoritz
    pcmoritz committed Aug 15, 2017
    Configuration menu
    Copy the full SHA
    49a4acb View commit details
    Browse the repository at this point in the history

  6. handle very long longs with custom serialization callback

    @pcmoritz
    pcmoritz committed Aug 15, 2017
    Configuration menu
    Copy the full SHA
    bd36c83 View commit details
    Browse the repository at this point in the history

  7. working version

    @pcmoritz
    pcmoritz committed Aug 15, 2017
    Configuration menu
    Copy the full SHA
    8b2ffe6 View commit details
    Browse the repository at this point in the history

  8. serialization of custom objects

    @pcmoritz
    pcmoritz committed Aug 15, 2017
    Configuration menu
    Copy the full SHA
    f229d8d View commit details
    Browse the repository at this point in the history

  9. rebase

    @pcmoritz
    pcmoritz committed Aug 15, 2017
    Configuration menu
    Copy the full SHA
    30bb960 View commit details
    Browse the repository at this point in the history

  10. fix python unicode string

    @pcmoritz
    pcmoritz committed Aug 15, 2017
    Configuration menu
    Copy the full SHA
    2171761 View commit details
    Browse the repository at this point in the history

  11. fix imports

    @pcmoritz
    pcmoritz committed Aug 15, 2017
    Configuration menu
    Copy the full SHA
    7069e20 View commit details
    Browse the repository at this point in the history

  12. fix

    @pcmoritz
    pcmoritz committed Aug 15, 2017
    Configuration menu
    Copy the full SHA
    c4782ac View commit details
    Browse the repository at this point in the history

  13. fix linting

    @pcmoritz
    pcmoritz committed Aug 15, 2017
    Configuration menu
    Copy the full SHA
    91b57d5 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2017

  1. fix namespaces

    @pcmoritz
    pcmoritz committed Aug 16, 2017
    Configuration menu
    Copy the full SHA
    2e08de4 View commit details
    Browse the repository at this point in the history

  2. clang-format

    @pcmoritz
    pcmoritz committed Aug 16, 2017
    Configuration menu
    Copy the full SHA
    802e739 View commit details
    Browse the repository at this point in the history

  3. lint fix

    @pcmoritz
    pcmoritz committed Aug 16, 2017
    Configuration menu
    Copy the full SHA
    a6105d2 View commit details
    Browse the repository at this point in the history

  4. fix first few comments

    @pcmoritz
    pcmoritz committed Aug 16, 2017
    Configuration menu
    Copy the full SHA
    080db03 View commit details
    Browse the repository at this point in the history

  5. convert DESERIALIZE_SEQUENCE to a template

    @pcmoritz
    pcmoritz committed Aug 16, 2017
    Configuration menu
    Copy the full SHA
    74b9e46 View commit details
    Browse the repository at this point in the history

  6. get rid of leaks and clarify reference counting for dicts

    @pcmoritz
    pcmoritz committed Aug 16, 2017
    Configuration menu
    Copy the full SHA
    c38c58d View commit details
    Browse the repository at this point in the history

  7. remove code duplication

    @pcmoritz
    pcmoritz committed Aug 16, 2017
    Configuration menu
    Copy the full SHA
    aaf6f09 View commit details
    Browse the repository at this point in the history

  8. increase Py_True refcount and hide helper methods

    @pcmoritz
    pcmoritz committed Aug 16, 2017
    Configuration menu
    Copy the full SHA
    3929273 View commit details
    Browse the repository at this point in the history

  9. make DictBuilder private

    @pcmoritz
    pcmoritz committed Aug 16, 2017
    Configuration menu
    Copy the full SHA
    e73c1ea View commit details
    Browse the repository at this point in the history

  10. mutable refs and small fixes

    @pcmoritz
    pcmoritz committed Aug 16, 2017
    Configuration menu
    Copy the full SHA
    3298329 View commit details
    Browse the repository at this point in the history

  11. cleanups

    @pcmoritz
    pcmoritz committed Aug 16, 2017
    Configuration menu
    Copy the full SHA
    99e2d1a View commit details
    Browse the repository at this point in the history

  12. clang-format

    @pcmoritz
    pcmoritz committed Aug 16, 2017
    Configuration menu
    Copy the full SHA
    3e94e6d View commit details
    Browse the repository at this point in the history

  13. more fixes

    @pcmoritz
    pcmoritz committed Aug 16, 2017
    Configuration menu
    Copy the full SHA
    c1f377b View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2017

  1. restructure

    @pcmoritz
    pcmoritz committed Aug 17, 2017
    Configuration menu
    Copy the full SHA
    e1fc0c5 View commit details
    Browse the repository at this point in the history

  2. make exported API more consistent

    @pcmoritz
    pcmoritz committed Aug 17, 2017
    Configuration menu
    Copy the full SHA
    faf9a3e View commit details
    Browse the repository at this point in the history

  3. fix api

    @pcmoritz
    pcmoritz committed Aug 17, 2017
    Configuration menu
    Copy the full SHA
    2f0760c View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2017

  1. documentation

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    389bfc6 View commit details
    Browse the repository at this point in the history

  2. fix callbacks

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    aeafd82 View commit details
    Browse the repository at this point in the history

  3. prevent possible memory leaks

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    c425978 View commit details
    Browse the repository at this point in the history

  4. convert DESERIALIZE_SEQUENCE back to a macro

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    a88d410 View commit details
    Browse the repository at this point in the history

  5. cleanups

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    f25f3f3 View commit details
    Browse the repository at this point in the history

  6. cleanup

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    4cc45cd View commit details
    Browse the repository at this point in the history

  7. fix longlong vs int64 and unsigned variant

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    bcebdfe View commit details
    Browse the repository at this point in the history

  8. shuffle stuff around

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    adcc8f7 View commit details
    Browse the repository at this point in the history

  9. fix GIL

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    95cb9da View commit details
    Browse the repository at this point in the history

  10. linting

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    aa1f300 View commit details
    Browse the repository at this point in the history

  11. make it compile on windows

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    49aba8a View commit details
    Browse the repository at this point in the history

  12. more fixes

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    84d62f6 View commit details
    Browse the repository at this point in the history

  13. fixes

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    fe56c73 View commit details
    Browse the repository at this point in the history

  14. make tests work

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    a6fdb76 View commit details
    Browse the repository at this point in the history

  15. more fixes

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    54af39b View commit details
    Browse the repository at this point in the history

  16. remove sequence.h

    @pcmoritz
    pcmoritz committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    831e2f2 View commit details
    Browse the repository at this point in the history

  17. Fix various Clang compiler warnings due to integer conversions. clang… 

    …-format

Change-Id: Id100134ed72a42ed2bba6cab0b5fd5b0f29030e8
    @wesm
    wesm committed Aug 18, 2017
    Configuration menu
    Copy the full SHA
    c8efef9 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2017

  1. Do not use ARROW_CHECK in production code. Consolidate python_to_arro… 

    …w code
    @wesm
    wesm committed Aug 19, 2017
    Configuration menu
    Copy the full SHA
    ce5784d View commit details
    Browse the repository at this point in the history

  2. Refactoring, address code review comments. fix flake8 issues 

    Change-Id: I9ced59de48f169b6609dd27f8239ceb22fd5ebeb
    @wesm
    wesm committed Aug 19, 2017
    Configuration menu
    Copy the full SHA
    a9522c5 View commit details
    Browse the repository at this point in the history

  3. Add doxygen comment to set_serialization_callbacks 

    Change-Id: If9ac54f494495186743b0a6929ea193ca5048ed0
    @wesm
    wesm committed Aug 19, 2017
    Configuration menu
    Copy the full SHA
    8a42f30 View commit details
    Browse the repository at this point in the history

  4. Use pytest tmpdir for large memory map fixture so works on Windows 

    Change-Id: Ia2d359a11fcecb9ce68af03554010acfc38de091
    @wesm
    wesm committed Aug 19, 2017
    Configuration menu
    Copy the full SHA
    8e59617 View commit details
    Browse the repository at this point in the history

  5. Add a Python container for the SerializedPyObject data, total_bytes m… 

    …ethod

Change-Id: I03b16c39951fedd069c935fff99f29f41dd5834c
    @wesm
    wesm committed Aug 19, 2017
    Configuration menu
    Copy the full SHA
    114a5fb View commit details
    Browse the repository at this point in the history

  6. Memory map fixture robustness on Windows 

    Change-Id: Ieaf56dc38769d1d407af27d14c5b78c4ecf5d49a
    @wesm
    wesm committed Aug 19, 2017
    Configuration menu
    Copy the full SHA
    a6a402e View commit details
    Browse the repository at this point in the history

  7. Add pyarrow.deserialize convenience method 

    Change-Id: Icfbee217fe4e0872f1b2bb306083596cdd62c992
    @wesm
    wesm committed Aug 19, 2017
    Configuration menu
    Copy the full SHA
    b70235c View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2017

  1. Add SerializedPyObject to public API 

    Change-Id: Id02790935e750554f6d71a730e543b37e412a1c9
    @wesm
    wesm committed Aug 20, 2017
    Configuration menu
    Copy the full SHA
    2164db7 View commit details
    Browse the repository at this point in the history

  2. Fix typo 

    Change-Id: I1c42641d6560d0815dce102e8481916b8bf1fe38
    @wesm
    wesm committed Aug 20, 2017
    Configuration menu
    Copy the full SHA
    31486ed View commit details
    Browse the repository at this point in the history