Skip to content

Commit

Permalink
Finish 0.6.0 blog post
Browse files Browse the repository at this point in the history
Change-Id: Iee16aac42268ab0d08320fd4931f267382f67c78
  • Loading branch information
wesm committed Aug 16, 2017
1 parent a78cb94 commit 1b7c4b6
Show file tree
Hide file tree
Showing 2 changed files with 49 additions and 8 deletions.
53 changes: 47 additions & 6 deletions site/_posts/2017-08-16-0.6.0-release.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,54 @@ memory format remains stable since the 0.3.x release.
See the [Install Page][2] to learn how to get the libraries for your
platform. The [complete changelog][5] is also available.

## Plasma Object Store
## Plasma Shared Memory Object Store

This release includes the [Plasma Store][7], which you can read more about in
the linked blog post. This system was originally developed as part of the [Ray
Project][8] at the [UC Berkeley RISELab][9]. We recognized that Plasma would be
highly valuable to the Arrow community as a tool for shared memory management
and zero-copy deserialization. Additionally, we believe we will be able to
develop a stronger software stack through sharing of IO and buffer management
code.

The Plasma store is a server application which runs as a separate process. A
reference C++ client, with Python bindings, is made available in this
release. Clients can be developed in Java or other languages in the future to
enable simple sharing of complex datasets through shared memory.

## Arrow Format Additions
## Arrow Format Addition: Map type

We added a Map logical type to represent ordered and unordered maps
in-memory. This corresponds to the `MAP` logical type annotation in the Parquet
format (where maps are represented as repeated structs).

Map is represented as a list of structs. It is the first example of a logical
type whose physical representation is a nested type. We have not yet created
implementations of Map containers in any of the implementations, but this can
be done in a future release.

As an example, the Python data:

```
data = [{'a': 1, 'bb': 2, 'cc': 3}, {'dddd': 4}]
```

Could be represented in an Arrow `Map<String, Int32>` as:

```
Map<String, Int32> = List<Struct<keys: String, values: Int32>>
is_valid: [true, true]
offsets: [0, 3, 4]
values: Struct<keys: String, values: Int32>
children:
- keys: String
is_valid: [true, true, true, true]
offsets: [0, 1, 3, 5, 9]
data: abbccdddd
- values: Int32
is_valid: [true, true, true, true]
data: [1, 2, 3, 4]
```
## Python Changes

Some highlights of Python development outside of bug fixes and general API
Expand All @@ -55,10 +97,6 @@ improvements include:

## Toward Arrow 1.0.0 and Beyond

We have much work ahead of us to build out Arrow integrations in other data
systems to improve their processing performance and interoperability with other
systems.

We are still discussing the roadmap to 1.0.0 release on the [developer mailing
list][6]. The focus of the 1.0.0 release will likely be memory format stability
and hardening integration tests across the remaining data types implemented in
Expand All @@ -69,3 +107,6 @@ Java and C++. Please join the discussion there.
[3]: http://github.com/apache/parquet-cpp
[5]: http://arrow.apache.org/release/0.6.0.html
[6]: http://mail-archives.apache.org/mod_mbox/arrow-dev/
[7]: http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-object-store/
[8]: https://ray-project.github.io/ray/
[9]: https://rise.cs.berkeley.edu/
4 changes: 2 additions & 2 deletions site/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ <h1>Apache Arrow</h1>
<p class="lead">Powering Columnar In-Memory Analytics</p>
<p>
<a class="btn btn-lg btn-success" href="mailto:dev-subscribe@arrow.apache.org" role="button">Join Mailing List</a>
<a class="btn btn-lg btn-primary" href="{{ site.baseurl }}/install/" role="button">Install (0.5.0 Release - July 23, 2017)</a>
<a class="btn btn-lg btn-primary" href="{{ site.baseurl }}/install/" role="button">Install (0.6.0 Release - August 14, 2017)</a>
</p>
</div>
<h4><strong>Latest News</strong>: <a href="{{ site.baseurl }}/blog/">Apache Arrow 0.5.0 release</a></h4>
<h4><strong>Latest News</strong>: <a href="{{ site.baseurl }}/blog/">Apache Arrow 0.6.0 release</a></h4>
<div class="row">
<div class="col-lg-4">
<h2>Fast</h2>
Expand Down

0 comments on commit 1b7c4b6

Please sign in to comment.