Apache Nemo (incubating) v0.3

Nemo release 0.3

Apache Nemo (incubating) v0.1

Introduction

Apache Nemo is an in-memory distributed data processing framework that supports flexible optimization of scheduling and communication according to resource and data characteristics. This release includes implementation of policy layer, modular runtime and several example policies.

Main Features

Policy Layer

• optimizer package includes compiler passes that can be used to compose a policy. In this release, we provide annotation passes that allow policy writers to annotate metadata in IR-level DAG, reshaping passes that modifies the structure of the DAG.

Example Policies

• examples package include policies that optimize scheduling and communication according to resource and data characteristics, such as TransientResourcePolicy and DataSkewPolicy.

Modular Runtime

• Runtime of Nemo has a modular nature, where each module can be configured according to the applied policy. IR-level DAG is translated to physical-level DAG, which is launched by a single Master and executed in parallel with multiple Executors. According to the optimization encoded in the applied policy, scheduling and communication module of the Runtime is auto-configured, and optimized physical-level DAG is executed.

v0.2-rc5

[NEMO-460] Setting coders in CombinePerKey transformation (#303)

JIRA: [NEMO-460: Setting coders in CombinePerKey transformation](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-460)

**Major changes:**
- Added the additional parameter "inputCoder" for GBKTransform constructor.
- Fixed the input coder and the output coder for the partial combine transform and the final combine transform.

**Minor changes to note:**
- Fixed the main output TupleTags for the partial combine transform and the final combine transform.

**Tests for the changes:**
- Current tests suffice.

**Other comments:**
- This needs to be merged after merging #302

Closes #303

Apache Nemo (incubating) v0.2

New Features

Off-Heap Memory Management

• Nemo supports off-heap memory: max_offheap_ratio and chunk_size_kb options added #234
• Users can write policies with off-heap SerializedPartition with writing DirectByteBufferOuputStream #213
• MemoryPoolAssigner used in block creation to use off-heap memory #222 #206

Smart Compiler

• XGBoostPolicy #203

Serverless

• Lambda execution on Nemo(ResourceLambdaProperty, LambdaPass and LambdaPolicy) #214

Streaming

• NEXMark applications run on Nemo #198

Runtime Data Plane

• gRPC (instead of NCS) as a default RPC implementation #218

Major refactorings

Refactoring of Existing Passes

• Skew handling #193 #189 #188
• RuntimePass #191
• IRDAG #190

Misc.

Committer's Guide

• Committer's Guide is added #186
  Refer to https://github.com/apache/incubator-nemo/blob/master/.github/COMMITTING.md for contribution!

Other Additions

• Refined task-level and job-level metrics #289
• Migrated to Java 11 #291
• Upgraded Beam version to 2.11.0 #204