Skip to content

Commit

Permalink
detailed description of MA
Browse files Browse the repository at this point in the history
  • Loading branch information
yongkun.wang committed Nov 15, 2013
1 parent c871f85 commit cb7c032
Showing 1 changed file with 10 additions and 3 deletions.
13 changes: 10 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,19 @@ It is forked from Cloudera's 0.9.3-cdh3u0 distribution (Flume-OG).

Two major enhancements:

1) Masterless Ack:
1) Masterless ACK (MA):
This enhancement aims to increase the reliability and throughput of the whole distributed collecting system.
Flume provides End-to-End delivery mode to guarantee the data delivery; an acknowledgement message (ACK) is sent back to original node to confirm the successful delivery of a group of messages. However, the ACKs are sent back through master, which could be a single-point-of-failure or bottle neck of the whole system.

Flume provides End-to-End delivery mode to guarantee the data delivery; an acknowledgement message (ACK) is sent back to original node to confirm the successful delivery of a group of messages.

However, the ACKs are sent back through master, which could be a single-point-of-failure or bottle neck of the whole system.
Therefore, I re-designed the ACKs system to let the ACK go back via the route of Event.
Main enhancements:
a) Distribute ACK traffic to each flume node, ACK goes back to original agent throught the route of event.
b) Reuse the connection for event transmission.
c) Buffer input/output stream for thrift to improve throughput. https://github.com/yongkun/thrift

You can also get some information here https://issues.apache.org/jira/browse/FLUME-640
Apache JIRA is here, with some documents https://issues.apache.org/jira/browse/FLUME-640
This enhancement was supposed to be merged into 0.10 if Flume was not upgraded to NG;

2) Append to HDFS with new file rotation method.
Expand Down

0 comments on commit cb7c032

Please sign in to comment.