-
Notifications
You must be signed in to change notification settings - Fork 171
Suro server
Suro server consists of three components: SuroInput, MessageRouter, and Sink.
Suro server is a THsHaServer server with TNonblockingServerTransport. The ThriftServer receives messages of TMessageSet type from Suro client and puts them in a queue managed by a MessageSetProcessor. Similar to Suro's asynchronous client, a MessageSetProcessor can choose to use a memory queue or a file-backed queue. The advantage of file-backed queue is that messages are written to disk first, so they will not get lost even if Suro server terminates unexpectedly.
Instead of receiving messages from thrift Suro client, Suro server can consume messages from Kafka broker 0.8. KafkaConsumer is implemented in suro-kafka and internally, KafkaConsumer is routing messages to sinks according to MessageRouter and sink configuration.
MessageRouter dispatches messages to sinks according to their routing key and filtering rules. User can define message routing map as Json string.
Sink is the destination of the data. It can be local file(LocalFileSink), remote file(RemoteFileSink such as S3FileSink), or other data service such as Kafka, Elasticsearch, and Cassandra. Currently, the following sinks are open sourced:
- LocalFileSink: text file or Hadoop sequence file is supported.
- S3FileSink: It embeds LocalFileSink and upload the file to AWS S3.
- HdfsFileSink: It embeds LocalFileSink and upload the file to HDFS.
- KafkaSink: Sends messages to Kafka 0.8 broker.
- ElasticSearchSink: Sends messages to ElasticSearch server v1.1.0.
We may also later open source sinks for HTTP/HTTPs endpoints, and Cassandra.
Instead of