Support Redis cluster in input and output plugins

[jordansissel]

(This issue was originally filed by @liuxingyishi at elastic/logstash#2979)

---

When I set an output to a node in redis-cluster(Redis 3.0.0),it can not direct to other nodes in redis-cluster automatically.Port 7000 to 7005 are used by redis cluster.
logstash logstash-simple.conf:
input { stdin { } }
output {
redis {
host => ["localhost:7000"]
data_type => "list"
key => "key_count"}
stdout { codec => rubydebug }
}
logstash logs:exception=>#Redis::CommandError: MOVED 15454 127.0.0.1:7002
I supposed that it may be caused by logstash(redis-client role) was not started in cluster mode.

[ChastinaLi]

👍

[splitice]

Just thought I would share a little bit of an experiement with cluster support.

Heres the hacked together plugin (https://github.com/splitice/logstash-input-redis), do not even think of running this is in production. Its just some early exploration of what would be needed (and some fiddling in ruby as I have never touched the language before). After I get our log producer to support redis-cluster I'll do some more work.

Only the queue/list based method has been tested.

Configure like:

redis_cluster {
host => "127.0.0.1"
data_type => "list"
keys => ["rsyslog_1"]
}

[guyboertje]

OK - All the bazillion plus ones have convinced me. I am also swayed by the fine experiment that @splitice has done with Jedis.

I will build a Jedis JRuby extension that supports pipelined lrange ltrim on clustered Redis.

[mp911de]

@guyboertje @splitice Did you test the redis input also with lettuce? Lettuce does not require connection pooling and pipelines commands by default.

[huseyincotuk]

@guyboertje @splitice Any progress on the development? I can test it as a volunteer right now. Please let me know when you finish.

[splitice]

We are now using a different intermediary queue, my work is complete / idle.

Imho redid cluster tooling still needs some work before it's ready for us.