Contexts

[splitice]

Do you have plans to introduce other contexts such as something similar to access or rewrite (start of TCP stream). Or filter contexts such as body filtering?

With filtering, the cost of marshalling between lua strings and the char*'s representing the payload should probably be considered as many streams could be considerably large. FFI?

Oh and thank you for this release, I have a few ideas for things to try with this.

[agentzh]

@splitice See https://github.com/openresty/stream-lua-nginx-module#todo for TODO items. There is an access handler in the NGINX stream subsystem, but it does not support yielding. And I don't see any output filter mechanism in the stream subsystem either. Yeah, Lua string conversions could probably become a concern. We could go the FFI route for extreme use cases.

[splitice]

Ah I totally missed that, thanks.

Hopefully its possible in the future, there are so many neat applications of applying filtering and access control to proxied streams.

[agentzh]

@splitice Well, patches welcome :)

[splitice]

I think the specifics of that would be above me for now.

Unfortunately I found the other limiting factor in the stream module (lack of variable/map support) today. For now I'll have to shelve my plans unfortunately.

I do hope however that some smart cookie will take up the challenge.

[splitice]

Although I could probably do the bare minimum of what I want to do with a balancer_by_lua_* context I suppose.

I might have to look into how that works.

[rshriram]

Picking up the conversation again, even if I were to be able to get hold of the payload, given that this is a tcp stream, there is actually no guarantee that you will get the actual payload you are after if you are operating on message oriented streams (redis, MySQL, memcached, thrift rpc, Kafka, etc). you could be receiving a portion of the payload, which might be a valid tcp segment because the sender couldn't send the entire message (eg Kafka message) in one send() call.

So, unless you know the application's message boundary, it will become hard to decode and do things like on the fly rewriting of content. Http made it easy with clear message boundaries.

if I want I protocol-based proxies, with a bit of Lua magic to do some intelligent stuff (just like what we do with http), how would you go about this @agentzh ? I could have a tcp stream server that speaks redis tcp protocol in content_by_lua , to clients downstream (in lua) and then load balance the redis queries to a pool of redis backends using the balance by lua block or even reuse redis lua module. Note that redis is just an example. A similar use case applies to MySQL, thrift rpc upstreams, etc.

We would basically need downstream modules for various protocols, and have a filtering subsystem, followed by existing upstream modules in lua for these protocols (lua redis, lua MySQL, etc).

Overheads aside, are my assumptions and the approach correct?

[agentzh]

@rshriram Yes, it's doable.

[splitice]

Stream module just got filter module support ;)

http://hg.nginx.org/nginx/rev/56fc55e32f23

[agentzh]

@splitice That's cool.