attempt to yield across C-call boundary in function 'require' with Openresty

[Blasterdick]

With uncommented line:

local http_request = require("http.request")

I'm seeing this error, while line:

local http_headers = require("http.headers")

doesn't lead to it.

I'm using docker openresty:alpine inage, lua-http installed with luarocks5.1.

lua_package_path "/usr/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?.lua;;";

Path to a /usr/local/share/lua/5.1/http/ provided with a symlink.

bash-4.3# ls -l /usr/local/share/lua/5.1/
total 56
-rw-r--r--    1 root     root          9559 Jul  5 11:07 basexx.lua
drwxr-xr-x    2 root     root          4096 Jul  5 11:06 compat53
drwxr-xr-x    3 root     root          4096 Jul  5 11:07 cqueues
-rw-r--r--    1 root     root          4105 Jul  5 11:07 cqueues.lua
-rw-r--r--    1 root     root          2097 Jul  5 11:07 fifo.lua
lrwxrwxrwx    1 root     root            41 Jul  5 11:07 http -> /usr/local/share/lua/5.1/http_0_2_0-http/
drwxr-xr-x    3 root     root          4096 Jul  5 11:07 http_0_2_0-http
drwxr-xr-x    2 root     root          4096 Jul  5 11:07 lpeg_patterns
drwxr-xr-x    4 root     root          4096 Jul  5 11:07 openssl
-rw-r--r--    1 root     root            25 Jul  5 11:07 openssl.lua
-rw-r--r--    1 root     root          6286 Jul  5 11:07 re.lua

[daurnimator]

Could you give me a traceback? I'm curious where this is coming from.

[Blasterdick]

2017/07/06 12:37:19 [error] 14#14: *1 lua entry thread aborted: runtime error: attempt to yield across C-call boundary
stack traceback:
coroutine 0:
        [C]: in function 'require'
        /var/www/config/nginx/lua/google_pubsub.lua:8: in function </var/www/config/nginx/lua/google_pubsub.lua:1>,

I'm using lua-http to sent push (POST) requests to Google Pub/Sub with function like this:

local http_request = require("http.request")
local http_headers = require("http.headers")

local req2 = function(uri, body, auth)
    local headers = http_headers.new()
    headers:append(":authorization", auth)
    headers:append(":content-length", #body)

    local request = http_request.new_from_uri(uri)
    request.version = 2
    request.tls = true
    request.port = 443
    request.headers = headers
    request:set_body(body)

    local resp_headers, stream = request:go()
    if resp_headers:get ":status" ~= "200" then
        ngx.log(ngx.ERR, stream:get_body_as_string())
    end
end

Also, here's a Dockerfile snippet with deps installing process:

FROM openresty/openresty:alpine
RUN \
	apk update && \
	apk add --no-cache unzip gcc make linux-headers bsd-compat-headers musl-dev m4
RUN \
	apk add --no-cache openssl openssl-dev lua5.1 lua5.1-dev luajit luarocks5.1 lua5.1-cjson lua5.1-socket lua5.1-sec lua5.1-crypto lua5.1-lpeg
RUN \
    luarocks-5.1 install http && \
    ln -s /usr/local/share/lua/5.1/http_0_2_0-http/ /usr/local/share/lua/5.1/http

Here's nginx.conf snippet of lua_package_path:

lua_package_path "/usr/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?.lua;;";
lua_package_cpath "/usr/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/?.so;;";

[daurnimator]

I'm using lua-http to sent push (POST) requests to Google Pub/Sub with function like this:

Note that cqueues (and hence lua-http) doesn't currently play nice with openresty: :go() and other lua-http blocking operations will block the nginx main loop. This is something I've tried to fix before ( see openresty/lua-nginx-module#450)

I'm using lua-http to sent push (POST) requests to Google Pub/Sub with function like this:
...
Also, here's a Dockerfile snippet with deps installing process:

What do I do with these? Could you give me complete instructions to replicate?

ln -s /usr/local/share/lua/5.1/http_0_2_0-http/ /usr/local/share/lua/5.1/http

It's odd that you need this....

[Blasterdick]

What do I do with these? Could you give me complete instructions to replicate?

I thought, maybe some packages have wrong versions or are missing.

It's odd that you need this....

Yup, but that's the way luarocks install this module in the alpine linux.

will block the nginx main loop

So, it's a bad idea for me to use Openresty for this purpose, or bare-metal Nginx with some special modules should do the trick? (sorry for a newbie question) Maybe, I should give some other web server a spin in that case?

[daurnimator]

What do I do with these? Could you give me complete instructions to replicate?

I thought, maybe some packages have wrong versions or are missing.

I mean: can you give me a sequence of commands to type into my shell to make that docker snippet + lua snippet throw the error? i.e. give me the full docker file; give me a full nginx.conf; etc.
It takes me far longer if I have to read through how the docker image you use is created+how it's meant to be configured, and then code up 100 lines of supporting config+code just to make your snippet run.

Yup, but that's the way luarocks install this module in the alpine linux.

Has this been brought up with the alpine luarocks maintainer? @jirutka ping.

So, it's a bad idea for me to use Openresty for this purpose,

It won't be performant if you use the default lua-http blocking API.
One solution is to not use any blocking APIs. i.e. you'd need to call things with a timeout of 0. This mode of use isn't fully fleshed out in the current lua-http release (e.g. there's no request-in-progress object (see https://github.com/daurnimator/lua-http/projects/2#card-552050)); but you should be able to get something basic working.

bare-metal Nginx with some special modules should do the trick

I'm not sure if there is a nginx http2 client module that supports this type of usage.

Maybe, I should give some other web server a spin in that case?

Maybe. I don't fully know what you're trying to do.
You could use lua-http to create a minimal http server (see https://github.com/daurnimator/lua-http/blob/master/examples/server_hello.lua) and then use it as an upstream from your nginx instance.

[Blasterdick]

Thank you for a suggestion, now I'm trying to use http-lua as a server, and I'm getting this error:

Pseudo-header fields are only valid in the context in which they are defined

with next code:

-- Make an HTTP/2.0 request
local req2 = function(uri, body, auth)

    local headers = http_headers.new()
    headers:upsert(":method", "POST")
    headers:upsert(":scheme", "https")
    headers:upsert(":path", uri)
    headers:upsert(":authorization", auth)
    headers:upsert(":content-length", #body)

    local request = http_request.new_from_uri(uri)
    request.version = 2
    request.tls = true
    request.port = 443
    request.headers = headers
    request:set_body(body)

    local resp_headers, stream = request:go(30)
    if resp_headers:get ":status" ~= "200" then
        print(stream:get_body_as_string())
    end
end

[daurnimator]

Pseudo-header fields are only valid in the context in which they are defined

headers:upsert(":authorization", auth)
headers:upsert(":content-length", #body)

":authorization" and ":content-length" are not a valid pseudo headers (you probably didn't mean to write the colon).

The only headers that can start with : are :path, :method, :authority, :scheme and :status. See http://httpwg.org/specs/rfc7540.html#PseudoHeaderFields for more info.

request.headers = headers

Avoid replacing the headers field: instead modify the existing one.
If you do so, you can avoid upserting scheme and path.

[Blasterdick]

It errors with attempt to index a nil value (field 'headers') if I try to add headers to an existing one.

Won't you, please, review my code? I'm a complete newbie in Lua and not a programmer at all, so only with a little help from my friends I can get something working.

Thanks in advance. comrade.

[Blasterdick]

After some editing, I came up with this version.

Now it hangs till timeout and errors with:

failed: pubsub_push.lua:17: attempt to index a nil value (local 'resp_headers')

While netstat shows an open connection to Google PubSub IP from appropriate PID.

[rokf]

The error happens because of the timeout. On timeout the go method returns nil, the error message and error number so your resp_headers variable gets the value nil. I'd say that there's something wrong with the headers. The possible errors are described here (section 3.1.3) if you're interested. Have you tried doing the same request with some other tool like curl?

[Blasterdick]

Thanks for a suggestion, comrade. Yup, I've tried with Postman with the same headers and raw POST data.

Duplicated question here.

[rokf]

Does your request body have the correct JSON format like required?(https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.topics/publish)

[Blasterdick]

Affirmative. With Postman everything works as expected.

[daurnimator]

I was able to make your code work.

Notes:

• Google responds with a RST_STREAM if your token contains a new line. consider using "*l" to read your token so you know you don't get a newline
• Consider caching your token rather than reading it every time
• Use assert with most lua-http calls (or handle your errors)
  • Around :get_body_as_string()
  • Around :go()
  • When using the lua file api, e.g. io.open