ngx.resp.get_headers(): Why is the response header incomplete?

[chenshanyi]

config:
location =/version {
default_type 'text/plain';

        content_by_lua_file /usr/local/lmrs/lua/api/version.lua;
    
        log_by_lua '
	ngx.log(ngx.ERR,"header: ", require"cjson".encode(ngx.resp.get_headers()))
    ';
    }

request:
curl -i "http://127.0.0.1:8088/version"
HTTP/1.1 200 OK
Server: nginx/1.12.2
Date: Tue, 10 Sep 2019 13:17:46 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive

log:
2019/09/10 09:12:14 [error] 123144#0: *3 [lua] log_by_lua(lmrs.conf:195):2: header: {"transfer-encoding":"chunked","content-type":"text/plain","connection":"close"} while logging request, client: 127.0.0.1, server: , request: "GET /version HTTP/1.1", host: "127.0.0.1:8088"

Why is the response header incomplete?

[dndx]

Hello,

Nginx in general stores headers in a linked list. But this has a drawback of changing and accessing a header requires traversing the list to find it which may be slow. As an optimization Nginx also defined a bunch of "commonly used" headers that are not stored in the list, but rather in the struct itself, this guarantees O(1) access while changing/reading those "commonly used" headers.

See definitions of ngx_http_headers_out_t struct

ngx.resp.get_headers calls ngx_http_lua_ngx_resp_get_headers. In that function we did not handle all the cases for "fast path" headers. AFAIK only content-type, content-length, connection and transfer-encoding were handled.

From Nginx source that handles $sent_http_* variables, the following header support needs to be added for ngx.resp.get_headers to behave correctly:

• Location
• Last-Modified
• Keep-Alive
• Cache-Control
• Link
• Date
• Server

Note that the last two headers, Date and Server were not even handled by the $sent_http_* variables properly, presumably they were overlooked. This was confirmed with my local testing by reading $sent_http_server and $sent_http_date in log phase. Both returned nil.

Is there any work around for this?
Our own requirements are to log in access logs the Location response header which works simply with $upstream_http_location.
However we have additional requirements to mask out some secure data from the parameters in that url. I have some lua script that does that masking successfully but we can't pass it ngx.resp.get_headers()["location"].

[bjuergens]

I don't know much about nginx or lua, but I noticed some weird behavior, which let me to find this issue.

I was playing around with luafilters for headers, when I noticed, that ngx.resp.get_headers gives a different number of headers when called from set_by_lua_block and from header_filter_by_lua

consider the following example

set_by_lua_block $resp_headers_short{
    local lua_resp_headers_short = ""
    for k, v in pairs(ngx.resp.get_headers()) do
        lua_resp_headers_short = lua_resp_headers_short .. k.."="..v.." "
    end
    return lua_resp_headers_short
}

set $resp_headers_full "";
header_filter_by_lua '
    for k, v in pairs(ngx.resp.get_headers()) do
        ngx.var.resp_headers_full = ngx.var.resp_headers_full .. k.."="..v.." "
    end
';

the first block only produces connection=keep-alive, while the latter block produces all of them last-modified=Tue, 13 Apr 2021 15:13:59 GMT etag=\"6075b537-264\" accept-ranges=bytes content-type=text/html connection=keep-alive content-length=612 (sans date and server)

considering the comment above, I guess that header_filter_by_lua is executed at a different time, where fast-path headers are already part of the request.

Is there any work around for this?

use header_filter_by_lua, apparently

[alexdowad]

Is there any work around for this? Our own requirements are to log in access logs the Location response header which works simply with $upstream_http_location. However we have additional requirements to mask out some secure data from the parameters in that url. I have some lua script that does that masking successfully but we can't pass it ngx.resp.get_headers()["location"].

I had the same problem of trying to access the Location header in a log_by_lua block (though a different use case from you). @dndx's comment gave me the needed clue.

I was able to get it with ngx.var.sent_http_location. I guess for you it would be ngx.var.upstream_http_location.