Log format improvements #6

itskingori · 2021-02-10T10:28:56Z

By doing the following:

Renamed main logging format to main_default.
Remove unnecessary double space between $time_local & $status from main_default logging format.
Add main_json logging format and configure it on access_log to log in JSON format.

See commits for details. Each commit is an idea.

* http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format * https://stackoverflow.com/questions/25049667/how-to-generate-a-json-log-from-nginx

itskingori

Some commentary ... 💬

itskingori · 2021-02-10T10:33:39Z

config/log.conf

  '}';

-access_log /dev/stdout main_default;
+access_log /dev/stdout main_json;


I dunno if changing the default logging is agreeable but figured in 2021 we might as well move to structured logging and so having this in JSON is a good default! 😅 Makes parsing it easier!!

And I should have mentioned that the reason I left main_default in was because it allows one to override access_log /dev/stdout main_default; elsewhere e.g. like in a location block.

dunno if changing the default logging is agreeable but figured in 2021 we might as well move to structured logging and so having this in JSON is a good default! 😅 Makes parsing it easier!!

It's bittersweet after seeing human-readable lines for so long. 😭

It's bittersweet after seeing human-readable lines for so long. 😭

🙈 🙊 🙉

itskingori · 2021-02-10T10:34:27Z

config/log.conf

@@ -1,4 +1,4 @@
-log_format main '$remote_addr - $remote_user [$time_local]  $status '
+log_format main '$remote_addr - $remote_user [$time_local] $status '


Removed because I think this was left here by mistake. Not easy to spot so it could have easily been missed.

itskingori · 2021-02-10T10:36:01Z

config/log.conf

+    '"body_bytes_sent":"$body_bytes_sent",'
+    '"http_referrer":"$http_referer",'
+    '"http_user_agent":"$http_user_agent",'
+    '"http_x_forwarded_for":"$http_x_forwarded_for",'
+    '"proxy_x_request_id":"$proxy_x_request_id",'
+    '"remote_addr":"$remote_addr",'
+    '"remote_user":"$remote_user",'
+    '"request":"$request",'
+    '"request_length":"$request_length",'
+    '"request_time":"$request_time",'
+    '"status": "$status",'
+    '"time_local":"$time_local"'


To note, the last attribute has no ,. Also; two questions:

Is there anything we'd like to add?

Is there anything we'd like to rename?

I've added $host, $proxy_x_forwarded_port, $proxy_x_forwarded_proto and $proxy_x_forwarded_ssl in 05abb67.

I think it's probably okay for now.

That may be useful for debugging.

itskingori

Tested just in case there are syntax errors ...

$ docker-compose up -d
Creating network "docker-nginx-proxy_local" with the default driver
Creating docker-nginx-proxy_app_1 ... done
Creating docker-nginx-proxy_proxy_1 ... done

$ docker-compose ps
           Name                         Command               State               Ports             
----------------------------------------------------------------------------------------------------
docker-nginx-proxy_app_1     caddy run --config /etc/ca ...   Up      2019/tcp, 443/tcp, 80/tcp     
docker-nginx-proxy_proxy_1   /docker-entrypoint.sh ngin ...   Up      80/tcp, 0.0.0.0:8080->8080/tcp

$ curl -I http://0.0.0.0:8080 
HTTP/1.1 200 OK
Date: Wed, 10 Feb 2021 12:31:06 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 12226
Connection: keep-alive
Accept-Ranges: bytes
Etag: "qlhhmd9fm"
Last-Modified: Thu, 17 Dec 2020 12:35:01 GMT
X-Robots-Tag: "noindex, nofollow"

$ docker-compose logs -f proxy
Attaching to docker-nginx-proxy_proxy_1
proxy_1  | {"body_bytes_sent":"0","host":"0.0.0.0","http_referrer":"","http_user_agent":"curl/7.73.0","http_x_forwarded_for":"","proxy_x_forwarded_port":"8080","proxy_x_forwarded_proto":"http","proxy_x_forwarded_ssl":"off","proxy_x_request_id":"fc4fa2bfb7b54ec917a462d6c9bb5442","remote_addr":"172.18.0.1","remote_user":"","request":"HEAD / HTTP/1.1","request_length":"77","request_time":"0.001","status": "200","time_local":"10/Feb/2021:12:31:06 +0000"}

And the output is valid JSON ...

{
  "body_bytes_sent": "0",
  "host": "0.0.0.0",
  "http_referrer": "",
  "http_user_agent": "curl/7.73.0",
  "http_x_forwarded_for": "",
  "proxy_x_forwarded_port": "8080",
  "proxy_x_forwarded_proto": "http",
  "proxy_x_forwarded_ssl": "off",
  "proxy_x_request_id": "fc4fa2bfb7b54ec917a462d6c9bb5442",
  "remote_addr": "172.18.0.1",
  "remote_user": "",
  "request": "HEAD / HTTP/1.1",
  "request_length": "77",
  "request_time": "0.001",
  "status": "200",
  "time_local": "10/Feb/2021:12:31:06 +0000"
}

itskingori · 2021-02-10T12:44:56Z

config/log.conf

+    '"request_length":"$request_length",'
+    '"request_time":"$request_time",'
+    '"status": "$status",'
+    '"time_local":"$time_local"'


By the way this returns 10/Feb/2021:12:31:06 +000 but there's also $time_iso8601. Thoughts?

$time_iso8601 local time in the ISO 8601 standard format $time_local local time in the Common Log Format

Options are:

We change to $time_iso8601.

We keep both $time_iso8601 and $time_local.

I'm leaning on the second one so that one can choose what they want.

I think $time_iso8601 is better as it is more common than CLF. 😬

LOL. Ok, changing.

Changed to $time_iso8601 in c2b081f.

itskingori added 2 commits February 10, 2021 13:02

Remove unnecessary double space

4f77960

Renamed main logging format to main_default

3b2b0c9

itskingori requested a review from a team as a code owner February 10, 2021 10:28

itskingori added 3 commits February 10, 2021 13:30

Add main_json logging format

839b200

* http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format * https://stackoverflow.com/questions/25049667/how-to-generate-a-json-log-from-nginx

Configure access_log with main_json

e54e574

Bump version and update CHANGELOG

ddd075d

itskingori force-pushed the log_format_improvements branch from 76e1ea8 to ddd075d Compare February 10, 2021 10:31

itskingori commented Feb 10, 2021

View reviewed changes

itskingori mentioned this pull request Feb 10, 2021

Add support for request tracing via X-Request-ID header #5

Merged

Add some more info to main_json

05abb67

That may be useful for debugging.

itskingori commented Feb 10, 2021

View reviewed changes

Prefer time_iso8601 over time_local for time format

c2b081f

zacblazic approved these changes Feb 10, 2021

View reviewed changes

itskingori merged commit 2c9b4cb into master Feb 10, 2021

itskingori deleted the log_format_improvements branch February 10, 2021 13:55

		@@ -1,4 +1,4 @@
		log_format main '$remote_addr - $remote_user [$time_local] $status '
		log_format main '$remote_addr - $remote_user [$time_local] $status '

Log format improvements #6

Log format improvements #6

Uh oh!

Conversation

itskingori commented Feb 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

itskingori left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

itskingori left a comment

Choose a reason for hiding this comment

Uh oh!

itskingori Feb 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zacblazic Feb 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

itskingori commented Feb 10, 2021 •

edited

Loading

itskingori Feb 10, 2021 •

edited

Loading

zacblazic Feb 10, 2021 •

edited

Loading