BUG: --enable-dynamic-configuration=true doesnt work with external auth #2268
Description
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT
NGINX Ingress controller version:master (0.346)
Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T19:48:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.10", GitCommit:"bebdeb749f1fa3da9e1312c4b08e439c404b3136", GitTreeState:"clean", BuildDate:"2017-11-03T16:31:49Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Environment:
- Cloud provider or hardware configuration: AWS
- OS (e.g. from /etc/os-release): Ubuntu
- Kernel (e.g.
uname -a): 4.4.65-k8s Basic structure #1 SMP Tue May 2 15:48:24 UTC 2017 x86_64 Linux - Install tools: Kops
What happened:
External auth feature stopped working when nginx is started with --enable-dynamic-configuration=true
What you expected to happen:
External auth to work properly
How to reproduce it (as minimally and precisely as possible):
- Create ingress file that uses auth-url
- Make a request to a url that should use the auth-url in ingress
Anything else we need to know:
Dynamic configuration works for everything beside external auth. The backends are correctly configured and requests succeed.
How external auth is configured in ingress (the annotation has correct prefix, and localhost means that the external auth server is running as a sidecar for nginx)
ingress.kubernetes.io/auth-url: http://localhost:8888/validate
ingress.kubernetes.io/auth-response-headers: X-SEC-ID
Nginx failure log:
"POST /path HTTP/1.1" 200 0 "-" "Main (Main)" 0 0.975 [external-authentication] 127.0.0.1:8886 0 0.975 200 a158d8a6e08c556a1fad5c39aeed28f6
2018/03/21 21:06:12 [error] 178#178: *728 failed to run balancer_by_lua*: /etc/nginx/lua/balancer.lua:31: attempt to index local 'backend' (a nil value)
stack traceback:
/etc/nginx/lua/balancer.lua:31: in function 'balance'
/etc/nginx/lua/balancer.lua:97: in function 'call'
balancer_by_lua:2: in function <balancer_by_lua:1> while connecting to upstream, client: IP, server: server, request: "POST /path HTTP/1.1", host: "host"
Nginx startup logs:
curl localhost:18080/configuration/backends
nil
More logs:
NGINX Ingress controller
Release: 0.12.0
Build: git-0398c410
Repository: https://github.com/aledbf/ingress-nginx
-------------------------------------------------------------------------------
W0321 19:50:09.284853 7 client_config.go:529] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0321 19:50:09.285014 7 main.go:181] Creating API client for https://REDUCTED:443
I0321 19:50:09.285641 7 main.go:201] trying to discover Kubernetes version
I0321 19:50:09.294493 7 main.go:225] Running in Kubernetes Cluster version v1.7 (v1.7.10) - git (clean) commit bebdeb749f1fa3da9e1312c4b08e439c404b3136 - platform linux/amd64
I0321 19:50:09.297059 7 main.go:84] validated nginx/nginx-default-backend as the default backend
I0321 19:50:09.299614 7 main.go:105] service nginx/ingress-nginx validated as source of Ingress status
I0321 19:50:09.549228 7 stat_collector.go:77] starting new nginx stats collector for Ingress controller running in namespace (class nginx)
I0321 19:50:09.549244 7 stat_collector.go:78] collector extracting information from port 18080
I0321 19:50:09.560763 7 nginx.go:281] starting Ingress controller
I0321 19:50:09.572054 7 store.go:404] adding configmap nginx/ingress-nginx to backend
I0321 19:50:10.677349 7 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"app-bff-staging", Name:"app-bff-ingress", UID:"5f491870-20bc-11e8-889c-0a91ad1abda6", APIVersion:"extensions", ResourceVersion:"39561205", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress app-bff-staging/app-bff-ingress
...
I0321 19:50:10.762101 7 store.go:614] running initial sync of secrets
I0321 19:50:10.765779 7 backend_ssl.go:68] adding secret app-bff-staging/app-bff-staging-tls to the local store
I0321 19:50:10.767969 7 nginx.go:302] starting NGINX process...
I0321 19:50:10.768770 7 controller.go:183] backend reload required
I0321 19:50:10.768804 7 stat_collector.go:34] changing prometheus collector from to vts
I0321 19:50:10.769177 7 util.go:64] system fs.file-max=2097152
I0321 19:50:10.769227 7 nginx.go:560] maximum number of open file descriptors : 523264
I0321 19:50:10.775212 7 leaderelection.go:174] attempting to acquire leader lease...
I0321 19:50:10.792038 7 status.go:196] new leader elected: ingress-nginx-2742315815-3kwgn
I0321 19:50:10.903607 7 nginx.go:658] NGINX configuration diff
I0321 19:50:10.903784 7 nginx.go:659] --- /etc/nginx/nginx.conf 2018-03-21 03:31:50.000000000 +0000
+++ /tmp/new-nginx-cfg724125397 2018-03-21 19:50:10.893639217 +0000
@@ -1,6 +1,1744 @@
-# A very simple nginx configuration file that forces nginx to start.
+
+daemon off;
+
+worker_processes 4;
+
pid /run/nginx.pid;
-events {}
-http {}
-daemon off;
\ No newline at end of file
+worker_rlimit_nofile 523264;
+
+worker_shutdown_timeout 10s ;
+
+events {
+ multi_accept on;
+ worker_connections 16384;
+ use epoll;
+}
+
+http {
+ lua_package_cpath "/usr/local/lib/lua/?.so;/usr/lib/x86_64-linux-gnu/lua/5.1/?.so;;";
+ lua_package_path "/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/?.lua;/usr/local/lib/lua/?.lua;;";
+
+ lua_shared_dict configuration_data 5M;
+ lua_shared_dict round_robin_state 1M;
+ lua_shared_dict locks 512k;
+
+ init_by_lua_block {
+ require("resty.core")
+ collectgarbage("collect")
+
+ -- init modules
+ local ok, res
+
+ ok, res = pcall(require, "configuration")
+ if not ok then
+ error("require failed: " .. tostring(res))
+ else
+ configuration = res
+ end
+
+ ok, res = pcall(require, "balancer")
+ if not ok then
+ error("require failed: " .. tostring(res))
+ else
+ balancer = res
+ end
+ }
+
+ init_worker_by_lua_block {
+ balancer.init_worker()
+ }
+
+ real_ip_header proxy_protocol;
+
+ real_ip_recursive on;
+
+ set_real_ip_from 0.0.0.0/0;
+
+ geoip_country /etc/nginx/geoip/GeoIP.dat;
+ geoip_city /etc/nginx/geoip/GeoLiteCity.dat;
+ geoip_org /etc/nginx/geoip/GeoIPASNum.dat;
+ geoip_proxy_recursive on;
+
+ vhost_traffic_status_zone shared:vhost_traffic_status:10m;
+ vhost_traffic_status_filter_by_set_key $server_name;
+
+ aio threads;
+ aio_write on;
+
+ tcp_nopush on;
+ tcp_nodelay on;
+
+ log_subrequest on;
+
+ reset_timedout_connection on;
+
+ keepalive_timeout 75s;
+ keepalive_requests 100;
+
+ client_header_buffer_size 1k;
+ client_header_timeout 60s;
+ large_client_header_buffers 4 8k;
+ client_body_buffer_size 8k;
+ client_body_timeout 60s;
+
+ http2_max_field_size 4k;
+ http2_max_header_size 16k;
+
+ types_hash_max_size 2048;
+ server_names_hash_max_size 1024;
+ server_names_hash_bucket_size 64;
+ map_hash_bucket_size 64;
+
+ proxy_headers_hash_max_size 512;
+ proxy_headers_hash_bucket_size 64;
+
+ variables_hash_bucket_size 128;
+ variables_hash_max_size 2048;
+
+ underscores_in_headers off;
+ ignore_invalid_headers on;
+
+ limit_req_status 503;
+
+ include /etc/nginx/mime.types;
+ default_type text/html;
+
+ gzip on;
+ gzip_comp_level 5;
+ gzip_http_version 1.1;
+ gzip_min_length 256;
+ gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
+ gzip_proxied any;
+ gzip_vary on;
+
+ # Custom headers for response
+
+ server_tokens on;
+
+ # disable warnings
+ uninitialized_variable_warn off;
+
+ # Additional available variables:
+ # $namespace
+ # $ingress_name
+ # $service_name
+ log_format upstreaminfo '%v - [$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $request_id';
+
+ map $request_uri $loggable {
+
+ default 1;
+ }
+
+ access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
+
+ error_log /var/log/nginx/error.log info;
+
+ resolver REDUCTED valid=30s;
+
+ # Retain the default nginx handling of requests without a "Connection" header
+ map $http_upgrade $connection_upgrade {
+ default upgrade;
+ '' close;
+ }
+
+ map $http_x_forwarded_for $the_real_ip {
+
+ # Get IP address from Proxy Protocol
+ default $proxy_protocol_addr;
+
+ }
+
+ # trust http_x_forwarded_proto headers correctly indicate ssl offloading
+ map $http_x_forwarded_proto $pass_access_scheme {
+ default $http_x_forwarded_proto;
+ '' $scheme;
+ }
+
+ # validate $pass_access_scheme and $scheme are http to force a redirect
+ map "$scheme:$pass_access_scheme" $redirect_to_https {
+ default 0;
+ "http:http" 1;
+ "https:http" 1;
+ }
+
+ map $http_x_forwarded_port $pass_server_port {
+ default $http_x_forwarded_port;
+ '' $server_port;
+ }
+
+ map $pass_server_port $pass_port {
+ 443 443;
+ default $pass_server_port;
+ }
+
+ # Obtain best http host
+ map $http_host $this_host {
+ default $http_host;
+ '' $host;
+ }
+
+ map $http_x_forwarded_host $best_http_host {
+ default $http_x_forwarded_host;
+ '' $this_host;
+ }
+
+ server_name_in_redirect off;
+ port_in_redirect off;
+
+ rewrite_log on;
+
+ ssl_protocols TLSv1.2;
+
+ # turn on session caching to drastically improve performance
+
+ ssl_session_cache builtin:1000 shared:SSL:10m;
+ ssl_session_timeout 10m;
+
+ # allow configuring ssl session tickets
+ ssl_session_tickets on;
+
+ # slightly reduce the time-to-first-byte
+ ssl_buffer_size 4k;
+
+ # allow configuring custom ssl ciphers
+ ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
+ ssl_prefer_server_ciphers on;
+
+ ssl_ecdh_curve auto;
+
+ proxy_ssl_session_reuse on;
+
+ upstream upstream_balancer {
+ server 0.0.0.1; # placeholder
+
+ balancer_by_lua_block {
+ balancer.call()
+ }
+
+ keepalive 1000;
+ }
+
+ ## start server _
+ server {
+ server_name _ ;
+
+ listen 80 proxy_protocol default_server backlog=511;
+
+ listen [::]:80 proxy_protocol default_server backlog=511;
+
+ set $proxy_upstream_name "-";
+
+ listen 443 proxy_protocol default_server backlog=511 ssl http2;
+
+ listen [::]:443 proxy_protocol default_server backlog=511 ssl http2;
+
+ # PEM sha: 06070084cadda62b8096e43ff43f464dfc4c57a7
+ ssl_certificate /ingress-controller/ssl/default-fake-certificate.pem;
+ ssl_certificate_key /ingress-controller/ssl/default-fake-certificate.pem;
+
+ location / {
+
+ if ($scheme = https) {
+ more_set_headers "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
+ }
+
+ access_log off;
+
+ port_in_redirect off;
+
+ set $proxy_upstream_name "upstream-default-backend";
+
+ set $namespace "";
+ set $ingress_name "";
+ set $service_name "";
+
+ client_max_body_size "1m";
+
+ proxy_set_header Host $best_http_host;
+
+ # Pass the extracted client certificate to the backend
+
+ proxy_set_header ssl-client-cert "";
+ proxy_set_header ssl-client-verify "";
+ proxy_set_header ssl-client-dn "";
+
+ # Allow websocket connections
+ proxy_set_header Upgrade $http_upgrade;
+
+ proxy_set_header Connection $connection_upgrade;
+
+ proxy_set_header X-Real-IP $the_real_ip;
+
+ proxy_set_header X-Forwarded-For $the_real_ip;
+
+ proxy_set_header X-Forwarded-Host $best_http_host;
+ proxy_set_header X-Forwarded-Port $pass_port;
+ proxy_set_header X-Forwarded-Proto $pass_access_scheme;
+ proxy_set_header X-Original-URI $request_uri;
+ proxy_set_header X-Scheme $pass_access_scheme;
+
+ # Pass the original X-Forwarded-For
+ proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
+
+ # mitigate HTTPoxy Vulnerability
+ # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
+ proxy_set_header Proxy "";
+
+ # Custom headers to proxied server
+
+ proxy_connect_timeout 5s;
+ proxy_send_timeout 60s;
+ proxy_read_timeout 60s;
+
+ proxy_buffering "off";
+ proxy_buffer_size "4k";
+ proxy_buffers 4 "4k";
+ proxy_request_buffering "on";
+
+ proxy_http_version 1.1;
+
+ proxy_cookie_domain off;
+ proxy_cookie_path off;
+
+ # In case of errors try the next upstream server before returning an error
+ proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
+
+ proxy_pass http://upstream_balancer;
+
+ proxy_redirect off;
+
+ }
+
+ # health checks in cloud providers require the use of port 80
+ location /healthz {
+ access_log off;
+ return 200;
+ }
+
+ # this is required to avoid error if nginx is being monitored
+ # with an external software (like sysdig)
+ location /nginx_status {
+ allow 127.0.0.1;
+ allow ::1;
+ deny all;
+
+ access_log off;
+ stub_status on;
+ }
+
+ }
+ ## end server _
+
+ ## start server REDUCTED
+ server {
+ server_name REDUCTED ;
+
+ listen 80 proxy_protocol;
+
+ listen [::]:80 proxy_protocol;
+
+ set $proxy_upstream_name "-";
+
+ listen 443 proxy_protocol ssl http2;
+
+ listen [::]:443 proxy_protocol ssl http2;
+
+ # PEM sha: 61ad3926d647a0de2c0fdad636f6c38e012a9dd3
+ ssl_certificate /ingress-controller/ssl/app-bff-staging-app-bff-staging-tls.pem;
+ ssl_certificate_key /ingress-controller/ssl/app-bff-staging-app-bff-staging-tls.pem;
+
+ location /.well-known/acme-challenge {
+
+ if ($scheme = https) {
+ more_set_headers "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
+ }
+
+ port_in_redirect off;
+
+ set $proxy_upstream_name "kube-lego-kube-lego-nginx-8080";
+
+ set $namespace "kube-lego";
+ set $ingress_name "kube-lego-nginx";
+ set $service_name "kube-lego-nginx";
+
+ client_max_body_size "1m";
+
+ proxy_set_header Host $best_http_host;
+
+ # Pass the extracted client certificate to the backend
+
+ proxy_set_header ssl-client-cert "";
+ proxy_set_header ssl-client-verify "";
+ proxy_set_header ssl-client-dn "";
+
+ # Allow websocket connections
+ proxy_set_header Upgrade $http_upgrade;
+
+ proxy_set_header Connection $connection_upgrade;
+
+ proxy_set_header X-Real-IP $the_real_ip;
+
+ proxy_set_header X-Forwarded-For $the_real_ip;
+
+ proxy_set_header X-Forwarded-Host $best_http_host;
+ proxy_set_header X-Forwarded-Port $pass_port;
+ proxy_set_header X-Forwarded-Proto $pass_access_scheme;
+ proxy_set_header X-Original-URI $request_uri;
+ proxy_set_header X-Scheme $pass_access_scheme;
+
+ # Pass the original X-Forwarded-For
+ proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
+
+ # mitigate HTTPoxy Vulnerability
+ # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
+ proxy_set_header Proxy "";
+
+ # Custom headers to proxied server
+
+ proxy_connect_timeout 5s;
+ proxy_send_timeout 60s;
+ proxy_read_timeout 60s;
+
+ proxy_buffering "off";
+ proxy_buffer_size "4k";
+ proxy_buffers 4 "4k";
+ proxy_request_buffering "on";
+
+ proxy_http_version 1.1;
+
+ proxy_cookie_domain off;
+ proxy_cookie_path off;
+
+ # In case of errors try the next upstream server before returning an error
+ proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
+
+ proxy_pass http://upstream_balancer;
+
+ proxy_redirect off;
+
+ }
+
+ location / {
+
+ if ($scheme = https) {
+ more_set_headers "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
+ }
+
+ port_in_redirect off;
+
+ set $proxy_upstream_name "app-bff-staging-app-bff-service-3000";
+
+ set $namespace "app-bff-staging";
+ set $ingress_name "app-bff-notls-ingress";
+ set $service_name "app-bff-service";
+
+ # enforce ssl on server side
+ if ($redirect_to_https) {
+
+ return 308 https://$best_http_host$request_uri;
+
+ }
+
+ client_max_body_size "1m";
+
+ proxy_set_header Host $best_http_host;
+
+ # Pass the extracted client certificate to the backend
+
+ proxy_set_header ssl-client-cert "";
+ proxy_set_header ssl-client-verify "";
+ proxy_set_header ssl-client-dn "";
+
+ # Allow websocket connections
+ proxy_set_header Upgrade $http_upgrade;
+
+ proxy_set_header Connection $connection_upgrade;
+
+ proxy_set_header X-Real-IP $the_real_ip;
+
+ proxy_set_header X-Forwarded-For $the_real_ip;
+
+ proxy_set_header X-Forwarded-Host $best_http_host;
+ proxy_set_header X-Forwarded-Port $pass_port;
+ proxy_set_header X-Forwarded-Proto $pass_access_scheme;
+ proxy_set_header X-Original-URI $request_uri;
+ proxy_set_header X-Scheme $pass_access_scheme;
+
+ # Pass the original X-Forwarded-For
+ proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
+
+ # mitigate HTTPoxy Vulnerability
+ # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
+ proxy_set_header Proxy "";
+
+ # Custom headers to proxied server
+
+ proxy_connect_timeout 5s;
+ proxy_send_timeout 60s;
+ proxy_read_timeout 60s;
+
+ proxy_buffering "off";
+ proxy_buffer_size "4k";
+ proxy_buffers 4 "4k";
+ proxy_request_buffering "on";
+
+ proxy_http_version 1.1;
+
+ proxy_cookie_domain off;
+ proxy_cookie_path off;
+
+ # In case of errors try the next upstream server before returning an error
+ proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
+
+ proxy_pass http://upstream_balancer;
+
+ proxy_redirect off;
+
+ }
+
+ }
+ ## end server REDUCTED
+
...
+ # default server, used for NGINX healthcheck and access to nginx stats
+ server {
+ # Use the port 18080 (random value just to avoid known ports) as default port for nginx.
+ # Changing this value requires a change in:
+ # https://github.com/kubernetes/ingress-nginx/blob/master/controllers/nginx/pkg/cmd/controller/nginx.go
+ listen 18080 default_server backlog=511;
+ listen [::]:18080 default_server backlog=511;
+ set $proxy_upstream_name "-";
+
+ location /healthz {
+ access_log off;
+ return 200;
+ }
+
+ location /nginx_status {
+ set $proxy_upstream_name "internal";
+
+ access_log off;
+ stub_status on;
+
+ }
+
+ location /configuration {
+ allow 127.0.0.1;
+
+ allow ::1;
+
+ deny all;
+ content_by_lua_block {
+ configuration.call()
+ }
+ }
+
+ location / {
+
+ set $proxy_upstream_name "upstream-default-backend";
+
+ proxy_pass http://upstream_balancer;
+
+ }
+
+ }
+}
+
+stream {
+ log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;
+
+ access_log /var/log/nginx/access.log log_stream;
+
+ error_log /var/log/nginx/error.log;
+
+ # TCP services
+
+ # UDP services
+
+}
+
I0321 20:06:44.696023 7 controller.go:192] ingress backend successfully reloaded...
I0321 20:06:45.584802 7 backend_ssl.go:174] updating local copy of ssl certificate app-bff-staging/app-bff-staging-tls with missing intermediate CA certs
I0321 20:06:45.697116 7 nginx.go:771] posting backends configuration: [{"name":"app-bff-staging-app-bff-service-3000","service":{"metadata":{"name":"app-bff-service","namespace":"app-bff-staging","selfLink":"/api/v1/namespaces/app-bff-staging/services/app-bff-service","uid":"5f2f79ae-20bc-11e8-a1f9-02d01e4a667a","resourceVersion":"39561187","creationTimestamp":"2018-03-05T21:30:05Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"name\":\"app-bff-service\",\"namespace\":\"app-bff-staging\"},\"spec\":{\"ports\":[{\"name\":\"backend\",\"port\":3000,\"protocol\":\"TCP\",\"targetPort\":3000}],\"selector\":{\"app\":\"app-bff\"},\"type\":\"ClusterIP\"}}\n"}},"spec":{"ports":[{"name":"backend","protocol":"TCP","port":3000,"targetPort":3000}],"selector":{"app":"app-bff"},"clusterIP":"REDUCTED","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":3000,"secure":false,"secureCACert":{"secret":"","caFilename":"","pemSha":""},"sslPassthrough":false,"endpoints":[{"address":"REDUCTED","port":"3000","maxFails":0,"failTimeout":0,"target":{"kind":"Pod","namespace":"app-bff-staging","name":"app-bff-3-2881575126-jwllz","uid":"7643c83d-2cea-11e8-889c-0a91ad1abda6","resourceVersion":"42266422"}},{"address":"REDUCTED","port":"3000","maxFails":0,"failTimeout":0,"target":{"kind":"Pod","namespace":"app-bff-staging","name":"app-bff-stateless-3511195472-7p332","uid":"c4bbdd20-2cf2-11e8-889c-0a91ad1abda6","resourceVersion":"42276219"}},{"address":"REDUCTED","port":"3000","maxFails":0,"failTimeout":0,"target":{"kind":"Pod","namespace":"app-bff-staging","name":"app-bff-2-2331681453-cfwgn","uid":"65e3132c-2cea-11e8-889c-0a91ad1abda6","resourceVersion":"42266325"}},{"address":"REDUCTED","port":"3000","maxFails":0,"failTimeout":0,"target":{"kind":"Pod","namespace":"app-bff-staging","name":"app-bff-1-1759178922-bv2m0","uid":"c5898379-2cf2-11e8-889c-0a91ad1abda6","resourceVersion":"42276416"}}],"sessionAffinityConfig":{"name":"","cookieSessionAffinity":{"name":"","hash":""}}},{"name":"upstream-default-backend","service":{"metadata":{"name":"nginx-default-backend","namespace":"nginx","selfLink":"/api/v1/namespaces/nginx/services/nginx-default-backend","uid":"462f4e80-81ab-11e7-9380-0a36945459aa","resourceVersion":"42292594","creationTimestamp":"2017-08-15T11:17:07Z","labels":{"k8s-addon":"ingress-nginx.addons.k8s.io"},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"k8s-addon\":\"ingress-nginx.addons.k8s.io\"},\"name\":\"nginx-default-backend\",\"namespace\":\"nginx\"},\"spec\":{\"ports\":[{\"port\":80,\"targetPort\":\"http\"}],\"selector\":{\"app\":\"nginx-default-backend\"}}}\n"}},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":"http"}],"selector":{"app":"nginx-default-backend"},"clusterIP":"REDUCTED","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":0,"secure":false,"secureCACert":{"secret":"","caFilename":"","pemSha":""},"sslPassthrough":false,"endpoints":[{"address":"REDUCTED","port":"8080","maxFails":0,"failTimeout":0,"target":{"kind":"Pod","namespace":"nginx","name":"nginx-default-backend-3184178138-hj7kt","uid":"b593b707-c2e3-11e7-95a9-06760f4b832a","resourceVersion":"21728925"}}],"sessionAffinityConfig":{"name":"","cookieSessionAffinity":{"name":"","hash":""}}}, REDUCTED_A_LOT_OF_OTHER_SERVICES]
- [] - - [21/Mar/2018:20:06:45 +0000] "POST /configuration/backends HTTP/1.1" 201 5 "-" "Go-http-client/1.1" 12906 0.000 [-] - - - -
I0321 20:06:45.700599 7 controller.go:202] dynamic reconfiguration succeeded
I0321 20:06:46.606372 7 controller.go:183] backend reload required
I0321 20:06:46.606441 7 util.go:64] system fs.file-max=2097152
I0321 20:06:46.606449 7 nginx.go:560] maximum number of open file descriptors : 523264
I0321 20:06:46.811802 7 nginx.go:658] NGINX configuration diff
I0321 20:06:46.811823 7 nginx.go:659] --- /etc/nginx/nginx.conf 2018-03-21 20:06:44.633528996 +0000
...
I0321 20:06:46.958516 7 controller.go:192] ingress backend successfully reloaded...
Related issue: #2231