Skip to content

NGINX in debug mode fails #3503

New issue
New issue
Open
Open
NGINX in debug mode fails#3503
Labels
bugSomething isn't working
@bjee19

Description

@bjee19
bjee19
opened on Jun 12, 2025

Describe the bug
When I deploy NGF, then modify the NginxProxy to include debug: true in the containerSpec, then deploy the cafe-example, the nginx conf is not updated and error messages saying "timed out waiting for agent to register nginx" are repeatedly outputted in the NGF logs.

To Reproduce
Steps to reproduce the behavior:

  1. Make image off of edge
  2. Deploy NGF
  3. modify NginxProxy to include debug:true in containerSpec
  4. deploy cafe-example
  5. Check logs for errors, check nginx conf of gateway to see its not updated.

Expected behavior
NGINX should be able to be deployed in Debug mode.

Your environment

  • Version of the NGINX Gateway Fabric - c0fe552
  • Version of Kubernetes:
  • Kubernetes platform (e.g. Mini-kube or GCP): kind
  • Details on how you expose the NGINX Gateway Fabric Pod (e.g. Service of type LoadBalancer or port-forward):
  • Logs of NGINX container: kubectl -n <nginx-deployment-namespace> logs deployments/<nginx-deployment>
  • NGINX Configuration: kubectl -n <nginx-deployment-namespace> exec -it deployments/<nginx-deployment> -- nginx -T

NGINX logs:

Defaulted container "nginx" out of: nginx, init (init)
+ trap handle_term TERM
+ trap handle_quit QUIT
+ rm -rf '/var/run/nginx/*.sock'
starting nginx ...
+ echo 'starting nginx ...'
+ '[' debug = debug ']'
+ + nginx_pid=12
/usr/sbin/nginx-debug -g 'daemon off;'
+ SECONDS=0
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
2025/06/12 00:26:18 [notice] 12#12: js vm init njs: 0000FFFF874B2A80
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
2025/06/12 00:26:18 [notice] 12#12: using the "epoll" event method
+ ps -ef
2025/06/12 00:26:18 [notice] 12#12: nginx/1.28.0
2025/06/12 00:26:18 [notice] 12#12: built by gcc 14.2.0 (Alpine 14.2.0)
2025/06/12 00:26:18 [notice] 12#12: OS: Linux 6.10.14-linuxkit
2025/06/12 00:26:18 [notice] 12#12: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2025/06/12 00:26:18 [notice] 12#12: start worker processes
2025/06/12 00:26:18 [notice] 12#12: start worker process 46
+ grep 'nginx: master process'
2025/06/12 00:26:18 [notice] 12#12: start worker process 47
+ grep -v grep
2025/06/12 00:26:18 [notice] 12#12: start worker process 48
2025/06/12 00:26:18 [notice] 12#12: start worker process 49
2025/06/12 00:26:18 [notice] 12#12: start worker process 50
2025/06/12 00:26:18 [notice] 12#12: start worker process 51
   12 nginx     0:00 {nginx-debug} nginx: master process /usr/sbin/nginx-debug -g daemon off;
   46 nginx     0:00 {nginx-debug} nginx: master process /usr/sbin/nginx-debug -g daemon off;
   47 nginx     0:00 {nginx-debug} nginx: master process /usr/sbin/nginx-debug -g daemon off;
+ echo 'starting nginx-agent ...'
2025/06/12 00:26:18 [notice] 12#12: start worker process 52
starting nginx-agent ...
+ agent_pid=53
+ '[' 0 '!=' 0 ']'
+ nginx-agent
+ wait_term
+ wait 53
2025/06/12 00:26:18 [notice] 12#12: start worker process 54
2025/06/12 00:26:18 [notice] 12#12: start worker process 55
2025/06/12 00:26:18 [notice] 12#12: start worker process 56
2025/06/12 00:26:18 [notice] 12#12: start worker process 57
2025/06/12 00:26:18 [notice] 12#12: start worker process 58
time=2025-06-12T00:26:18.176Z level=INFO msg="Configured allowed directories" allowed_directories="[/etc/nginx-agent/ /etc/nginx/ /usr/share/nginx/ /var/run/nginx/]"
time=2025-06-12T00:26:18.177Z level=INFO msg="Configured labels" labels=map[]
time=2025-06-12T00:26:18.177Z level=INFO msg="Excluded files from being watched for file changes" exclude_files=[^.*(\.log|.swx|~|.swp)$]
time=2025-06-12T00:26:18.177Z level=INFO msg="Starting NGINX Agent" version=v3.0.1 commit=a45c50b
time=2025-06-12T00:26:18.177Z level=INFO msg="Dialing grpc server" server_addr=my-release-nginx-gateway-fabric.nginx-gateway.svc:443
time=2025-06-12T00:26:18.181Z level=INFO msg="Finished registering plugins" plugins="[resource command file collector watcher]"
time=2025-06-12T00:26:18.183Z level=INFO msg="Starting OTel Collector plugin"
time=2025-06-12T00:26:18.183Z level=INFO msg="Starting OTel collector"
time=2025-06-12T00:26:23.199Z level=INFO msg="Reloading OTel collector config, resource updated" correlation_id=de90d29b-4723-11f0-8ffc-deb652b2156b
time=2025-06-12T00:26:23.199Z level=INFO msg="No Data Plane Instance found" correlation_id=de90d29b-4723-11f0-8ffc-deb652b2156b
time=2025-06-12T00:26:23.201Z level=INFO msg="Closing OTel Collector plugin" correlation_id=de90d29b-4723-11f0-8ffc-deb652b2156b
time=2025-06-12T00:26:23.201Z level=INFO msg="Shutting down OTel Collector" state=Running correlation_id=de90d29b-4723-11f0-8ffc-deb652b2156b
time=2025-06-12T00:26:23.202Z level=INFO msg="OTel collector run finished"
time=2025-06-12T00:26:23.246Z level=INFO msg="Connection created" response=response:{status:COMMAND_STATUS_OK} correlation_id=de90d29b-4723-11f0-8ffc-deb652b2156b
time=2025-06-12T00:26:23.246Z level=INFO msg="Agent connected" correlation_id=de90d29b-4723-11f0-8ffc-deb652b2156b
time=2025-06-12T00:26:23.973Z level=INFO msg="OTel Collector shutdown" state=Closed correlation_id=de90d29b-4723-11f0-8ffc-deb652b2156b
time=2025-06-12T00:26:23.973Z level=INFO msg="Restarting OTel collector" correlation_id=de90d29b-4723-11f0-8ffc-deb652b2156b
time=2025-06-12T00:26:53.251Z level=ERROR msg="Failed to receive message from subscribe stream" error="rpc error: code = Unknown desc = timed out waiting for agent to register nginx" correlation_id=de90d29b-4723-11f0-8ffc-deb652b2156b
time=2025-06-12T00:27:24.109Z level=ERROR msg="Failed to receive message from subscribe stream" error="rpc error: code = Unknown desc = timed out waiting for agent to register nginx" correlation_id=de90d29b-4723-11f0-8ffc-deb652b2156b
time=2025-06-12T00:27:56.673Z level=ERROR msg="Failed to receive message from subscribe stream" error="rpc error: code = Unknown desc = timed out waiting for agent to register nginx" correlation_id=de90d29b-4723-11f0-8ffc-deb652b2156b
time=2025-06-12T00:28:31.534Z level=ERROR msg="Failed to receive message from subscribe stream" error="rpc error: code = Unknown desc = timed out waiting for agent to register nginx" correlation_id=de90d29b-4723-11f0-8ffc-deb652b2156b
time=2025-06-12T00:29:19.751Z level=ERROR msg="Failed to receive message from subscribe stream" error="rpc error: code = Unknown desc = timed out waiting for agent to register nginx" correlation_id=de90d29b-4723-11f0-8ffc-deb652b2156b
time=2025-06-12T00:30:03.472Z level=ERROR msg="Failed to receive message from subscribe stream" error="rpc error: code = Unknown desc = timed out waiting for agent to register nginx" correlation_id=de90d29b-4723-11f0-8ffc-deb652b2156b
time=2025-06-12T00:30:49.157Z level=ERROR msg="Failed to receive message from subscribe stream" error="rpc error: code = Unknown desc = timed out waiting for agent to register nginx" correlation_id=de90d29b-4723-11f0-8ffc-deb652b2156b

NGF logs:

{"level":"info","ts":"2025-06-12T00:25:19Z","msg":"Starting the NGINX Gateway Fabric control plane","version":"edge","commit":"c0fe5525cfa7b10d3d7a601c3d1a83f54b7b13f8","date":"2025-06-11T22:37:52Z","dirty":"false"}
{"level":"info","ts":"2025-06-12T00:25:19Z","msg":"Starting manager"}
{"level":"info","ts":"2025-06-12T00:25:19Z","msg":"starting server","name":"health probe","addr":"[::]:8081"}
{"level":"info","ts":"2025-06-12T00:25:19Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2025-06-12T00:25:19Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":9113","secure":false}
{"level":"info","ts":"2025-06-12T00:25:19Z","msg":"attempting to acquire leader lease nginx-gateway/my-release-nginx-gateway-fabric-leader-election..."}
{"level":"info","ts":"2025-06-12T00:25:19Z","logger":"eventLoop.eventHandler","msg":"Reconfigured control plane.","batchID":6}
{"level":"info","ts":"2025-06-12T00:25:37Z","msg":"successfully acquired lease nginx-gateway/my-release-nginx-gateway-fabric-leader-election"}
{"level":"info","ts":"2025-06-12T00:25:37Z","logger":"telemetryJob","msg":"Starting cronjob"}
{"level":"info","ts":"2025-06-12T00:26:14Z","logger":"eventHandler","msg":"NGINX configuration was successfully updated"}
{"level":"info","ts":"2025-06-12T00:26:14Z","logger":"provisioner","msg":"Creating/Updating nginx resources","namespace":"default","name":"gateway-nginx"}
{"level":"info","ts":"2025-06-12T00:26:14Z","logger":"provisioner","msg":"Creating/Updating nginx resources","namespace":"default","name":"gateway-nginx"}
{"level":"info","ts":"2025-06-12T00:26:14Z","logger":"provisioner","msg":"Creating/Updating nginx resources","namespace":"default","name":"gateway-nginx"}
{"level":"info","ts":"2025-06-12T00:26:14Z","logger":"provisioner","msg":"Creating/Updating nginx resources","namespace":"default","name":"gateway-nginx"}
{"level":"info","ts":"2025-06-12T00:26:15Z","logger":"eventHandler","msg":"NGINX configuration was successfully updated"}
{"level":"info","ts":"2025-06-12T00:26:18Z","logger":"eventHandler","msg":"NGINX configuration was successfully updated"}
{"level":"info","ts":"2025-06-12T00:26:18Z","logger":"eventHandler","msg":"NGINX configuration was successfully updated"}
{"level":"info","ts":"2025-06-12T00:26:19Z","logger":"eventHandler","msg":"NGINX configuration was successfully updated"}
{"level":"info","ts":"2025-06-12T00:26:19Z","logger":"eventHandler","msg":"NGINX configuration was successfully updated"}
{"level":"info","ts":"2025-06-12T00:26:23Z","logger":"nginxUpdater.commandService","msg":"Creating connection for nginx pod: gateway-nginx-6c46ff959c-hr626"}
{"level":"error","ts":"2025-06-12T00:26:53Z","logger":"nginxUpdater.commandService","msg":"error waiting for connection","error":"timed out waiting for agent to register nginx","stacktrace":"github.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent.(*commandService).Subscribe\n\tgithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/command.go:138\ngithub.com/nginx/agent/v3/api/grpc/mpi/v1._CommandService_Subscribe_Handler\n\tgithub.com/nginx/agent/v3@v3.0.1/api/grpc/mpi/v1/command_grpc.pb.go:233\ngithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/grpc/interceptor.(*ContextSetter).Stream.ContextSetter.Stream.func1\n\tgithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/grpc/interceptor/interceptor.go:64\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1702\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1819\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1035"}
{"level":"error","ts":"2025-06-12T00:27:24Z","logger":"nginxUpdater.commandService","msg":"error waiting for connection","error":"timed out waiting for agent to register nginx","stacktrace":"github.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent.(*commandService).Subscribe\n\tgithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/command.go:138\ngithub.com/nginx/agent/v3/api/grpc/mpi/v1._CommandService_Subscribe_Handler\n\tgithub.com/nginx/agent/v3@v3.0.1/api/grpc/mpi/v1/command_grpc.pb.go:233\ngithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/grpc/interceptor.(*ContextSetter).Stream.ContextSetter.Stream.func1\n\tgithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/grpc/interceptor/interceptor.go:64\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1702\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1819\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1035"}
{"level":"error","ts":"2025-06-12T00:27:56Z","logger":"nginxUpdater.commandService","msg":"error waiting for connection","error":"timed out waiting for agent to register nginx","stacktrace":"github.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent.(*commandService).Subscribe\n\tgithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/command.go:138\ngithub.com/nginx/agent/v3/api/grpc/mpi/v1._CommandService_Subscribe_Handler\n\tgithub.com/nginx/agent/v3@v3.0.1/api/grpc/mpi/v1/command_grpc.pb.go:233\ngithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/grpc/interceptor.(*ContextSetter).Stream.ContextSetter.Stream.func1\n\tgithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/grpc/interceptor/interceptor.go:64\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1702\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1819\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1035"}
{"level":"error","ts":"2025-06-12T00:28:31Z","logger":"nginxUpdater.commandService","msg":"error waiting for connection","error":"timed out waiting for agent to register nginx","stacktrace":"github.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent.(*commandService).Subscribe\n\tgithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/command.go:138\ngithub.com/nginx/agent/v3/api/grpc/mpi/v1._CommandService_Subscribe_Handler\n\tgithub.com/nginx/agent/v3@v3.0.1/api/grpc/mpi/v1/command_grpc.pb.go:233\ngithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/grpc/interceptor.(*ContextSetter).Stream.ContextSetter.Stream.func1\n\tgithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/grpc/interceptor/interceptor.go:64\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1702\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1819\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1035"}
{"level":"error","ts":"2025-06-12T00:29:19Z","logger":"nginxUpdater.commandService","msg":"error waiting for connection","error":"timed out waiting for agent to register nginx","stacktrace":"github.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent.(*commandService).Subscribe\n\tgithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/command.go:138\ngithub.com/nginx/agent/v3/api/grpc/mpi/v1._CommandService_Subscribe_Handler\n\tgithub.com/nginx/agent/v3@v3.0.1/api/grpc/mpi/v1/command_grpc.pb.go:233\ngithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/grpc/interceptor.(*ContextSetter).Stream.ContextSetter.Stream.func1\n\tgithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/grpc/interceptor/interceptor.go:64\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1702\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1819\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1035"}
{"level":"error","ts":"2025-06-12T00:30:03Z","logger":"nginxUpdater.commandService","msg":"error waiting for connection","error":"timed out waiting for agent to register nginx","stacktrace":"github.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent.(*commandService).Subscribe\n\tgithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/command.go:138\ngithub.com/nginx/agent/v3/api/grpc/mpi/v1._CommandService_Subscribe_Handler\n\tgithub.com/nginx/agent/v3@v3.0.1/api/grpc/mpi/v1/command_grpc.pb.go:233\ngithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/grpc/interceptor.(*ContextSetter).Stream.ContextSetter.Stream.func1\n\tgithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/grpc/interceptor/interceptor.go:64\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1702\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1819\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1035"}
{"level":"error","ts":"2025-06-12T00:30:49Z","logger":"nginxUpdater.commandService","msg":"error waiting for connection","error":"timed out waiting for agent to register nginx","stacktrace":"github.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent.(*commandService).Subscribe\n\tgithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/command.go:138\ngithub.com/nginx/agent/v3/api/grpc/mpi/v1._CommandService_Subscribe_Handler\n\tgithub.com/nginx/agent/v3@v3.0.1/api/grpc/mpi/v1/command_grpc.pb.go:233\ngithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/grpc/interceptor.(*ContextSetter).Stream.ContextSetter.Stream.func1\n\tgithub.com/nginx/nginx-gateway-fabric/internal/controller/nginx/agent/grpc/interceptor/interceptor.go:64\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1702\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1819\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\tgoogle.golang.org/grpc@v1.72.2/server.go:1035"}

Additional context
When I check the nginx/gateway pod and run ps it correctly shows nginx debug processes running:

Defaulted container "nginx" out of: nginx, init (init)
/ $ ps
PID   USER     TIME  COMMAND
    1 nginx     0:00 {entrypoint.sh} /bin/bash /agent/entrypoint.sh debug
   12 nginx     0:00 {nginx-debug} nginx: master process /usr/sbin/nginx-debug -g daemon off;
   46 nginx     0:00 {nginx-debug} nginx: worker process
   47 nginx     0:00 {nginx-debug} nginx: worker process
   48 nginx     0:00 {nginx-debug} nginx: worker process
   49 nginx     0:00 {nginx-debug} nginx: worker process
   50 nginx     0:00 {nginx-debug} nginx: worker process
   51 nginx     0:00 {nginx-debug} nginx: worker process
   52 nginx     0:00 {nginx-debug} nginx: worker process
   53 nginx     0:00 nginx-agent
   54 nginx     0:00 {nginx-debug} nginx: worker process
   55 nginx     0:00 {nginx-debug} nginx: worker process
   56 nginx     0:00 {nginx-debug} nginx: worker process
   57 nginx     0:00 {nginx-debug} nginx: worker process
   58 nginx     0:00 {nginx-debug} nginx: worker process
   80 nginx     0:00 sh
   87 nginx     0:00 ps

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    NGINX Gateway Fabric

    Status

    🆕 New

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions