Kong1.3 with pgsql in k8s generate a large number of dropping event logs #5578

w313062006 · 2020-02-18T11:49:37Z

Summary

I deploy kong1.3(also test kong 1.4) in k8s(1.15.5) with a postgres,
I See a large number of dropping event logs periodically， hundreds of thousands...
someone know what happen? I find a issue #3417, but in this issue, dropping events is generated when kong is start up

Additional Details & Logs

Kong version (1.3.0)

2020/02/18 11:40:17 [verbose] Kong: 1.3.0
2020/02/18 11:40:17 [debug] ngx_lua: 10015
2020/02/18 11:40:17 [debug] nginx: 1015008
2020/02/18 11:40:17 [debug] Lua: LuaJIT 2.1.0-beta3
2020/02/18 11:40:17 [verbose] reading config file at /etc/kong/kong.conf
2020/02/18 11:40:17 [debug] reading environment variables
2020/02/18 11:40:17 [debug] KONG_ADMIN_LISTEN ENV found with "off"
2020/02/18 11:40:17 [debug] KONG_PG_HOST ENV found with "10.90.8.49"
2020/02/18 11:40:17 [debug] KONG_PROXY_LISTEN ENV found with "0.0.0.0:8000,0.0.0.0:8443 ssl"
2020/02/18 11:40:17 [debug] KONG_LUA_PACKAGE_PATH ENV found with "/usr/local/custom/?.lua;;"
2020/02/18 11:40:17 [debug] KONG_ADMIN_ERROR_LOG ENV found with "/dev/stderr"
2020/02/18 11:40:17 [debug] KONG_PG_PASSWORD ENV found with "******"
2020/02/18 11:40:17 [debug] KONG_LOG_LEVEL ENV found with "notice"
2020/02/18 11:40:17 [debug] KONG_PROXY_ERROR_LOG ENV found with "/dev/stderr"
2020/02/18 11:40:17 [debug] KONG_PG_PORT ENV found with "5500"
2020/02/18 11:40:17 [debug] KONG_ADMIN_ACCESS_LOG ENV found with "/dev/stdout"
2020/02/18 11:40:17 [debug] admin_access_log = "/dev/stdout"
2020/02/18 11:40:17 [debug] admin_error_log = "/dev/stderr"
2020/02/18 11:40:17 [debug] admin_listen = {"off"}
2020/02/18 11:40:17 [debug] anonymous_reports = true
2020/02/18 11:40:17 [debug] cassandra_consistency = "ONE"
2020/02/18 11:40:17 [debug] cassandra_contact_points = {"127.0.0.1"}
2020/02/18 11:40:17 [debug] cassandra_data_centers = {"dc1:2","dc2:3"}
2020/02/18 11:40:17 [debug] cassandra_keyspace = "kong"
2020/02/18 11:40:17 [debug] cassandra_lb_policy = "RequestRoundRobin"
2020/02/18 11:40:17 [debug] cassandra_port = 9042
2020/02/18 11:40:17 [debug] cassandra_repl_factor = 1
2020/02/18 11:40:17 [debug] cassandra_repl_strategy = "SimpleStrategy"
2020/02/18 11:40:17 [debug] cassandra_schema_consensus_timeout = 10000
2020/02/18 11:40:17 [debug] cassandra_ssl = false
2020/02/18 11:40:17 [debug] cassandra_ssl_verify = false
2020/02/18 11:40:17 [debug] cassandra_timeout = 5000
2020/02/18 11:40:17 [debug] cassandra_username = "kong"
2020/02/18 11:40:17 [debug] client_body_buffer_size = "8k"
2020/02/18 11:40:17 [debug] client_max_body_size = "0"
2020/02/18 11:40:17 [debug] client_ssl = false
2020/02/18 11:40:17 [debug] database = "postgres"
2020/02/18 11:40:17 [debug] db_cache_ttl = 0
2020/02/18 11:40:17 [debug] db_cache_warmup_entities = {"services","plugins"}
2020/02/18 11:40:17 [debug] db_resurrect_ttl = 30
2020/02/18 11:40:17 [debug] db_update_frequency = 5
2020/02/18 11:40:17 [debug] db_update_propagation = 0
2020/02/18 11:40:17 [debug] dns_error_ttl = 1
2020/02/18 11:40:17 [debug] dns_hostsfile = "/etc/hosts"
2020/02/18 11:40:17 [debug] dns_no_sync = false
2020/02/18 11:40:17 [debug] dns_not_found_ttl = 30
2020/02/18 11:40:17 [debug] dns_order = {"LAST","SRV","A","CNAME"}
2020/02/18 11:40:17 [debug] dns_resolver = {}
2020/02/18 11:40:17 [debug] dns_stale_ttl = 4
2020/02/18 11:40:17 [debug] error_default_type = "text/plain"
2020/02/18 11:40:17 [debug] headers = {"server_tokens","latency_tokens"}
2020/02/18 11:40:17 [debug] log_level = "notice"
2020/02/18 11:40:17 [debug] lua_package_cpath = ""
2020/02/18 11:40:17 [debug] lua_package_path = "/usr/local/custom/?.lua;;"
2020/02/18 11:40:17 [debug] lua_socket_pool_size = 30
2020/02/18 11:40:17 [debug] lua_ssl_verify_depth = 1
2020/02/18 11:40:17 [debug] mem_cache_size = "128m"
2020/02/18 11:40:17 [debug] nginx_admin_directives = {}
2020/02/18 11:40:17 [debug] nginx_daemon = "on"
2020/02/18 11:40:17 [debug] nginx_http_directives = {{value="TLSv1.1 TLSv1.2 TLSv1.3",name="ssl_protocols"},{value="/config/danlu-kong.conf",name="include"}}
2020/02/18 11:40:17 [debug] nginx_http_ssl_protocols = "TLSv1.1 TLSv1.2 TLSv1.3"
2020/02/18 11:40:17 [debug] nginx_http_upstream_directives = {{value="60s",name="keepalive_timeout"},{value="100",name="keepalive_requests"},{value="60",name="keepalive"}}
2020/02/18 11:40:17 [debug] nginx_http_upstream_keepalive = "60"
2020/02/18 11:40:17 [debug] nginx_http_upstream_keepalive_requests = "100"
2020/02/18 11:40:17 [debug] nginx_http_upstream_keepalive_timeout = "60s"
2020/02/18 11:40:17 [debug] nginx_optimizations = true
2020/02/18 11:40:17 [debug] nginx_proxy_directives = {}
2020/02/18 11:40:17 [debug] nginx_sproxy_directives = {}
2020/02/18 11:40:17 [debug] nginx_stream_directives = {}
2020/02/18 11:40:17 [debug] nginx_user = "nobody nobody"
2020/02/18 11:40:17 [debug] nginx_worker_processes = "auto"
2020/02/18 11:40:17 [debug] origins = {}
2020/02/18 11:40:17 [debug] pg_database = "kong"
2020/02/18 11:40:17 [debug] pg_host = "10.90.8.49"
2020/02/18 11:40:17 [debug] pg_max_concurrent_queries = 0
2020/02/18 11:40:17 [debug] pg_password = "******"
2020/02/18 11:40:17 [debug] pg_port = 5500
2020/02/18 11:40:17 [debug] pg_semaphore_timeout = 60000
2020/02/18 11:40:17 [debug] pg_ssl = false
2020/02/18 11:40:17 [debug] pg_ssl_verify = false
2020/02/18 11:40:17 [debug] pg_timeout = 5000
2020/02/18 11:40:17 [debug] pg_user = "kong"
2020/02/18 11:40:17 [debug] prefix = "/usr/local/kong/"
2020/02/18 11:40:17 [debug] proxy_access_log = "logs/access.log"
2020/02/18 11:40:17 [debug] proxy_error_log = "/dev/stderr"
2020/02/18 11:40:17 [debug] proxy_listen = {"0.0.0.0:8000","0.0.0.0:8443 ssl"}
2020/02/18 11:40:17 [debug] real_ip_header = "X-Real-IP"
2020/02/18 11:40:17 [debug] real_ip_recursive = "off"
2020/02/18 11:40:17 [debug] router_consistency = "strict"
2020/02/18 11:40:17 [debug] stream_listen = {"off"}
2020/02/18 11:40:17 [debug] trusted_ips = {}
2020/02/18 11:40:17 [debug] upstream_keepalive = 60
2020/02/18 11:40:17 [verbose] prefix in use: /usr/local/kong

Kong error logs

the period is some minutes

Kong configuration (the output of a GET request to Kong's Admin port - see

{
    "plugins":{
        "enabled_in_cluster":[
            "prometheus"
        ],
        "available_on_server":{
            "correlation-id":true,
            "pre-function":true,
            "cors":true,
            "ldap-auth":true,
            "loggly":true,
            "hmac-auth":true,
            "zipkin":true,
            "request-size-limiting":true,
            "azure-functions":true,
            "request-transformer":true,
            "oauth2":true,
            "response-transformer":true,
            "ip-restriction":true,
            "statsd":true,
            "jwt":true,
            "proxy-cache":true,
            "basic-auth":true,
            "key-auth":true,
            "http-log":true,
            "danlu-istio-header":true,
            "datadog":true,
            "tcp-log":true,
            "rate-limiting":true,
            "post-function":true,
            "prometheus":true,
            "acl":true,
            "kubernetes-sidecar-injector":true,
            "syslog":true,
            "file-log":true,
            "udp-log":true,
            "response-ratelimiting":true,
            "aws-lambda":true,
            "session":true,
            "bot-detection":true,
            "request-termination":true
        }
    },
    "tagline":"Welcome to kong",
    "configuration":{
        "plugins":[
            "bundled",
            "danlu-istio-header"
        ],
        "client_ssl":false,
        "proxy_access_log":"logs/access.log",
        "trusted_ips":{

        },
        "prefix":"/usr/local/kong",
        "loaded_plugins":{
            "correlation-id":true,
            "pre-function":true,
            "cors":true,
            "ldap-auth":true,
            "loggly":true,
            "hmac-auth":true,
            "zipkin":true,
            "request-size-limiting":true,
            "azure-functions":true,
            "request-transformer":true,
            "oauth2":true,
            "response-transformer":true,
            "syslog":true,
            "statsd":true,
            "jwt":true,
            "proxy-cache":true,
            "basic-auth":true,
            "key-auth":true,
            "http-log":true,
            "rate-limiting":true,
            "datadog":true,
            "tcp-log":true,
            "bot-detection":true,
            "post-function":true,
            "session":true,
            "acl":true,
            "kubernetes-sidecar-injector":true,
            "ip-restriction":true,
            "file-log":true,
            "udp-log":true,
            "response-ratelimiting":true,
            "aws-lambda":true,
            "danlu-istio-header":true,
            "prometheus":true,
            "request-termination":true
        },
        "cassandra_username":"kong",
        "admin_ssl_cert_key":"/api-server-cert/tls.key",
        "dns_resolver":{

        },
        "pg_user":"kong",
        "mem_cache_size":"128m",
        "cassandra_data_centers":[
            "dc1:2",
            "dc2:3"
        ],
        "nginx_admin_directives":{

        },
        "nginx_http_upstream_directives":[
            {
                "value":"60s",
                "name":"keepalive_timeout"
            },
            {
                "value":"100",
                "name":"keepalive_requests"
            },
            {
                "value":"60",
                "name":"keepalive"
            }
        ],
        "nginx_http_directives":[
            {
                "value":"TLSv1.1 TLSv1.2 TLSv1.3",
                "name":"ssl_protocols"
            },
            {
                "value":"prometheus_metrics 5m",
                "name":"lua_shared_dict"
            }
        ],
        "pg_host":"10.90.8.49",
        "nginx_acc_logs":"/usr/local/kong/logs/access.log",
        "pg_semaphore_timeout":60000,
        "proxy_listen":[
            "off"
        ],
        "client_ssl_cert_default":"/usr/local/kong/ssl/kong-default.crt",
        "cassandra_ssl":false,
        "db_update_frequency":5,
        "db_update_propagation":0,
        "stream_listen":[
            "off"
        ],
        "nginx_err_logs":"/usr/local/kong/logs/error.log",
        "cassandra_port":9042,
        "dns_order":[
            "LAST",
            "SRV",
            "A",
            "CNAME"
        ],
        "dns_error_ttl":1,
        "headers":[
            "server_tokens",
            "latency_tokens"
        ],
        "cassandra_lb_policy":"RequestRoundRobin",
        "nginx_optimizations":true,
        "nginx_http_upstream_keepalive_timeout":"60s",
        "pg_timeout":5000,
        "nginx_http_upstream_keepalive_requests":"100",
        "database":"postgres",
        "pg_database":"kong",
        "nginx_worker_processes":"auto",
        "router_consistency":"strict",
        "lua_package_cpath":"",
        "admin_ssl_cert":"/api-server-cert/tls.crt",
        "admin_acc_logs":"/usr/local/kong/logs/admin_access.log",
        "ssl_cert_key_default":"/usr/local/kong/ssl/kong-default.key",
        "nginx_conf":"/usr/local/kong/nginx.conf",
        "lua_package_path":"/usr/local/custom/?.lua;;",
        "nginx_pid":"/usr/local/kong/pids/nginx.pid",
        "upstream_keepalive":60,
        "pg_password":"******",
        "dns_stale_ttl":4,
        "nginx_kong_stream_conf":"/usr/local/kong/nginx-kong-stream.conf",
        "pg_ssl_verify":false,
        "admin_access_log":"/dev/stdout",
        "proxy_error_log":"/dev/stderr",
        "stream_listeners":{

        },
        "proxy_listeners":{

        },
        "proxy_ssl_enabled":false,
        "nginx_http_upstream_keepalive":"60",
        "db_cache_warmup_entities":[
            "services",
            "plugins"
        ],
        "enabled_headers":{
            "latency_tokens":true,
            "X-Kong-Proxy-Latency":true,
            "Via":true,
            "server_tokens":true,
            "Server":true,
            "X-Kong-Upstream-Latency":true,
            "X-Kong-Upstream-Status":false
        },
        "nginx_http_ssl_protocols":"TLSv1.1 TLSv1.2 TLSv1.3",
        "origins":{

        },
        "db_resurrect_ttl":30,
        "error_default_type":"text/plain",
        "cassandra_consistency":"ONE",
        "client_max_body_size":"0",
        "admin_error_log":"/dev/stderr",
        "admin_ssl_cert_default":"/usr/local/kong/ssl/admin-kong-default.crt",
        "dns_not_found_ttl":30,
        "pg_ssl":false,
        "nginx_stream_directives":{

        },
        "kong_env":"/usr/local/kong/.kong_env",
        "cassandra_repl_strategy":"SimpleStrategy",
        "ssl_cipher_suite":"modern",
        "cassandra_schema_consensus_timeout":10000,
        "log_level":"notice",
        "pg_max_concurrent_queries":0,
        "nginx_kong_conf":"/usr/local/kong/nginx-kong.conf",
        "real_ip_header":"X-Real-IP",
        "dns_hostsfile":"/etc/hosts",
        "admin_listeners":[
            {
                "listener":"0.0.0.0:8001",
                "proxy_protocol":false,
                "reuseport":false,
                "transparent":false,
                "ssl":false,
                "ip":"0.0.0.0",
                "deferred":false,
                "http2":false,
                "port":8001,
                "bind":false
            },
            {
                "listener":"0.0.0.0:8444 ssl",
                "proxy_protocol":false,
                "reuseport":false,
                "transparent":false,
                "ssl":true,
                "ip":"0.0.0.0",
                "deferred":false,
                "http2":false,
                "port":8444,
                "bind":false
            }
        ],
        "dns_no_sync":false,
        "cassandra_timeout":5000,
        "admin_ssl_cert_key_default":"/usr/local/kong/ssl/admin-kong-default.key",
        "cassandra_ssl_verify":false,
        "lua_ssl_verify_depth":1,
        "cassandra_contact_points":[
            "127.0.0.1"
        ],
        "real_ip_recursive":"off",
        "cassandra_repl_factor":1,
        "client_ssl_cert_key_default":"/usr/local/kong/ssl/kong-default.key",
        "nginx_daemon":"off",
        "anonymous_reports":true,
        "nginx_proxy_directives":{

        },
        "nginx_sproxy_directives":{

        },
        "pg_port":5500,
        "admin_ssl_enabled":true,
        "client_body_buffer_size":"8k",
        "ssl_preread_enabled":true,
        "ssl_cert_csr_default":"/usr/local/kong/ssl/kong-default.csr",
        "admin_listen":[
            "0.0.0.0:8001",
            "0.0.0.0:8444 ssl"
        ],
        "cassandra_keyspace":"kong",
        "ssl_cert_default":"/usr/local/kong/ssl/kong-default.crt",
        "lua_socket_pool_size":30,
        "db_cache_ttl":0
    },
    "version":"1.3.0",
    "node_id":"49aee94c-a69a-44e6-8550-4cb219f39344",
    "lua_version":"LuaJIT 2.1.0-beta3",
    "prng_seeds":{
        "pid: 59":206991824076,
        "pid: 79":313837194247,
        "pid: 53":197140212302,
        "pid: 41":482447122229,
        "pid: 61":241662543511,
        "pid: 73":117109212167,
        "pid: 54":240212271173,
        "pid: 80":121146194139,
        "pid: 66":161151124156,
        "pid: 74":175819819083,
        "pid: 51":206192052917,
        "pid: 43":426920723055,
        "pid: 63":361614516557,
        "pid: 71":220601092717,
        "pid: 56":233153165225,
        "pid: 82":241112153110,
        "pid: 64":131719823123,
        "pid: 76":149102108214,
        "pid: 49":238901942161,
        "pid: 69":582032011872,
        "pid: 55":253101971753,
        "pid: 47":245252246170,
        "pid: 67":232228542481,
        "pid: 75":207371399124,
        "pid: 52":130146107150,
        "pid: 40":432175418610,
        "pid: 60":396763207173,
        "pid: 1":241314419712,
        "pid: 72":561472371141,
        "pid: 44":561802542065,
        "pid: 58":251183719872,
        "pid: 78":371283679205,
        "pid: 85":741912511143,
        "pid: 81":742131971941,
        "pid: 86":217187721108,
        "pid: 84":552286396100,
        "pid: 46":826934732521,
        "pid: 48":173137174131,
        "pid: 39":249211951781,
        "pid: 45":197937218112,
        "pid: 68":223391201233,
        "pid: 57":124174151782,
        "pid: 83":174168241217,
        "pid: 65":122737422020,
        "pid: 77":821661031317,
        "pid: 50":109774513934,
        "pid: 42":187202165208,
        "pid: 62":281271231292,
        "pid: 70":221111248190
    },
    "timers":{
        "pending":122,
        "running":0
    },
    "hostname":"kong-control-plane-5dd6796fbb-4tzvg"
}

Operating system
kong in k8s

The text was updated successfully, but these errors were encountered:

Tieske · 2020-02-18T16:42:13Z

looks like eventdata gets evicted before being used from the SHM. So either shm too small, or number of events too high.

Do you use a custom nginx configuration template? did you modify the SHM sizes? If the shm size is too small, events may get evicted prematurely.

How many worker-processes do you have? In the logs above I see

2020/02/18 11:40:17 [debug] nginx_worker_processes = "auto"

That seems to be a bad idea, iirc in a k8s environment you need to set the number of CPU's assigned to the docker container. By default the value will be set to the number of CPU core's in the machine, as in hardware, not the number assigned to the docker container. So if you get way too many workers, you may also get way more events, that would fill up the shm, and cause some to be evicted.

p0pr0ck5 · 2020-02-18T16:44:57Z

Possibly relevant: thibaultcha/lua-resty-mlcache#88

w313062006 · 2020-02-19T01:12:38Z

@Tieske thank you apply
yes, I set nginx_worker_processes = auto, and the worker processes num is 48 because the machina core is 48. I will modify worker processes and shm for test again.

Tieske · 2020-02-19T08:10:02Z

@w313062006 thx for trying. Please provide feedback on how that goes, because we might need to bump the default shm size if it fails on a 48core machine.

@p0pr0ck5 Not sure that is relevant, iirc that issue is an issue if a worker process crashes down the line, and it still tries to replay events from hours, or even days ago. This issue is during start up. Just to verify @w313062006 ; it only happens during start up right?

w313062006 · 2020-02-19T09:06:01Z

@Tieske
I have started the test, after one day of testing, I will feedback results, at present, there are no errors.
these errors occur periodically along with kong lifecycle, not only start up

Tieske · 2020-02-19T12:04:04Z

@w313062006 if they happen again, can you post the full logs? (as an attachment)

w313062006 · 2020-02-20T10:07:16Z

@Tieske hi, I come back to report my test result. first, your suggestion is correct, thank you!
there are several cases I have test:
1, change workprocess to 16 & mount shm to kong pod (https://docs.okd.io/latest/dev_guide/shared_memory.html, after mount, size of pod /dev/shm change to 126G) -> result: no error
2, change workprocess to 16 -> result: no error
3, change workprocess to auto(48) & mount shm to kong pod -> result: error
4, change workprocess to auto(48) & less kong configuration(service, routes, upstream, targets) -> no error
5, change workprocess to auto(48) & change nginx_kong.lua some config larger(like kong_process_events,kong_cluster_events,kong_healthchecks...) -> error (but both frequency and quantity are reduced)

from these test, I hava some questions and view
1, how can I increase shm in k8s? mount to /dev/shm maybe not working.
2,are these events synchronize kong's configuration between nginx worker processes? maybe larger kong configuration with larger workprocess can cause error occur.
3,at present, these errors are positively related to workprocess number

Tieske · 2020-02-20T15:48:48Z

@w313062006 thx for testing it. It confirms the hypothesis.

Regarding your tests:

less workerprocesses does it, this shm setting has no effect
same as 1. since that shm had no effect
shm has no effect, so the same errors as before, as expected
reducing the number of configuration elements reduces the pressure on the shm, and hence works in reducing the errors (even no error at all)
the kong_process_events is the SHM setting that matters, increasing that size reduces the errors.

Your questions:

I don't know actually, but @hbagdi probably knows
yes, see answer 3.
each worker will start and add some events, hence the later workers will have more to deal with, and will be the first to notice missing events that got evicted.

So in your case:

make sure the number of worker processes configured for Kong matches the number of CPU cores assigned to the container you are running (there is no sense in running more worker processes, it will only reduce performance).
increase the SHM named kong_process_events to prevent any errors from happening.

Tieske · 2020-02-20T18:14:58Z

@w313062006 would you mind telling me how many entities you have configured in the system? (routes/services/consumers/plugins/upstreams ?)

w313062006 · 2020-02-21T01:21:21Z

@w313062006 would you mind telling me how many entities you have configured in the system? (routes/services/consumers/plugins/upstreams ?)

@Tieske
services, routes, upstreams -> every 100+
consumers -> no one
plugins -> 10+ (define in services)

w313062006 · 2020-02-21T01:37:58Z

@Tieske
add a new test case:
change workprocess to 16 & not set resource request and limit in k8s & kong is on a heavily loaded machine -> result: error (but at present did not cause the request 500 error code, oh, I submit this issue because while kong logs so much dropping events, the requests may return 500 error code)

so, is machine load related to errors occur?

w313062006 · 2020-02-21T01:59:49Z

@Tieske

increase the SHM named kong_process_events to prevent any errors from happening.

I watch one kong instance :8001/status，when error happen, kong_process_events not full

Tieske · 2020-02-21T07:50:37Z

getting these errors is problematic, it leaves the system in an undetermined state, so should be avoided in any case. At some point you'll start seeing other anomalies originating from this.

When looking at :8001/status it might not be full, but fragmented. What level of utilisation did you see , eg in %?

w313062006 · 2020-02-21T10:07:17Z

@Tieske
in one cese, I set kong_process_events 128m, when allocated reach 70+m or 80+m (sorry, i forget accurate num), the error occur, but frequency and quantity in a low level

Tieske · 2020-02-21T15:25:16Z

are you sure they are the same errors? as in your original post?

can you post the logs somehwere?

w313062006 · 2020-02-22T13:48:53Z

@Tieske

Tieske · 2020-02-22T20:18:58Z

That's weird, difference seems to big for fragmentation

Tieske · 2020-02-22T20:23:57Z

the worker-events lib had a fix for some memory issues in version 1.0.0, that version was already in Kong 1.3.0. So that cannot be the issue.

Can you send the full logs? even if they are big, just zip'em and upload them somewhere

w313062006 · 2020-02-25T02:05:59Z

@Tieske
sorry, because I deploy kong in k8s, the log of pod is removed when I finish test...
What else can I do to help you？

Tieske · 2020-04-13T08:41:14Z

Let's close this for now. We can always reopen when the problem returns.

w313062006 · 2021-04-23T02:46:24Z

@Tieske
reopen this issue, because I think this dropping event error relate to Kong/kubernetes-ingress-controller#1212,
because of kic update routes&upstream always, the event cause by the resource change will burst

when I hard code like Kong/kubernetes-ingress-controller#1212, the dropping event error is miss and the kong_process_event is not full (before the hard code, when kong start up, kong_process_event will full quickly )

if the number of upstreams & routes is large, the resource change event is large, the dropping event error more likely happen

hbagdi · 2021-04-23T04:03:33Z

@w313062006 To be clear, do you see the problem happening even after filling in the missing defaults in the controller?

w313062006 · 2021-04-23T04:29:59Z

@w313062006 To be clear, do you see the problem happening even after filling in the missing defaults in the controller?

@hbagdi yes, before hard code, dropping event will happen, but not cause request error, so I don't handle them. but when hard code done, the dropping event error not happen, so I think this two issue is related

hbagdi · 2021-04-23T04:33:02Z

@hbagdi yes, before hard code, dropping event will happen, but not cause request error, so I don't handle them. but when hard code done, the dropping event error not happen, so I think this two issue is related

Could be. You are using Kong 1.5 which is relatively older. If you observe the problem with a relatively recent version of Kong, please provide reproducible steps and open a new issue (link this one).

w313062006 · 2021-04-23T04:36:47Z

@hbagdi yes, before hard code, dropping event will happen, but not cause request error, so I don't handle them. but when hard code done, the dropping event error not happen, so I think this two issue is related

Could be. You are using Kong 1.5 which is relatively older. If you observe the problem with a relatively recent version of Kong, please provide reproducible steps and open a new issue (link this one).

ok, I will, thx reply

Tieske closed this as completed Apr 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kong1.3 with pgsql in k8s generate a large number of dropping event logs #5578

Kong1.3 with pgsql in k8s generate a large number of dropping event logs #5578

w313062006 commented Feb 18, 2020 •

edited by hishamhm

Loading

Tieske commented Feb 18, 2020

p0pr0ck5 commented Feb 18, 2020

w313062006 commented Feb 19, 2020

Tieske commented Feb 19, 2020

w313062006 commented Feb 19, 2020

Tieske commented Feb 19, 2020

w313062006 commented Feb 20, 2020 •

edited

Loading

Tieske commented Feb 20, 2020

Tieske commented Feb 20, 2020

w313062006 commented Feb 21, 2020 •

edited

Loading

w313062006 commented Feb 21, 2020

w313062006 commented Feb 21, 2020

Tieske commented Feb 21, 2020

w313062006 commented Feb 21, 2020 •

edited

Loading

Tieske commented Feb 21, 2020

w313062006 commented Feb 22, 2020

Tieske commented Feb 22, 2020

Tieske commented Feb 22, 2020

w313062006 commented Feb 25, 2020

Tieske commented Apr 13, 2020

w313062006 commented Apr 23, 2021

hbagdi commented Apr 23, 2021

w313062006 commented Apr 23, 2021 •

edited

Loading

hbagdi commented Apr 23, 2021

w313062006 commented Apr 23, 2021

Kong1.3 with pgsql in k8s generate a large number of dropping event logs #5578

Kong1.3 with pgsql in k8s generate a large number of dropping event logs #5578

Comments

w313062006 commented Feb 18, 2020 • edited by hishamhm Loading

Summary

Additional Details & Logs

Tieske commented Feb 18, 2020

p0pr0ck5 commented Feb 18, 2020

w313062006 commented Feb 19, 2020

Tieske commented Feb 19, 2020

w313062006 commented Feb 19, 2020

Tieske commented Feb 19, 2020

w313062006 commented Feb 20, 2020 • edited Loading

Tieske commented Feb 20, 2020

Tieske commented Feb 20, 2020

w313062006 commented Feb 21, 2020 • edited Loading

w313062006 commented Feb 21, 2020

w313062006 commented Feb 21, 2020

Tieske commented Feb 21, 2020

w313062006 commented Feb 21, 2020 • edited Loading

Tieske commented Feb 21, 2020

w313062006 commented Feb 22, 2020

Tieske commented Feb 22, 2020

Tieske commented Feb 22, 2020

w313062006 commented Feb 25, 2020

Tieske commented Apr 13, 2020

w313062006 commented Apr 23, 2021

hbagdi commented Apr 23, 2021

w313062006 commented Apr 23, 2021 • edited Loading

hbagdi commented Apr 23, 2021

w313062006 commented Apr 23, 2021

w313062006 commented Feb 18, 2020 •

edited by hishamhm

Loading

w313062006 commented Feb 20, 2020 •

edited

Loading

w313062006 commented Feb 21, 2020 •

edited

Loading

w313062006 commented Feb 21, 2020 •

edited

Loading

w313062006 commented Apr 23, 2021 •

edited

Loading