bug: Routing configuration not taking effect, and the backend reports an etcd error. #2071

wumengchao · 2023-11-25T02:37:29Z

Current Behavior

Occasionally, changes to the routing configuration do not take effect, and the deleted routing configurations are still active. At the same time, the backend reports the following errors, and I am not sure if these errors are related to this bug.

2023-11-25T04:31:57+08:00 �[31merror�[0m adapter/etcd.go:141 failed to create object, ignore it{error 26 0 rpc error: code = InvalidArgument desc = etcdserver: duplicate key given in txn request} {revision 11 26 } {key 15 0 /apisix/routes/6d611fe0 }

Expected Behavior

No response

Error Logs

2023-11-25T04:31:57+08:00 �[31merror�[0m adapter/etcd.go:141 failed to create object, ignore it{error 26 0 rpc error: code = InvalidArgument desc = etcdserver: duplicate key given in txn request} {revision 11 26 } {key 15 0 /apisix/routes/6d611fe0 }

Steps to Reproduce

Deployed using the latest version 1.7.1, through the Composite Architecture method.

Environment

APISIX Ingress controller version (apisix-ingress-controller 1.7.1)
Kubernetes cluster version (1.20.6) tke
OS version if running APISIX Ingress controller in a bare-metal environment (Linux VM-1-200-tlinux 4.14.105-19-0020.1 misc: some basic goals #1 SMP Fri Apr 23 10:22:59 CST 2021 x86_64 x86_64 x86_64 GNU/Linux) tos

Revolyssup · 2023-11-26T21:01:14Z

@wumengchao What are the logs from Ingress when you apply the configuration which doesn't take effect?

wumengchao · 2023-11-28T02:00:31Z

@wumengchao当您应用未生效的配置时，Ingress 的日志是什么？

2023-11-28T09:55:18+08:00 �[31merror�[0m adapter/etcd.go:141 failed to create object, ignore it{error 26 0 rpc error: code = InvalidArgument desc = etcdserver: duplicate key given in txn request} {revision 11 354 } {key 15 0 /apisix/routes/35232d71 }

I have launched 15 pods.

wumengchao · 2023-11-28T05:46:11Z

From the logs, it appears that an error is reported every 5 minutes.
Could it be that a synchronization service error is causing concurrent writes to etcd?

wumengchao · 2023-11-28T18:07:48Z

Today, it was also discovered that some pods remain in the "create object" state during the startup process, and all access attempts to enter these pods result in a 404 error with the message {"error_msg":"404 Route Not Found"}.
When the routing configuration of a pod becomes ineffective, it can only be restored by deleting the routing configuration and then re-adding it.

The last log entry is: adapter/etcd.go:147 created object{revision 11 126 } {key 15 0 /apisix/routes/ac8106c4 }

s there a probe that can detect whether the route object has been successfully created?

Revolyssup · 2023-11-29T08:23:56Z

Looking into this...

Revolyssup · 2023-11-29T08:56:56Z

@wumengchao Composite mode simulates the etcd, so the log is coming from etcd-adapter that is used by ingress controller which simulates etcd. The data is stored in in-mem b-tree.

When you say, "i have launched 15 pods", which pods are you talking about?
The log created object{revision 11 126 } {key 15 0 /apisix/routes/ac8106c4 } means that this route was created.
You can use etcdctl to query the ingress controller pod at port 12379 to check the route configuration.

So let's do this. After you create the route, can you query the apisix-ingress-controller pod by querying with etcdctl on port 12379 to confirm that the route was created. Or establish a watch to confirm that the route was created. If etcdctl reports positive then that means route was actually created and apisix could not watch properly.

Revolyssup · 2023-11-29T09:04:48Z

something like this

wumengchao · 2023-11-29T14:03:23Z

15 pods refer to ingress-apisix-composite.
When the pod routing configuration became invalid, I connected to the problematic pod's 12379 port using etcdkeeper and found that the data was normal.At this point, the only way to recover was to delete the route and then rebuild it. I tried deleting the pod and creating a new one, but it was ineffective. I changed the deployment to a DaemonSet.

wumengchao · 2023-11-30T06:58:33Z

I intentionally configured an incorrect path. After encountering "failed to find route in cache," the following error immediately appears: "failed to create object, ignore it{error 26 0 rpc error: code = InvalidArgument desc = etcdserver: duplicate key given in txn request} {revision 11 76 } {key 15 0 /apisix/routes/eff2f3e2 }".

wumengchao · 2023-11-30T13:09:37Z

“failed to create object, ignore it{error 26 0 rpc error: code = InvalidArgument ”
How can one find an InvalidArgument?

Revolyssup · 2023-12-01T05:18:09Z

@wumengchao A similar issue with reproduction steps has been reported here - #2076. I am working on this,

Revolyssup · 2023-12-11T03:41:12Z

fixed by #2078

wumengchao · 2023-12-14T15:12:43Z

After compiling and releasing the master branch, the issue still persists.

Revolyssup · 2023-12-15T03:45:39Z

@wumengchao Same errors?

wumengchao · 2023-12-16T06:15:59Z

@wumengchao Same errors?

yes. This is a screenshot after compilation.

wumengchao · 2023-12-16T06:17:10Z

This is a screenshot of the error message.

wumengchao · 2023-12-16T06:21:30Z

I deliberately made incorrect changes to the routing configuration, such as the service name or port, and then changed it back to normal. This issue can be reproduced immediately.

Revolyssup · 2024-01-02T20:22:10Z

@wumengchao Yes this log can be ignored. I can confirm that the latest master doesn't have this bug.
I followed the following repro steps.

Build the image on master
deployed samples/deploy/composite.yml (with my image)
created wrong route then corrected it, it was updated in apisix successfully. Similarly deleted and created again, same thing happened.

wumengchao · 2024-01-03T13:09:30Z

OK

EtleDev · 2024-06-06T13:11:19Z

Hello,

I still have this issue with composite ingress controller, and routing isn't operate. I have a log like this every minute :

2024-06-06 15:09:43 2024-06-06T21:09:43+08:00 error adapter/etcd.go:141 failed to create object, ignore it{error 26 0 rpc error: code = InvalidArgument desc = etcdserver: duplicate key given in txn request} {revision 11 4 <nil>} {key 15 0 /apisix/upstreams/f2a515ca <nil>} 2024-06-06 15:09:43 2024-06-06T21:09:43+08:00 error adapter/etcd.go:141 failed to create object, ignore it{error 26 0 rpc error: code = InvalidArgument desc = etcdserver: duplicate key given in txn request} {revision 11 4 <nil>} {key 15 0 /apisix/routes/eec74e34 <nil>}

Do you know why ?

damoshushu · 2024-06-26T08:59:25Z

hello @Revolyssup ,
I still get this issue

kubectl -n apisix get pods apisix-7bc4f47f47-gcvws -o yaml | grep image -C 4
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    image: apache/apisix-ingress-controller:1.8.2
    imagePullPolicy: IfNotPresent

logs:

2024-06-26T16:41:13+08:00	info	adapter/etcd.go:147	created object{revision 11 27  <nil>} {key 15 0 /apisix/routes/d19e6f8f <nil>}
2024-06-26T16:41:13+08:00	info	adapter/etcd.go:147	created object{revision 11 28  <nil>} {key 15 0 /apisix/upstreams/4405a246 <nil>}
2024-06-26T16:41:13+08:00	info	adapter/etcd.go:147	created object{revision 11 29  <nil>} {key 15 0 /apisix/plugin_configs/6b2eb039 <nil>}
2024-06-26T16:41:13+08:00	info	adapter/etcd.go:147	created object{revision 11 30  <nil>} {key 15 0 /apisix/routes/8659c925 <nil>}
2024-06-26T16:41:13+08:00	error	adapter/etcd.go:141	failed to create object, ignore it{error 26 0  rpc error: code = InvalidArgument desc = etcdserver: duplicate key given in txn request} {revision 11 30  <nil>} {key 15 0 /apisix/upstreams/4405a246 <nil>}
2024-06-26T16:41:13+08:00	info	adapter/etcd.go:147	created object{revision 11 31  <nil>} {key 15 0 /apisix/routes/2d3cc270 <nil>}

boscard · 2024-08-02T14:23:36Z

@Revolyssup I've tested this with versions 1.7.0, 1.7.1, 1.8.0, 1.8.1 and 1.8.2 and ApiSix version 3.9.1 and I'm always getting the same errors:

2024-08-02T22:19:38+08:00       error   adapter/etcd.go:141     failed to create object, ignore it{error 26 0  rpc error: code = InvalidArgument desc = etcdserver: duplicate key given in txn request} {revision 11 115  <nil>} {key 15 0 /apisix/upstreams/5c607e35 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 116  <nil>} {key 15 0 /apisix/routes/ebba555c <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 117  <nil>} {key 15 0 /apisix/routes/a08b23a9 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 118  <nil>} {key 15 0 /apisix/ssls/1aaae1cf <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 119  <nil>} {key 15 0 /apisix/upstreams/133835d9 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 120  <nil>} {key 15 0 /apisix/routes/87d32b8f <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 121  <nil>} {key 15 0 /apisix/ssls/15c53338 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 122  <nil>} {key 15 0 /apisix/upstreams/e673e843 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 123  <nil>} {key 15 0 /apisix/routes/b8a67001 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 124  <nil>} {key 15 0 /apisix/ssls/9e952c6f <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 125  <nil>} {key 15 0 /apisix/upstreams/d561e564 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 126  <nil>} {key 15 0 /apisix/routes/24f80351 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 127  <nil>} {key 15 0 /apisix/ssls/ad479896 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 128  <nil>} {key 15 0 /apisix/upstreams/d7275b3d <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 129  <nil>} {key 15 0 /apisix/routes/2270db75 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 130  <nil>} {key 15 0 /apisix/ssls/d1c3f9a3 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 131  <nil>} {key 15 0 /apisix/upstreams/8b2bcb52 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 132  <nil>} {key 15 0 /apisix/routes/7220df47 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 133  <nil>} {key 15 0 /apisix/routes/46a46b3e <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 134  <nil>} {key 15 0 /apisix/routes/d77bd714 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 135  <nil>} {key 15 0 /apisix/routes/a2e2c60a <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 136  <nil>} {key 15 0 /apisix/routes/d64f4c89 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 137  <nil>} {key 15 0 /apisix/routes/adf55859 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 138  <nil>} {key 15 0 /apisix/ssls/792ba877 <nil>}
2024-08-02T22:19:38+08:00       error   adapter/etcd.go:141     failed to create object, ignore it{error 26 0  rpc error: code = InvalidArgument desc = etcdserver: duplicate key given in txn request} {revision 11 138  <nil>} {key 15 0 /apisix/upstreams/8b2bcb52 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 139  <nil>} {key 15 0 /apisix/routes/c3caca1d <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 140  <nil>} {key 15 0 /apisix/ssls/6c28a184 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 141  <nil>} {key 15 0 /apisix/upstreams/51554a79 <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 142  <nil>} {key 15 0 /apisix/routes/6f69657d <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 143  <nil>} {key 15 0 /apisix/ssls/ac2bf411 <nil>}
2024-08-02T22:19:38+08:00       error   adapter/etcd.go:141     failed to create object, ignore it{error 26 0  rpc error: code = InvalidArgument desc = etcdserver: duplicate key given in txn request} {revision 11 143  <nil>} {key 15 0 /apisix/upstreams/65afa96b <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:147     created object{revision 11 144  <nil>} {key 15 0 /apisix/routes/964b0c6a <nil>}
2024-08-02T22:19:38+08:00       info    adapter/etcd.go:185     updated object{revision 11 145  <nil>} {key 15 0 /apisix/ssls/d7f19a17 <nil>}

I can help with debugging if needed.

rknightion · 2024-08-14T19:20:43Z

We're getting the same over in #2276 (also using apisixtls resources)

Revolyssup added the bug Something isn't working label Nov 29, 2023

Revolyssup self-assigned this Nov 29, 2023

Revolyssup closed this as completed Dec 11, 2023

Revolyssup reopened this Dec 15, 2023

Revolyssup closed this as completed Jan 3, 2024

rknightion mentioned this issue Aug 14, 2024

bug: Can't define multiple k8s ingress definitions with pathType: Prefix pointing to single service when using combined mode #2276

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bug: Routing configuration not taking effect, and the backend reports an etcd error. #2071

bug: Routing configuration not taking effect, and the backend reports an etcd error. #2071

wumengchao commented Nov 25, 2023

Revolyssup commented Nov 26, 2023

wumengchao commented Nov 28, 2023

wumengchao commented Nov 28, 2023 •

edited

Loading

wumengchao commented Nov 28, 2023 •

edited

Loading

Revolyssup commented Nov 29, 2023

Revolyssup commented Nov 29, 2023 •

edited

Loading

Revolyssup commented Nov 29, 2023

wumengchao commented Nov 29, 2023

wumengchao commented Nov 30, 2023

wumengchao commented Nov 30, 2023

Revolyssup commented Dec 1, 2023

Revolyssup commented Dec 11, 2023

wumengchao commented Dec 14, 2023

Revolyssup commented Dec 15, 2023

wumengchao commented Dec 16, 2023

wumengchao commented Dec 16, 2023

wumengchao commented Dec 16, 2023

Revolyssup commented Jan 2, 2024

wumengchao commented Jan 3, 2024

EtleDev commented Jun 6, 2024

damoshushu commented Jun 26, 2024 •

edited

Loading

boscard commented Aug 2, 2024

rknightion commented Aug 14, 2024

bug: Routing configuration not taking effect, and the backend reports an etcd error. #2071

bug: Routing configuration not taking effect, and the backend reports an etcd error. #2071

Comments

wumengchao commented Nov 25, 2023

Current Behavior

Expected Behavior

Error Logs

Steps to Reproduce

Environment

Revolyssup commented Nov 26, 2023

wumengchao commented Nov 28, 2023

wumengchao commented Nov 28, 2023 • edited Loading

wumengchao commented Nov 28, 2023 • edited Loading

Revolyssup commented Nov 29, 2023

Revolyssup commented Nov 29, 2023 • edited Loading

Revolyssup commented Nov 29, 2023

wumengchao commented Nov 29, 2023

wumengchao commented Nov 30, 2023

wumengchao commented Nov 30, 2023

Revolyssup commented Dec 1, 2023

Revolyssup commented Dec 11, 2023

wumengchao commented Dec 14, 2023

Revolyssup commented Dec 15, 2023

wumengchao commented Dec 16, 2023

wumengchao commented Dec 16, 2023

wumengchao commented Dec 16, 2023

Revolyssup commented Jan 2, 2024

wumengchao commented Jan 3, 2024

EtleDev commented Jun 6, 2024

damoshushu commented Jun 26, 2024 • edited Loading

boscard commented Aug 2, 2024

rknightion commented Aug 14, 2024

wumengchao commented Nov 28, 2023 •

edited

Loading

wumengchao commented Nov 28, 2023 •

edited

Loading

Revolyssup commented Nov 29, 2023 •

edited

Loading

damoshushu commented Jun 26, 2024 •

edited

Loading