Open
Description
Expected behavior:
When clicking on Users and User Roles in the Teleport Web UI, the system should retrieve and display the information without any errors, even with a large number of roles in the backend.
Current behavior:
Clicking on Users and User Roles in the Teleport Web UI triggers an error with a red banner showing the message grpc: received message larger than max (13589253 vs. 4194304)
The following appear in the proxy logs at the same time:
Jan 10 14:36:02 example.com teleport[2202215]: 2024-01-10T14:36:02Z DEBU [PGBK] Fetched change feed events. elapsed:1.983769ms messages:1 pgbk/background.go:262
Jan 10 14:36:03 example.com teleport[2202215]: 2024-01-10T14:36:03Z WARN [NODE:1:CA] Re-init the cache on error error:[
Jan 10 14:36:03 example.com teleport[2202215]: ERROR REPORT:
Jan 10 14:36:03 example.com teleport[2202215]: Original Error: *trace.LimitExceededError grpc: received message larger than max (13589253 vs. 4194304)
Jan 10 14:36:03 example.com teleport[2202215]: Stack Trace:
Jan 10 14:36:03 example.com teleport[2202215]: github.com/gravitational/teleport/api@v0.0.0/client/client.go:1585 github.com/gravitational/teleport/api/client.(*Client).GetRoles
Jan 10 14:36:03 example.com teleport[2202215]: github.com/gravitational/teleport/lib/cache/collections.go:1173 github.com/gravitational/teleport/lib/cache.roleExecutor.getAll
Jan 10 14:36:03 example.com teleport[2202215]: github.com/gravitational/teleport/lib/cache/collections.go:97 github.com/gravitational/teleport/lib/cache.(*genericCollection[...]).fetch
Jan 10 14:36:03 example.com teleport[2202215]: github.com/gravitational/teleport/lib/cache/cache.go:1551 github.com/gravitational/teleport/lib/cache.(*Cache).fetch.func2
Jan 10 14:36:03 example.com teleport[2202215]: golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1
Jan 10 14:36:03 example.com teleport[2202215]: runtime/asm_amd64.s:1650 runtime.goexit
Jan 10 14:36:03 example.com teleport[2202215]: User Message: failed to fetch resource: "role"
Jan 10 14:36:03 example.com teleport[2202215]: grpc: received message larger than max (13589253 vs. 4194304)] cache/cache.go:1071
Jan 10 14:36:03 example.com teleport[2202215]: 2024-01-10T14:36:03Z DEBU [NODE:1:CA] Reloading cache. cache/cache.go:1075
Jan 10 14:36:04 example.com teleport[2202215]: 2024-01-10T14:36:04Z DEBU [PGBK] Fetched change feed events. elapsed:2.12619ms messages:1 pgbk/background.go:262
Bug details:
- Teleport version: 14.3.0
- Recreation steps:
- Log into Teleport Web UI on a Cluster with thousands of roles. The affected system had 11,529 roles. The overall size of the role objects likely has more to do with the maximum grpc size rather than simply the count of roles.
- Navigate to Access Management.
- Click on Users or User Roles.
- Observe the error message indicating a gRPC size limit issue.
- Interestingly, the paginated list of roles in the access request screen seems to be working correctly (which is where the 11,529 role count came from)
The log entries indicated suggest that the grpc message received while fetching roles exceeded the maximum allowed message size, causing the error.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
