ERROR_CODE_CANCELLED, "context canceled" errors in logs after enabling the distributed mode on

[fryderykdyc-sennder]

Hi, up until recently we were running permify in a few k8s replicas but without enabling the distributed caching mode. Everything was running fine and the logs were looking good.

Earlier this week we enabled the distributed mode caching. Attaching the respective part of configuration below.
Everything apparently works fine. Depending services don't report errors, the http traces in datadog look good (we talk to permify using http protocol).

However I see:

• errors in datadog when I select "grpc.server.request" as the trace type.
• in logs I get a lot of following errors:

{
  "msg": "finished call",
  "protocol": "grpc",
  "level": "ERROR",
  "peer": {
    "address": "100.104.77.54:9983"
  },
  "time": "2025-12-16T12:24:01.044956289Z",
  "grpc": {
    "start_time": "2025-12-16T12:24:01Z",
    "request": {
      "deadline": "2025-12-16T12:24:05Z"
    },
    "component": "server",
    "code": "Internal",
    "method": "Check",
    "service": "base.v1.Permission",
    "method_type": "unary",
    "time_ms": "24.775",
    "error": "rpc error: code = Internal desc = ERROR_CODE_CANCELLED"
  }
}

{
  "msg": "ERROR_CODE_CANCELLED",
  "level": "ERROR",
  "time": "2025-12-16T12:24:01.044936916Z"
}

{
  "msg": "rpc error: code = Canceled desc = context canceled",
  "level": "ERROR",
  "time": "2025-12-16T12:24:01.044859883Z"
}

{
  "msg": "finished call",
  "protocol": "grpc",
  "level": "ERROR",
  "peer": {
    "address": "100.104.77.54:9983"
  },
  "time": "2025-12-16T12:24:00.86811204Z",
  "grpc": {
    "start_time": "2025-12-16T12:24:00Z",
    "request": {
      "deadline": "2025-12-16T12:24:04Z"
    },
    "component": "server",
    "code": "Internal",
    "method": "Check",
    "service": "base.v1.Permission",
    "method_type": "unary",
    "time_ms": "0.487",
    "error": "rpc error: code = Internal desc = ERROR_CODE_CANCELLED"
  }
}

{
  "msg": "finished call",
  "protocol": "grpc",
  "level": "ERROR",
  "peer": {
    "address": "100.104.77.54:9983"
  },
  "time": "2025-12-16T12:24:00.582466849Z",
  "grpc": {
    "start_time": "2025-12-16T12:24:00Z",
    "request": {
      "deadline": "2025-12-16T12:24:04Z"
    },
    "component": "server",
    "code": "Internal",
    "method": "Check",
    "service": "base.v1.Permission",
    "method_type": "unary",
    "time_ms": "0.805",
    "error": "rpc error: code = Internal desc = ERROR_CODE_CANCELLED"
  }
}

and so on.

Is it some actual issue I shall worry about? Or just some internals are mistakenly interpreted as errors while they are really not?

My configuration of the distributed mode:

      distributed:
        enabled: true
        address: "permify-service.permify-service.svc.cluster.local:5000"
        port: "5000"

[omer-topal]

Hello @fryderykdyc-sennder

First quick check: how are you exposing Permify internally? Is permify-service.permify-service.svc.cluster.local:5000 a Headless Service that load-balances across all Permify replicas?

To help us reproduce this on our side, can you also share:

-The Permify version you are running
-The schema you are using (or a minimal schema + sample tuples) and an example request that triggers the error
-Anything in the request path that might cancel requests

[shmuelarditi]

@omer-topal same as what happend here: #2559 - in my opinion it needs to be reopened.
we are running version v1.6.0
we are using a headless service.
Endpoints resolve correctly

getting those errors:

level=ERROR msg="rpc error: code = Canceled desc = context canceled"
level=ERROR msg=ERROR_CODE_CANCELLED
level=ERROR msg="finished call" grpc.service=base.v1.Permission grpc.method=Check grpc.code=Internal grpc.error="rpc error: code = Internal desc = ERROR_CODE_CANCELLED" grpc.time_ms=4.9

maybe it will help:

• Invoker gRPC server starts successfully: invoker grpc server successfully started: 5000.
• RBAC configured: ServiceAccount with get/list/watch on services, endpoints, endpointslices.
• Endpoints resolve correctly: Verified pod IPs match headless service endpoints.
• Works fine with distributed cache disabled (but answers are not reliable we faced inconsistencies).
• Requests fail in 2-40ms with context canceled.
• Both pods log errors simultaneously during Permission.Check coordination.

configuration summary:
2 replicas

PERMIFY_DISTRIBUTED_ENABLED: "true"
PERMIFY_DISTRIBUTED_ADDRESS: "permify-headless.permify.svc.cluster.local:5000"
PERMIFY_DISTRIBUTED_PORT: "5000"

headless-service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: stg-platform-new-infra-permify:/Service:permify/permify-headless
  name: permify-headless
  namespace: permify
spec:
  clusterIP: None
  ports:
    - name: distributed
      port: 5000
      targetPort: 5000
  publishNotReadyAddresses: true
  selector:
    app: permify

[omer-topal]

@shmuelarditi are you seeing any failures client-side (HTTP/gRPC responses, timeouts, retries), or are these only showing up in Permify logs / grpc.server.request traces?

At the moment we’re trying to confirm whether ERROR_CODE_CANCELLED is just internal coordination noise being logged as ERROR vs an actual functional issue.

If you have an example where the caller receives an error, can you share the request (or minimal repro), plus schema/tuples.

[shmuelarditi]

@omer-topal Hi sorry for the late response, we do not see any errors on the client side.
i hope that indeed its just noise, would love to get your confirmation on that whenever possible.

[fryderykdyc-sennder]

Hello @fryderykdyc-sennder

First quick check: how are you exposing Permify internally? Is permify-service.permify-service.svc.cluster.local:5000 a Headless Service that load-balances across all Permify replicas?

To help us reproduce this on our side, can you also share:

-The Permify version you are running -The schema you are using (or a minimal schema + sample tuples) and an example request that triggers the error -Anything in the request path that might cancel requests

Hi @omer-topal!
Also sorry for the late reply, xmass period and stuff, you know :)
On our side it also looks harmless as no issues on the client side it seems, but I will try to take some time either tomorrow or early next week and provide a complete reproduceable example. For now we just filtered out errors in datadog and it seems everything works fine.
Although it would definitely make sense to fix it as now our APM in datadog is full of errors that are not really errors.

[fryderykdyc-sennder]

Hi @omer-topal I tried to reproduce the issue locally today but no success so far :( I don't see the errors in the logs.
I used a local k8s in docker setup, attaching it here: https://github.com/fryderykdyc-sennder/permify/tree/local-k8s/k8s-local

To run that:

brew install kind
# check out my fork locally 
git clone https://github.com/fryderykdyc-sennder/permify.git 
cd permify/k8s-local

kind create cluster --name permify-local --config kind-config.yaml
./deploy.sh

# check if it works, should give {"status":"SERVING"}
curl http://localhost:3476/healthz

make ensure-tenant
make schema-upload
make insert-tuples

# in one terminal:
kubectl logs -f deployment/permify-service -n permify-service

# in another one:
make check-permission

but unfortunately no CONTEXT CANCELLED errors visible.

First quick check: how are you exposing Permify internally? Is permify-service.permify-service.svc.cluster.local:5000 a Headless Service that load-balances across all Permify replicas?

I am 99% sure that yes but will talk to my infra colleague that helped me set that up and will get back to you.

-The Permify version you are running

1.5.3

The schema you are using (or a minimal schema + sample tuples) and an example request that triggers the error

I am not able to isolate the particular queries that cause the error, but we have indeed many rules in the schema that contain OR and AND conditions

-Anything in the request path that might cancel requests

No idea tbh but we see that error A LOT

[shmuelarditi]

i can also confirm im getting the following log messages, seems like its working and the context error is indeed just noise...
but again i would appreciate if you could confirm that.

time=2026-01-11T09:26:34.547Z level=INFO msg="finished call" protocol=grpc grpc.component=server grpc.service=grpc.health.v1.Health grpc.method=Check grpc.method_type=unary peer.address=127.0.0.1:33678 grpc.start_time=2026-01-11T09:26:34Z grpc.code=OK grpc.time_ms=0.061
time=2026-01-11T09:26:36.571Z level=INFO msg="Received new ClientConn state" state="{ResolverState:{Addresses:[{Addr: \"10.204.3.79:5000\", ServerName: \"permify-headless.permify\", } {Addr: \"10.204.5.18:5000\", ServerName: \"permify-headless.permify\", }] Endpoints:[{Addresses:[{Addr: \"10.204.3.79:5000\", ServerName: \"permify-headless.permify\", }] Attributes:<nil>} {Addresses:[{Addr: \"10.204.5.18:5000\", ServerName: \"permify-headless.permify\", }] Attributes:<nil>}] ServiceConfig:<nil> Attributes:<nil>} BalancerConfig:0xc000619d10}"
time=2026-01-11T09:26:36.572Z level=INFO msg="Current consistent members" member_count=2
time=2026-01-11T09:26:36.572Z level=INFO msg="Consistent member" member=permify-headless.permify|10.204.3.79:5000
time=2026-01-11T09:26:36.572Z level=INFO msg="Consistent member" member=permify-headless.permify|10.204.5.18:5000
time=2026-01-11T09:26:36.572Z level=INFO msg="Creating new picker" width=1
time=2026-01-11T09:26:36.572Z level=INFO msg="Updating ClientConn state" connectivity_state=READY

[shmuelarditi]

@omer-topal i just found out it isnt just noise, tried using this in production it caused issues, even if they are not visible from the clients side, services speaking with permify got 429 errors, and we had to revert back to single replica.

[omer-topal]

@shmuelarditi , @fryderykdyc-sennder Thanks for the update!

We will investigate the root cause behind the context canceled errors and the 429s you are seeing, and follow up here.

[fryderykdyc-sennder]

First quick check: how are you exposing Permify internally? Is permify-service.permify-service.svc.cluster.local:5000 a Headless Service that load-balances across all Permify replicas?

Double confirmed, the answer is yes. Thanks for looking into this!