bug: phase_func(): failed to connect to the unix socket unix:/usr/local/apisix/conf/apisix-1.sock: permission denied

[zar3bski]

Current Behavior

I can't use a python plugin as a ext-plugin-post-req. When I deactivate the plugin,the request reaches the upstream service.
Here are the processes in the container

ps 
PID   USER     TIME  COMMAND
    1 root      0:00 {openresty} nginx: master process /usr/local/openresty/bin/openresty -p /usr/local/apisix -g daemon off;
   46 nobody    0:00 {openresty} nginx: worker process
   .....
   75 nobody    0:00 {openresty} nginx: worker process
   76 nobody    0:00 {openresty} nginx: worker process
   77 nobody    0:00 {openresty} nginx: worker process
   78 nobody    0:00 {openresty} nginx: cache manager process
   80 root      0:00 {openresty} nginx: privileged agent process
   81 root      0:00 python3 /usr/share/extras/py-apisix/py-runner start

It seems that both apisix and my plugin run as root so they should be able to communicate through /usr/local/apisix/conf/apisix-1.sock:

ls -la /usr/local/apisix/conf/apisix-1.sock
srwxr-xr-x    1 root     root             0 Sep 14 13:17 /usr/local/apisix/conf/apisix-1.sock

I just can't make sense of this behavior

Expected Behavior

Requests should reach the upstream service enriched with the headers defined in the plugin module

apisix-python-plugin-runner/apisix/plugins/rewrite.py

    def filter(self, conf: Any, request: Request, response: Response):
        """
        The plugin executes the main function
        :param conf:
            plugin configuration after parsing
        :param request:
            request parameters and information
        :param response:
            response parameters and information
        :return:
        """

        # print plugin configuration
        print(conf)

        # Fetch request nginx variable `host`
        host = request.get_var("host")
        print(host)

        # Fetch request body
        body = request.get_body()
        print(body)

        # Rewrite request headers
        request.set_header("X-Resp-A6-Runner", "Python")

        # Rewrite request args
        request.set_arg("a6_runner", "Python")

        # Rewrite request path
        request.set_uri("/a6/python/runner")

Error Logs

2022/09/14 13:47:01 [crit] 47#47: *795708 connect() to unix:/usr/local/apisix/conf/apisix-1.sock failed (13: Permission denied), client: 192.168.1.139, server: _, request: "GET /mock/ HTTP/1.1", host: "localhost:32674"
2022/09/14 13:47:01 [error] 47#47: *795708 [lua] init.lua:726: phase_func(): failed to connect to the unix socket unix:/usr/local/apisix/conf/apisix-1.sock: permission denied, client: 192.168.1.139, server: _, request: "GET /mock/ HTTP/1.1", host: "localhost:32674"
2022/09/14 13:47:01 [warn] 47#47: *795708 [lua] plugin.lua:759: run_plugin(): ext-plugin-post-req exits with http status code 503, client: 192.168.1.139, server: _, request: "GET /mock/ HTTP/1.1", host: "localhost:32674"

Steps to Reproduce

1. Deploy the chart with the following values.yaml

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apisix:
  # Enable or disable Apache APISIX itself
  # Set it to false and ingress-controller.enabled=true will deploy only ingress-controller
  enabled: true

  admin_key:
    - name: admin
      key: edd1c9f034335f136f87ad84b625c8f1  # using fixed API token has security risk, please update it when you deploy to production environment
      role: admin

  # Enable nginx IPv6 resolver
  enableIPv6: true

  # Use Pod metadata.uid as the APISIX id.
  setIDFromPodUID: false

  customLuaSharedDicts: []
    # - name: foo
    #   size: 10k
    # - name: bar
    #   size: 1m
  luaModuleHook:
    enabled: false
    # extend lua_package_path to load third party code
    luaPath: ""
    # the hook module which will be used to inject third party code into APISIX
    # use the lua require style like: "module.say_hello"
    hookPoint: ""
    # configmap that stores the codes
    configMapRef:
      name: ""
      # mounts decides how to mount the codes to the container.
      mounts:
        - key: ""
          path: ""

  enableCustomizedConfig: false
  customizedConfig: {}

  image:
    repository: apisix/apisix #apache/apisix
    pullPolicy: Always
    # Overrides the image tag whose default is the chart appVersion.
    tag: 2.13.3-alpine
    
  # Use a `DaemonSet` or `Deployment`
  kind: Deployment
  # kind is DaemonSet, replicaCount not become effective
  replicaCount: 1

  podAnnotations: {}
  podSecurityContext: {}
    # fsGroup: 2000
  securityContext: {}
    # capabilities:
    #   drop:
    #   - ALL
    # readOnlyRootFilesystem: true
    # runAsNonRoot: true
    # runAsUser: 1000

  # See https://kubernetes.io/docs/tasks/run-application/configure-pdb/ for more details
  podDisruptionBudget:
    enabled: false
    minAvailable: 90%
    maxUnavailable: 1

  resources: {}
    # We usually recommend not to specify default resources and to leave this as a conscious
    # choice for the user. This also increases chances charts run on environments with little
    # resources, such as Minikube. If you do want to specify resources, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
    # limits:
    #   cpu: 100m
    #   memory: 128Mi
    # requests:
    #   cpu: 100m
    #   memory: 128Mi
  hostNetwork: false

  nodeSelector: {}
  tolerations: []
  affinity: {}
  # If true, it will sets the anti-affinity of the Pod.
  podAntiAffinity:
    enabled: false

  # timezone is the timezone where apisix uses.
  # For example: "UTC" or "Asia/Shanghai"
  # This value will be set on apisix container's environment variable TZ.
  # You may need to set the timezone to be consistent with your local time zone,
  # otherwise the apisix's logs may used to retrieve event maybe in wrong timezone.
  timezone: ""

  # extraEnvVars An array to add extra env vars
  # e.g:
  # extraEnvVars:
  #   - name: FOO
  #     value: "bar"
  #   - name: FOO2
  #     valueFrom:
  #       secretKeyRef:
  #         name: SECRET_NAME
  #         key: KEY
  extraEnvVars: []

nameOverride: ""
fullnameOverride: ""

gateway:
  type: NodePort
  # If you want to keep the client source IP, you can set this to Local.
  # ref: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip
  externalTrafficPolicy: Cluster
  # type: LoadBalancer
  # annotations:
  #   service.beta.kubernetes.io/aws-load-balancer-type: nlb
  externalIPs: []
  http:
    enabled: true
    servicePort: 80
    containerPort: 9080
  tls:
    enabled: false
    servicePort: 443
    containerPort: 9443
    existingCASecret: ""
    certCAFilename: ""
    http2:
      enabled: true
    sslProtocols: "TLSv1.2 TLSv1.3"
  # L4 proxy (TCP/UDP)
  stream:
    enabled: false
    only: false
    tcp: []
    udp: []
  ingress:
    enabled: false
    annotations: {}
      # kubernetes.io/ingress.class: nginx
      # kubernetes.io/tls-acme: "true"
    hosts:
      - host: apisix.local
        paths: []
    tls: []
  #  - secretName: apisix-tls
  #    hosts:
  #      - chart-example.local

admin:
  # Enable Admin API
  enabled: true
  # admin service type
  type: ClusterIP
  # loadBalancerIP: a.b.c.d
  # loadBalancerSourceRanges:
  #   - "143.231.0.0/16"
  externalIPs: []
  #
  port: 9180
  servicePort: 9180
  # Admin API support CORS response headers
  cors: true
  # Admin API credentials
  credentials:
    admin: edd1c9f034335f136f87ad84b625c8f1
    viewer: 4054f7cf07e344346cd3f287985e76a2

  allow:
    # The ip range for allowing access to Apache APISIX
    ipList:
      - 127.0.0.1/24

nginx:
  workerRlimitNofile: "20480"
  workerConnections: "10620"
  workerProcesses: auto
  enableCPUAffinity: true

# APISIX plugins to be enabled
plugins:
  - api-breaker
  - authz-keycloak
  - basic-auth
  - batch-requests
  - consumer-restriction
  - cors
  - echo
  - fault-injection
  - grpc-transcode
  - hmac-auth
  - http-logger
  - ip-restriction
  - ua-restriction
  - jwt-auth
#  - kafka-logger
  - key-auth
  - limit-conn
  - limit-count
  - limit-req
  - node-status
  - openid-connect
  - authz-casbin
  - prometheus
  - proxy-cache
  - proxy-mirror
  - proxy-rewrite
  - redirect
  - referer-restriction
  - request-id
  - request-validation
  - response-rewrite
  - serverless-post-function
  - serverless-pre-function
  - sls-logger
  - syslog
  - tcp-logger
  - udp-logger
  - uri-blocker
  - wolf-rbac
  - zipkin
  - traffic-split
  - gzip
  - real-ip
  - ext-plugin-pre-req
  - ext-plugin-post-req
stream_plugins:
  - mqtt-proxy
  - ip-restriction
  - limit-conn

pluginAttrs:  {}

extPlugin:
  enabled: true
  cmd: ["python3","/usr/share/extras/py-apisix/py-runner", "start"]

# customPlugins allows you to mount your own HTTP plugins.
customPlugins:
  enabled: false
  # the lua_path that tells APISIX where it can find plugins,
  # note the last ';' is required.
  luaPath: "/opts/custom_plugins/?.lua"
  plugins:
    # plugin name.
    - name: ""
      # plugin attrs
      attrs: {}
      # plugin codes can be saved inside configmap object.
      configMap:
        # name of configmap.
        name: ""
        # since keys in configmap is flat, mountPath allows to define the mount
        # path, so that plugin codes can be mounted hierarchically.
        mounts:
          - key: ""
            path: ""
          - key: ""
            path: ""

updateStrategy: {}
  # type: RollingUpdate

extraVolumes: 
  - name: apisix-lib
    hostPath:
      path: /home/zar3bski/Documents/Code/octaave/test_files/apisix-python-plugin-runner/apisix
  - name: apisix-bin
    hostPath:
      path: /home/zar3bski/Documents/Code/octaave/test_files/apisix-python-plugin-runner/bin
  - name: apisix-conf
    hostPath:
      path: /home/zar3bski/Documents/Code/octaave/test_files/apisix-python-plugin-runner/conf

extraVolumeMounts:
  - name: apisix-lib
    mountPath: /usr/lib/python3.9/site-packages/apisix
    readOnly: true
  - name: apisix-bin
    mountPath: /usr/share/extras/py-apisix
    readOnly: true
  - name: apisix-conf
    mountPath: /usr/share/extras/conf
    readOnly: true

discovery:
  enabled: false
  registry:
    # Integration service discovery registry. E.g eureka\dns\nacos\consul_kv
    # reference:
    # https://apisix.apache.org/docs/apisix/discovery#configuration-for-eureka
    # https://apisix.apache.org/docs/apisix/discovery/dns#service-discovery-via-dns
    # https://apisix.apache.org/docs/apisix/discovery/consul_kv#configuration-for-consul-kv
    # https://apisix.apache.org/docs/apisix/discovery/nacos#configuration-for-nacos
    #
    # an eureka example:
    # eureka:
    #   host:
    #     - "http://${username}:${password}@${eureka_host1}:${eureka_port1}"
    #     - "http://${username}:${password}@${eureka_host2}:${eureka_port2}"
    #   prefix: "/eureka/"
    #   fetch_interval: 30
    #   weight: 100
    #   timeout:
    #     connect: 2000
    #     send: 2000
    #     read: 5000

# access log and error log configuration
logs:
  enableAccessLog: true
  accessLog: "/dev/stdout"
  accessLogFormat: '$remote_addr - $remote_user [$time_local] $http_host \"$request\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\"'
  accessLogFormatEscape: default
  errorLog: "/dev/stderr"
  errorLogLevel: "warn"

dns:
  resolvers:
    - 127.0.0.1
    - 172.20.0.10
    - 114.114.114.114
    - 223.5.5.5
    - 1.1.1.1
    - 8.8.8.8
  validity: 30
  timeout: 5

initContainer:
  image: busybox
  tag: 1.28

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  targetMemoryUtilizationPercentage: 80

# Custom configuration snippet.
configurationSnippet:
  main: |
  httpStart: |
  httpEnd: |
  httpSrv: |
  httpAdmin: |
  stream: |
# Observability configuration.
# ref: https://apisix.apache.org/docs/apisix/plugins/prometheus/
serviceMonitor:
  enabled: false
  # namespace where the serviceMonitor is deployed, by default, it is the same as the namespace of the apisix
  namespace: ""
  # name of the serviceMonitor, by default, it is the same as the apisix fullname
  name: ""
  # interval at which metrics should be scraped
  interval: 15sfalse
  metricPrefix: apisix_
  # container port where the metrics are exposed
  containerPort: 9091
  # @param serviceMonitor.labels ServiceMonitor extra labels
  labels: {}
  # @param serviceMonitor.annotations ServiceMonitor annotations
  annotations: {}

# etcd configuration
# use the FQDN address or the IP of the etcd
etcd:
  # install etcd(v3) by default, set false if do not want to install etcd(v3) together
  enabled: true
  host:
    # host or ip e.g. http://172.20.128.89:2379
    - http://etcd.host:2379
  prefix: "/apisix"
  timeout: 30

  # if etcd.enabled is true, set more values of bitnami/etcd helm chart
  auth:
    rbac:
      # No authentication by default
      create: false
      user: ""
      password: ""
    tls:
      enabled: false
      existingSecret: ""
      certFilename: ""
      certKeyFilename: ""
      verify: true
      sni: ""

  service:
    port: 2379

  replicaCount: 3

dashboard:
  enabled: true
  replicaCount: 1

  image:
    repository: apache/apisix-dashboard
    pullPolicy: IfNotPresent
    # Overrides the image tag whose default is the chart appVersion.
    tag: 2.13-alpine

  imagePullSecrets: []
  nameOverride: ""
  fullnameOverride: ""

  serviceAccount:
    # Specifies whether a service account should be created
    create: true
    # Annotations to add to the service account
    annotations: {}
    # The name of the service account to use.
    # If not set and create is true, a name is generated using the fullname template
    name: ""

  podAnnotations: {}

  podSecurityContext: {}
    # fsGroup: 2000

  securityContext: {}
    # capabilities:
    #   drop:
    #   - ALL
    # readOnlyRootFilesystem: true
    # runAsNonRoot: true
    # runAsUser: 1000

  config:
    conf:
      listen:
        host: 0.0.0.0
        port: 9000
      etcd:
        # Supports defining multiple etcd host addresses for an etcd cluster
        endpoints:
          - apisix-etcd:2379
        # apisix configurations prefix
        prefix: "/apisix"
        # Etcd basic auth info
        username: ~
        password: ~
      log:
        errorLog:
          level: warn
          filePath: /dev/stderr
        accessLog:
          filePath: /dev/stdout

    authentication:
      secret: secret
      expireTime: 3600
      users:
        - username: admin
          password: admin

  service:
    type: ClusterIP
    port: 80

    ingress:
      enabled: false
      # Kubernetes 1.18+ support ingressClassName attribute
      className: ""
      annotations: {}
        # kubernetes.io/ingress.class: nginx
        # kubernetes.io/tls-acme: "true"
      # domain access apisix example:
      # hosts:
      # - host: apisix-dashboard.local
      #    paths:
      #     - /*
      hosts:
        - host: apisix-dashboard.local
          paths: []
      tls: []
      #  - secretName: chart-example-tls
      #    hosts:
      #      - chart-example.local

    resources: {}
      # We usually recommend not to specify default resources and to leave this as a conscious
      # choice for the user. This also increases chances charts run on environments with little
      # resources, such as Minikube. If you do want to specify resources, uncomment the following
      # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
      # limits:
      #   cpu: 100m
      #   memory: 128Mi
      # requests:
      #   cpu: 100m
      #   memory: 128Mi

    autoscaling:
      enabled: false
      minReplicas: 1
      maxReplicas: 100
      targetCPUUtilizationPercentage: 80
      # targetMemoryUtilizationPercentage: 80

    nodeSelector: {}

    tolerations: []

    affinity: {}
  
ingress-controller:
  enabled: false

2. create a route

{
  "uri": "/mock/*",
  "name": "mock",
  "methods": [
    "GET",
    "POST",
    "PUT",
    "DELETE",
    "PATCH",
    "HEAD",
    "OPTIONS",
    "CONNECT",
    "TRACE"
  ],
  "plugins": {
    "ext-plugin-post-req": {
      "conf": [
        {
          "name": "rewrite",
          "value": "{\"enable\":\"feature\"}"
        }
      ],
      "disable": false
    },
    "ext-plugin-pre-req": {
      "disable": true
    }
  },
  "upstream": {
    "nodes": [
      {
        "host": "172.17.0.1",
        "port": 8081,
        "weight": 1
      }
    ],
    "timeout": {
      "connect": 6,
      "send": 6,
      "read": 6
    },
    "type": "roundrobin",
    "scheme": "http",
    "pass_host": "pass",
    "keepalive_pool": {
      "idle_timeout": 60,
      "requests": 1000,
      "size": 320
    }
  },
  "status": 1
}

3. query the route curl http://localhost:${mapped port of apisix gateway}/mock/

Environment

• APISIX version (run apisix version): 2.13.3
• Operating system (run uname -a): Linux apisix-6878f46cb5-hps57 5.15.0-47-generic doc: added document of plugin prometheus. apisix#51-Ubuntu SMP Thu Aug 11 07:51:15 UTC 2022 x86_64 Linux (????? The image is supposed to be Alpine based)
• OpenResty / Nginx version (run openresty -V or nginx -V): 1.21.4.1
• Plugin runner version, for issues related to plugin runners: python plugin v 0.2.0
• LuaRocks version, for installation issues (run luarocks --version):

[tokers]

Your APISIX worker processes run as the nobody, who doesn't has the write permission for the UNIX domain socket.

FYI:

On Linux, connecting to a stream socket object requires write permission on that socket; sending a datagram to a datagram socket likewise requires write permission on that socket.

[spacewander]

Does python runner has a similar code like
https://github.com/apache/apisix-go-plugin-runner/blob/d6805db1083d1958ccc0d1fb1c0d015dc6d64d15/internal/server/server.go#L212

[spacewander]

Look like we need to fix it in

apisix-python-plugin-runner/apisix/runner/server/server.py

Line 79 in 3de14f8

def __init__(self, config: NewServerConfig):

?

CC @SkyeYoung

[SkyeYoung]

Look like we need to fix it in

apisix-python-plugin-runner/apisix/runner/server/server.py

Line 79 in 3de14f8

def __init__(self, config: NewServerConfig):

Let me try to fix it.

[zar3bski]

Your APISIX worker processes run as the nobody, who doesn't has the write permission for the UNIX domain socket.

FYI:

On Linux, connecting to a stream socket object requires write permission on that socket; sending a datagram to a datagram socket likewise requires write permission on that socket.

Yeah, I was surprised to see it. Why is it the case? I am using https://hub.docker.com/r/apache/apisix/tags?page=1&name=alpine v. 2.13.3-alpine

[zar3bski]

    def __init__(self, config: NewServerConfig):
        self.fd = config.socket.file
        if os.path.exists(self.fd):
            os.remove(self.fd)
        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
        self.sock.bind(self.fd)
        os.chmod(self.fd, 766) # HERE
        self.sock.listen(1024)

Well, this works but, for security reasons, I am not quite happy with this permissive approach. Would it be safer to get nobody's uid and chown the socket path?

[zar3bski]

I have a fix. MR in a few minutes