Namespace-Scoped pgAdmin 4 is not accessible via Service

[FNSdev]

Please ensure you do the following when reporting a bug:

• Provide a concise description of what the bug is.
• Provide information about your environment.
• Provide clear steps to reproduce the bug.
• Attach applicable logs. Please do not attach screenshots showing logs unless you are unable to copy and paste the log data.
• Ensure any code / output examples are properly formatted for legibility.

Note that some logs needed to troubleshoot may be found in the /pgdata/<CLUSTERNAME>/pg_log directory on your Postgres instance.

An incomplete bug report can lead to delays in resolving the issue or the closing of a ticket, so please be as detailed as possible.

If you are looking for general support, please view the support page for where you can ask questions.

Thanks for reporting the issue, we're looking forward to helping you!

Overview

pgAdmin can't be accessed using the Service object, e.g. curl rhino-pgadmin:5050.

Environment

Please provide the following details:

• Platform: AKS
• Platform Version: 1.26
• PGO Image Tag: 5.5.0
• Postgres Version 16

Steps to Reproduce

REPRO

Provide steps to get to the error condition:

1. Follow the example and create a standalone pgadmin statefulset.
2. Expose it kubectl expose -n postgres-operator ${PGADMIN_POD} --name rhino-pgadmin
3. Try to curl the service curl rhino-pgadmin:5050 (for example, from another pod in the cluster)

EXPECTED

1. pgadmin is accessible via service

ACTUAL

1. http 503

Logs

# curl http://pg-admin:5050 --verbose
* Host pg-admin:5050 was resolved.
* IPv6: (none)
* IPv4: 10.0.34.40
*   Trying 10.0.34.40:5050...
* Connected to pg-admin (10.0.34.40) port 5050
> GET / HTTP/1.1
> Host: pg-admin:5050
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 503 Service Unavailable
< content-length: 175
< content-type: text/plain
< date: Wed, 20 Dec 2023 12:18:30 GMT
< server: envoy
< 
* Connection #0 to host pg-admin left intact

Additional Information

• Running curl 0.0.0.0:5050 from the pgadmin pod itself works fine
• I have tried to explicitly set PGADMIN_LISTEN_ADDRESS to 0.0.0.0 - nothing changes
• Using pod's IP instead of service also doesn't work

[FNSdev]

upd.

to ensure that Istio isn't causing the problem, I have deployed another pod without a sidecar.
results are similar:

using service

/ # curl http://pg-admin:5050 --verbose
* Host pg-admin:5050 was resolved.
* IPv6: (none)
* IPv4: 10.0.1.169
*   Trying 10.0.1.169:5050...
* connect to 10.0.1.169 port 5050 from 10.244.2.17 port 53174 failed: Connection refused
* Failed to connect to pg-admin port 5050 after 4 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to pg-admin port 5050 after 4 ms: Couldn't connect to server

using pod IP

/ # curl 10.244.5.49:5050 --verbose
*   Trying 10.244.5.49:5050...
* connect to 10.244.5.49 port 5050 from 10.244.2.17 port 57800 failed: Connection refused
* Failed to connect to 10.244.5.49 port 5050 after 1 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to 10.244.5.49 port 5050 after 1 ms: Couldn't connect to server

IP addresses here are different because I had to recreate the service & pgadmin statefulset.

[benjaminjb]

Hi @FNSdev, sorry to hear you're running into this. Can you share the spec for the rhino-pgadmin svc that is created by kubectl expose?

[FNSdev]

hello @benjaminjb

sure, here it is.

please note that I created it for my pgadmin resource, not the one from example, but I believe it shouldn't make a difference.

pgadmin

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PGAdmin
metadata:
  name: fdp
  namespace: default
spec:
  metadata:
    labels:
      sidecar.istio.io/inject: "false"
  dataVolumeClaimSpec:
    accessModes:
      - "ReadWriteOnce"
    resources:
      requests:
        storage: 1Gi
  serverGroups:
    - name: default
      postgresClusterSelector: {} # automatically detect all clusters in the namespace

service

-> % kubectl expose pod pgadmin-afb7e764-1e14-42d7-af80-8822065998de-0 --name test-pgadmin
service/test-pgadmin exposed

-> % kubectl get service test-pgadmin -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2023-12-20T15:41:57Z"
  labels:
    controller-revision-hash: pgadmin-afb7e764-1e14-42d7-af80-8822065998de-6f9dffbcbc
    postgres-operator.crunchydata.com/data: pgadmin
    postgres-operator.crunchydata.com/pgadmin: fdp
    postgres-operator.crunchydata.com/role: pgadmin
    sidecar.istio.io/inject: "false"
    statefulset.kubernetes.io/pod-name: pgadmin-afb7e764-1e14-42d7-af80-8822065998de-0
  name: test-pgadmin
  namespace: default
  resourceVersion: "106674245"
  uid: ea0abf32-c636-42d0-ac7a-5bb276afef42
spec:
  clusterIP: 10.0.112.172
  clusterIPs:
  - 10.0.112.172
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 5050
    protocol: TCP
    targetPort: 5050
  selector:
    controller-revision-hash: pgadmin-afb7e764-1e14-42d7-af80-8822065998de-6f9dffbcbc
    postgres-operator.crunchydata.com/data: pgadmin
    postgres-operator.crunchydata.com/pgadmin: fdp
    postgres-operator.crunchydata.com/role: pgadmin
    sidecar.istio.io/inject: "false"
    statefulset.kubernetes.io/pod-name: pgadmin-afb7e764-1e14-42d7-af80-8822065998de-0
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

[benjaminjb]

Thanks! Hmmm, that service spec looks good (or at least, it looks like mine). Just to double-check, are you running kubectl port-forward on the service after creating it?

[FNSdev]

Port forwarding works fine, that is not the issue.

I can't access pgadmin using service name as host from a different pod, for example.

That leads to not being able to make pgadmin part of our service mesh. If I add istio-proxy sidecar and create a Virtual Service, it doesn't work, because istio can't reach pgadmin via service.

And that issue is specific to pgadmin, we have no issues with other applications in the cluster. I was able to modify the selector to point to one of our applications (instead of pgadmin) - and it works fine.

[benjaminjb]

Ah, OK, I think I'm following and now I can reproduce this issue.

To restate the issue:

(a) inside the pgadmin pod, I can curl localhost:5050 and get a result
(b) inside the pgadmin pod or another pod, I cannot curl the IP of the pod (or the service)
(c) for comparison, inside a PG pod, I can curl localhost:5432; and I get the same result if I curl the IP (inside the same pod or a different pod).

So I think the issue isn't with the service (since I get the same bad result when I try to access the pod directly). I wonder if the issue is with how the image is serving pgadmin4. (Though why does it work fine for kubectl expose/kubectl port-forward?) I'm going to run some tests with other images.

[FNSdev]

Yeah, that's a nice summary.

I also got to the same conclusion about pgadmin image being the problem, but unfortunately I am not sure if there is something to be done on my side.

The only setting that caught my attention was PGADMIN_LISTEN_ADDRESS, but setting it to 0.0.0.0 didn't solve the issue.

[benjaminjb]

OK, @FNSdev, can you try putting this in your pgadmin spec:

  config:
    settings:
      DEFAULT_SERVER: "0.0.0.0"

It seems to work for me (i.e., I can curl localhost, curl the pod's IP, curl the service; and port-forward the service and access the website), but I feel uncomfortable with exposing a server like that. Really depends on the security of the network. Well, first, let's see if this even works for you.

My other thoughts on this issue are

(a) We have a ticket in our backlog re: the server on the pgadmin image; let me attach my notes from this conversation;
(b) I have minimal hands-on experience with Istio, but I wonder if we could work this problem from the other side: leave the pgadmin server serving on 127.0.0.1, and set Istio to route to localhost. (

(I'm not sure that's possible with Istio, though this article seems promising. If you want to explore that option, please share your Istio sidecar/virtual service specs.)

So, to recap:

• I think DEFAULT_SERVER: "0.0.0.0" might work in the short-term
• I wonder if changing the Istio configurations might offer a more secure solution
• I'm making notes for changing the image for a different solution

[FNSdev]

Thanks a lot @benjaminjb, I will try that out after returning from holidays. Merry Christmas 🙂

I will also check the other approach with Istio, didn't know that it is possible.

maybe you could also explain to me what is PGADMIN_LISTEN_ADDRESS (set as env variable) and how is it different from DEFAULT_SERVER that we set in the config?

[benjaminjb]

Hmmm, I think PGADMIN_LISTEN_ADDRESS would be the right var to change but only if you were running the pgadmin Docker image -- that's part of their entrypoint script.

Have a good break!

[andreasgeisslerdt]

Hi,
I have the same issue and tested successfully the proposed solution (this article).
Here is my setup:

---
apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PGAdmin
metadata:
  labels:
    app: my-pgadmin
  name: my-pgadmin
spec:
  dataVolumeClaimSpec:
    accessModes:
      - 'ReadWriteOnce'
    resources:
      requests:
        storage: 1Gi
  serverGroups:
    - name: supply
      postgresClusterSelector: {}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: my-pgadmin
  name: my-pgadmin
  namespace: onap
spec:
  ports:
    - name: pgadmin
      port: 5050
      protocol: TCP
      targetPort: pgadmin
  selector:
    postgres-operator.crunchydata.com/pgadmin: my-pgadmin
  type: ClusterIP
---
# fix for pgadmin (https://github.com/CrunchyData/postgres-operator/issues/3809)
apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: my-pgadmin
spec:
  workloadSelector:
    labels:
      postgres-operator.crunchydata.com/pgadmin: my-pgadmin
  ingress:
  - port:
      number: 5050
      protocol: TCP
      name: pgadmin
    defaultEndpoint: 127.0.0.1:5050

[tjmoore4]

@andreasgeisslerdt Thanks for providing that information. Glad to hear that the solution is working for you!

@FNSdev were you able to test the proposed solution in your environment?