bug: failed to delete ingress rule if it was prepared for the external service

[vlashchanka]

Describe the bug

Hi, I was working on preparing the admin part of my hosted swiftwave instance using this guildeline: https://github.com/orgs/swiftwave-org/discussions/779

I however misconfigured originally the rules for the subdomain to live on 80 and 443 port, later wanted to use HTTPS redirect feature, but it failed as there was a rule defined on the port 80, so I decided to delete both rules, however nothing happens and I get the failed status:

I successfully prepared everything which I need at console.domain.dom, but still want to make a cleanup and avoid any change I could go to the http version. + now I can't delete the admin.domain.dom domain.

Could you , please, help?

cc: @tanmoysrt

Are you working on this issue?

No

[tanmoysrt]

You can delete the domain at port 80, then try to enable https redirection.

Let me know if you face issue there

[vlashchanka]

@tanmoysrt hey, I tried to delete both rules for 80 and 443 ports, no luck.
Domain deletion doesn't work, due to the ingress rules associated:

[tanmoysrt]

Can you try to delete the ingress rule on port 80 ?
Let me know what's the error it's showing

[vlashchanka]

@tanmoysrt could you please elaborate more? Do you mean from the UI? If yes, as I wrote before, doesn't work.

In browser I see 200 status and {"data":{"deleteIngressRule":true}} response.

Then on ui it is shown as Failed and graphql returns back this:

            {
                "id": 8,
                "status": "failed",
                "protocol": "http",
                "domain": {
                    "name": "admin.my.domain",
                    "__typename": "Domain"
                },
                "httpsRedirect": false,
                "port": 80,
                "targetType": "externalService",
                "externalService": "172.17.0.1",
                "application": {
                    "name": "",
                    "__typename": "Application"
                },
                "authenticationType": "none",
                "basicAuthAccessControlListName": "",
                "targetPort": 3333,
                "__typename": "IngressRule"
            },

[tanmoysrt]

@vlashchanka Try one thing

1. For both 80, 443 ingress rule, recreate the ingress rules by Actions > Recreate & Fix
2. Wait for changing the status for both of the rules
3. Then, delete both of them one by one
4. Then again add whatever ingress rule you want.

Let me knoe, if it's failing also

[vlashchanka]

@tanmoysrt , tried. Nothing changed:

• recreated, became green 🟢
• tried to delete, first status deleting
• after some time failed 🔴

For both.

[tanmoysrt]

@vlashchanka
Then, I need to check the deletion flow, maybe some unhandled case.

I need few infos to find the root cause -

1. Can you provide a list of steps to reproduce/simulate the scenario

2. SSH to your server and grab the haproxy config by

   cat /var/lib/swiftwave/haproxy/haproxy.cfg

   ⚠️ Please obfuscate domain name, service names in config before sharing

3. Does swiftwave able to generate SSL cert for your domain , what's the status in domain list ?

[vlashchanka]

I used this guideline to configure following the described steps:
https://github.com/orgs/swiftwave-org/discussions/779#discussioncomment-9768040

I think this part is suspicious to start from, adding external services:

I before was trying to setup the application ingress rules and they were deleted without any issues, I would say.

Here is the config, hopefully I removed everything important 😄 :

global
  chroot /var/lib/haproxy
  user haproxy
  group haproxy
  master-worker
  maxconn 100000
  stats socket /var/run/haproxy.sock user haproxy group haproxy mode 660 level admin expose-fd listeners

defaults unnamed_defaults_1
  mode http
  maxconn 4000
  log global
  option forwardfor
  option tcp-smart-accept
  timeout http-request 10s
  timeout check 10s
  timeout connect 10s
  timeout client 1m
  timeout queue 1m
  timeout server 1m
  timeout http-keep-alive 10s
  retries 3
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http

userlist 01JC3YRW4AS4N0RDPDK84GM3RR

resolvers docker
  nameserver ns1 127.0.0.11:53
  hold nx 30s
  hold obsolete 30s
  hold other 30s
  hold refused 30s
  hold timeout 30s
  hold valid 10s
  timeout resolve 2s
  timeout retry 2s
  accepted_payload_size 8192
  resolve_retries 5

frontend fe_http from unnamed_defaults_1
  mode http
  bind :80
  acl letsencrypt-acl path_beg /.well-known
  http-request redirect scheme https code 301 if { hdr(host) -i redacted1.redacted.domain }
  http-request redirect scheme https code 301 if { hdr(host) -i redacted2.redacted.domain }
  use_backend letsencrypt_backend if letsencrypt-acl
  use_backend be_172.17.0.1_3333 if { hdr(host) -i admin.redacted.domain }
  default_backend error_backend

frontend fe_https from unnamed_defaults_1
  mode http
  bind :443 ssl crt /etc/haproxy/ssl/ alpn h2,http/1.1
  acl letsencrypt-acl path_beg /.well-known
  http-request set-header X-Forwarded-Proto https
  use_backend letsencrypt_backend if letsencrypt-acl
  use_backend be_172.17.0.1_3333 if { hdr(host) -i redacted1.redacted.domain }
  use_backend be_redacted2_3000 if { hdr(host) -i redacted2.redacted.domain }
  use_backend be_172.17.0.1_3333 if { hdr(host) -i admin.redacted.domain }
  default_backend error_backend

backend be_172.17.0.1_3333 from unnamed_defaults_1
  balance roundrobin
  server-template 172.17.0.1_container- 1 172.17.0.1:3333 no-check init-addr none resolvers docker

backend be_redacted2_3000 from unnamed_defaults_1
  balance roundrobin
  server-template redacted2_container- 1 redacted2:3000 no-check init-addr none resolvers docker

backend error_backend from unnamed_defaults_1
  mode http
  http-request deny deny_status 502

backend letsencrypt_backend from unnamed_defaults_1
  option httpchk
  http-check send meth GET uri /healthcheck hdr Host "$SWIFTWAVE_SERVICE_ADDRESS"
  http-check expect status 200
  http-request set-header Host "$SWIFTWAVE_SERVICE_ADDRESS"
  server swiftwave_service_https "$SWIFTWAVE_SERVICE_ENDPOINT" check ssl verify required ca-file /etc/ssl/certs/ca-certificates.crt check-sni "$SWIFTWAVE_SERVICE_ADDRESS" sni str("$SWIFTWAVE_SERVICE_ADDRESS")
  server swiftwave_service_http "$SWIFTWAVE_SERVICE_ENDPOINT" check

program api
  command /dataplaneapi.sh
  no option start-on-reload

@tanmoysrt

[vlashchanka]

@tanmoysrt, yes, those ssl work fine, I guess:

[tanmoysrt]

@vlashchanka thanks for providing the info.
Will try to simulate the issue.
I will share the updates here

[vlashchanka]

@tanmoysrt also hard to say if this is related issue (sharing just in case it could help):

Image : redacted.1redacted.domain:3334/6b485e80-870d-4fe4-a982-a75edd5d22c8:e6221bd3-9a44-4bf5-95dc-0b3b142c05c6
Started pushing image to registry
Get "https://redacted1.redacted.domain:3334/v2/": http: server gave HTTP response to HTTPS client


Image push failed
Failed to build application
image push failed

Seems that my image registry is not working too, noticed now when I tried to redeploy application. Is it also a known issue, btw? Should something be changed in some config?

[tanmoysrt]

Yeah it's normal, as you have changed the config.yml and your domain mappings in ingress not working correctly

[tanmoysrt]

You can just revert the overriden config in swiftwave config to fix it

[vlashchanka]

You can just revert the overriden config in swiftwave config to fix it - how to do this? Could you help?