v2.2.2 connection refused when loading pads

[Artim96]

Describe the bug
Updated from 2.1.1 to 2.2.2, the admin page opens without issues, but loading any pad results in 2024/08/28 11:19:12 [error] 985096#985096: *14 connect() failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: pad.domain.de, request: "GET /socket.io/?EIO=4&transport=websocket HTTP/1.1", upstream: "http://127.0.0.1:9001/socket.io/?EIO=4&transport=websocket", host: "pad.domain.de"

Also, during running bin/run.sh the script claims issues with the socketTransportProtocols setting and falling back to defaults. But for all I can tell, it's identical to what's in the settings template: "socketTransportProtocols" : ["websocket", "polling"],

Full log from running the pad from CLI: etherpad_log.txt

To Reproduce
Steps to reproduce the behavior:

1. Update to v2.2.2
2. open any pad

Expected behavior
Pad opens

Screenshots
If applicable, add screenshots to help explain your problem.

Server (please complete the following information):

• Etherpad version: v2.2.2
• OS: Debian 12.6
• Node.js version (node --version): v20.17.0
• npm version (npm --version): 10.8.2
• Is the server free of plugins: yes

[SamTV12345]

Describe the bug Updated from 2.1.1 to 2.2.2, the admin page opens without issues, but loading any pad results in 2024/08/28 11:19:12 [error] 985096#985096: *14 connect() failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: pad.domain.de, request: "GET /socket.io/?EIO=4&transport=websocket HTTP/1.1", upstream: "http://127.0.0.1:9001/socket.io/?EIO=4&transport=websocket", host: "pad.domain.de"

Also, during running bin/run.sh the script claims issues with the socketTransportProtocols setting and falling back to defaults. But for all I can tell, it's identical to what's in the settings template: "socketTransportProtocols" : ["websocket", "polling"],

Full log from running the pad from CLI: etherpad_log.txt

To Reproduce Steps to reproduce the behavior:

1. Update to v2.2.2
2. open any pad

Expected behavior Pad opens

Screenshots If applicable, add screenshots to help explain your problem.

Server (please complete the following information):

• Etherpad version: v2.2.2
• OS: Debian 12.6
• Node.js version (node --version): v20.17.0
• npm version (npm --version): 10.8.2
• Is the server free of plugins: yes

Hi the bug with the invalid socketIOTransportProtocols is already fixed. It is not a bug in your configuration. For reproduction I updated my pad server and updated to v2.2.2 . So far it works without a problem: https://pad.samtv.fyi/p/test

[Artim96]

Hi the bug with the invalid socketIOTransportProtocols is already fixed. It is not a bug in your configuration. For reproduction I updated my pad server and updated to v2.2.2 . So far it works without a problem: https://pad.samtv.fyi/p/test

Then how come I'm getting that exact issue with v2.2.2? If it keeps appearing it kinda can't be fixed, don't you think?

[SamTV12345]

Hi the bug with the invalid socketIOTransportProtocols is already fixed. It is not a bug in your configuration. For reproduction I updated my pad server and updated to v2.2.2 . So far it works without a problem: https://pad.samtv.fyi/p/test

Then how come I'm getting that exact issue with v2.2.2? If it keeps appearing it kinda can't be fixed, don't you think?

Oh might be a misunderstanding. I fixed the message in the develop branch. I deployed my test Etherpad to demonstrate that it works for me with v2.2 with connecting to pads. Maybe something is wrong on your side? Was there a change to your reverse proxy?

[Artim96]

Hi the bug with the invalid socketIOTransportProtocols is already fixed. It is not a bug in your configuration. For reproduction I updated my pad server and updated to v2.2.2 . So far it works without a problem: https://pad.samtv.fyi/p/test

Then how come I'm getting that exact issue with v2.2.2? If it keeps appearing it kinda can't be fixed, don't you think?

Oh might be a misunderstanding. I fixed the message in the develop branch. I deployed my test Etherpad to demonstrate that it works for me with v2.2 with connecting to pads. Maybe something is wrong on your side? Was there a change to your reverse proxy?

nope, nothing. Just the usual path of git pull origin and git checkout tags/v2.2.2, followed by bin/run.sh (obviously all as the etherpad user).

[SamTV12345]

Hi the bug with the invalid socketIOTransportProtocols is already fixed. It is not a bug in your configuration. For reproduction I updated my pad server and updated to v2.2.2 . So far it works without a problem: https://pad.samtv.fyi/p/test

Then how come I'm getting that exact issue with v2.2.2? If it keeps appearing it kinda can't be fixed, don't you think?

Oh might be a misunderstanding. I fixed the message in the develop branch. I deployed my test Etherpad to demonstrate that it works for me with v2.2 with connecting to pads. Maybe something is wrong on your side? Was there a change to your reverse proxy?

nope, nothing. Just the usual path of git pull origin and git checkout tags/v2.2.2, followed by bin/run.sh (obviously all as the etherpad user).

Could you give me access to your environment? Would be worthy to check why this happened. Maybe it is an undiscovered bug in Etherpad.

[Artim96]

Could you give me access to your environment? Would be worthy to check why this happened. Maybe it is an undiscovered bug in Etherpad.

Sorry, but that's out of the question. There's just way too much other stuff on the same system I can't just have anyone poke through.

[SamTV12345]

Could you give me access to your environment? Would be worthy to check why this happened. Maybe it is an undiscovered bug in Etherpad.

Sorry, but that's out of the question. There's just way too much other stuff on the same system I can't just have anyone poke through.

Maybe try to replicate the issue somehow. I need some sort of reproducible system where I can safely check logs, debug code etc.

[Artim96]

Could you give me access to your environment? Would be worthy to check why this happened. Maybe it is an undiscovered bug in Etherpad.

Sorry, but that's out of the question. There's just way too much other stuff on the same system I can't just have anyone poke through.

Maybe try to replicate the issue somehow. I need some sort of reproducible system where I can safely check logs, debug code etc.

Question is how much is needed to reproduce the issue. The actual content of the pads is saved in a mariadb, so if it's a setup issue, theoretically handing you an archive of the installation - with the config file redacted - including everything used to bring it online, should do the trick, doesn't it?

[SamTV12345]

Could you give me access to your environment? Would be worthy to check why this happened. Maybe it is an undiscovered bug in Etherpad.

Sorry, but that's out of the question. There's just way too much other stuff on the same system I can't just have anyone poke through.

Maybe try to replicate the issue somehow. I need some sort of reproducible system where I can safely check logs, debug code etc.

Question is how much is needed to reproduce the issue. The actual content of the pads is saved in a mariadb, so if it's a setup issue, theoretically handing you an archive of the installation - with the config file redacted - including everything used to bring it online, should do the trick, doesn't it?

Yes let's try it that way. The database shouldn't be a problem we have tests for every used database but I'll give it a try :)

[Artim96]

I've created an archive, it should have preserved all permissions and ownerships except where I had to edit files. Is there a way to privately message you the link?

[SamTV12345]

I've created an archive, it should have preserved all permissions and ownerships except where I had to edit files. Is there a way to privately message you the link?

Thanks. I'll have a look after work. You can email me the link to samelus1998@outlook.de

[SamTV12345]

I managed to start Etherpad on bare metal with your configuration. I used the following configuration for the database:

# Use root/example as user/password credentials
version: '3.1'

services:

  db:
    image: mysql
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: EtherpadProto3
    # (this is just an example, not intended to be a production configuration)
      MYSQL_USER: etherpad_user
      MYSQL_DATABASE: etherpad_db
      MYSQL_PASSWORD: EtherpadProto3
    ports:
      - "3306:3306"

I'll continue now with the reverse proxy configuration.

[SamTV12345]

I've created an archive, it should have preserved all permissions and ownerships except where I had to edit files. Is there a way to privately message you the link?

Do you have instructions on how to setup a new Etherpad instance with your nginx configuration? My nginx is crashing:

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/09/09 17:58:39 [emerg] 1#1: dlopen() "/etc/nginx/modules/ngx_http_brotli_filter_module.so" failed (/etc/nginx/modules/ngx_http_brotli_filter_module.so: cannot open shared object file: No such file or directory) in /etc/nginx/nginx.conf:6
nginx: [emerg] dlopen() "/etc/nginx/modules/ngx_http_brotli_filter_module.so" failed (/etc/nginx/modules/ngx_http_brotli_filter_module.so: cannot open shared object file: No such file or directory) in /etc/nginx/nginx.conf:6

[Artim96]

You are missing the libraries for brotli support. Either comment out the two lines towards the beginning of the nginx conf or follow this guide: https://linuxhint.com/enable-brotli-compression-nginx/

[SamTV12345]

You are missing the libraries for brotli support. Either comment out the two lines towards the beginning of the nginx conf or follow this guide: https://linuxhint.com/enable-brotli-compression-nginx/

Thanks that fixed the issue. I'll continue debugging this.

[Artim96]

The issue is still present in v2.2.5.

[SamTV12345]

Is that your problem?

[SamTV12345]

The issue is still present in v2.2.5.

This works. I also don't know why you don't have a server section in your configuration:

events {
  worker_connections 1024;
}



http {
  server {
    listen 80;
    server_name 192.168.2.50;

    location / {
        proxy_set_header X-WEBAUTH-USER testuseradmin;
        proxy_pass http://192.168.2.50:8000;
    }
}
}
nginx-basic-auth:~# cat nginx.conf 
events {
  worker_connections 1024;
}



http {
  server {
    listen 80;
    server_name 192.168.2.50;

    location / {
        proxy_set_header X-WEBAUTH-USER testuseradmin;
        proxy_pass http://192.168.2.50:8000;
    }
}
}

[Artim96]

Is that your problem?

I'll have to double check, but I don't think it shows any kind of error message.

[Artim96]

The issue is still present in v2.2.5.

This works.

Not for me, same issue of not loading any pads. I'll try to set up etherpad via docker, but I won't be able before Thursday.

I also don't know why you don't have a server section in your configuration:

events {
  worker_connections 1024;
}



http {
  server {
    listen 80;
    server_name 192.168.2.50;

    location / {
        proxy_set_header X-WEBAUTH-USER testuseradmin;
        proxy_pass http://192.168.2.50:8000;
    }
}
}
nginx-basic-auth:~# cat nginx.conf 
events {
  worker_connections 1024;
}



http {
  server {
    listen 80;
    server_name 192.168.2.50;

    location / {
        proxy_set_header X-WEBAUTH-USER testuseradmin;
        proxy_pass http://192.168.2.50:8000;
    }
}
}

Because that's only the general nginx config, the server sections should be under sites-available.

[SamTV12345]

Is that your problem?

I'll have to double check, but I don't think it shows any kind of error message.

But you see something? So you get the white page with the pad not loading right? Do you have a playground or something so I can navigate to the site and check my browser logs. It could be that the new js files don't load correctly but I'd like to be certain.

[SamTV12345]

The issue is still present in v2.2.5.

This works.

Not for me, same issue of not loading any pads. I'll try to set up etherpad via docker, but I won't be able before Thursday.

I also don't know why you don't have a server section in your configuration:

events {
  worker_connections 1024;
}



http {
  server {
    listen 80;
    server_name 192.168.2.50;

    location / {
        proxy_set_header X-WEBAUTH-USER testuseradmin;
        proxy_pass http://192.168.2.50:8000;
    }
}
}
nginx-basic-auth:~# cat nginx.conf 
events {
  worker_connections 1024;
}



http {
  server {
    listen 80;
    server_name 192.168.2.50;

    location / {
        proxy_set_header X-WEBAUTH-USER testuseradmin;
        proxy_pass http://192.168.2.50:8000;
    }
}
}

Because that's only the general nginx config, the server sections should be under sites-available.

I added most of the stuff from the sites-available folder. I'd ditch the other things for a first iteration. Also on your production instance you deny all json documents which causes your instance not having support for PWA apps which are a really nice feature to bookmark things and have a desktop like experience

events {}

http {

 map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
    }

server {
    listen 80;
    server_name your_domain.com;

    location / {
        proxy_pass http://192.168.2.122:9001;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }


    location /socket {
                 rewrite  ^/socket/(.*)  /$1 break; #used to send request to base url
                proxy_pass http://192.168.2.122:9001;
                 proxy_redirect off;
                 proxy_pass_request_headers on;
                 proxy_set_header X-Real-IP $remote_addr;
                 proxy_set_header Host $http_host;
                 proxy_set_header X-NginX-Proxy true;
                 proxy_set_header X-Forwarded-Host $host;
                 proxy_set_header X-Forwarded-Server $host;
                 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                 proxy_http_version 1.1;
                 proxy_set_header Upgrade $http_upgrade;
                 proxy_set_header Connection "upgrade";
                 proxy_set_header Host $host;
        }


}
} 

[Artim96]

But you see something? So you get the white page with the pad not loading right? Do you have a playground or something so I can navigate to the site and check my browser logs. It could be that the new js files don't load correctly but I'd like to be certain.

I wish I'd get that far, but no, I only see this loading screen:

As you can see, no error message is displayed (the blue color is from dark reader, but even with no extensions enabled and in various browsers the result is the same and it only happens since after 2.1.1.)

[Artim96]

I added most of the stuff from the sites-available folder. I'd ditch the other things for a first iteration.

That shouldn't make any differences unless there was as big of a change how various parts from the etherpad are reached like with the transition to v2. In that case I could only urge to keep things the way they are as they do work, and not have users constantly rewrite their http server config. And if you remember, the last time we tried rewriting that to solve the transition to v2, things broke even harder. Also, back then there actually were error messages. This time around there has yet a way to be found to get any messages about what's going wrong.

Also on your production instance you deny all json documents which causes your instance not having support for PWA apps which are a really nice feature to bookmark things and have a desktop like experience

In general yes, but I do not see any benefit on any of our web services, especially not with etherpad. The whole point of it is to be an online collaboration tool. So a desktop-like experience is at no point useful or even desired. Also, the way we use etherpad, it wouldn't get used like that either way, as it's more or less just used as a more comfortable tool to write formatted text for another system. It pushes a template to a pad where it creates the link for via the APIKEY, it's getting filled out and afterward imported back into the other system via the same path. It's almost never used in a way that one pad is reused on several occasions over and over again. It's just for several people to be writing on the same text file.

[Artim96]

events {}

http {

 map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
    }

server {
    listen 80;
    server_name your_domain.com;

    location / {
        proxy_pass http://192.168.2.122:9001;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }


    location /socket {
                 rewrite  ^/socket/(.*)  /$1 break; #used to send request to base url
                proxy_pass http://192.168.2.122:9001;
                 proxy_redirect off;
                 proxy_pass_request_headers on;
                 proxy_set_header X-Real-IP $remote_addr;
                 proxy_set_header Host $http_host;
                 proxy_set_header X-NginX-Proxy true;
                 proxy_set_header X-Forwarded-Host $host;
                 proxy_set_header X-Forwarded-Server $host;
                 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                 proxy_http_version 1.1;
                 proxy_set_header Upgrade $http_upgrade;
                 proxy_set_header Connection "upgrade";
                 proxy_set_header Host $host;
        }


}
} 

It seems you where right. No idea why there never was a clear error message, but I just adapted this config and it works. Lets hope it will also do so with the next few versions.