[BUG] After upgrade to 4.9.0-1 (debian12) stuck with "Response Error" in log and "Wazuh dashboard server is not ready yet"

[DaLynxx]

Describe the bug

A clear and concise description of what the bug is.

To Reproduce
I upgraded from 4.8 (4.8.2 I believe), following https://documentation.wazuh.com/current/upgrade-guide/upgrading-central-components.html

Steps to reproduce the behavior:

1. Upgrade to 4.9.0.
2. Try to reach the dashboard login screen via browser
3. Web page shows "Wazuh dashboard server is not ready yet"

Expected behavior

1. Dashboard login screen when browsing to the server.

OpenSearch Version

wazuh-indexer/now 4.9.0-1 amd64 [installed,local]
wazuh-manager/now 4.9.0-1 amd64 [installed,local]

Dashboards Version

wazuh-dashboard/now 4.9.0-1 amd64 [installed,local]

Plugins

Please list all plugins currently enabled.

Not sure how I find out. I think I run more or less "out of the box" installation.

Screenshots

If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: debian 12, Linux 6.8.12-1-pve (LXC container on proxmox)
• Browser and version: Firefox 130.0

Additional context

I have updated before. Started with a 4.7.x version, got through a couple of 4.8.x steps. All worked well previously following the instructions.

wazuh-manager and wazuh-indexer and wazuh-dashboard all show "active (running)" when asking systemctl status .

However, wazuh-dashboard fills the "log" (journalctl) with

Sep 05 21:35:47 wazuh opensearch-dashboards[2809]: {"type":"log","@timestamp":"2024-09-05T19:35:47Z","tags":["error","opensearch","data"],"pid":2809,"message":"[ResponseError]: Response Error"}
Sep 05 21:35:49 wazuh opensearch-dashboards[2809]: {"type":"log","@timestamp":"2024-09-05T19:35:49Z","tags":["error","opensearch","data"],"pid":2809,"message":"[ResponseError]: Response Error"}
Sep 05 21:35:52 wazuh opensearch-dashboards[2809]: {"type":"log","@timestamp":"2024-09-05T19:35:52Z","tags":["error","opensearch","data"],"pid":2809,"message":"[ResponseError]: Response Error"}
Sep 05 21:35:54 wazuh opensearch-dashboards[2809]: {"type":"log","@timestamp":"2024-09-05T19:35:54Z","tags":["error","opensearch","data"],"pid":2809,"message":"[ResponseError]: Response Error"}
Sep 05 21:35:57 wazuh opensearch-dashboards[2809]: {"type":"log","@timestamp":"2024-09-05T19:35:57Z","tags":["error","opensearch","data"],"pid":2809,"message":"[ResponseError]: Response Error"}

Restarting wazuh-dashboard generates the following log-flow

Sep 05 21:37:14 wazuh systemd[1]: Started wazuh-dashboard.service - wazuh-dashboard.
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-service"],"pid":3109,"message":"Plugin \"dataSourceManagement\" has been disabled since the following >
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-service"],"pid":3109,"message":"Plugin \"applicationConfig\" is disabled."}
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-service"],"pid":3109,"message":"Plugin \"cspHandler\" is disabled."}
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-service"],"pid":3109,"message":"Plugin \"dataSource\" is disabled."}
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-service"],"pid":3109,"message":"Plugin \"visTypeXy\" is disabled."}
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","plugins-system"],"pid":3109,"message":"Setting up [48] plugins: [usageCollection,opensearchDashboardsUsageColl>
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["info","savedobjects-service"],"pid":3109,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch>
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["error","opensearch","data"],"pid":3109,"message":"[ResponseError]: Response Error"}
Sep 05 21:37:16 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:16Z","tags":["error","savedobjects-service"],"pid":3109,"message":"Unable to retrieve version information from OpenSearch nodes."}
Sep 05 21:37:19 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:19Z","tags":["error","opensearch","data"],"pid":3109,"message":"[ResponseError]: Response Error"}
Sep 05 21:37:21 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:21Z","tags":["error","opensearch","data"],"pid":3109,"message":"[ResponseError]: Response Error"}
Sep 05 21:37:24 wazuh opensearch-dashboards[3109]: {"type":"log","@timestamp":"2024-09-05T19:37:24Z","tags":["error","opensearch","data"],"pid":3109,"message":"[ResponseError]: Response Error"}

Curl to the dashboard does not respond either.

Content of opensearch_dashboards.yml

server.host: 0.0.0.0
server.port: 443
opensearch.hosts: https://127.0.0.1:9200
opensearch.ssl.verificationMode: certificate
opensearch.requestHeadersAllowlist: ["securitytenant","Authorization"]
opensearch_security.multitenancy.enabled: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
server.ssl.enabled: true
server.ssl.key: "/etc/wazuh-dashboard/certs/wazuh-dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/wazuh-dashboard.pem"
opensearch.ssl.certificateAuthorities: ["/etc/wazuh-dashboard/certs/root-ca.pem"]
uiSettings.overrides.defaultRoute: /app/wz-home
opensearch_security.cookie.secure: true

netstat

root@wazuh:/# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:1515            0.0.0.0:*               LISTEN      603/wazuh-authd
tcp        0      0 0.0.0.0:1514            0.0.0.0:*               LISTEN      814/wazuh-remoted
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      3491/node
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      418/master
tcp        0      0 0.0.0.0:55000           0.0.0.0:*               LISTEN      555/python3
tcp6       0      0 :::22                   :::*                    LISTEN      1/init
tcp6       0      0 ::1:25                  :::*                    LISTEN      418/master
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      164/java
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      164/java
udp        0      0 0.0.0.0:68              0.0.0.0:*                           89/dhclient

[DaLynxx]

Hm, just added the netstat output.
Weird. Why is tcp6 listed for the 127.0.0.1:9200 and 9300

[DaLynxx]

9200 and 9300 is tied to the indexer rather than the dashboard, or?

[kslintubs]

Did the update too on Ubuntu 24.04.1 LTS , same error message

[kullarkert]

I'm also having exact same issue and same messages.

I have Ubuntu 22.04.4 LTS and did apt-get upgrade

Some log files

cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

{"date":"2024-09-05T12:00:19.865Z","level":"error","location":"monitoring:fetchAllAgentsFromApiHost","message":"ApiID: default, Error request with offset/limit 0/500: Request failed with status code 400"}
{"date":"2024-09-05T12:00:19.881Z","level":"error","location":"monitoring:fetchAllAgentsFromApiHost","message":"ApiID: default, Error request with offset/limit 0/500: Request failed with status code 400"}
{"date":"2024-09-05T12:00:19.895Z","level":"error","location":"monitoring:fetchAllAgentsFromApiHost","message":"ApiID: default, Error request with offset/limit 0/500: Request failed with status code 400"}
{"date":"2024-09-05T12:00:19.910Z","level":"error","location":"monitoring:fetchAllAgentsFromApiHost","message":"ApiID: default, Error request with offset/limit 0/500: Request failed with status code 400"}

cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

[2024-09-06T07:19:59,574][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:19:59,576][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37318}], closing connection
[2024-09-06T07:19:59,581][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:19:59,583][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37326}], closing connection
[2024-09-06T07:19:59,588][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:19:59,590][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37340}], closing connection
[2024-09-06T07:20:02,069][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:20:02,072][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37352}], closing connection
[2024-09-06T07:20:02,077][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:20:02,078][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37362}], closing connection
[2024-09-06T07:20:02,085][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:20:02,087][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37372}], closing connection
[2024-09-06T07:20:02,092][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
[2024-09-06T07:20:02,094][WARN ][o.o.t.TcpTransport       ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:37380}], closing connection
[2024-09-06T07:20:02,499][WARN ][o.o.p.c.u.JsonConverter  ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])

[BooopLJ]

I also have the same problem after updating to version 4.9.0 on Ubuntu 22.04.4 LTS

[kullarkert]

This command did the trick. Got answer from slack channel.

/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p '<Secr3tP4ssw*rd>'

In all services in one server deployment, changing the password should solve the issue.

[tkolaski]

in wazuh-cluster.log I had entries:

Authentication finally failed for kibanaserver from 127.0.0.1:51086

I found:

https://groups.google.com/g/wazuh/c/rdCF0MBR6oU

It helped

[BooopLJ]

@kullarkert thank you for suggesting a solution, but I still have this error. The server was reset after the password was changed.
opensearch-dashboards[5609]: {"type":"log","@timestamp":"2024-09-06T08:03:00Z","tags":["error","opensearch","data"],"pid":5609,"message":"[ResponseError]: Response Error"}

[PeterKnotek]

Hi

have problem after update

service wazuh-dashboard status
● wazuh-dashboard.service - wazuh-dashboard
     Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-09-06 17:07:15 CEST; 13min ago
   Main PID: 14160 (node)
      Tasks: 11 (limit: 9371)
     Memory: 166.8M
        CPU: 18.727s
     CGroup: /system.slice/wazuh-dashboard.service
             └─14160 /usr/share/wazuh-dashboard/node/bin/node /usr/share/wazuh-dashboard/src/cli/dist -c /etc/wazuh-dashboard/opensearch_dashboards.yml

Sep 06 17:20:04 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:04Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:06 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:06Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:09 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:09Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:11 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:11Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:14 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:14Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:16 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:16Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:19 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:19Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:21 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:21Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:24 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:24Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}
Sep 06 17:20:26 wazuh opensearch-dashboards[14160]: {"type":"log","@timestamp":"2024-09-06T15:20:26Z","tags":["error","opensearch","data"],"pid":14160,"message":"[ResponseError]: Response Error"}

Can help me

Thanks

[asteriscos]

As part of Wazuh 4.9.0, we published Wazuh dashboard revision 1 packages:
wazuh-dashboard-4.9.0-1.deb
wazuh-dashboard-4.9.0-1.rpm

These packages introduced a change in the keystore location to /usr/share/wazuh-dashboard/config, which caused issues when upgrading from previous versions, leading to the "Dashboard is not ready" error. Additionally, the kibanaserver user displayed failed authentication messages in the indexer logs.

To resolve this, we have rolled out Wazuh dashboard revision 2 of these packages:
wazuh-dashboard-4.9.0-2.deb
wazuh-dashboard-4.9.0-2.rpm

Wazuh 4.x repositories are available again.

Important: If you upgraded using wazuh-dashboard-4.9.0-1 and afterward changed the passwords, follow these steps:

1. Install the new Wazuh dashboard revision 2 packages.
2. Backup your current keystore:
   cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak
3. Backup the keystore from the new location:
   cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak
4. Move the keystore to the correct location:
   mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore
5. Restart the Wazuh dashboard.

[borkedporcupine]

As part of Wazuh 4.9.0, we published Wazuh dashboard revision 1 packages: wazuh-dashboard-4.9.0-1.deb wazuh-dashboard-4.9.0-1.rpm

These packages introduced a change in the keystore location to /usr/share/wazuh-dashboard/config, which caused issues when upgrading from previous versions, leading to the "Dashboard is not ready" error. Additionally, the kibanaserver user displayed failed authentication messages in the indexer logs.

To resolve this, we have rolled out Wazuh dashboard revision 2 of these packages: wazuh-dashboard-4.9.0-2.deb wazuh-dashboard-4.9.0-2.rpm

Wazuh 4.x repositories are available again.

Important: If you upgraded using wazuh-dashboard-4.9.0-1 and afterward changed the passwords, follow these steps:

1. Install the new Wazuh dashboard revision 2 packages.

2. Backup your current keystore:
   `cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak`

3. Backup the keystore from the new location:
   `cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak`

4. Move the keystore to the correct location:
   `mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore`

5. Restart the Wazuh dashboard.

I can confirm that I was able to upgrade to 4.9.0 now without issue. Thanks for the fix.

[stefardi]

As part of Wazuh 4.9.0, we published Wazuh dashboard revision 1 packages: wazuh-dashboard-4.9.0-1.deb wazuh-dashboard-4.9.0-1.rpm
These packages introduced a change in the keystore location to /usr/share/wazuh-dashboard/config, which caused issues when upgrading from previous versions, leading to the "Dashboard is not ready" error. Additionally, the kibanaserver user displayed failed authentication messages in the indexer logs.
To resolve this, we have rolled out Wazuh dashboard revision 2 of these packages: wazuh-dashboard-4.9.0-2.deb wazuh-dashboard-4.9.0-2.rpm
Wazuh 4.x repositories are available again.
Important: If you upgraded using wazuh-dashboard-4.9.0-1 and afterward changed the passwords, follow these steps:

1. Install the new Wazuh dashboard revision 2 packages.

2. Backup your current keystore:
   `cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak`

3. Backup the keystore from the new location:
   `cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak`

4. Move the keystore to the correct location:
   `mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore`

5. Restart the Wazuh dashboard.

I can confirm that I was able to upgrade to 4.9.0 now without issue. Thanks for the fix.

this is working for me. Thank you.