Error: JsonRpcConnection: Error while sending JSON-RPC message for identity

[cde0]

Describe the bug

I start icinga but i get this Error: JsonRpcConnection: Error while sending JSON-RPC message for identity 'satellite2'
Error: Broken pipe
And it show that Server is Down.

Topology: master -> satellite -> monitor_server

[2022-01-03 14:34:59 +0000] information/ApiListener: Finished syncing endpoint 'satellite2' in zone 'satellite2'.
[2022-01-03 14:34:59 +0000] warning/JsonRpcConnection: Error while sending JSON-RPC message for identity 'satellite2'
Error: Broken pipe

Stacktrace:
 0# __cxa_throw in /usr/lib64/icinga2/sbin/icinga2
 1# boost::asio::detail::coro_async_result<boost::asio::executor_binder<void (*)(), boost::asio::executor>, unsigned long>::get() in /usr/lib64/icinga2/sbin/icinga2
 2# boost::asio::async_result<std::decay<boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >&>::type, void (boost::system::error_code, unsigned long)>::return_type boost::asio::buffered_write
_stream<icinga::UnbufferedAsioTlsStream>::async_flush<boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >&>(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::as
io::executor> >&) in /usr/lib64/icinga2/sbin/icinga2
 3# icinga::JsonRpcConnection::WriteOutgoingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) in /usr/lib64/icinga2/sbin/icinga2
 4# 0x0000000000AFE697 in /usr/lib64/icinga2/sbin/icinga2
 5# 0x0000000000B06D89 in /usr/lib64/icinga2/sbin/icinga2
 6# make_fcontext in /lib64/libboost_context.so.1.69.0
[2022-01-03 14:34:59 +0000] warning/JsonRpcConnection: API client disconnected for identity 'satellite2'
[2022-01-03 14:34:59 +0000] warning/ApiListener: Removing API client for endpoint 'satellite2'. 0 API clients left.

To Reproduce

IDK :)

Expected behavior

Work, With out a crash

Screenshots

If applicable, add screenshots to help explain your problem.

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version):
  icinga2 - The Icinga 2 network monitoring daemon (version: 2.13.2-1)

Copyright (c) 2012-2022 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later https://gnu.org/licenses/gpl2.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
Platform: CentOS Linux
Platform version: 7 (Core)
Kernel: Linux
Kernel version: 3.10.0-1160.25.1.el7.x86_64
Architecture: x86_64

Build information:
Compiler: GNU 4.8.5
Build host: runner-hh8q3bz2-project-322-concurrent-0
OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017

• Operating System and version: CentOS 7
• Enabled features (icinga2 feature list):
  Enabled features: api checker mainlog
• Icinga Web 2 version and modules (System - About):

Icinga Web 2 Version
2.9.5
Git commit
053971c99dc1a4510beb64a888ea695cc14032dc
PHP Version
7.2.24-0ubuntu0.18.04.10
Git commit date
2021-11-18
Copyright
© 2013-2022 Icinga GmbH

• Config validation (icinga2 daemon -C):
  is return with out Error
• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

[loosi]

it seems to be an issue with icinga2 master > 2.13.0 in combination with older certain slaves/clients (older OS but up-to-date icinga2 version).
I downgraded our master to 2.12.5 and now its working again. Nothing to hold on forever, but maybe someone else can figure out the issue (possibly something related to #8718 ?)

[Al2Klimov]

@cde0 Indeed, what are your Icinga 2 versions per node?

[Wintermute2k6]

[ref/NC] #781426
Agent Version: v2.11.2
Master Version: v2.13.6-1

[Al2Klimov]

No 2.12 satellite in-between? That’s not supported!

[Al2Klimov]

it seems to be an issue with icinga2 master > 2.13.0 in combination with older certain slaves/clients (older OS but up-to-date icinga2 version).

Please be more specific on the versions.

the issue (possibly something related to #8718 ?)

Unlikely. Since v2.11, only TLSv1.2 is supported. That support hasn't been dropped by #8718.

[Al2Klimov]

I'm afraid such log messages indicate a remotely closed connection and only the respective peer's log at that time can say why it's been closed.

[ulyszk]

It seems I hit something similar problem with these icinga clients:

Debug log is like:

[2025-04-08 10:57:07 +0200] information/ApiListener: Finished sending replay log for endpoint 'mon.domain' in zone 'master'.
[2025-04-08 10:57:07 +0200] information/ApiListener: Finished syncing endpoint 'mon.domain' in zone 'master'.
[2025-04-08 10:57:07 +0200] notice/JsonRpcConnection: Received 'icinga::Hello' message from identity 'mon.domain'.
[2025-04-08 10:57:07 +0200] notice/JsonRpcConnection: Error while reading JSON-RPC message for identity 'mon.domain': Error: End of file

Stacktrace:
 0# __cxa_throw in /usr/lib64/icinga2/sbin/icinga2
 1# 0x00000000008CA6C8 in /usr/lib64/icinga2/sbin/icinga2
 2# icinga::JsonRpc::ReadMessage(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) in /usr/lib64/icinga2/sbin/icinga2
 3# icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) in /usr/lib64/icinga2/sbin/icinga2
 4# 0x0000000000B6D1DF in /usr/lib64/icinga2/sbin/icinga2
 5# 0x0000000000B742EB in /usr/lib64/icinga2/sbin/icinga2
 6# make_fcontext in /lib64/libboost_context.so.1.66.0
[2025-04-08 10:57:07 +0200] warning/JsonRpcConnection: API client disconnected for identity 'mon.domain'
[2025-04-08 10:57:07 +0200] warning/ApiListener: Removing API client for endpoint 'mon.domain'. 0 API clients left.
[2025-04-08 10:57:07 +0200] debug/EndpointDbObject: update is_connected=0 for endpoint 'mon.domain'

From the server I see these:

[2025-04-08 04:17:56 +0200] notice/ApiListener: Setting log position for identity 'host02.domain': 2025/04/01 16:57:57
[2025-04-08 04:17:56 +0200] information/ApiListener: New client connection for identity 'host02.domain' from [::ffff:172.10.10.57]:9042
[2025-04-08 04:17:56 +0200] notice/ApiListener: Ignoring JSON-RPC connection from [::ffff:172.10.10.57]:9042. We're already connected to Endpoint 'host02.domain'.

icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.14.3-1)

Copyright (c) 2012-2025 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-1160.90.1.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 11.2.1
  Build host: runner-hh8q3bz2-project-575-concurrent-0
  OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

How can I resolve this problem?

[Al2Klimov]

Sounds like you need v2.14.4 which addresses exactly this. Actually v2.14.5 because v2.14.4 is broken.

[ulyszk]

Thank you for the lightning fast answer. I will try it.

[ulyszk]

I still have this problem:

[2025-04-10 13:19:06 +0200] notice/JsonRpcConnection: Error while reading JSON-RPC message for identity 'mon.domain': Error: End of file

Stacktrace:
 0# __cxa_throw in /usr/lib64/icinga2/sbin/icinga2
 1# 0x00000000008C59DE in /usr/lib64/icinga2/sbin/icinga2
 2# icinga::JsonRpc::ReadMessage(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) in /usr/lib64/icinga2/sbin/icinga2
 3# icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) in /usr/lib64/icinga2/sbin/icinga2
 4# 0x0000000000B71AD8 in /usr/lib64/icinga2/sbin/icinga2
 5# make_fcontext in /lib64/libboost_context.so.1.66.0
[2025-04-10 13:19:06 +0200] notice/JsonRpcConnection: Disconnecting API client for identity 'mon.domain'

After both node are updated to v2.14.5 version. Do you have other ideas?