You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Should you find it insufficient, please let me know.
on-prem machine
VM (Virtualbox, KVM, etc. please specify)
VM running on a cloud service, please be explicit and add details
container (Kubernetes, Docker, containerd, etc. please specify)
or a combination, please be explicit
jails if it is FreeBSD
classic packaging
onedir packaging
used bootstrap to install
Steps to Reproduce the behavior
Install salt-master 3007 on remote
Install salt-minion 3007 locally
Run salt-call -l info grains.get fqdn on local machine
Running it you should see something akin to:
[ERROR ] Caught exception in PubChannel connect callback AttributeError("'NoneType' object has no attribute 'fire_event'")
Traceback (most recent call last):
File "/opt/salt/channel/client.py", line 542, in connect_callback
self.event.fire_event({"master": self.opts["master"]}, "__master_connected")
^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'fire_event'
...
local:
salt-minion.example.org
Task was destroyed but it is pending!
task: <Task pending name='Task-7' coro=<RequestClient._stream_return() running at /opt/salt/transport/tcp.py:1761> wait_for=<Future finished exception=StreamClosedError('Stream is closed')>>
Expected behavior
Closing channels and transports should properly wait on underlying streams before closing them fully.
Screenshots
Probably inapplicable.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
Salt: 3007.0Python Version:
Python: 3.12.3 (main, May 13 2024, 10:19:24) [Clang 16.0.6 ]Dependency Versions:
cffi: 1.16.0cherrypy: Not Installeddateutil: 2.9.0.post0docker-py: 7.0.0gitdb: Not Installedgitpython: Not InstalledJinja2: 3.1.4libgit2: Not Installedlooseversion: 1.3.0M2Crypto: 0.38.0Mako: 1.3.3msgpack: 1.0.8msgpack-pure: Not Installedmysql-python: 1.4.6packaging: 21.3pycparser: 2.22pycrypto: Not Installedpycryptodome: Not Installedpygit2: Not Installedpython-gnupg: 0.5.2PyYAML: 5.4.1PyZMQ: 25.1.2relenv: Not Installedsmmap: 5.0.1timelib: 0.3.0Tornado: 6.4ZMQ: 4.1.2Salt Package Information:
Package Type: Not InstalledSystem Versions:
dist: ubuntu 22.04 jammylocale: utf-8machine: x86_64release: 5.15.0-30-genericsystem: Linuxversion: Ubuntu 22.04 jammy
Additional context
I was able to trace examples of code lines, where the errors happen.
Somewhat same thing happens with SyncWrapper, where, as seen in RemotePillar, it both calls close method on AsyncReqChannel and destroys io_loop passed to it. However, as seen in error, underlying TCP transport fails to process StreamClosedError in time, leaving it as a Task in closed io_loop
Technically it also happens in TCP transport, but try catch wrapper simply catches StreamClosedError and puts it out as debug message.
The text was updated successfully, but these errors were encountered:
Also a seperate issue that doesn't really deserve a full write up (in my eyes), but TCP RequestClient needs to set self._closing = True to avoid TransportWarning: Unclosed transport! in logs
AppCrashExpress
changed the title
[BUG] Channels are closed while listened to in 3007.0
[BUG] [3007.0] Channels are closed while listened to
Jun 13, 2024
Yeah, this causes issues with TCP transport. Returns and jobs aren't working properly. There's some TCP close loop or something another. This is even more problematic in that the internal messaging is hard coded to tcp [1].
Unfortunately I can't figure out how to fix /this/ issue. It's beyond me.
[1] Probably because zeromq was broken .. (#66751) and chose to just disable zeromq.
Description
Hello!
We are currently trying to update Saltstack to 3007.0.
This issue was tested on the commit ID: 31c9d0d
When using TCP transport to run something like
salt-call grains.get fqdn
, logs will contain two types of errors:When TCP PubChannel is closed:
When TCP ReqChannel (the one wrapped in SyncWrapper) is closed:
Setup
This is shortened configuration, since we use patched custom installation and I'm not sure what I'm allowed to show:
Should you find it insufficient, please let me know.
Steps to Reproduce the behavior
salt-call -l info grains.get fqdn
on local machineRunning it you should see something akin to:
Expected behavior
Closing channels and transports should properly wait on underlying streams before closing them fully.
Screenshots
Probably inapplicable.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Additional context
I was able to trace examples of code lines, where the errors happen.
For PubChannel, it happens in SMinion during master evaluation. Closing channel will both close underlying transport and destroy event queue as seen here: https://github.com/saltstack/salt/blob/v3007.0/salt/channel/client.py#L463-L470
However
connect_callback
and, by extension,self.event.fire_event
still seem to run, causingNone.fire_event
Somewhat same thing happens with SyncWrapper, where, as seen in RemotePillar, it both calls
close
method onAsyncReqChannel
and destroysio_loop
passed to it. However, as seen in error, underlying TCP transport fails to process StreamClosedError in time, leaving it as a Task in closedio_loop
Technically it also happens in TCP transport, but
try catch
wrapper simply catches StreamClosedError and puts it out as debug message.The text was updated successfully, but these errors were encountered: