-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] "Minion did not return. [No response]" with salt-proxy in 3005. #62694
Comments
As an aside, you should not allow unattended-upgrades to install new major versions of anything. Things will break. Does the minion eventually return, allowing you to lookup the job result later? |
Hi @OrangeDog, Thanks for the reply. Yes, this is the danger of unattended upgrades (I do maintain a blacklist of critical packages not to update, so it is mainly system/background packages which get updated). I need to review whether Salt should be on that list, but that is an aside, since if it is the upgrade which caused it, I would have had the same issue on a manual upgrade too. I can't see anything obvious in the release notes that stands out as an obvious cause. As I noted above, running With regards to logs, nothing is appended to the proxy minion logs when this issue happens. I included what looks like the relevant snippet from the master (with With regards to state, I can say with certainty which state is is, as I only use one :) This is my state file:
Then switch.jinja has includes to many other jinja templates. Thanks, Ian |
An update on this. I downgraded salt on my proxy machine to 3004.2 and it fixes the problem.
The salt master is still running 3005:
Interestingly, state.highstate commands feel like they take much longer to return than it did to give the 'Minion did not return' error prior to upgrade, so it could just be that a timeout has been lowered in 3005?
|
That only sets the log level of the master, not the (proxy) minion.
Almost certainly. Things are frequently changed, and you want to be able to test things first before you take down your whole infra. That's also why Salt provides repos for a single major version so you only get the minor updates. |
Any suggestions as to whether there is a timeout which may be adjustable, which I could increase as a workaround, instead of having to use the older version? |
I don't know which one specifically would apply, but you could start with More useful would be getting the debug log from the minion, though if it already doesn't even have any warnings then it's probably just slowed down a bit. |
Are you using the default timers ? In my environment it does not work at all. If the job cache returns the correct result . You should increase the master timeout. |
Description
We have had a working Salt, salt-proxy, NAPALM network automation setup for around a year.
Suddenly, we find that any
state.highstate
call fails withMinion did not return. [No response]
.If I run
salt-run jobs.lookup_jid
, it shows the expected output.Other calls, like
test.ping
or querying grains etc works fine.Looking at our unattended upgrades logs, it seems there was an update from 3004 to 3005 recently, so wondering if that's the cause?
Setup
Using Debian packages on Debian stable (bullseye).
Running on our own VMs.
The salt-proxy processes are on a separate VM, in the same VLAN.
Steps to Reproduce the behavior
Running
$ sudo salt mydevice state.highstate test=true
fails, as above.Using
-l debug
, shows:Versions Report
salt --versions-report
The text was updated successfully, but these errors were encountered: