Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proxy Minion does not return schedule result #62995

Open
TheBirdsNest opened this issue Nov 1, 2022 · 0 comments
Open

Proxy Minion does not return schedule result #62995

TheBirdsNest opened this issue Nov 1, 2022 · 0 comments
Labels
Bug broken, incorrect, or confusing behavior needs-triage Proxy-Minion

Comments

@TheBirdsNest
Copy link

TheBirdsNest commented Nov 1, 2022

Description
I believe this is related to issue #39668 which was closed in 2017.
When implementing a schedule with a Salt Proxy Minion, the result of the scheduled task is not returned to the master or to the Event Bus.

Setup
x1 Salt Master running on a Cloud VM with 8GB RAM and 4 vCPU's
x5 Syndic Masters running on a Cloud VM with 256GB RAM and 64 vCPU's serving 1500 proxy minions each.

Steps to Reproduce the behavior
Create a schedule that targets a proxy minion:

c73b4f53-2528-4ff9-af51-82837bfd18e2:
    schedule:
      enabled: true
      example:
        cron: 0 * * * *
        enabled: true
        function: test.ping
        name: example
      switchport_utilization:
        cron: 0 10,14 * * 1-5
        enabled: true
        function: interface.utilization_summary
        name: switchport_utilization
        splay: 1800

Wait for the job to be executed. Below is the output from the proxy debug:

[DEBUG   ] LazyLoaded interface.utilization_summary
[DEBUG   ] LazyLoaded test.ping
[DEBUG   ] LazyLoaded mine.update
[DEBUG   ] Subprocess Thread-263-Schedule-__proxy_keepalive cleaned up
[DEBUG   ] schedule: Job __proxy_keepalive was scheduled with jid_include, adding to cache (jid_include defaults to True)
[DEBUG   ] schedule: Job __proxy_keepalive was scheduled with a max number of 1
[INFO    ] Running scheduled job: __proxy_keepalive with jid 20221101172955648942
[DEBUG   ] Subprocess Thread-264-Schedule-__proxy_keepalive added
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded status.proxy_reconnect
[DEBUG   ] schedule.handle_func: adding this job to the jobcache with data {'id': 'c73b4f53-2528-4ff9-af51-82837bfd18e2', 'fun': 'status.proxy_reconnect', 'fun_args': [{'proxy_name': 'napalm'}], 'schedule': '__proxy_keepalive', 'jid': '20221101172955648942', 'pid': 11939}
[DEBUG   ] schedule.handle_func: Removing /var/cache/salt/proxy/c73b4f53-2528-4ff9-af51-82837bfd18e2/proc/20221101172955648942
[DEBUG   ] LazyLoaded interface.utilization_summary
[DEBUG   ] LazyLoaded test.ping
[DEBUG   ] LazyLoaded mine.update
[DEBUG   ] Subprocess Thread-264-Schedule-__proxy_keepalive cleaned up

Check the Event Bus.. There is no event for __schedule_return:

20221101173104365994    {
    "_stamp": "2022-11-01T17:31:04.366385",
    "minions": [
        "c73b4f53-2528-4ff9-af51-82837bfd18e2"
    ]
}
salt/job/20221101173104365994/new       {
    "_stamp": "2022-11-01T17:31:04.367102",
    "arg": [],
    "fun": "schedule.list",
    "jid": "20221101173104365994",
    "minions": [
        "c73b4f53-2528-4ff9-af51-82837bfd18e2"
    ],
    "missing": [],
    "tgt": "*",
    "tgt_type": "glob",
    "user": "root"
}
salt/stats/MWorker-3    {
    "_stamp": "2022-11-01T17:31:04.372164",
    "stats": {
        "publish": {
            "mean": 0.006384372711181641,
            "runs": 1
        }
    },
    "time": 1014.6155388355255,
    "worker": "MWorker-3"
}
salt/job/20221101173104365994/ret/c73b4f53-2528-4ff9-af51-82837bfd18e2 {
    "_stamp": "2022-11-01T17:31:04.448417",
    "cmd": "_return",
    "fun": "schedule.list",
    "fun_args": [],
    "id": "c73b4f53-2528-4ff9-af51-82837bfd18e2",
    "jid": "20221101173104365994",
    "retcode": 0,
    "return": "schedule:\n  enabled: true\n  example:\n    cron: 0 * * * *\n    enabled: true\n    function: test.ping\n    name: example\n  switchport_utilization:\n    cron: 0 10,14 * * 1-5\n    enabled: true\n    function: interface.utilization_summary\n    name: switchport_utilization\n    splay: 1800\n",
    "success": true
}
salt/stats/MWorker-0    {
    "_stamp": "2022-11-01T17:31:04.450653",
    "stats": {
        "_return": {
            "mean": 0.002768278121948242,
            "runs": 1
        }
    },
    "time": 1012.1096804141998,
    "worker": "MWorker-0"
}

Check the Job Cache.. There are no jobs for the schedule:

20221101171415775118:
    ----------
    Arguments:
    Function:
        schedule.list
    StartTime:
        2022, Nov 01 17:14:15.775118
    Target:
        *
    Target-type:
        glob
    User:
        root
20221101173104365994:
    ----------
    Arguments:
    Function:
        schedule.list
    StartTime:
        2022, Nov 01 17:31:04.365994
    Target:
        *
    Target-type:
        glob
    User:
        root

Expected behavior
Once a scheduled job is executed, the return is published on the event bus and is exposed to the master.

Versions Report
Salt Version:
Salt: 3005

Dependency Versions:
cffi: 1.14.4
cherrypy: 5.6.0
dateutil: 2.8.1
docker-py: 4.4.4
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.0.2
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: 2.6.1
pycryptodome: 3.9.9
pygit2: Not Installed
Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
python-gnupg: Not Installed
PyYAML: 5.3.1
PyZMQ: 18.0.1
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4

System Versions:
dist: centos 7 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1160.76.1.el7.x86_64
system: Linux
version: CentOS Linux 7 Core

@TheBirdsNest TheBirdsNest added Bug broken, incorrect, or confusing behavior needs-triage labels Nov 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug broken, incorrect, or confusing behavior needs-triage Proxy-Minion
Projects
None yet
Development

No branches or pull requests

2 participants