AMOS 2025 - Worker mangement and orchestration

[ClProsser]

What kind of change does this PR introduce?
feature

What is the current behavior?
Currently, worker nodes are not supported by EMBArk.

What is the new behavior (if this is a feature change)?
In this PR, we provide a basic implementation of our project mission: Orchestration in EMBArk, including worker setup/configuration. As of now, it is possible to create workers, configure/update them, run an analysis on them and retrieve results.

This PR includes:

• Settings app:
  • Allows to dynamically set settings using the EMBArk webinterface
  • Currently only supports enabling orchestrators/workers. If enabled, a worker app is added to the sidebar (once the page is reloaded)
• Worker management:
  • Offline workers can be created using configurations, by specifying a SSH user name, password and an IP range. EMBArk then creates every reachable worker in this IP range
  • Workers are then base-configured, i.e. a sudoers file is created if the user isn't root
  • Once a worker is base-configured, an offline installation can be triggered. EMBArk fetches all necessary artifacts, transfers them to the worker and initiates the installation
  • If something is bugged, a worker can be soft-resetted (stop running docker containers, delete analysis related files) or hard-resetted (uninstall and remove EMBA and all its dependencies)
  • Every two minutes, worker related system information is fetched (Available CPU cores, RAM, available/total storage)
• Update management:
  • EMBArk keeps track which versions of dependencies (EMBA repo, EMBA docker image, APT dependencies, external dependencies) are installed on each worker
  • EMBArk can check if new dependency versions are available for download
  • If new versions are available, a diff can be shown (available vs installed) and individual dependencies can be updated. If a worker is free, this is done immediately, otherwise the update is delayed until the worker is free again (e.g. after the current update, or after the current analysis)
• Orchestrator:
  • Schedules queued analysis tasks and assigns them to workers, once free workers are available
  • Keeps track which workers are busy/free
  • Starts the analysis on the worker. To do this, the firmware is copied to the worker. Once this is done, EMBArk checks if the analysis is still running (e.g. if the docker container is still up). In addition, EMBArk zips and transfers partial results to the EMBArk host.
• Async tasks: To reduce response times and thus waiting times in the UI, we added Celery

Does this PR introduce a breaking change?
No

Other information:
This is the third and main release of the AMOS 2025 EMBArk project.

As we are still testing the system as a whole, this is a draft PR. The prevention of overlapping IP ranges and SSH key auth is not included yet.

[ClProsser]

I could not find a word-splitting issue here, what did you find?

However, I noticed that the result of the awk command includes quotes, thus the comparison is always false. I updated the check.

[ClProsser]

Now, this PR is not in draft-state anymore.

Three problems should be addressed soon:

1. Configuration overlapping: Currently, ip ranges in configurations might overlap, resulting in workers having multiple related configurations. In this case, multiple features might break each other (e.g. Transferring an SSH key (as password auth was deactivated after the first configuration scan), ...)
2. Dependency state is set as available even if a dependency setup fails: EMBArk caches all worker dependencies for later use (Repo, docker image, external dir, apt dependencies). As of now, EMBArk has no strategy to deal with failing dependency setup on the EMBArk host. Assuming e.g. that the EMBArk host has no internet connection, the docker image can't be downloaded. Thus the setup might fail. A solution might be to retry, and stop if it failed 3 times.
3. Pipfile.lock updates: As we added multiple python packages, the state of the Pipfile.lock does not reflect the state in the main branch. We did not just update all packages, as we are not familiar with your strategy in selecting and verifying packages.

Furthermore, it might be reasonable to refactor the BoundedExecutor class to celery tasks, and if needed, benefit from multiple celery processes.

If there are any change requests in this PR, feel free to comment them in this PR, depending on its size we'll fix it.

[BenediktMKuehne]

Great work

Looking forward to merging this
...once I have tested it

[BenediktMKuehne]

unsure why this file is called that

[ClProsser]

In this file we define the function new_autoadd_client, which uses paramiko.AutoAddPolicy. However, CodeQL does not allow this (https://codeql.github.com/codeql-query-help/python/py-paramiko-missing-host-key-validation/). The only way to define exceptions is to exclude whole files. Thus a file codeql_ignore.py was created, and added to /codeql-config.yml.