Data and tmp standardization

[adamoutler]

This pull request implements a major simplification of the NetAlertX Docker filesystem, consolidating all persistent data into a single /data volume and all ephemeral runtime data into a single /tmp tmpfs mount.

🧐 Why is this necessary?

The previous mount structure was complex, requiring users to manage two separate persistent volumes (netalertx_config, netalertx_db) and a list of up to five tmpfs mounts. This created a high barrier to entry and was prone to misconfiguration.

This new structure aligns with best practices and simplifies the user experience for all use cases, from a simple docker compose up at home to complex, air-gapped enterprise deployments.

---

🛠️ Summary of Changes

This PR touches nearly every layer of the container to achieve this simplification, from the Dockerfile to the application-level PHP code.

1. Filesystem & Mounts

• docker-compose.yml:
  • Removed netalertx_config and netalertx_db volumes.
  • Added a single persistent volume named data mounted to /data.
  • Removed all individual tmpfs mounts (e.g., /tmp/log, /tmp/api, /tmp/run).
  • Added a single tmpfs mount for /tmp, which now contains all runtime/ephemeral subdirectories.
• Dockerfile:
  • Creates /data and /tmp base directories with correct 20211:20211 / 700 permissions.
  • Environment variables (NETALERTX_CONFIG_DIR, NETALERTX_DB_DIR, NETALERTX_LOG_DIR, etc.) have been updated to point to the new paths (e.g., /data/config, /data/db, /tmp/log).

2. Path Abstraction & Code Updates

• Hard-coded paths have been eliminated. All PHP and Python services were refactored to use environment variables and centralized path helpers (like server/config_paths.py) instead of hard-coded paths like /app/config, /app/db, or /app/api.
• Fixed Startup Hang: Corrected a critical bug where front/php/server/util.php was still referencing /app/api/table_settings.json, causing PHP-FPM to hang on startup. It now correctly uses the abstracted path.

3. Entrypoint & Migration Logic

• entrypoint.d/01-data-migration.sh (New):
  • A new migration script intelligently handles users with legacy mounts. See "Migration Path" section below for details.
• entrypoint.d/25-mandatory-folders.sh (New):
  • This script now runs on every startup to create the entire required directory tree inside both /data and /tmp (e.g., /data/config, /data/db, /tmp/log, /tmp/api, /tmp/run, /tmp/nginx/active-config). This ensures the container functions correctly whether the user mounts the root /data and /tmp or individual sub-mounts.
• entrypoint.sh (Updated):
  • Implements "fail-fast" logic. If any entrypoint.d script exits with status 1 (indicating a critical, system-breaking error), the container will print a standardized error message with a troubleshooting link and exit.
  • Non-critical warnings (status 2-255) will still be printed but will not halt the container.
  • Removed redundant chown commands per review feedback.

---

🗺️ Migration Path for Existing Users

This change includes an assisted migration to move data from old volumes.

1. A user with an existing instance must add the new /data volume to their docker-compose.yml while keeping their old netalertx_config and netalertx_db volumes.
2. On restart, 01-data-migration.sh will detect this hybrid state.
3. It will copy all data from /app/config and /app/db into the new /data/config and /data/db directories.
4. It will create .migration files in the old volumes and print a persistent warning on every startup, instructing the user to shut down, remove the old netalertx_config and netalertx_db volume definitions, and restart.
5. Once the user's docker-compose.yml is updated to only use the /data volume, the warnings will cease.

---

✅ Testing & Validation

• All unit tests are passing.
• Test Fixtures: All test configurations in test/docker_tests/configurations/ were updated to reflect the new mount structure.
• Test Logic:
  • test_mount_diagnostics was updated to parse the new warning tables and assert their contents directly.
  • test_missing_host_network_warning is now passing.
  • Tests that became unreachable due to the new "fail-fast" logic have been commented out with notes.
• Docker Tests: Verified that a container launched with both sub-mounts (e.g., -v /tmp/log:/tmp/log) and root mounts (-v data:/data) functions correctly and devices.php is available.

---

New Features

Some additional items were added to reduce development complexity.

1. lint: flake8 was added to improve code quality of any file I needed to work with.
2. zsh: a change from default shell "fish" to "zsh" was made to allow proper BASH syntax while retaining auto-complete functionality.
3. a .devcontainer/NetAlertX.code-workspace Workspace was added which allows quick access to logs and other out-of-source inspections during normal operation

4. Build Tasks are assigned a description which can help to determine the use case for each build task 5. New build task "Build & Launch Prodcution Docker Container" helps to perform an evaluation on the full user-experience container. 6. New build task "Wipe and Regenerate Database and Config" will do just what's on the tin, allowing to evaluate fresh-launch within the devcontainer environment. 7. Task Docker system and build Prune will now prompt using the vscode GUI for a "YES" before continuing 8. Before unit testing, a fresh copy of the test container is now built.

💬 Addressed Code Review Feedback

• Successfully implemented the "fail-fast" logic for exit status 1 in entrypoint.sh. which reduced complexity in implementation

Future enhancements

1. Currently unit testing ensures the environment is prepared to launch the app. However, some automated web testing should be conducted to ensure the default configuration and db are correct and the application is functional. Selenium or Playwrite testing should be implemented.
2. Tests which were failing in the repository are still failing and should be cleaned up. I did not touch these unit tests. I had considered commenting them out since they've been failing for a long time now.
3. Add some/all unit testing to Actions on Github stages. Each build should include some or all of the unit tests. The test suite takes a long time to run so maybe it's best to not run ALL tests every time, but a few select tests should be run. eg. a docker build, then some selected pytest and Playwrite tests to ensure the UI operates and provides expected results before publishing to /dev branch.