Kubo/prj 16 custom by theMladyPan · Pull Request #8 · metrotech-sk/xerxes-node

theMladyPan · 2026-01-13T17:33:07Z

No description provided.

Copilot

Pull request overview

This PR represents a substantial refactoring of the Xerxes node data collection and upload system for project "PRJ-16 custom". The changes move from a pickle-based single-collection approach to a JSON-based multi-collection architecture with separate handling for accelerometer, FFT, and strain/temperature data.

Changes:

Refactored uploader from pickle serialization to JSON with support for multiple MongoDB collections (ACCs, FFTs, Ss_Ts)
Introduced new Worker/Job threading system in system2.py with flexible leaf configuration
Updated configuration structure to support multiple workers and roots with custom periods
Added deployment scripts for Huawei modem setup and improved systemd service configuration

Reviewed changes

Copilot reviewed 21 out of 23 changed files in this pull request and generated 36 comments.

Show a summary per file

File	Description
xerxes_node/uploader.py	Complete rewrite: JSON-based processing, multi-collection inserts, improved data parsing
xerxes_node/timeutil.py	New utility module for timestamp generation with ISO format
xerxes_node/system2.py	New Worker/Job architecture for flexible sensor polling and data collection
xerxes_node/system.py	Code formatting improvements and import of shared timeutil
xerxes-worker.py	Major refactor: new config structure, multiple workers, custom Accelerometer class with FFT support
tests/test_uploader_refactor.py	New unit tests for refactored uploader functionality
requirements.txt	Updated pymongo (3.12.0→4.8.0) and xerxes-protocol (1.4.1→1.4.5)
script/setup.sh	Interactive network ID prompt, updated user to 'ubuntu', new system configs
script/etc/systemd/system/*	Updated service files for xerxes-node and new huawei-modem service
script/etc/udev/rules.d/*	New udev rules for USB latency and Huawei modem
script/etc/netplan/50-cloud-init.yaml	Static network configuration for eth0 and Huawei modem
example/config*.yaml	Updated config structure with workers and roots
.env.gpg	Added encrypted environment file
.gitignore	Added config file patterns

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-13T17:52:19Z

+    @property
    def alive(self) -> bool:
        return self.worker.is_alive()


The alive property will raise an AttributeError if accessed before start() is called, since self.worker is not initialized in __init__. Consider initializing self.worker = None in __init__ and returning False when worker is None, or document that this property should only be accessed after calling start().

Copilot · 2026-01-13T17:52:20Z

+# read network id from the stdin:
+printf 'Enter the network id: '
+read network_id
+echo "Using network id: $network_id"    


Trailing whitespace detected at the end of the line. This should be removed for code cleanliness.

Suggested change

echo "Using network id: $network_id"

echo "Using network id: $network_id"

Copilot · 2026-01-13T17:52:20Z

+    def __init__(
+        self, 
+        collections: Dict[str, Collection], 
+        workdir: str, 
+        upload_period: int = 10,
+        sensor_root_map: Dict[str, str] = None
+    ):
+        """
+        Args:
+            collections (Dict[str, Collection]): Dictionary containing 'ACCs', 'FFTs', 'Ss_Ts' collections.
+            workdir (str): Directory where .dat files are stored.
+            upload_period (int): How often to check for new files (in seconds).
+            sensor_root_map (Dict[str, str]): Map of sensor label (e.g., 'A33') to root label (e.g., 'usb0').
+                                              Acts as 'hub' metadata.
+        """
+        self.cols = collections

        # check if measurements directory exists, if not create it
-        if not os.path.isdir(directory):
+        if not os.path.isdir(workdir):
            try:
-                os.mkdir(directory)
+                os.mkdir(workdir)
            except:
-                raise AttributeError(f"Directory {directory} does not exist.")
-
-        self._directory = directory
-
+                raise AttributeError(f"Directory {workdir} does not exist.")
+
+        self._directory = workdir
+        self._upload_period = upload_period
+        self.sensor_root_map = sensor_root_map or {}


No validation that required collections are provided. The constructor accepts a collections dictionary but doesn't validate that it contains the required keys ('ACCs', 'FFTs', 'Ss_Ts'). If a required collection is missing, the uploader will silently skip inserting data for that type, which could lead to data loss without clear error messages.

Copilot · 2026-01-13T17:52:20Z

+)
+
+from .timeutil import add_timestamp
+from threading import Thread, Lock


Duplicate import statement. The imports Thread and Lock from threading are imported twice on lines 7 and 18.

Suggested change

from threading import Thread, Lock

Copilot · 2026-01-13T17:52:21Z

+    @property
+    def reset(self):
+        self.reset_soft()
+        return None


The reset property has a side effect (calls reset_soft()) but doesn't actually return any meaningful value. Properties should generally not have side effects. Consider renaming this to a method like do_reset() or trigger_reset() to make the side effect more explicit.

Suggested change

@property

def reset(self):

self.reset_soft()

return None

def trigger_reset(self) -> None:

"""Trigger a soft reset on the accelerometer device."""

self.reset_soft()

Copilot · 2026-01-13T17:52:31Z

 import time
-import pickle
+import json
+from typing import Dict, List, Optional, Any


Import of 'List' is not used.

Suggested change

from typing import Dict, List, Optional, Any

from typing import Dict, Optional, Any

Copilot · 2026-01-13T17:52:31Z

+            log.warning(f"Uploader encountered empty file {filename}")
+            try:
+                os.remove(filename)
+            except OSError:


'except' clause does nothing but pass and there is no explanatory comment.

Copilot · 2026-01-13T17:52:31Z

+                log.warning(f"No timestamp in {filename}, skipping.")
+                try:
+                    os.remove(filename)
+                except OSError:


'except' clause does nothing but pass and there is no explanatory comment.

Copilot · 2026-01-13T17:52:31Z

+            log.warning(f"Invalid timestamp in {filename}: {e}")
+            try:
+                os.remove(filename)
+            except OSError:


'except' clause does nothing but pass and there is no explanatory comment.

Copilot · 2026-01-13T17:52:32Z

+Before=network-online.target
+
+[Service]
+ExecStart=/home/ubuntu/xerxes-node/script/huawei_boot.sh


The ExecStart in huawei-modem.service runs /home/ubuntu/xerxes-node/script/huawei_boot.sh as root, even though /home/ubuntu is typically writable by the unprivileged ubuntu user. If an attacker compromises the ubuntu account, they can modify this script and obtain arbitrary code execution as root on the next boot or whenever the huawei-modem.service unit is started. To mitigate this, move the script to a root-owned, non-user-writable path (for example under /usr/local/sbin) and/or configure the unit to run as a non-privileged user while ensuring the script and its directory are not writable by less-privileged accounts.

Suggested change

ExecStart=/home/ubuntu/xerxes-node/script/huawei_boot.sh

ExecStart=/usr/local/sbin/huawei_boot.sh

theMladyPan and others added 20 commits July 12, 2024 22:19

WIP

87c2096

Refactor setup.sh script to use 'ubuntu' as the user instead of 'xerxes'

17b9f51

update config.yaml

62159ba

WIP

05b3ac2

wip

a89a7a4

Refactor logging format to include timestamp and log level

a2edae0

Added lock for network access to

208db70

chore: Add USB serial latency rule and update setup script

84f8cac

Refactor logging format to include timestamp and log level

ef2895d

chore: Add Huawei USB mode switch UDEV rule

f4f652d

Huawei MODEM udev + netplan fix

dbb0411

chore: Update Huawei MODEM udev and netplan configuration

8a8682a

autoboot huawei stick WIP

5121c79

chore: Update systemd-networkd-wait-online configuration

7e80d4c

chore: Remove unnecessary systemd-networkd-wait-online configuration

423bcb1

chore: Update systemd-networkd-wait-online service configuration

d994159

chore: Update setup script and systemd configuration

d5d3ed5

chore: Update system configuration and logging format

131aada

fix: remove pprint from requirements (it is a built-in module)

137230f

tried to change code

67e1dc0

theMladyPan assigned Copilot Jan 13, 2026

theMladyPan marked this pull request as ready for review January 13, 2026 17:42

Copilot AI review requested due to automatic review settings January 13, 2026 17:42

Copilot started reviewing on behalf of theMladyPan January 13, 2026 17:43 View session

Copilot AI reviewed Jan 13, 2026

View reviewed changes

added functionality to add other sensors as well.

2844a21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubo/prj 16 custom#8

Kubo/prj 16 custom#8
theMladyPan wants to merge 21 commits into
mainfrom
kubo/PRJ-16-custom

theMladyPan commented Jan 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	echo "Using network id: $network_id"
	echo "Using network id: $network_id"

	from typing import Dict, List, Optional, Any
	from typing import Dict, Optional, Any

	ExecStart=/home/ubuntu/xerxes-node/script/huawei_boot.sh
	ExecStart=/usr/local/sbin/huawei_boot.sh

Conversation

theMladyPan commented Jan 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants