v4.2.0

This release implements a major architectural overhaul - going fully parallel - which enabled many responsivity and usability improvements. This change is transparent to the user, and you can upgrade normally - using the setup tool. The logs tab also separates the logs by thread. We rebuilt the communication to OPUS from DDE to HTTP. Furthermore, PYRA can now automatically save the peak position of recorded interferograms. In case CamTracker does not shut down gracefully, we will now forcefully shut it down as a last resort.

PYRA 4.2.0 was tested on two separate Windows 10 systems with OPUS 7.8 and 8.2. If the new OPUS communication does not work for your case, please let us know - the setup tool has a rollback command.

It is possible that "Windows Defender Firewall" pops up the first time OPUS gets started by PYRA 4.2.0 (due to the activation of the HTTP interface) - please accept the preselected options. It is also possible that something else is already using the localhost (visit "localhost" in your browser) - please deactivate this or move it to another port.

Contributors: Moritz Makowski (@dostuffthatmatters)

Details

New Parallel Architecture

Up to version 4.1.4, PYRA had "threads" (upload and helios) which were executed in parallel to the main thread and "modules" (opus, camtracker, measurement conditions, system checks, enclosure) which all ran sequentially on the main thread. This sequential architecture of otherwise independent modules was simple, but became increasingly hard to work with. For example, the CamTracker logic was only run once the opus logic finished. Furthermore, when a module had an exception, the following modules were not executed. Additionally, when an error happened in one of the modules, all the modules paused.

By making all modules parallel, we could significantly reduce the reaction time of PYRA, simplify many things and add features that would have caused long delays in the other modules - or hideous state machines.

OPUS Communication

Up to version 4.1.4, PYRA used DDE to communicate with OPUS. Since Microsoft replaced it internally with Windows95, the interface has only been kept for compatibility reasons. This means that there is almost no documentation or libraries for DDE. Since OPUS also has an HTTP interface, we switched to using that one. We added the OPUS HTTP Client to our Python utility library, so one can use it outside of PYRA to: see tum-esm-utils.

With this new way, we can finally detect if the macro is still running - and restart it in case it stopped. We can also detect whether any other macro is running and log which OPUS version is being used.

Automatic Peak Positioning:

Whenever powering up the spectrometer, the EM27/SUN does not know position of the pendulum fully accurate. That is why it is recommended to save the center position of recorded interferograms after every power up so that no points on one ifg side get lost because it is not stored centered. In OPUS, you can do this in "Measure > Setup Measurement Parameters > Check Signal". Or you can let PYRA do this after every power up by configuring it in the OPUS Configuration and save a few minutes per system per day.

All Changes

OPUS Improvements

• Add possibility to set peak position automatically #235
• Figure out why OPUS answers in Chinese #124
• Solve OPUS communication issues #238
• Opus Macro is not restarted, when OPUS crashes #228

Camtracker

• CamTracker does not always close #176
• Force-close CamTracker if it does not close within some timelimit #230

Operational

• Don't measure in manual mode when sun elevation is below minimum #237
• Improve data yield and performance in fast switching conditions #232
• Test-Email not sent when emails are deactivated #224

TUM Enclosure/Helios

• Make the Helios circle crop fraction configurable #233
• Improve Helios more #236
• Early-return from enclosure procedure when PLC is not connected #123

v4.1.4

Stability improvements for the upload.

Contributors: Moritz Makowski (@dostuffthatmatters)

All Changes

• Update the upload library circadian_scp_upload from v0.4.6 to v0.5.0 because this update brought a lot of stability improvements (#229)

v4.1.3

Developer improvements (migrate package managers), more logs, more control about the upload, bugfixes.

Contributors: Moritz Makowski (@dostuffthatmatters)

All Changes

• Add option upload.only_upload_when_not_measuring - before this was set to true by default (#219)
• Migrate from Poetry to PDM - now the installation does not require Poetry anymore, but only pip (#214)
• Migrate the UI from Yarn to Bun
• Add more logs from Helios thread (#218)
• Various bugfixes (#216, #217, #221)

v4.1.2

Usability improvements.

Contributors: Moritz Makowski (@dostuffthatmatters)

All Changes

• Update the overview tab state variables (#212) - Replace "reset needed" and "motor failed" with "heater power" and "helios edge pixels"

v4.1.1

Bugfixes, more control about the upload.

Contributors: Moritz Makowski (@dostuffthatmatters)

All Changes

• Add option to only run uploads during night (#210)
• Bugfixes and improvements after big update 4.1.0 (#209)

v4.1.0

A ton of tiny improvements and bug fixes over 4.0.8. A new upload that works for arbitrary directories. A completely redesigned UI. Many explanations inside the configuration tab. More user-friendly logs tab. Error email with any SMTP provider (not just Gmail). Buttons to test Pyra parts from the UI. Enriched activity history. Links to issue tracker and docs in UI.

Contributors: Moritz Makowski (@dostuffthatmatters)

All Changes

Code Quality Improvements:

• Now using Pydantic objects instead of TypedDicts and stricter types: #174, #122
• Refactor Helios code #194
• Merge persistent and temporary state into one object: #190
• No longer keep info logs (just the debug logs). Write log lines to archive right away. Only keep the last 5-10 minutes of logs in the debug log file to decrease system load and UI parsing logic: #189
• The UI now parses the loaded state and config using zod (https://zod.dev) for better runtime type safety. It sends error notifications if the parsing is unsuccessful but can display logs, nevertheless.
• Remove many UI loading states in favor of using toast.promise from react-hot-toast (https://react-hot-toast.com/docs/toast#advanced)
• Add a central UI hook that prevents multiple CLI commands from being run at the same time

Core Improvements:

• Prevent failing modules from blocking other modules from executing when there is no dependency between them: #173
• Add more links to PLC documentation: #172
• Add power cycle for the camera at midnight: #179
• Make the error email server configurable (still supports the Gmail one): #191
• Store GPS and Sun Elevation in state and render them in UI: #26
• Log current sun angle: #192
• In error emails, send debug-logs instead of info-logs: #183
• Save data from TUM PLC in files: #201
• Do not start measurements when PLC indicates rain: #203
• Fix bug in CLI when force-closing cover: #207

Opus Improvements:

• If we have the info about the cover angle, only run the Opus Macro when the cover is open: #198
• Fix Pyra launching multiple Opus windows: #202
• Possibly fix DDE errors due to multiple OPUS windows: #155

Helios Improvements:

• Showing the current Helios images in the UI: #199
• Add a minimum time between Helios decision changes: #195
• Add more log lines to clarify buffer logic: #178
• Make Canny filter threshold and target pixel brightness configurable: #200

Upload Improvements:

• Refactor the Upload: Now supports arbitrary directories: #193
• Do not run upload when measuring: #177
• Make upload work for datalogger files: #92
• Improve the observability of the upload process: the UI now has a dedicated logs filter for upload logs, and the logs are more verbose now: #116

CamTracker Improvements:

• Deal with large CamTracker files by only reading the last few lines: #107
• Add two more checks for a possible restart to detect whether CamTracker is stuck at initialization: #184

UI Improvements:

• The UI can now be used when the core is not running
• Divide logs into four categories: main, helios, upload, ui
• On file text inputs, add a button to show the parent directory in the File Explorer: #171
• Add button to copy config.json: #182
• Show long error messages in UI instead of printing "details in console": #196
• Add buttons to test upload and emailing: #120, #109
• Add buttons to test OPUS and CamTracker connectivity: #204, #205
• Fix the "config has changed" popup: #106
• Add links to documentation and issue tracker: #163
• Improve the activity history: #188
• Improve the design of everything: #175
• Debug notifications: #34

Sneak Peek

The two tabs, "overview" and "automation" from 4.0.8, have been merged into one overview tab. The new activity history and system status show more information. The measurement status is more apparent now (big green/gray bars).

Before:

After:

The config now explains all non-obvious settings. OPUS, CamTracker, Upload, and Error Emails have a self-test that can be run from the UI. The navigation bar has links to the issue tracker and the documentation (top right).

Before:

After:

The logs tab navigation has been improved so that each thread's logs can be seen separately.

Before:

After:

v4.0.8

Many tiny improvements and bug fixes over 4.0.7. Now, also usable offline. Migrated from astropy to skyfield. Removed local utility implementations in favor of the better tested tum-esm-utils. Improve reliability of Helios camera connection.

Contributors: Moritz Makowski (@dostuffthatmatters)

Details

Fix issues with PLC communication:

The underlying library (snap7) changed their internal error types from the custom Snap7Exception to the built-in RuntimeError. We had a logic built into Pyra to only report PLC errors if they persist for more than 10 minutes, but this was only considering any Snap7Exception.

• Closes #151
• Freeze the poetry dependencies to the one used in 4.0.7
• Closes #157

Fix astronomy utilities:

The previously used Astronomy library (astropy) does not work offline. It also has a lot of side effects (weird errors from internal logging) and bad types annotations and documentation. So we switched to the Skyfield library (https://rhodesmill.org/skyfield/) and included the astronomical dataset to be downloaded directly in the repository (~ 16 MB, but would have to be downloaded anyway).

• Closes #126
• Closes #125
• Closes #154

Helios Improvements:

We made the Helios implementation more resilient to camera issues - e.g. when changing the exposure or other settings frequently.

• Closes #167
• Close #169

More use of tum-esm-utils:

We have merged utility functions of many of our projects into a utility library (https://github.com/tum-esm/utils). We removed some of the utility classes/functions from the Pyra codebase and use the better tested tum-esm-utils library.

• Closes #165

Miscellaneous:

• Removed the git commit SHA from emails since this is redundant now
• Closes #156

v4.0.7

This release improves and refines Helios - our system of detecting direct vs. indirect sunlight. It also improves Pyra's utility functions and tests.

Contributors: Moritz Makowski (@dostuffthatmatters)

Details

Helios Improvements:

Refactor the decision circle buffer. Refactor the Ringlist datastructure and add elaborate tests. Reduce the number of quickly alternating Helios decisions. Add a script to determine optimal Helios parameters. Refactor the Helios code a bit. Simplify the Helios config.

• Closes #147
• Closes #148
• Closes #139
• Closes #98

Miscellaneous:

Refactor the dependency groups and add caching to Tauri CI workflow.

• Closes #141
• Closes #111

v4.0.6

This release adds a lot of final touches to our first non-prerelease version. This includes a lot of improvements to the documentation, bugfixes and tiny features making it easier to use Pyra.

Contributors: Moritz Makowski (@dostuffthatmatters), Patrick Aigner (@patrickjaigner)

v4.0.5

• Upload
• Debugging
• Full static typing
• Rebranding "VBDSD" -> "Helios"