Release notes from pupil

v3.6

2023-02-16T16:10:34Z

v3.6

Pupil Capture, Player, and Service release

2022-11-11T10:21:54Z

We are pleased to announce the release of Pupil Core software v3.5!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord! 😄

Overview

In Pupil v3.5, we cleaned up log messages, improved software stability, fixations are now cached between Pupil Player sessions, and we have fine tuned 3d pupil confidence.

Pupil v3.5 is the first release that supports the new macOS Monterey (with a small caveat). Read more below.

New

Recording-specific annotation hotkeys
Add option to filter out low-confidence values in raw data export
Add option to disable world video recording
macOS Monterey support

Improved

Reduced number of user-facing log messages
Export-in-progress file-name suffixes
Windows: Handle exceptions during driver installation gracefully
Windows: Option to skip automatic driver installation
Windows: Create start menu entries and desktop shortcuts for all users on install
Add Euler angles to headpose tracker result
Normalize IMU pitch/roll values to [-180, 180] degrees
Confidence adjustment for pye3d results
Cache post-hoc fixation detector results
Documentation

New

Recording-specific annotation hotkeys - #2170

Until now, the keyboard-shortcut setup for annotations was stored in Pupil Player's session settings. These were reset if you updated Pupil Core software to a new version.

Starting with this version, annotation keyboard shortcuts will be stored persistently within the recording.

The definitions are stored in <recording dir>/offline_data/annotation_definitions.json with the format:

{
    "version": 1,
    "definitions": {
        "<label>": "<hotkey>"
    }
}

This file can be shared between recordings.

When Pupil Player is opened, it will attempt to load the annotation definitions from the recording-specific file. If the file is not found or is invalid, Pupil Player will fallback to the session settings which contain the last known annotation definitions.

Add option to filter out low-confidence values in raw data export - #2210

Pupil Core software assigns "confidence" values to its pupil detections and gaze estimations. They indicate the quality of the measurement. For visualization and some analysis Plugins, Pupil Player hides gaze data below the "Minimum data confidence" threshold (adjustable in the general settings menu).

Prior to this release, the raw data exporter plugin exported all data, independently of its confidence. New users, therefore, often wonder why their exported data looks noisier than what they have seen in Pupil Player.

Starting with this release, users will have the option to filter out low-confidence data (i.e. below the "Minimum data confidence" threshold) in the export. The option can be enabled in the raw data exporter's menu.

Add option to disable world video recording - #2169

If you are not interested in recording the scene video, you can now disable the scene video recording in the Recorder menu of Pupil Capture's world window.

macOS Monterey support

Due to new technical limitations, Pupil Capture and Pupil Service need to be started with administrator privileges to get access to the video camera feeds. To do that, copy the applications into your /Applications folder and run the corresponding command from the terminal:

Pupil Capture: sudo /Applications/Pupil\ Capture.app/Contents/MacOS/pupil_capture
Pupil Service: sudo /Applications/Pupil\ Service.app/Contents/MacOS/pupil_service

Note: The terminal will prompt you for your administrator password. It will not preview any typed keys. Simply hit enter once the password has been typed.

Note: When recording with administrator privileges, the resulting folder inherits admin file permissions. Pupil Player will detect these and ask you for the administrator password to reset the file permissions. This will be only necessary once per recording.

Improved

Export-in-progress file-name suffixes - #2167, #2186

Exporting video can take a long time. Pupil Player displays the progress as a growing circle around the menu icon while writing the result incrementally to disk. Specifically, it writes the result directly to the final export location. While the export is ongoing, the video file is invalid but still available to the user. This can lead to users opening the video before it has finished exporting and then encountering playback issues.

Starting with this release, Pupil Core software changes the video exports to include a .writing file name suffix during the export. This prevents other applications from recognizing the partially exported video file as complete. Once the export is done, the suffix is removed and other applications are able to recognize the file type as video.

Reduced number of user-facing log messages - #2190

Over the years, Pupil Capture's functionality has grown and with it the number of user-facing log messages. They can be overwhelming to the point where important log messages are overlooked or ignored. For this release, we have reviewed the log messages that are being displayed in the UI during typical actions, e.g. calibration and recordings, and have reduced them significantly.

Windows: Handle exceptions during driver installation gracefully - #2173

Previously, errors during the driver installation could cause the software to crash. Now, errors will be logged to the log file and the user will be asked to install the drivers manually.

Windows: Option to skip automatic driver installation - #2207

Using the -skip-driv or --skip-driver-installation flags, one can now launch Pupil Capture and Pupil Service without running the automatic driver installation on Windows. This can be helpful to speed up the start up time or to avoid unwanted driver installations.

Windows: Create start menu entries and desktop shortcuts for all users on install - #2166

Pupil Core software is installed to C:\Program Files (x86)\Pupil-Labs by default. This directory should be accessible by all users but the start menu entry and desktop shortcut are only installed for the current user. Starting with this release, the installer will create start menu entries and desktop shortcuts for all users on install.

Add Euler angles to headpose tracker result - #2175

Next to camera poses as Rodrigues' rotation vectors, the headpose tracker now also exports the camera orientation in Euler angles.

We also fixed a typo in the exported file name.

- head_pose_tacker_model.csv
+ head_pose_tracker_model.csv

Normalize IMU pitch/roll values to [-180, 180] - #2171

Orientation values exported by the IMU Timeline plugin will now be normalized to a range of [-180, 180] degrees.

Confidence adjustment for pye3d results - #2195

In some cases, pye3d was overestimating the confidence of its pupil detections. These were visible as noisy gaze estimations in the Pupil Core applications. This release includes an updated version of pye3d with adjusted confidence estimations. As a result, the gaze signal should look more stable during lower confidence situations, e.g. while squinting or looking down. For more information, see the Developer notes below.

Cache post-hoc fixation detector results - #2194

Post-hoc fixation detector results no longer need to be recalculated if you close and reopen a recording. Pupil Player stores the results together with the used configuration (version of the gaze data and fixation detection parameters) within the recording directory. The cached data will be restored if the Player configuration matches the configuration stored on disk. If it does not match, Player will recalculate the fixations to avoid inconsistencies.

Documentation

We have improved our documentation in several places:

Pupillometry best practices - pupil-docs#453
Blink detection - pupil-docs#463
Pupil Core API - pupil-docs#455

Fixed

Fixed typo in exported headpose tracker files - #2185
Fix confidence graphs in Pupil Player for pre-2.0 recordings - #2191
Use correct socket API to retrieve local IP address for Service - #2201
Fix loading 200 Hz gaze data in Pupil Invisible recordings - #2204
Fix loading short audio streams - #2206

Known issues

Windows: Connecting remotely to Pupil Remote on Pupil Service does not work in some cases
Pupil Groups: Users have reported issues with Pupil Groups connectivity when running Pupil Capture on different devices.

We are investigating these issues and will try to resolve them with an update as soon as possible.

Developer notes

Dependencies updates

Python dependencies can be updated using pip and the requirements.txt file:

python -m pip install --upgrade pip wheel
pip install -r requirements.txt

pye3d v0.2.0 and v0.3.0

Version 0.2.0 of pye3d has lowered 3d-search-result confidence by 40%. When the confidence of the input 2d pupil datum is lower than a specific threshold (0.7), pye3d attempts to reconstruct the 2d pupil ellipse based on its existing 3d eye model. We noticed that the confidence for these results was systematically overestimated given the actual detection result quality and have therefore adjusted the overall confidence assignment for them. See pye3d-detector#36 for reference.

Version 0.3.0 uses a new serialization format for the refractionizer parameters but is otherwise functionally equivalent to version 0.2.0.

pyglui v1.31.0

This new version of pyglui introduces support for multi-line thumb and icon labels.

Example:

from pyglui import ui

context = {"state": False}

ui.Icon(
    "state",
    context,
    label="Multi\nLine",
    label_offset_size=-10,  # smaller font
    label_offset_x=0,  # no horizontal adjustments
    label_offset_y=-10,  # move label upwards
    label_line_height=0.8,  # reduce line height
)

zeromq-pyre >= v0.3.4

Starting with version v0.3.4, zeromq-pyre is able to handle cases gracefully where it is not able to access any network interfaces. See #2187 for reference.

Plugin API

Easy plugin icon customization - #2196

Plugins can now overwrite the following class attributes to customize the menu icon:

icon_pos_delta: relative positioning
icon_size_delta: relative font resizing
icon_line_height (multi-line icons only): line distance

Requires pyglui v1.31.0 or higher

Real-time Network API

Add `surf_to_dist_img_trans` and `dist_img_to_surf_trans` to surface events #2168

These two homographies are necessary to transform points from the surface coordinate system to the 2d image space of the scene camera, and vice versa. They have been available as part of the surface location exports in Pupil Player. With this release, they also become available to other plugins in Pupil Capture or via its Network API.

Example plugin that access the new homographies in real-time:

import numpy as np
from plugin import Plugin

class Example(Plugin):
    def recent_events(self, events):
        # access surfaces or fallback to empty list if not available
        surfaces = events.get("surfaces", [])
        for surface in surfaces:
            # access raw homography data, a list of lists
            homography = surface["surf_to_dist_img_trans"]
            # convert to a matrix
            homography = np.array(homography)
            # use homography, e.g. printing it
            print(f"{surface['name']}: {homography}")

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, such as WinRAR or 7-Zip (both are available for free).

Supported operating systems

macOS Mojave 10.14, macOS Catalina 10.15, macOS Big Sur 11
macOS Monterey 12 requires administrator privileges to access the video cameras
Ubuntu 16.04 or newer
Windows 10

Checksums

$ shasum -a 256 pupil_v3.5*
41428173c33d07171bb7bf8f0c83b7debfabb66a1dfc7530d1ae302be70ccc9c  pupil_v3.5-1-g1cdbe38_windows_x64.msi.rar
608aceaf4bf450254812b3fcaf8ea553450fcf2d626ae9c2ed0404348918d6ad  pupil_v3.5-7-g651931ba_macos_x64.dmg
786feb5e937a68a21821ab6b6d86be915bcbb7c5e24c11664dbf8a95e5fcb8cc  pupil_v3.5-8-g0c019f6_linux_x64.zip

Pupil Capture, Player, and Service release

2021-10-27T08:26:46Z

We are pleased to announce the release of Pupil Core software v3.4!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord! 😄

Overview

Pupil v3.4 introduces a new color scheme for pupil detection algorithms, improved gaze-estimation stability for frozen eye models, and a new IMU visualization and export plugin for Pupil Invisible recordings within Pupil Player.

New

Dynamic pye3d model confidence
Pupil detection color scheme
Added IMU Timeline plugin

Improved

Gaze-estimation stability with frozen eye models
Annotation log message
Prevent log flood in Surface Tracker related to finding homographies
Support upgrading legacy recordings

New

Dynamic pye3d model confidence

Pupil Core software version v3.4 ships the new pye3d version 0.1.1.

In the previous versions of pye3d model_confidence was fixed to a value of 1.0. Starting with version 0.1.0, pye3d assigns different model_confidence values based on the long-term model's parameters. Specifically, model_confidence is set to 1.0 by default and set to 0.1 if at least one model output exceeds its physiologically reasonable range. If the ranges are exceeded, it is likely that the model is either not fit well or the 2d input ellipse was a false detection.

The assumed physiological bounds are:

phi: [-80, 80] degrees
theta: [-80, 80] degrees
pupil diameter: [1.0, 9.0] mm
eyeball center x: [-10, 10] mm
eyeball center y: [-10, 10] mm
eyeball center z: [20, 75] mm

Phi and theta ranges are relative to the eye camera's optical axis. The eye ball center ranges are defined relative to the origin of the eye camera's 3d coordinate system.

The model_confidence will be set to 0.0 if the gaze direction cannot be calculated.

Pupil detection color scheme

The new pupil detection color scheme makes use of the dynamic model confidence values by visualizing within-bounds and out-of-bounds models in different colors.

Color legend:

#FAC800: Pupil ellipse (2d)
#F92500: Pupil ellipse (3d)
#0777FA: Long-term model outline (within-bounds)
#80E0FF: Long-term model outline (out-of-bounds)

Note: Even though the detected pupil ellipses fit the pupil well on the right-side picture, the model is outside of the physiological ranges defined above. This can lead to errors in gaze-direction and pupil-size estimates. You can improve such a model by looking into different directions.

While the 3d debug window is opened, pye3d also visualizes the short- and ultra-long-term model outlines:

#FFB876: Short-term model outline
#FF5C16: Ultra-long-term model outline

Added IMU Timeline plugin - #2151

This is a new plugin designed to visualize and export IMU data available in Pupil Invisible recordings.

In addition to the raw angular velocity and linear acceleration, the plugin also uses Madgwick's algorithm to estimate pitch and roll.

To export the data, the plugin must be enabled and a raw data export must be triggered. Once the export is complete, there should be a imu_timeline.csv file with the following columns:

Key	Description
`imu_timestamp`	Timestamp of the IMU reading
`world_index`	Associated_frame: closest world video frame
`gyro_x`	Angular velocity about the x axis in degrees/s
`gyro_y`	Angular velocity about the y axis in degrees/s
`gyro_z`	Angular velocity about the z axis in degrees/s
`accel_x`	Linear acceleration along the x axis in G (9.80665 m/s^2)
`accel_y`	Linear acceleration along the y axis in G (9.80665 m/s^2)
`accel_z`	Linear acceleration along the z axis in G (9.80665 m/s^2)
`pitch`	Orientation expressed as Euler angles
`roll`	Orientation expressed as Euler angles

Improved

Improve gaze-estimation stability with frozen eye models

Freezing the eye models is the best way to reduce noisy data when in a controlled environment. Prior to this release, freezing the eye model only affected eye ball center and pupil diameter measurements. Starting with this version, freezing the eye model also stabilizes gaze direction estimates.

In order to adapt to slippage, pye3d estimates eyeball positions on different time scales. These serve specific downstream purposes. By default, the short-term model, which integrates only the most recent pupil observations, is used for calculating raw gaze-direction estimates.

In highly controlled environments with few eye movements (e.g. subject uses head rest and is asked to fixate a static target), the most recent pupil observations might not be sufficient to build a stable short-term model. If this is the case, the gaze direction varies more than one would expect given the controlled environment.
Starting in pye3d version 0.1.0, the gaze direction will be inferred from the long-term model if it is frozen.

Freezing the eye model is only recommended in controlled environments as this prevents the model from adapting to slippage. In turn, it is able to provide very stable data.

Improve annotation log message - #2148

When creating a new annotation, the log message will no longer show the Pupil Timestamp. Instead, Pupil Capture will show how old the annotation is, while Pupil Player will show the frame index of the world video.

This change should help avoid the confusion associated with displaying Pupil Timestamp.

Prevent log flood in Surface Tracker related to finding homographies - #2150

Previously, failing to locate a surface because of missing homographies would print an error message in the main window. In some cases, this would result in a flood of error logs.

This change ensures that failures of such type are only issued once, and are written to the log file directly.

Support upgrading legacy recordings - #2161

Pupil Core software version v1.16 is no longer necessary to upgrade deprecated recording formats. Instead, Pupil v3.4 performs all necessary upgrades by itself. This release also includes a bug fix for an issue with pre v0.7.4 recordings.

In v1.16, recordings made with Pupil Capture v1.2 or earlier, and
Pupil Mobile r0.22.x or earlier have been deprecated due to the fact that these recordings are missing meta information that is required for the upgrade to the Pupil Player Recording Format 2.0. For details see "Missing Meta Information" in the v1.16 release notes.

Fixed

Use UTF-8 decoding for known UTF-8 encoded text files - #2146
Update circle drawing to display correctly on Apple Silicon devices - #2147
Fix crash on surface tracker export if filling the marker cache is not complete - #2157
Fix surface tracker marker export to use export range - #2158

Developer notes

Dependencies updates

Python dependencies can be updated using pip and the requirements.txt file:

python -m pip install --upgrade pip wheel
pip install -r requirements.txt

pyglui v1.30.0

This new version of pyglui includes the menu icon for the new IMU timeline plugin as well as the new Color_Legend UI element.

from pyglui import ui

color_rgb_red = (1.0, 0.0, 0.0)
label = "red color label"
ui.Color_Legend(color_rgb_red, label)

pye3d v0.1.1

This version improves gaze-estimation stability for frozen eye models. For details see above and the pye3d changelog.

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, such as WinRAR or 7-Zip (both are available for free).

Supported operating systems

Windows 10
Ubuntu 16.04+
macOS Mojave 10.14+
macOS Monterey 12 is not supported by this release

Pupil Capture, Player, and Service release

2021-06-25T10:23:25Z

We are pleased to announce the release of Pupil Core software v3.3!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord! 😄

Overview

Pupil v3.3 focuses on improving software usability, stability and performance. We have made improvements to how product-specific plugins are loaded in Pupil Player, updated keyboard shortcuts, and fixed several stability issues. Additionally, the surface tracker now exports marker detection results.

Note: Prior to this release,diameter_3d data for all Pupil Core v3 recordings did not have corneal refraction applied. See pye3d v0.0.7 notes below for updates and post-hoc fixes.

Improved

pye3d v0.0.7

This release comes with our latest pye3d version including the following improvements

Performance improvements - #27
Simplification of Conic parameter calculation - #26
Correctly apply corneal-refraction correction to diameter_3d result - #28

All pye3d versions prior to 0.0.7 do not correctly apply corneal refraction to the diameter_3d field. To check if you are affected by this bug, take a look at the method column of your pupil_positions.csv export.

You have two options to get correctly calculated pupil size data:

Use the circle_3d_radius values and multiply them by 2, or
Re-run the post-hoc pupil detection in Pupil Player v3.3 or later

Marker detection export - #2140

The Surface Tracker plugin requires fiducial markers for defining and tracking areas of interest. Until now, the detection result of these was only stored as an intermediate result. With this release, the data is also stored as a CSV file next to other surface-tracking related exports.

The data is exported to surfaces/marker_detections.csv.

Column name	Description
world_index	Scene video frame index
marker_uid	Label for identifying the marker
corner_0/1/2/3_x/y	Corner coordinates in pixels

Improve surface detection consistency - #2139

Surfaces can now be defined as long as one or more markers are detected in the scene video. Previously, this was only sufficient in some cases -- usually, at least two markers were needed.

Note: Surface tracking works best if you use multiple markers spread-out over the area of interest to define and track your surface. The surface position export files contains information about how many markers were detected for each surface position. You can use this information as an estimation of the tracking stability.

Product-specific plugin loading in Pupil Player, part 2 - #2125

We continue work on finetuning the list of enabled plugins for Pupil Invisible recordings.

Starting with this release we have removed the “Pupil from recording” option, as there is no pupillometry data to load from Pupil Invisible recordings. Pupil Player will no longer generate an empty pupil_positions.csv file after export.

Keyboard shortcuts update - #2129

The default annotation keyboard shortcut was changed to x in both Pupil Capture and Pupil Player to avoid collision with other plugin shortcuts.

We've posted tables of keyboard shortcuts for Pupil Capture and Pupil Player below. You can find these tables in our Capture and Player documentation.

Fixed

Fixed HMD calibration crashing Pupil Service - #2120
Fixed Pupil Groups serialization issue - #2124
Fixed toggling pupil detection on and off - #2126
Fixed calibration area calculation errors - #2128
Fixed video source replacement in eye process - #2131
Fixed gaze visualization on Apple Silicon computers - #2132
Fixed Pupil Invisible timestamp update procedure precision loss - #2134
Made pye3d plugin's debug visualization colors consistent - #2136
Fixed validation crash when not enough reference data is collected - #2137
Fixed a crash when using Time Sync on macOS - #2138

Developer notes

Dependencies updates

Python dependencies can be updated using pip and the requirements.txt file:

python -m pip install --upgrade pip wheel
pip install -r requirements.txt

We have specifically updated the version requirement for pye3d.

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, such as WinRAR or 7-Zip (both are available for free).

Pupil Capture, Player, and Service release (Updated)

2021-03-21T18:56:20Z

We are pleased to announce the release of Pupil Core software v3.2!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord! 😄

Overview

Pupil v3.2 focuses on quality-of-life and stability.

New

We added accuracy calculation and visualisation for VR/AR calibrations.
Enable plugins in Pupil Player only if they are able to be used with the product that created the recordings.

Improved

Pupil Player has improved playback for Pupil Invisible recordings.

New

Accuracy visualisation and calculation for HMD calibrations - #2112

Prior to this release, when performing HMD calibration, accuracy was not calculated nor visualised in Pupil Capture. This release makes accuracy calculation and visualisation consistent across setups.

Product specicific Pupil Player plugin loading - #2100

Load plugins in Pupil Player only if they are able to be used with the product that created the recordings.

Pupil Player is a very flexible tool. It can load, playback, visualize, and export data from recordings made with Pupil Core, Pupil Mobile, and Pupil Invisible! Each of these products has different capabilities and data formats for their recordings.

We have evaluated all Pupil Player plugins and their validity for each product. Starting with this release, Pupil Player will only enable plugins that are suitable for the active recording type.

Here is a list of what plugins are available based on the product that generated the recording:

Pupil Player Plugin	Pupil Core	Pupil Invisible	Pupil Mobile
Blink detection	✔️	➖	✔️
Fixation detection	✔️	➖	✔️
Post-hoc pupil detection	✔️	➖	✔️
Post-hoc gaze calibration	✔️	➖	✔️
Pupil from recording	✔️	➖	➖
Gaze from recording	✔️	✔️	➖

Improved

pye3d detection speed improvement

We are aware that pye3d requires more computational resources than its predecessor, and are actively working to improve the speed. For some users, it has become more difficult to reach a consistent detection rate of 200Hz.

The first step we have taken to speed up pye3d is to disable some optional computations that are not currently used by Pupil Capture and Pupil Player (jump to developer notes for details).

Fixed

Support recordings with decoding issues - #2107
Support recordings when an audio device fails - #2103
Fixed laggy playback after seeking on Pupil Invisible recordings - #2108

Developer notes

Dependencies updates

Python dependencies can be updated using pip and the requirements.txt file:

python -m pip install --upgrade pip wheel
pip install -r requirements.txt

pye3d v0.0.6

pye3d v0.0.6 allows disabling optional computations. Pupil Core is not making use of these optional computations, they have been disabled, effectively reducing the computational load of pye3d pupil detections. Read about the changes here.

Release Note Updates

15.03.2021 - v3.2-16

Fixes an issue causing Pupil Player to crash on Windows when enabling the eye video overlay or post-hoc pupil detection - #2117

21.03.2021 - v3.2-20

Improves HMD Calibration stability in Pupil Service, fixes #2119 - eb8c232

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, such as WinRAR or 7-Zip (both are available for free).

Pupil Capture, Player, and Service release

2021-02-10T10:43:03Z

We are pleased to announce the release of Pupil Core software v3.1!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord! 😄

Motivation

Pupil Core software is now compatible with macOS 11 Big Sur! In addition, we have added support in Pupil Player for 200Hz gaze data downloaded from Pupil Cloud.

Improvements

macOS 11 Big Sur compatibility - #2086, #2087, #2090, #2095

macOS 11 Big Sur introduced backwards-incompatible changes that caused earlier versions of Pupil Core and other software to not run properly.

There is currently only a single Python version, Python 3.9.1, that is fully compatible with macOS Big Sur. As a result, our new macOS Pupil Core bundle ships with this new Python version. This may have an effect on user plugin dependencies. For more information, check out the developer notes below.

Support for Pupil Cloud's 200Hz gaze recordings - #2082

Due to the computational limitations on the Companion Device, gaze data is recorded at 66Hz. Recent updates to Pupil Cloud automatically "upgrades" the gaze data recordings when they are uploaded to Pupil Cloud providing you with 200Hz gaze data. In version 3.1, Pupil Player can read and export the new data.

Note: Recordings that were downloaded prior to this Pupil Cloud update need to be re-downloaded in order to include the 200Hz gaze data.

Changes

Visual calibration marker appearance - #2095

The visual appearance of the calibration marker changed slightly, due to how we render the calibration marker on screen. The change was necessary to support screen-based calibrations on Apple Silicon Macs. As a result, the marker edges do not appear as smooth as the previous version. The overall design and detection process remains the same.

Bug Fixes

Gracefully handle Pupil Invisible video files starting with a null frame - #2077, #2089
Handle zero-byte video files when looking up Pupil Mobile intrinsics #2083

Developer notes

Dependencies updates

Python dependencies can be updated using pip and the requirements.txt file:

python -m pip install --upgrade pip wheel
pip install -r requirements.txt

Support latest GLEW 2.2 - pyglui #117

This release also adds support for the latest version of GLEW. This means that the pyglui dependency needs to be updated to version 1.29.

pye3d v0.0.5

We have included pye3d==0.0.5 which fixes compatibility issues with numpy==1.20 and scikit-learn==0.24.1.

macOS bundle shipping with Python 3.9.1

To simplify the usage of the application bundles, each bundle is shipped with their own Python environment. All plugins and their dependencies will run in the context of this environment.

Especially compiled dependencies, e.g. pyrealsense2, are always compiled against a specific Python version. If you are running custom user plugins, you will have to reinstall compiled dependencies in order to make them work with the new Python version.

Bundles ship with the following Python versions:

macOS: Python 3.9.1
Windows bundle: Python 3.6
Linux bundle: Python 3.6

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, such as WinRAR or 7-Zip (both are available for free).

macOS compatibility - #2086, #2087

Note: This is the first release to support macOS 11 Big Sur. We no longer support macOS 10.13 High Sierra since the macOS package manager homebrew no longer provides pre-compiled binaries for Pupil Core software dependencies.

Pupil Capture, Player, and Service release

2021-01-07T09:23:21Z

We are pleased to announce the release of Pupil Core software v3.0!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord! 😄

Motivation

We are excited to introduce pye3d - a complete overhaul of the 3D pupil detector employed by Pupil Core software. In building pye3d, we have left no stone unturned, reimagining many of the algorithmic building blocks constituting the final gaze-estimation pipeline. pye3d is the result of three years of hard work and we are excited to get it into your hands!

Improvements

Improved built-in 3D pupil detector - #2011, #2065

pye3d implements a novel eye-model fitting strategy that incorporates a corneal refraction-correction step, accounting for nonlinear effects in ocular optics. Gaze estimates and pupil diameter estimates are now more accurate than ever before [2,3]. Thanks to a new sphere center estimation methodology [3], it is also considerably faster than our previous 3D detector [1] in regards to fitting robust and accurate eye models. Read more about how it works in our documentation.

Academic references

[1] L. Świrski and N. A. Dodgson. A Fully-Automatic, Temporal Approach to Single Camera, Glint-Free 3D Eye Model Fitting. In Proceedings of ECEM 2013. "L. Świrski and N. A. Dodgson. A Fully-Automatic, Temporal Approach to Single Camera, Glint-Free 3D Eye Model Fitting. In Proceedings of ECEM 2013."
[2] K. Dierkes, M. Kassner, A. Bullling, A novel approach to single camera, glint-free 3D eye model fitting including corneal refraction. In ETRA ’18: Symposium on Eye Tracking Research and Applications, 2018.
[3] K. Dierkes, M. Kassner, A. Bulling. A fast approach to refraction-aware eye-model fitting and gaze prediction. In ETRA ’19: Symposium on Eye Tracking Research and Applications, 2019.

Changes

Removal of old Pupil 3D detector

With this release, we also stop the maintainance of the previous 3D detector. Starting with the new 2.0 release, our pupil-detectors package no longer includes the previous 3D detector. This allows us to remove Ceres as a dependency, making the project much easier to maintain and build from source.

Should you rely on this detector, you can still download previous Pupil Core software releases or install previous pupil-detectors via PyPI.

Bug Fixes

Fixed issue that would leave an unresponsive eye window due to plugin crash - #2059
Fixed crash on canceling single marker calibration window mode - #2062
Fixed crash on post-hoc pupil detection when the frame count is zero - #2063

Developer notes

Network API Changes

Reworked pupil detector plugin and network API

In this release we reworked the pupil detector plugin base class. It is now easier than ever to implement a custom pupil detector and accompanying pupil detector plugin.

We have also introduced breaking changes to our network API for pupil detector plugins. Here is a list of possible notification payload descriptions that are handled by detector plugins:

# Notification for enabling/disabling any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.set_enabled',
    'subject': 'pupil_detector.set_enabled',
    'value': <is_enabled: bool>,
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}

# Notification for setting the Region-Of-Interest (ROI) for any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.set_roi',
    'subject': 'pupil_detector.set_roi',
    'value': <roi: (min_x: int, min_y: int, max_x: int, max_y: int)>,
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}

# Notification for updating a partial set of pupil detector properties for a specific pupil detector plugin
{
    'topic': 'notify.pupil_detector.set_properties',
    'subject': 'pupil_detector.set_properties',
    'values': <detector_properties: dict>,
    'eye_id': <eye_id: int>,  # required; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # required
}

# Notification for requesting pupil detector properties broadcast for any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.broadcast_properties',
    'subject': 'pupil_detector.broadcast_properties',
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}

For more information about custom pupil detector plugins, please read the documentation. You can also see examples of custom pupil detection plugins here.

For an example script that showcases the network API, please consult this helper script.

Changed 3D pupil datum format

pye3d uses a slightly different pupil format than our previous 3D detector. Because pye3d continously updates a single model instead of fitting multiple parallel ones, the new 3D pupil datum no longer contains the following keys: model_id and model_birth_timestamp.

Our latest Hmd-eyes release handles this by falling back to "0" and 0.0, respectively.

Binocular 3D gaze data with string-only keys - #2068

In order to be compatible with the msgpack-python v1.0.0 (specifially the new strict_map_key=True default), this release uses strings only as dictionary keys for all msgpack-encoded data that is published via the Network API.

This change affects primarily binocular 3D gaze data which used to contain integer dictionary keys.

Raw Data Exporter - Optional `model_id` key - #2061

Some of the fields in the pupil datum related to the 3D model no longer exist in pye3d (see above). This change ensures the raw data export handles missing fields and outputs a compatible csv file.

Updated `msgpack` dependency to version `1.0` - #2068

We continue to streamline and simplify the installation of Pupil Core dependencies. In this release we updated the msgpack to the latest 1.0. To ensure you have the latest required versions of all dependencies, please install them via requirements.txt.

python -m pip install --upgrade pip wheel
pip install -r requirements.txt

Added support for custom receiver high-water mark for HMD Streaming source - #2058

This change adds the ability to set a custom ZMQ high watermark value for HMD Streaming source. This gives users more control over the streaming behaviour when using Pupil Core with an HMD headset.

Added `launcher_process.should_stop` notification handling #2070

This change adds the ability to request stopping Capture, Player or Service with the same notification.

import zmq
import msgpack


# create and connect PUB socket to IPC
pub_socket = zmq.Socket(zmq.Context(), zmq.PUB)
pub_socket.connect(ipc_pub_url)


subject = "launcher_process.should_stop"
topic = "notify." + subject
payload = {
    'topic': topic,
    'subject': subject,
}

pub_socket.send_string(topic, flags=zmq.SNDMORE)
pub_socket.send(msgpack.dumps(payload, use_bin_type=True))

Simplified running from source on Windows - #2073

We have removed the necessity to run Pupil from source using the run_<app>.bat files on Windows. Instead of extending the PATH environment variable via the .bat file, we extend it via the application launcher. This also makes it easier to attach a debugger, e.g. via Visual Studio Code.

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, such as WinRAR or 7-Zip (both are available for free).

macOS (11) Big Sur compatibility - #2079

Note: Pupil Core is not currently supported on Mac OS Big Sur. We aim to release a compatability update within the next few weeks. 👷

v3.0.dev1

2020-12-09T09:37:54Z

v3.0.dev1

Pupil Capture, Player, and Service release

2020-11-16T16:58:36Z

We are pleased to announce the release of Pupil Core software v2.6!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord 😄

Motivation

In v2.6, we are taking a big step towards our modularity goal. We now support custom user pupil detector plugins. This change makes it easier than ever for you to modify the existing pupil detection algorithms or to build your own! You will no longer have to run Pupil Core software from source to get started with your custom pupil detector! Read more about it in our documentation.

Improvements

Support custom pupil detection plugins - #2033

You can now run your own pupil detection plugins that run in the eye processes. These plugins can be used to implement custom algorithms for extracting pupillometry data from the eye images and feeding it to the rest of the gaze mapping pipeline.

Pupil detection plugins are supported in Pupil Capture, Pupil Player, and Pupil Service. Similar to regular user plugins, custom pupil detection plugins should be placed inside pupil_capture_settings/plugins for Pupil Capture, pupil_player_settings/plugins for Pupil Player, and pupil_service_settings/plugin for Pupil Service. The plugins are automatically loaded when the applications starts.

Support HMD Streaming Source in eye process - #2052

We have added support for using HMD Streaming Source in the eye process. This also adds support for BGR and Gray frame format for HMD streams.

Support Pupil Invisible recording format v1.4 - #2054

This new recording format has been introduced in Pupil Companion 1.0.0. This version provides support for the OnePus 8 phone and Android 11. On the OnePlus 8, eye videos will be h264-encoded.

Bug Fixes

Removed import * statements - #2036
Fixed typo in circle detector benchmark - #2038
Fixed CPU graph and UI refresh performance - #2040, #2041
Fixed gaze with negative z-component - #2043
Fixed issue when minimizing 3D pupil detector debug window on Windows - #2047
Fixed UVC camera auto-selection priority in case of multiple pattern matches - #2051

Developer Notes

Added ability to stop eye process plugins via notifications - #2039

Another small improvement for eye process plugin management in this release is the ability to stop plugins via notifications. This gives better control over the list of plugins running in the eye process.

import zmq
import msgpack

# class name of the plugin to stop
plugin_name = "..."

# create and connect PUB socket to IPC
pub_socket = zmq.Socket(zmq.Context(), zmq.PUB)
pub_socket.connect(ipc_pub_url)


for eye_id in (0, 1):
    subject = "stop_eye_plugin"
    topic = "notify." + subject
    payload = {
        'topic': topic
        'subject': subject,
        'target': f"eye{eye_id}",
        'name': plugin_name,
    }

    pub_socket.send_string(topic, flags=zmq.SNDMORE)
    pub_socket.send(msgpack.dumps(payload, use_bin_type=True))

Added `requirements.txt` file - #2029

We continue to streamline and simplify the installation of Pupil Core dependencies. You can now use the requirements.txt file from the root of the pupil repository to install and update all Python dependencies.

python -m pip install --upgrade pip wheel
pip install -r requirements.txt

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, e.g. WinRAR.

Pupil Capture, Player, and Service release

2020-10-20T09:00:17Z

We are pleased to announce the release of Pupil Core software v2.5!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord 😄

Motivation

Pupil v2.5 focuses on further improvements to software stability. We have fixed multiple issues that prevented very old recordings from being opened in Pupil Player.

Improvements

Replacing internal GLFW bindings with pyGLFW - #2015

In this release we replaced the internal glfw.py bindings to GLFW with the external pyGLFW package. This change simplifies our dependency list, making easier to install GLFW on Windows.

New default camera intrinsics - #2023

We use camera intrinsics in multiple places to transform between the 2d distorted image space and the 3d undistorted camera space. With this release, we have updated the default scene camera intrinsics to perform this tansformation more accuractely at the image borders.

Note: Each camera is slightly different. Therefore, it is recommended to run your own camera intrinsics estimation to get the best possible transformations for your camera.

Bug Fixes

Fixed a crash in Pupil Player when attempting to visualize invalid data - #2021
Fixed an edge-case when handling recordings with negative time jumps - #2026, #2028
Fixed a frame drop issue on macOS when windows were occluded - #2027
Fixed an issue that caused instabilities when starting the Head Pose tracker - #2022
Surface Tracker stability improvements - #2025

Developer Notes

Improved Observable class - #2022

Improvements to the Observable class enable you to observe monkey patched methods (e.g. replaced methods that were not in the original class definition). Additionally, trying to add private methods as observers now fails with a more explicit error.

Changed Requirements

Please install the new external glfw dependency:

pip install glfw

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, e.g. WinRAR.

Release notes from pupil

v3.6

Pupil Capture, Player, and Service release

Overview

New

Recording-specific annotation hotkeys - #2170

Add option to filter out low-confidence values in raw data export - #2210

Add option to disable world video recording - #2169

macOS Monterey support

Improved

Export-in-progress file-name suffixes - #2167, #2186

Reduced number of user-facing log messages - #2190

Windows: Handle exceptions during driver installation gracefully - #2173

Windows: Option to skip automatic driver installation - #2207

Windows: Create start menu entries and desktop shortcuts for all users on install - #2166

Add Euler angles to headpose tracker result - #2175

Normalize IMU pitch/roll values to [-180, 180] - #2171

Confidence adjustment for pye3d results - #2195

Cache post-hoc fixation detector results - #2194

Documentation

Fixed

Known issues

Developer notes

Dependencies updates

pye3d v0.2.0 and v0.3.0

pyglui v1.31.0

zeromq-pyre >= v0.3.4

Plugin API

Easy plugin icon customization - #2196

Real-time Network API

Add surf_to_dist_img_trans and dist_img_to_surf_trans to surface events #2168

Downloads

Supported operating systems

Checksums

Pupil Capture, Player, and Service release

Overview

New

Dynamic pye3d model confidence

Pupil detection color scheme

Added IMU Timeline plugin - #2151

Improved

Improve gaze-estimation stability with frozen eye models

Improve annotation log message - #2148

Prevent log flood in Surface Tracker related to finding homographies - #2150

Support upgrading legacy recordings - #2161

Fixed

Developer notes

Dependencies updates

pyglui v1.30.0

pye3d v0.1.1

Downloads

Supported operating systems

Pupil Capture, Player, and Service release

Overview

Improved

pye3d v0.0.7

Marker detection export - #2140

Improve surface detection consistency - #2139

Product-specific plugin loading in Pupil Player, part 2 - #2125

Keyboard shortcuts update - #2129

Fixed

Developer notes

Dependencies updates

Downloads

Pupil Capture, Player, and Service release (Updated)

Overview

New

Improved

New

Accuracy visualisation and calculation for HMD calibrations - #2112

Product specicific Pupil Player plugin loading - #2100

Improved

pye3d detection speed improvement

Fixed

Developer notes

Dependencies updates

pye3d v0.0.6

Release Note Updates

15.03.2021 - v3.2-16

21.03.2021 - v3.2-20

Add `surf_to_dist_img_trans` and `dist_img_to_surf_trans` to surface events #2168

Raw Data Exporter - Optional `model_id` key - #2061

Updated `msgpack` dependency to version `1.0` - #2068

Added `launcher_process.should_stop` notification handling #2070

Added `requirements.txt` file - #2029