ataraxis-video-system

Interfaces with a wide range of cameras to flexibly record visual stream data as video files.

Detailed Description

This library abstracts all necessary steps for acquiring and saving video data. During each runtime, it interfaces with one or more cameras to grab the raw frames and encodes them as video files stored in the non-volatile memory. The library is specifically designed for working with multiple cameras at the same time and supports fine-tuning the acquisition and saving parameters to precisely balance the resultant video quality and real-time throughput for a wide range of applications. This library is part of the Ataraxis framework for AI-assisted scientific hardware control.

Features

Supports Windows, Linux, and macOS.
Uses OpenCV or GeniCam (Harvesters) to interface with a wide range of consumer, industrial, and scientific cameras.
Uses FFMPEG to efficiently encode acquired data as videos in real time using CPU or GPU.
Highly customizable and can be extensively fine-tuned for quality or throughput.
Supports inspecting, modifying, saving, and loading GenICam camera configurations for reproducible setups.
Provides a log data processing pipeline for extracting frame acquisition timestamps from runtime log archives, with post-processing tools for frame timing analysis and frame drop detection.
Generates camera manifest files that tag DataLogger output directories with source-to-name mappings, enabling downstream tools to identify which log archives were produced by ataraxis-video-system.
Includes an MCP server for AI agent integration (compatible with Claude Desktop and other MCP clients).
Apache 2.0 License.

Dependencies

FFMPEG version n8.1. The installed FFMPEG must be available on the system’s path and callable from Python processes.
A GenTL Producer interface compatible with the Harvesters library if the target camera requires the 'harvesters' camera interface. It is recommended to use the CTI interface supplied by the camera’s vendor, if possible, as this typically ensures that the camera performs as advertised. If the camera-specific CTI file is not available, it is possible to instead use a general interface, such as MvImpactAcquire. This library has been tested using MvImpactAcquire version 2.9.2.

For users, all other library dependencies are installed automatically by all supported installation methods. For developers, see the Developers section for information on installing additional development dependencies.

Installation

Source

Note, installation from source is highly discouraged for anyone who is not an active project developer.

Download this repository to the local machine using the preferred method, such as git-cloning. Use one of the stable releases that include precompiled binary and source code distribution (sdist) wheels.
If the downloaded distribution is stored as a compressed archive, unpack it using the appropriate decompression tool.
cd to the root directory of the prepared project distribution.
Run pip install . to install the project and its dependencies.

pip

Use the following command to install the library and all of its dependencies via pip: pip install ataraxis-video-system

Usage

OS Support Status

While this library works on all major operating systems, it is largely up to the maintainers of the low-level library components (OpenCV, Harvesters, FFMPEG) to ensure that the operation is smooth on each supported OS. That, in turn, is not always possible for many nuanced reasons. This section summarizes the current state of the library for the three explicitly supported operating systems: macOS, Windows, and Linux.

Linux

This library was primarily written on and for Linux systems. It is extensively tested on Linux and performs well under all test conditions. It is very likely that Linux users will not experience any issues specific to this library.

Windows

The library is mostly stable on Windows systems, but requires additional setup to ensure smooth operation. First, the FFMPEG has to be updated to the latest stable version, as older versions may have a drastically reduced encoding speed even with hardware acceleration. Additionally, some of the advanced OpenCV’s features, such as the MSMF HW transformations, have to be disabled to support smooth runtimes on the Windows platform. Typically, the information of which features to disable is readily available from the OpenCV’s Windows community.

macOS

macOS mostly works as expected except for live frame displaying, which does not work for modern macOS devices. The issue is due to the OS restriction on drawing certain GUI elements outside the main thread of the application. The restriction interferes with the library, as it displays the acquired frames from the same process that interfaces with the camera to minimize the visual lag between grabbing and displaying the frame. This is a persistent issue that is unlikely to be fixed any time soon.

Quickstart

This is a minimal example of how to use this library. This example is intentionally kept minimal, consult the API documentation for all available VideoSystem configuration parameters.

Most library functionality is accessible through the VideoSystem class:

import tempfile
from pathlib import Path

import numpy as np
import polars as pl
from ataraxis_time import PrecisionTimer
from ataraxis_data_structures import DataLogger, assemble_log_archives
from ataraxis_base_utilities import console, LogLevel

from ataraxis_video_system import VideoSystem, VideoEncoders, CameraInterfaces, run_log_processing_pipeline

# Since the VideoSystem and DataLogger classes use multiprocessing under-the-hood, the runtime must be protected by the
# __main__ guard.
if __name__ == "__main__":

    # Enables the console module to communicate the example's runtime progress via the terminal.
    console.enable()

    # Specifies the directory where to save the acquired video frames and timestamps.
    tempdir = tempfile.TemporaryDirectory()  # Creates a temporary directory for illustration purposes
    output_directory = Path(tempdir.name)

    # The DataLogger is used to save frame acquisition timestamps to disk as uncompressed .npy files.
    logger = DataLogger(output_directory=output_directory, instance_name="webcam")

    # The DataLogger has to be started before it can save any log entries.
    logger.start()

    # The VideoSystem requires a unique system_id, a DataLogger instance, a human-readable name, and an output
    # directory. The system_id identifies this camera's log entries. The name is written to the camera manifest
    # file alongside the system_id to support downstream log processing discovery.
    vs = VideoSystem(
        system_id=np.uint8(101),
        data_logger=logger,
        name="webcam",
        output_directory=output_directory,
        camera_interface=CameraInterfaces.OPENCV,  # OpenCV interface for webcameras
        display_frame_rate=15,  # Displays the acquired data at a rate of 15 frames per second
        color=False,  # Acquires images in MONOCHROME mode
        video_encoder=VideoEncoders.H264,  # Uses H264 CPU video encoder.
        quantization_parameter=25,  # Increments the default qp parameter to reflect using the H264 encoder.
    )

    # Calling this method arms the video system and starts frame acquisition. However, the frames are not initially
    # saved to disk.
    vs.start()
    console.echo(f"VideoSystem: Started", level=LogLevel.SUCCESS)

    console.echo(f"Acquiring frames without saving...")
    timer = PrecisionTimer("s")
    timer.delay(delay=5, block=False)  # During this delay, camera frames are displayed to the user but are not saved

    # Begins saving frames to disk as an MP4 video file.
    console.echo(f"Saving the acquired frames to disk...")
    vs.start_frame_saving()
    timer.delay(delay=5, block=False)  # Records frames for 5 seconds, generating ~150 frames
    vs.stop_frame_saving()

    # Frame acquisition can be started and stopped as needed, although all frames are written to the same output
    # video file.

    # Stops the VideoSystem runtime and releases all resources.
    vs.stop()
    console.echo(f"VideoSystem: Stopped", level=LogLevel.SUCCESS)

    # Stops the DataLogger and assembles all logged data into a single .npz archive file. This step is required to be
    # able to extract the timestamps for further analysis.
    logger.stop()
    console.echo(f"Assembling the frame timestamp log archive...")
    assemble_log_archives(remove_sources=True, log_directory=logger.output_directory, verbose=True)

    # Runs the log processing pipeline to extract frame acquisition timestamps from the assembled log archive. The
    # pipeline writes results as Feather files in a camera_timestamps/ subdirectory under the specified output
    # directory. Source IDs are validated against the camera_manifest.yaml auto-generated by the VideoSystem.
    console.echo(f"Extracting frame acquisition timestamps from the assembled log archive...")
    processed_directory = output_directory / "processed"
    run_log_processing_pipeline(
        log_directory=logger.output_directory,
        output_directory=processed_directory,
        log_ids=["101"],
    )

    # Reads the processed timestamps from the output Feather file and computes the camera frame rate. The pipeline
    # writes output files into the camera_timestamps/ subdirectory.
    timestamps_directory = processed_directory / "camera_timestamps"
    dataframe = pl.read_ipc(source=timestamps_directory / "camera_101_timestamps.feather")
    timestamp_array = dataframe["frame_time_us"].to_numpy()
    time_diffs = np.diff(timestamp_array)
    fps = 1 / (np.mean(time_diffs) / 1e6)
    console.echo(
        message=(
            f"According to the extracted timestamps, the interfaced camera had an acquisition frame rate of "
            f"approximately {fps:.2f} frames / second."
        ),
        level=LogLevel.SUCCESS,
    )

    # Cleans up the temporary directory before shutting the runtime down.
    tempdir.cleanup()

Data Logging

This library relies on the DataLogger class to save frame acquisition timestamps to disk during runtime. Each saved frame’s acquisition timestamp is serialized and saved as an uncompressed .npy file.

The same DataLogger instance as used by the VideoSystem instances may be shared by multiple other Ataraxis assets that generate log entries, such as MicroControllerInterface instances. To support using the same logger instance for multiple concurrently active sources, each source has to use a unique identifier value (system id) when sending data to the logger instance.

Camera Manifest

Each VideoSystem instance automatically writes a camera_manifest.yaml file into the DataLogger output directory during initialization. The manifest associates the system_id with the human-readable name provided to the VideoSystem constructor. When multiple VideoSystem instances share the same DataLogger, each instance appends its entry to the same manifest file. The manifest is required by the log processing pipeline to identify which .npz archives were produced by ataraxis-video-system and to resolve source IDs for processing.

Log Format

Each frame’s acquisition timestamp is logged as a one-dimensional numpy uint8 array, saved as an .npy file. Inside the array, the data is organized in the following order:

The uint8 id of the data source (video system instance). The ID occupies the first byte of each log entry.
The uint64 timestamp that specifies the number of microseconds elapsed since the acquisition of the onset timestamp (see below). The timestamp occupies 8 bytes following the ID byte. This value communicates when each saved camera frame has been acquired.

Note, timestamps are generated at frame acquisition but are only submitted to the logger when the corresponding frame is saved to disk. Therefore, the timestamps always match the order that the saved frames appear in the video file.

Onset Timestamp

Each VideoSystem generates an onset timestamp as part of its start() method runtime. This log entry uses a modified data order and stores the current UTC time, accurate to microseconds, as the total number of microseconds elapsed since the UTC epoch onset. All further log entries for the same source use the timestamp section of their payloads to communicate the number of microseconds elapsed since the onset timestamp acquisition.

The onset log entry uses the following data organization order:

The uint8 id of the data source (video system instance).
The uint64 value 0 that occupies 8 bytes following the source id. A 'timestamp' value of 0 universally indicates that the log entry stores the onset timestamp.
The uint64 value that stores the number of microseconds elapsed since the UTC epoch onset. This value specifies the current time when the onset timestamp was generated.

Working with VideoSystem Logs

See the quickstart example above for a demonstration on how to assemble and parse the frame acquisition log archives generated by the VideoSystem instance at runtime.

Note, the parsed frame acquisition timestamps are stored as a contiguous numpy uint64 array (or as a Polars DataFrame column in Feather output files within the camera_timestamps/ subdirectory) that matches the order in which the frames were saved to disk as an .mp4 file. Each timestamp is given as the number of microseconds elapsed since the UTC epoch onset.

Log Processing

This library includes a log data processing pipeline for extracting frame acquisition timestamps from the .npz log archives generated by VideoSystem instances at runtime. The pipeline reads archives produced by the DataLogger, extracts timestamps for each saved frame, and writes the results as Polars DataFrames in Apache Feather (IPC) format.

The pipeline uses the camera manifest to identify which .npz archives were produced by ataraxis-video-system. A camera_manifest.yaml file must be present in the log directory for processing to succeed. When source IDs are not explicitly provided, the pipeline resolves all registered source IDs from the manifest automatically. When source IDs are provided explicitly, they are validated against the manifest to prevent accidental processing of non-video log archives.

The processing pipeline supports two execution modes. In local mode, all requested log archives are processed sequentially with automatic job tracking. In remote mode, a single job is executed by its unique identifier, enabling distributed processing across multiple compute nodes. Both modes use a YAML-based processing tracker to manage job lifecycle (scheduled, running, succeeded, or failed). All output files (tracker and Feather) are written into a camera_timestamps/ subdirectory under the specified output directory.

For single-directory processing, use the axvs process CLI command or the run_log_processing_pipeline() function directly. For multi-directory batch workflows, use the MCP server log processing tools, which handle archive discovery, batch preparation, concurrent execution, and status monitoring across multiple recording directories.

CLI

This library provides the axvs CLI that exposes the following commands:

Command	Description
`cti set`	Configures the library to use a specified GenTL Producer (.cti) file
`cti check`	Checks whether a valid .cti file is configured
`check devices`	Discovers all compatible cameras on the system
`check compatibility`	Verifies FFMPEG and GPU availability for video encoding
`run`	Starts an interactive video capture session
`process`	Processes log archives to extract frame acquisition timestamps
`mcp`	Starts the MCP server for AI agent integration
`configure read`	Reads a GenICam node value from a connected camera
`configure write`	Writes a value to a GenICam node on a connected camera
`configure dump`	Dumps GenICam configuration from a camera to a YAML file
`configure load`	Loads GenICam configuration from a YAML file to a camera

Use axvs --help or axvs COMMAND --help for detailed usage information.

MCP Server

This library provides an MCP server that exposes camera discovery, configuration, video recording, camera manifest management, and log data processing functionality for AI agent integration.

Starting the Server

Start the MCP server using the CLI:

axvs mcp

Available Tools

Tool	Description
`list_cameras`	Discovers all cameras compatible with OpenCV and Harvesters interfaces
`get_cti_status`	Checks whether the library is configured with a valid GenTL Producer (.cti) file
`set_cti_file`	Configures the library to use a specified CTI file for GeniCam camera support
`check_runtime_requirements`	Checks FFMPEG and GPU availability for video encoding
`start_video_session`	Starts a video capture session with specified camera and encoding parameters
`stop_video_session`	Stops the active video capture session and releases resources
`start_frame_saving`	Begins saving captured frames to a video file
`stop_frame_saving`	Stops saving frames while keeping the session active
`get_session_status`	Returns the current status of the video session
`assemble_log_archives_tool`	Consolidates raw .npy log entries into .npz archives by source ID
`validate_video_file_tool`	Validates a video file and extracts metadata using ffprobe
`read_genicam_node`	Reads a GenICam node value from a connected camera
`write_genicam_node`	Writes a value to a GenICam node on a connected camera
`dump_genicam_config`	Dumps GenICam configuration from a camera to a YAML file
`load_genicam_config`	Loads GenICam configuration from a YAML file to a camera
`read_camera_manifest_tool`	Reads a camera manifest file and returns its contents
`write_camera_manifest_tool`	Writes or updates a camera manifest file in a log directory
`discover_camera_data_tool`	Discovers confirmed camera recordings under a root directory via manifests
`prepare_log_processing_batch_tool`	Prepares a batch of log processing jobs across multiple directories
`execute_log_processing_jobs_tool`	Executes prepared log processing jobs with concurrent worker threads
`get_log_processing_status_tool`	Returns the current status of the active log processing session
`get_log_processing_timing_tool`	Returns timing information for all jobs in the active session
`cancel_log_processing_tool`	Cancels the active log processing execution session
`reset_log_processing_jobs_tool`	Resets failed or all jobs in a processing tracker for re-execution
`get_batch_status_overview_tool`	Summarizes processing status for all log directories under a root directory
`analyze_camera_frame_statistics_tool`	Computes frame timing statistics and frame drop analysis from feather files
`clean_log_processing_output_tool`	Deletes processed output directories for clean re-processing

Client Registration

MCP server registration and Claude Code skill assets for this library are distributed through the ataraxis marketplace as part of the video plugin. Install the plugin from the marketplace to automatically register the MCP server with compatible clients and make all associated skills available.

Using GeniCam Compatible Cameras

This library supports all cameras compatible with the GeniCam standard, which includes most GigE+ scientific and machine vision cameras.

Note, before using the library with a GeniCam camera, it must be provided with the path to the .cti GenTL Producer Interface file. Without an interface, the library is not able to interface with the GeniCam cameras. Use the axvs cti set CLI command to configure the library to use the .cti file provided by the camera vendor (preferred) or a general .cti file, such as mvImpactAcquire. This command only needs to be called once, as the library remembers and reuses the provided .cti file for all future runtimes.

GenICam Configuration

GenICam-compatible cameras expose many configurable parameters (exposure time, gain, resolution, trigger mode, etc.) through the GenICam standard. This library provides tools to inspect, modify, save, and load these parameters via the CLI (axvs configure) or the MCP server.

The axvs configure read command lists all writable nodes on a connected camera, or displays detailed metadata for a specific node (type, current value, valid range, unit, and description). The axvs configure write command sets a single node to a new value, with automatic type conversion for integer, float, boolean, and enumeration nodes.

To support reproducible configurations, the axvs configure dump command saves all current camera parameters to a human-readable YAML file, tagged with the camera model and serial number. The axvs configure load command restores a saved configuration onto a camera, with optional --strict mode that aborts on camera identity mismatches instead of issuing a warning. Saved configuration files can also be edited manually before loading.

Note, configurations are independent of video capture sessions. The camera is configured first, and the VideoSystem respects whatever configuration is already active on the camera when a session starts.

API Documentation

See the API documentation for the detailed description of the methods and classes exposed by components of this library.

Note, the API documentation also includes the details about the axvs CLI interface exposed by this library.

Developers

This section provides installation, dependency, and build-system instructions for the developers that want to modify the source code of this library.

Installing the Project

Note, this installation method requires mamba version 2.3.2 or above. Currently, all Sun lab automation pipelines require that mamba is installed through the miniforge3 installer.

Download this repository to the local machine using the preferred method, such as git-cloning.
If the downloaded distribution is stored as a compressed archive, unpack it using the appropriate decompression tool.
cd to the root directory of the prepared project distribution.
Install the core Sun lab development dependencies into the base mamba environment via the mamba install tox uv tox-uv command.
Use the tox -e create command to create the project-specific development environment followed by tox -e install command to install the project into that environment as a library.

Additional Dependencies

In addition to installing the project and all user dependencies, install the following dependencies:

Python distributions, one for each version supported by the developed project. Currently, this library supports the three latest stable versions. It is recommended to use a tool like pyenv to install and manage the required versions.

Development Automation

This project uses tox for development automation. The following tox environments are available:

Environment	Description
`lint`	Runs ruff formatting, ruff linting, and mypy type checking
`stubs`	Generates py.typed marker and .pyi stub files
`{py312,...}-test`	Runs the test suite via pytest for each supported Python
`coverage`	Aggregates test coverage into an HTML report
`docs`	Builds the API documentation via Sphinx
`build`	Builds sdist and wheel distributions
`upload`	Uploads distributions to PyPI via twine
`install`	Builds and installs the project into its mamba environment
`uninstall`	Uninstalls the project from its mamba environment
`create`	Creates the project’s mamba development environment
`remove`	Removes the project’s mamba development environment
`provision`	Recreates the mamba environment from scratch
`export`	Exports the mamba environment as .yml and spec.txt files
`import`	Creates or updates the mamba environment from a .yml file

Run any environment using tox -e ENVIRONMENT. For example, tox -e lint.

Note, all pull requests for this project have to successfully complete the tox task before being merged. To expedite the task’s runtime, use the tox --parallel command to run some tasks in parallel.

AI-Assisted Development

Claude Code skills and AI development assets for this project are distributed through the ataraxis marketplace across two plugins:

video plugin: Provides MCP server registration, video-specific skills for camera setup, pipeline orchestration, log processing, and post-recording verification. Install this plugin to register the axvs mcp server with compatible MCP clients and make all video workflow skills available.
automation plugin: Provides shared development skills that enforce Sun Lab coding conventions (Python style, README style, commit messages, pyproject.toml, tox configuration) and general-purpose codebase exploration tools.

Install both plugins from the marketplace to make all associated skills and development tools available to compatible AI coding agents.

Automation Troubleshooting

Many packages used in tox automation pipelines (uv, mypy, ruff) and tox itself may experience runtime failures. In most cases, this is related to their caching behavior. If an unintelligible error is encountered with any of the automation components, deleting the corresponding cache directories (.tox, .ruff_cache, .mypy_cache, etc.) manually or via a CLI command typically resolves the issue.

Versioning

This project uses semantic versioning. See the tags on this repository for the available project releases.

Authors

Ivan Kondratyev (Inkaros)
Jacob Groner (Jgroner11)
Natalie Yeung

License

This project is licensed under the Apache 2.0 License: see the LICENSE file for details.

Acknowledgments

All Sun lab members for providing the inspiration and comments during the development of this library.
The creators of all other dependencies and projects listed in the pyproject.toml file.

Name		Name	Last commit message	Last commit date
Latest commit History 260 Commits
.idea		.idea
docs		docs
envs		envs
src/ataraxis_video_system		src/ataraxis_video_system
tests		tests
.DS_Store		.DS_Store
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
README.md		README.md
pyproject.toml		pyproject.toml
tox.ini		tox.ini

Folders and files

Latest commit

History

Repository files navigation

ataraxis-video-system

Detailed Description

Features

Table of Contents

Dependencies

Installation

Source

pip

Usage

OS Support Status

Linux

Windows

macOS

Quickstart

Data Logging

Camera Manifest

Log Format

Onset Timestamp

Working with VideoSystem Logs

Log Processing

CLI

MCP Server

Starting the Server

Available Tools

Client Registration

Using GeniCam Compatible Cameras

GenICam Configuration

API Documentation

Developers

Installing the Project

Additional Dependencies

Development Automation

AI-Assisted Development

Automation Troubleshooting

Versioning

Authors

License

Acknowledgments

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 6

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Packages