Create a new core library for DataLab: `sigima`

[PierreRaybaut]

🎉 New repository

Here is the associated repository: https://github.com/DataLab-Platform/sigima

✨ Objective

Create a standalone Python package named datalab-processing that consolidates the computation engine, data I/O logic (except HDF5), and the simple XML-RPC client for DataLab.

This package will serve as the non-GUI backend for DataLab, and provide reusable functionality for headless applications and remote control workflows.

---

🔧 Scope of sigima

The datalab-processing package will include:

• 🧠 Signal and image processing logic (BaseProcessor, SignalProcessor, ImageProcessor)
• 🗃️ I/O and file format support for signals and images (except HDF5)
• 🧩 I/O plugin system (moved out of the GUI package)
• 🧪 Internal validation and test datasets (optional)
• 🚦 Type annotations and compatibility with mypy, pytest, and other quality tools

Options (to be discussed):

• 🛰️ Lightweight XML-RPC client to control a remote instance of DataLab

Note: HDF5 file handling will temporarily remain in datalab-app, but all other input/output logic (including the plugin mechanism for custom readers/writers) will be moved to sigima.

---

✅ To Extract into sigima

1. cdl.algorithms

• ✅ All core functions operating on raw NumPy arrays
• 🔁 Refactored under sigima.algorithms

2. cdl.computation

• ✅ All functions operating on SignalObj and ImageObj
• 🔁 Refactored into two top-level modules:
  • sigima.signal → for SignalObj-based operations
  • sigima.image → for ImageObj-based operations

3. cdl.core.model

• ⏳ Partially extract:
  • SignalObj, ImageObj
  • Other basic computation-related objects (e.g. wrappers for result types)
• ❌ Leave GUI- or application-specific classes in DataLab
• 🔁 Refactored under sigima.datatypes

4. cdl.core.io

• ⏳ Partially extract:
  • I/O for signals/images that are not tied to HDF5 or GUI persistence
  • ❌ Exclude HDF5-specific or GUI-related I/O
• 🔁 Refactored under sigima.io

5. cdl.plugins

• ❌ Do not reuse existing plugin classes (tightly coupled to DataLab GUI)
• ✅ Instead, define a clean plugin interface under sigima.plugins
  • For extendable compute operations

6. cdl.tests

• ✅ Migrate tests related to:
  • cdl.algorithms
  • cdl.computation
• 🔁 Refactor under tests/ in sigima, preserving coverage

7. cdl.utils

• ⏳ Only migrate utility functions strictly required by above modules
• 🔁 Refactor and isolate as sigima.utils (minimal)

8. Test Infrastructure & Data

• ✅ Migrate reusable parts:
  • tests.data/
  • Core test helpers
  • CI-compatible test layout

---

🔄 Refactoring Considerations

• ✅ Isolate any dependency on cdl.config, cdl.qt, or cdl.app
• ✅ Ensure standalone execution with no Qt/UI import chain
• ✅ Prepare pyproject.toml with optional plugin extras

---

📂 Proposed Package Layout

sigima/
├── algorithms/           # NumPy-based processing functions
├── signal/               # SignalObj-based operations (e.g., normalize, filter, resample)
│   ├── __init__.py
│   └── processing.py
├── image/                # ImageObj-based operations (e.g., ROI, threshold, morphology)
│   ├── __init__.py
│   └── filters.py
├── datatypes/            # SignalObj, ImageObj, Result wrappers
├── io/                   # Portable I/O for signal/image
├── plugins/              # Plugin architecture for compute functions
├── utils/                # Minimal utilities used by core modules
├── tests/                # Pytest-compatible test suite
│   ├── data/             # Test images/signals
│   └── ...
├── __init__.py
└── pyproject.toml

---

📌 Notes

• The name sigima emphasizes the unification of signal and image processing logic.
• GUI-specific functionality remains in datalab-app.
• The library will be MIT-licensed and documented as a reusable scientific computing core.
• With this structure, common imports become clean and readable:

from sigima.signal import normalize
from sigima.image import enhance_contrast

---

🚧 Tasks

• Extract computation and non-HDF5 I/O logic from cdl/ into sigima/
• Migrate the plugin-based I/O mechanism for signals/images
• Clean up internal dependencies and prepare pyproject.toml structure
• (eventually) Integrate the simple XML-RPC client (previously cdlclient)
• Write unit tests and CI configuration (GitHub Actions)
• Add Sphinx documentation and usage examples
• Publish to PyPI as sigima

---

📌 Context

This package replaces and unifies:

• the computational backend of the original DataLab codebase (cdl)
• (eventually) the external cdlclient RPC wrapper
• the future programmatic API surface of DataLab (headless usage, scripting, integration)

It complements:

• datalab-app: GUI frontend (HDF5 support remains here temporarily)
• sigimax: reusable GUI components and infrastructure

---

🔗 Related

• Naming and Packaging Strategy for the DataLab Ecosystem #181
• Create a new GUI tools library for DataLab: sigimax #182
• Archive and deprecate cdlclient package #183

[PierreRaybaut]

New name for the core library: datalab-processing (instead of datalab-core)