Add BP5 Variable Chunking Support: Adios MCP

[sohamvsonar]

Problem Statement

The current read_variable_at_step tool loads entire variables into memory, causing significant issues when working with large scientific datasets:

• Memory Exhaustion: Variables with huge elements cause out-of-memory errors
• AI Token Limits: Large arrays overwhelm AI context windows, making analysis impossible
• Performance Issues: Loading massive multi-dimensional datasets blocks processing
• Scalability Problems: Time-series and spatial data from simulations are often too large to process

This is particularly problematic for scientific workflows where BP5 files commonly contain variables with millions of elements from computational simulations.

Proposed Solution

Implement a new read_variable_chunk tool that enables memory-efficient reading of large BP5 variables through intelligent chunking:

Core Features Needed

1. Automatic Chunking: Split large variables along the first dimension while preserving other dimensions
2. Dynamic Sizing: Calculate optimal chunk sizes based on variable characteristics (suggested: 100-500 elements per chunk)
3. Iteration Support: Provide metadata for progressive reading with clear navigation
4. Memory Safety: Use ADIOS2's selection API to read only requested chunks
5. AI-Friendly Output: Include rich metadata for intelligent processing decisions

Impact

This feature would unlock AI-driven analysis of large scientific datasets that are currently impossible to process due to memory constraints. It's essential for enabling modern AI workflows in computational science and research environments.

[onestardao]

Super important request — but heads-up:
even if this gets implemented perfectly, it might still break in AI workflows, unless you're tracking semantic drift across chunks.

Why? Because memory-efficient chunking ≠ semantic-preserving chunking.
Large arrays (esp. time-series or spatial) can get sliced in ways that fragment meaning, not just shape — and that ruins any downstream reasoning or retrieval.

We’ve documented 16 such failure modes in modern RAG systems, including ones exactly like this (chunking by shape, losing semantic integrity).
Backed by Tesseract.js creator, MIT licensed, all open-source. No pitch — just trying to save others from painful detours.

Happy to share the map or point you to the relevant module if you’re curious.