Market Data Source

Financial precision with Decimal types | Python & Rust dual ecosystem support | Extensible architecture

Market Data Source is a Rust library for generating realistic synthetic market data with Python bindings. It provides a unified interface to generate real-time and historical market data, making it easier for developers to integrate market data into their applications for backtesting, research, and development.

Why Market Data Source?

• No API limits - Generate unlimited data without rate limiting
• No costs - Completely free and open source
• Reproducible - Deterministic generation with seed support
• Native bindings - Use from both Rust and Python
• Flexible - Export to CSV, JSON, PNG charts, or stream to CouchDB

Key features:

• Rust implementation with Python bindings via PyO3
• Realistic data generation using advanced algorithms to simulate market conditions
• Multiple export formats including CSV, JSON, CouchDB, and PNG charts
• Extensible architecture allowing users to easily add custom algorithms

Features

• Configurable synthetic market data generation
• Support for OHLC candles and tick data
• Random walk with drift algorithm
• Multiple export formats:
  • CSV export for data analysis
  • JSON and JSON Lines export
  • CouchDB integration for NoSQL storage
  • PNG chart generation for data visualization
• Deterministic generation with seed support
• Streaming generation for large datasets
• Extensible architecture for adding new algorithms

Installation

Quick Install

Python (pip)

pip install market-data-source

Rust (Cargo)

[dependencies]
market-data-source = "0.3.0"

Detailed Installation

Python Installation

Option 1: Install from PyPI (Recommended)

# Using pip
pip install market-data-source

# Using uv (faster)
uv pip install market-data-source

Option 2: Build from source

# Clone the repository
git clone https://github.com/yourusername/market-data-source.git
cd market-data-source

# Install with maturin
pip install maturin
maturin develop --release

Rust Installation

Add to your Cargo.toml:

[dependencies]
market-data-source = "0.3.0"

# Or with specific features
market-data-source = { 
    version = "0.3.0", 
    features = ["csv_export", "json_export", "png_export"] 
}

Available features:

• csv_export - CSV file export support
• json_export - JSON/JSONL export support
• png_export - PNG chart generation
• couchdb - CouchDB database export
• dotenvy - Environment variable configuration
• serde - Serialization support

Quick Start

Python Example

import market_data_source as mds

# Create generator with configuration
generator = mds.MarketDataGenerator(
    initial_price=100.0,
    volatility=0.02,
    seed=42
)

# Generate 100 OHLC candles
data = generator.generate_series(100)

# Export to various formats
generator.to_csv("data.csv", count=1000)
generator.to_json("data.json", count=1000)
generator.to_png("chart.png", count=500)

Rust Example

use market_data_source::{MarketDataGenerator, ConfigBuilder};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create generator with custom config
    let config = ConfigBuilder::new()
        .initial_price_f64(100.0)
        .volatility_f64(0.02)
        .seed(42)
        .build();
    
    let mut generator = MarketDataGenerator::with_config(config)?;
    
    // Generate OHLC data
    let candles = generator.generate_series(100);
    
    // Export to CSV
    #[cfg(feature = "csv_export")]
    {
        use market_data_source::export::to_csv_ohlc;
        to_csv_ohlc(&candles, "output.csv")?;
    }
    
    Ok(())
}

See examples/ directory for more comprehensive examples.

Environment Variables

Market Data Source supports configuration through environment variables. Copy .env.example to .env and configure your settings:

cp .env.example .env
# Edit .env with your actual values

Key environment variables:

• COUCHDB_URL: CouchDB server URL (default: http://localhost:5984)
• COUCHDB_USERNAME / COUCHDB_PASSWORD: Optional authentication
• COUCHDB_DATABASE: Database name (default: market_data)
• EXPORT_BATCH_SIZE: Batch size for bulk operations (default: 1000)
• API keys for future data providers (Alpha Vantage, Polygon, Finnhub, etc.)

See .env.example for the complete list of available environment variables.

Usage

Python Usage

Market Data Source provides a Pythonic API for generating market data:

import market_data_source as mds
import pandas as pd

# Create generator with configuration
generator = mds.MarketDataGenerator(
    initial_price=100.0,
    volatility=0.02,      # 2% volatility
    trend=0.0001,         # Slight upward trend
    volume_base=1000000,  # Base volume
    interval="1m",        # 1-minute bars
    seed=42              # Reproducible results
)

# Generate OHLC data
ohlc_data = generator.generate_series(100)

# Convert to pandas DataFrame
df = pd.DataFrame(ohlc_data)
df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')

# Generate tick data
ticks = generator.generate_ticks(1000)

# Use preset configurations
volatile_gen = mds.volatile_config()
stable_gen = mds.stable_config()
bull_gen = mds.bull_market_config()
bear_gen = mds.bear_market_config()

# Export data
generator.to_csv("data.csv", count=1000)
generator.to_json("data.json", count=1000)
generator.to_png("chart.png", count=500, width=1200, height=800)

See examples/python/ for more complete examples including pandas integration.

Rust Usage

Here's a simple example of how to use Market Data Source in Rust:

use market_data_source::{MarketDataGenerator, ConfigBuilder, TrendDirection};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a generator with default configuration
    let mut generator = MarketDataGenerator::new();
    
    // Generate 10 OHLC candles
    let candles = generator.generate_series(10);
    
    for candle in &candles[..3] {
        println!("{:?}", candle);
    }
    
    // Create a generator with custom configuration
    let config = ConfigBuilder::new()
        .initial_price_f64(100.0)
        .volatility_f64(0.02)  // 2% volatility
        .trend_f64(0.001)  // Slight upward trend
        .seed(42)  // For reproducible results
        .build();
    
    let mut custom_generator = MarketDataGenerator::with_config(config)?;
    let custom_candles = custom_generator.generate_series(5);
    
    Ok(())
}

Data Export

Market Data Source supports multiple export formats for different use cases:

CSV Export

Export generated data to CSV files for analysis in Excel, pandas, or other tools:

use market_data_source::{MarketDataGenerator, ConfigBuilder, TrendDirection};
use market_data_source::export::{to_csv_ohlc, to_csv_ticks};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = ConfigBuilder::new()
        .starting_price(100.0)
        .volatility(0.02)
        .trend(TrendDirection::Bullish, 0.001)
        .build()?;
    
    let mut generator = MarketDataGenerator::with_config(config)?;
    let ohlc_data = generator.generate_series(100);
    
    // Export OHLC data to CSV using convenience function
    to_csv_ohlc(&ohlc_data, "market_data.csv")?;
    
    // Or use the exporter directly for custom options
    use market_data_source::export::csv::CsvExporter;
    use market_data_source::export::DataExporter;
    
    let csv_exporter = CsvExporter::new()
        .delimiter(b';')  // Use semicolon delimiter
        .include_headers(true);
    csv_exporter.export_ohlc(&ohlc_data, "market_data_custom.csv")?;
    
    Ok(())
}

JSON Export

Export data in JSON or JSON Lines format:

use market_data_source::export::{to_json_ohlc, to_jsonl_ohlc};

// Standard JSON format using convenience function
to_json_ohlc(&ohlc_data, "market_data.json")?;

// JSON Lines format (one JSON object per line)
to_jsonl_ohlc(&ohlc_data, "market_data.jsonl")?;

// Or use the exporter directly for custom options
use market_data_source::export::json::{JsonExporter, JsonOptions};
use market_data_source::export::DataExporter;

// Pretty-printed JSON
let json_exporter = JsonExporter::with_options(JsonOptions::pretty());
json_exporter.export_ohlc(&ohlc_data, "pretty_data.json")?;

// JSON Lines format
let jsonl_exporter = JsonExporter::with_options(JsonOptions::json_lines());
jsonl_exporter.export_ohlc(&ohlc_data, "streaming_data.jsonl")?;

CouchDB Export

Export data directly to CouchDB for NoSQL storage and replication:

use market_data_source::export::{to_couchdb_ohlc, to_couchdb_ticks};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut generator = MarketDataGenerator::new();
    let ohlc_data = generator.generate_series(100);
    
    // Export using convenience functions
    to_couchdb_ohlc(&ohlc_data, "http://localhost:5984", "market_data")?;
    
    // Or use environment variables (requires dotenvy feature)
    #[cfg(all(feature = "couchdb", feature = "dotenvy"))]
    {
        use market_data_source::export::{to_couchdb_ohlc_env, to_couchdb_ticks_env};
        to_couchdb_ohlc_env(&ohlc_data)?;
    }
    
    // For custom options, use the exporter directly
    use market_data_source::export::couchdb::{CouchDbExporter, CouchDbOptions};
    use market_data_source::export::DataExporter;
    
    let options = CouchDbOptions::new()
        .timeout_seconds(30)
        .batch_size(100)
        .auto_create_database(true);
        
    let couchdb_exporter = CouchDbExporter::new_with_options(
        "http://localhost:5984", 
        "market_data", 
        options
    );
    couchdb_exporter.export_ohlc(&ohlc_data, "")?;
    
    Ok(())
}

Running Examples

The library includes comprehensive examples for all export formats:

# CSV export example
cargo run --example export_csv --features csv_export

# JSON export example  
cargo run --example export_json --features json_export

# CouchDB export example (requires CouchDB running)
cargo run --example export_couchdb --features couchdb

# PNG chart export example
cargo run --example export_charts --features png_export

# All export formats example
cargo run --example export_all --all-features

Current Status

✅ v0.1.0 Foundation Complete

• Library structure implemented
• Core data types (OHLC, Tick, Volume)
• Market data generator with configurable parameters
• Random walk with drift algorithm
• Builder pattern for configuration
• Comprehensive export infrastructure:
  • CSV export with streaming support and custom options
  • JSON and JSON Lines export with pretty printing
  • CouchDB integration with bulk operations
  • PNG chart generation with candlestick charts, line charts, volume bars, and moving averages
• Proper error handling with structured error types
• Serde serialization support
• 54+ tests passing (unit tests + integration tests + comprehensive export tests)
• Complete example suite demonstrating all export formats

🚧 In Development

• Additional generation algorithms
• More sophisticated market patterns
• API emulation features

Automated Releases

Market Data Source uses GitHub Actions with trusted publishing (OIDC) for secure, automated releases to both crates.io and PyPI without storing API tokens.

Release Process

To create a new release:

1. Update version in Cargo.toml and pyproject.toml
2. Update CHANGELOG.md with release notes
3. Commit and push changes
4. Create and push a version tag:

   git tag v0.3.1
   git push origin v0.3.1

The automated workflow will:

• Validate version consistency
• Run full test suite across all platforms
• Build and publish to crates.io
• Build Python wheels for multiple platforms
• Publish to PyPI
• Create GitHub release with artifacts

Testing Publication Setup

Test the publishing configuration without releasing:

# Via GitHub Actions UI:
# Actions → Test Publishing Setup → Run workflow

For more details, see .github/PUBLISHING.md.

Release Process

Market Data Source uses GitHub Actions for automated releases to both crates.io and PyPI. For complete documentation, see docs/RELEASE.md.

Quick Release

# 1. Prepare a patch release (e.g., 0.3.0 -> 0.3.1)
python scripts/prepare-release.py --bump patch

# 2. Review and update CHANGELOG.md
$EDITOR CHANGELOG.md

# 3. Commit and push the tag
git add -A
git commit -m "chore: prepare release v0.3.1"
git push origin v0.3.1  # Triggers automated release

Version Management

Versions are synchronized between Cargo.toml and pyproject.toml:

# Check version consistency
python scripts/sync-version.py --check

# Set specific version
python scripts/sync-version.py --set-version 0.3.1

Release Types

• Patch: Bug fixes (python scripts/prepare-release.py --bump patch)
• Minor: New features (python scripts/prepare-release.py --bump minor)
• Major: Breaking changes (python scripts/prepare-release.py --bump major)

Automated Workflow

The release workflow automatically:

1. Validates version consistency and runs all tests
2. Builds packages for all platforms
3. Publishes to crates.io and PyPI
4. Creates GitHub release with artifacts
5. Prepares PR for next development version

For detailed procedures and troubleshooting, see docs/RELEASE.md.

Contributing

Contributions are welcome! If you'd like to contribute to Market Data Source, please fork the repository and submit a pull request.

License

This project is licensed under the MIT License - see the LICENSE file for details.