-
Notifications
You must be signed in to change notification settings - Fork 1
Data Integration
- Implementation Requirements for Abstract Methods
- TVLibrarySymbolInfo Structure for Instrument Metadata
- TVBar Data Model for OHLCV Candlestick Data
- Historical Data Retrieval and Real-Time Streaming Patterns
- Error Handling and Disconnection Recovery
- Integration Examples with Common Data Sources
- Performance Considerations
The TVDatafeed system requires implementation of several abstract methods to enable custom data source integration. These methods are defined in the TVIExternalDatafeed, TVIDatafeedChartApi, and TVIDatafeedQuotesApi interfaces and must be overridden in subclasses to provide actual data functionality.
The resolveSymbol method is responsible for converting a symbol name into complete symbol information by constructing a TVLibrarySymbolInfo object with all required metadata. This method must handle symbol validation and return appropriate error responses through the onError callback when symbol resolution fails.
The getBars method retrieves historical OHLCV data based on the provided symbol information, resolution, and period parameters. It must process the TVPeriodParams which include time range specifications and countBack requirements, then return the appropriate bar data through the onResult callback along with metadata indicating whether more data is available.
The subscribeBars method establishes real-time data subscriptions for continuous bar updates. It stores subscription information including the symbol, resolution, and callback function in the internal subscribers dictionary, enabling the system to push new bar data as it becomes available. The method also accepts an onResetCacheNeededCallback for handling cache invalidation scenarios.
The unsubscribeBars method terminates real-time subscriptions by removing the specified listener from the subscribers dictionary, effectively stopping the flow of real-time updates for that particular subscription.
The TVLibrarySymbolInfo structure defines comprehensive metadata for trading instruments, containing both required and optional fields that describe various aspects of a symbol. The required parameters include fundamental information such as the symbol name, description, instrument type, trading session, exchange details, timezone, price format, pricescale, and minimum movement units.
For resolution support, the structure includes boolean flags and resolution arrays that specify which timeframes are available for the instrument. The has_intraday flag indicates intraday data availability, while supported_resolutions lists the specific resolution strings supported. Additional resolution multipliers are provided for intraday, daily, weekly, and monthly periods through their respective multiplier arrays.
The structure also defines display configuration options such as has_empty_bars for handling non-trading periods, visible_plots_set to specify which price components are visible (ohlcv, ohlc, or c), and volume_precision to control volume decimal places. Data status and timing information is captured through data_status, delay, expired, and expiration_date fields.
Classification metadata includes sector and industry categorization, while currency and unit conversion capabilities are supported through currency_code, original_currency_code, unit_id, and related conversion type fields. Extended trading sessions are handled via subsession_id and subsessions arrays, and price source information is managed through price_source_id and price_sources collections.
The TVBar data model represents OHLCV (Open, High, Low, Close, Volume) candlestick data for a specific time period. The model requires a time attribute expressed in milliseconds since Unix epoch in UTC timezone. For daily, weekly, and monthly bars, the time field should represent the trading day at 00:00 UTC rather than the session start time.
The model includes five core price attributes: open, high, low, and close prices as floating-point values, along with an optional volume attribute. The volume field can be omitted when volume data is not available for the instrument. Each bar instance encapsulates the complete price action for its designated time period, providing the fundamental building block for chart visualization.
The TVBar class provides a to_dict method that converts the bar data into a dictionary format suitable for serialization and transmission. This method ensures all attributes are properly formatted and included in the output, maintaining consistency across different data operations and integrations.
Historical data retrieval in the TVDatafeed system follows a callback-based pattern using the getBars method. The method receives a TVPeriodParams object containing the requested time range (from_ and to timestamps), countBack parameter, and a firstDataRequest flag. Implementations should prioritize countBack when supported, otherwise use the time range to fetch the appropriate number of historical bars.
Real-time streaming is managed through the subscription pattern implemented in subscribeBars and unsubscribeBars methods. When a client subscribes to real-time updates, the system stores the subscription details including the callback function (onTick) that will receive new bar data. The implementation should establish connections to real-time data sources such as WebSocket streams and push new TVBar instances to the registered callbacks as they become available.
The system maintains a dictionary of subscribers indexed by listenerGuid, allowing efficient management of multiple concurrent subscriptions. For real-time data delivery, implementations should handle connection lifecycle events, including automatic reconnection after disconnections and proper cleanup during unsubscription.
sequenceDiagram
participant Client
participant Datafeed
participant DataSource
Client->>Datafeed : subscribeBars(symbolInfo, resolution, onTick, listenerGuid)
Datafeed->>Datafeed : Store subscription in _subscribers
Datafeed->>DataSource : Establish real-time connection
loop Data Streaming
DataSource->>Datafeed : New bar data
Datafeed->>Datafeed : Validate and format TVBar
Datafeed->>Client : onTick(newBar)
end
Client->>Datafeed : unsubscribeBars(listenerGuid)
Datafeed->>Datafeed : Remove subscription from _subscribers
Datafeed->>DataSource : Close connection if no remaining subscriptions
Error handling in the TVDatafeed system follows a consistent callback-based approach across all operations. Each method that may encounter errors accepts an onError callback parameter that should be invoked with descriptive error messages when exceptional conditions occur. The TVDatafeedErrorCallback is defined as a callable that accepts a string parameter containing the error description.
For symbol resolution failures, the resolveSymbol method should catch exceptions during symbol information construction and invoke the onError callback with appropriate error messages. Similarly, the getBars method should handle data retrieval exceptions and communicate errors through the onError callback rather than allowing exceptions to propagate.
Disconnection recovery mechanisms should be implemented at the data source integration level. When real-time connections are lost, the implementation should attempt automatic reconnection with exponential backoff strategies. Upon successful reconnection, the system should resynchronize data and notify subscribers of any gaps through the onResetCacheNeededCallback.
The framework provides built-in logging through the standard Python logging module, with log messages at INFO level for subscription events and WARNING level for unimplemented methods. Custom implementations should maintain consistent logging practices to facilitate debugging and monitoring.
The BADatafeed example demonstrates integration patterns with various data sources. For REST API integration, the getBars method would be implemented to make HTTP requests to external APIs, transform the response data into TVBar objects, and return them through the onResult callback. The implementation should handle pagination, rate limiting, and response parsing specific to the target API.
WebSocket stream integration is exemplified in the subscribeBars method, where a real-time connection would be established to receive streaming data. The implementation should manage the WebSocket lifecycle, including connection establishment, message parsing, and error handling. Incoming data would be transformed into TVBar instances and pushed to the registered onTick callbacks.
For database-backed systems, both historical and real-time data access patterns can be implemented. Historical data retrieval would involve querying time-series databases using the period parameters, while real-time updates could be delivered through database change data capture (CDC) mechanisms or message queues. The implementation should optimize queries for time-range based access patterns and maintain appropriate indexing.
flowchart TD
A[Data Request] --> B{Request Type}
B --> |Historical| C[REST API Call]
B --> |Real-time| D[WebSocket Connection]
B --> |Database| E[Database Query]
C --> F[Parse Response]
D --> G[Process Stream]
E --> H[Format Results]
F --> I[Create TVBar Objects]
G --> I
H --> I
I --> J[Return via Callback]
Performance optimization for high-frequency data delivery requires careful consideration of several factors. Memory management of historical data caches should implement efficient data structures such as circular buffers or time-based eviction policies to prevent unbounded memory growth. For frequently accessed instruments, pre-fetching and caching of recent bars can reduce latency for client requests.
When handling multiple concurrent subscriptions for the same instrument and resolution, implementations should deduplicate data source connections to avoid redundant network requests. A fan-out pattern can be used where a single data stream serves multiple subscribers, reducing load on both the client and data source.
For high-frequency data delivery, batch processing of bar updates can improve efficiency by reducing the number of callback invocations. However, this must be balanced against the need for low-latency updates in trading applications. Implementations should provide configuration options to tune this trade-off based on use case requirements.
Network efficiency can be improved through compression of data payloads and efficient serialization formats when communicating with external data sources. Connection pooling and keep-alive strategies should be employed to minimize connection establishment overhead for REST APIs.