Replace deprecated datetime.utcnow() with datetime.now(timezone.utc)

.github/FUNDING.yml

@@ -0,0 +1,12 @@
+# These are supported funding model platforms
+
+github: [cuemacro]
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: # Replace with a single Ko-fi username
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: # Replace with a single Liberapay username
+issuehunt: # Replace with a single IssueHunt username
+otechie: # Replace with a single Otechie username
+custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

.gitignore

@@ -70,3 +70,8 @@ target/
 
 # Output Files
 
+#Enviroments
+venv/
+
+#ide
+.vscode

MANIFEST.in

@@ -1,3 +1,5 @@
 recursive-include findatapy/ *.*
 include findatapy/conf/*
-recursive-include findatapy_examples/ *.*
+recursive-include findatapy_examples/ *.*
+
+global-exclude *.py[cod] __pycache__ *.so *.pdf ~$*

README.md

@@ -2,7 +2,9 @@
 
 # [findatapy](https://github.com/cuemacro/findatapy)
 
-findatapy creates an easy to use Python API to download market data from many sources including Quandl, Bloomberg, Yahoo, Google etc. using
+[![Downloads](https://pepy.tech/badge/findatapy)](https://pepy.tech/project/findatapy)
+
+findatapy creates an easy to use Python API to download market data from many sources including ALFRED/FRED, Bloomberg, Yahoo, Google etc. using
 a unified high level interface. Users can also define their own custom tickers, using configuration files. There is also functionality which
 is particularly useful for those downloading FX market data. Below example shows how to download AUDJPY data from Quandl (and automatically 
 calculates this via USD crosses).
@@ -14,7 +16,11 @@ from findatapy.market import Market, MarketDataRequest, MarketDataGenerator
 
 market = Market(market_data_generator=MarketDataGenerator())
 
-md_request = MarketDataRequest(start_date='year', category='fx', data_source='quandl', tickers=['AUDJPY'])
+# Get you FRED API key from https://fred.stlouisfed.org/docs/api/api_key.html
+fred_api_key = "WRITE YOUR KEY HERE" 
+
+md_request = MarketDataRequest(start_date='year', category='fx', data_source='alfred', tickers=['AUDJPY'],
+                               fred_api_key=fred_api_key)
 
 df = market.fetch_market(md_request)
 print(df.tail(n=10))
@@ -54,24 +60,28 @@ To appear
 # Requirements
 
 Major requirements
-* Required: Python 3.6
+* Required: Python 3.10 (Python 2 is not supported)
 * Required: pandas, numpy etc.
-* Recommended: blpapi - Bloomberg Python Open API
+* Recommended: blpapi - Bloomberg Python Open API - install by separately running `pip install --index-url=https://blpapi.bloomberg.com/repository/releases/python/simple blpapi`
 * Recommended: chartpy - for funky interactive plots ([https://github.com/cuemacro/chartpy](https://github.com/cuemacro/chartpy)) and
-* Recommended: arctic - AHL library for managing time series in MongoDB
-* Recommended: multiprocessor_on_dill - standard multiprocessing library pickle causes issues ([https://github.com/sixty-north/multiprocessing_on_dill](https://github.com/sixty-north/multiprocessing_on_dill))
-* Recommended: fredapi - ALFRED/FRED has been rewritten and added to the project directly (from https://github.com/mortada/fredapi)
 
 # Installation
 
 For detailed installation instructions for chartpy, findatapy & finmarketpy and its associated Python libraries go to
 [https://github.com/cuemacro/finmarketpy/blob/master/INSTALL.md](https://github.com/cuemacro/finmarketpy/blob/master/INSTALL.md). The tutorial includes details on how to setup your entire Python environment.
 
 You can install the library using the below. After installation:
-* Make sure you edit the dataconstants class for the correct Quandl API and Twitter API keys etc.
+* Make sure you edit the dataconstants class for the correct Eikon API, Quandl API and Twitter API keys etc.
 * Or you can run set_api_keys.py script to set the API keys via storing in your keyring
 * Or you can create a datacred.py file which overwrites these keys
+* Or some of these API keys can be passed via MarketDataRequest on demand
+
+To install via pip (latest release):
+```
+pip install findatapy
+```
 
+To install newest repo copy:
 ```
 pip install git+https://github.com/cuemacro/findatapy.git
 ```
@@ -101,14 +111,241 @@ although it lacks some functionality of later Redis versions.
 
 # findatapy examples
 
-In findatapy/examples you will find several demos
+In findatapy/examples you will find several demos on how to download data from many different sources. Note, 
+for some such as Bloomberg or Eikon, you'll need to have a licence/subscription for it to work. Also there might be 
+certain limits of the history you can download for intraday data from certain sources (you will need to check with
+individual data providers)
 
 # Release Notes
 
+* 0.1.41 - findatapy (02 Jan 2026)
+* 0.1.40 - findatapy (08 Mar 2025)
+* 0.1.39 - findatapy (08 Mar 2025)
+* 0.1.38 - findatapy (19 Feb 2025)
+* 0.1.37 - findatapy (15 Jan 2025)
+* 0.1.36 - findatapy (27 Apr 2024)
+* 0.1.35 - findatapy (10 Apr 2024)
+* 0.1.34 - findatapy (08 Apr 2024)
+* 0.1.33 - findatapy (01 Apr 2024)
+* 0.1.32 - findatapy (17 Feb 2024)
+* 0.1.31 - findatapy (01 Dec 2023)
+* 0.1.30 - findatapy (12 Oct 2023)
+* 0.1.29 - findatapy (14 May 2023)
+* 0.1.28 - findatapy (19 Jul 2022)
+* 0.1.27 - findatapy (20 May 2022)
+* 0.1.26 - findatapy (07 Oct 2021)
+* 0.1.25 - findatapy (07 Oct 2021)
+* 0.1.24 - findatapy (06 Oct 2021)
+* 0.1.23 - findatapy (03 Jun 2021)
+* 0.1.22 - findatapy (01 Jun 2021)
+* 0.1.21 - findatapy (04 May 2021)
+* 0.1.20 - findatapy (11 Feb 2021)
+* 0.1.19 - findatapy (22 Jan 2021)
+* 0.1.18 - findatapy (02 Oct 2020)
+* 0.1.17 - findatapy (01 Oct 2020)
+* 0.1.16 - findatapy (13 Sep 2020)
+* 0.1.15 - findatapy (10 Sep 2020)
+* 0.1.14 - findatapy (25 Aug 2020)
+* 0.1.13 - findatapy (24 Aug 2020)
 * 0.1.12 - findatapy (06 May 2020)
 
 # Coding log
 
+* 11 Jan 2026
+  * Speeding up ticker parsing in ConfigManager
+* 02 Jan 2026
+  * Refactored ConfigManager so it can take DataFrames as input (not just CSVs)
+* 29 Oct 2025
+  * Added flag for conversion of datetime in IOEngine
+* 18 Jul 2025
+  * Speeded up Redis read/write of DataFrames
+* 08 Mar 2025
+  * Fixed various deprecation warnings for Pandas
+* 19 Feb 2025
+  * Fixed bug with Yahoo
+* 15 Jan 2024
+  * Fixed various issues accessing data on s3 related to credentials
+* 09 Nov 2024
+  * Added ALFRED/FRED FX tickers to time series mapping CSV files
+  * Fixed bug when downloading ALFRED/FRED FX tickers
+  * Added intraday downloading for Yahoo (and example for FX)
+  * Refactored out crypto downloaders into datavendorcrypto.py
+  * Refactored out ALFRED/FRED into datavendorfred.py
+  * Starting to add Databento Historical API
+  * Fixed deprecation error in calculations.py for 
+* 27 Apr 2024
+  * Removed additional list typecheck (to make code Python 3.8 compatible)
+* 09 Apr 2024
+  * Bugfix for reading Parquet files with columns
+* 08 Apr 2024
+  * Removed list typecheck (to make code Python 3.8 compatible)
+* 01 Apr 2024
+  * Added support for ArcticDB to store market data with findatapy
+* 17 Feb 2024
+  * Fixed ALFRED/FRED wrapper so now compatible with Python 3.10
+* 01 Dec 2023
+  * Remove fxcmpy dependency (package no longer exists?)
+  * Added type hinting in ioengine.py
+* 26 Jul 2023
+  * Fixed overrides for BBG
+* 12 May 2023
+  * Fixed Dukascopy downloader
+* 29 Aug 2022
+  * Reformatted text for line length
+* 23 Aug 2022
+  * Simplified MarketDataRequest
+* 12 Aug 2022
+  * Minor changes in MarketDataGenerator for non-standard columns
+* 11 Aug 2022
+  * Fixed various bug in MarketDataRequest, when updating constants and 
+  freeform str ticker queries with kwargs
+* 19 Jul 2022
+  * Various fixes for data download
+* 20 May 2022
+  * Added more customisation for data vendors
+* 25 Jan 2022
+  * Fixed delete key problem with Redis
+* 20 Jan 2022
+  * Fixed path join for s3
+* 17 Jan 2022
+  * Fixed bug in BBG wrapper
+* 16 Jan 2022
+  * Started to refactor code for PEP8 standards
+  * Added BBG tests
+* 15 Jan 2022
+  * Changed read/write on Redis to use Pickle instead of Arrow
+* 14 Dec 2021
+  * Fixed bug in overrides for BBG
+  * Patched ticker for EUR1Y deposit rate
+* 07 Oct 2021
+  * Fixed bug in downloading data for unusual categories 
+  * Fixed missing ticker in time_series_tickers.csv
+* 27 Sep 2021
+  * Fixed bug in numeric conversion in DataFrame
+  * Error trapping when downloading web pages to DataFrame
+  * Added ignore case in filter columns
+  * Removed lz4 compression for Arrow caching
+* 23 Sep 2021
+  * Fixed bug in YoY calculation
+* 29 Jul 2021
+  * Minor changes to `Market` for managing tickers
+* 28 Jul 2021
+  * Improved freeform ticker queries and fixed bug with downloading whole categories
+* 22 Jul 2021
+  * Fixed S3 credentials management and added S3 file copy method
+  * Added roll costs
+* 19 Jul 2021
+  * Added delete file method in `IOEngine` for S3
+* 12 Jul 2021
+  * Can now read CSV conf files for tickers from S3 buckets and improved S3 support (can now specify AWS credentials, as parameter)
+  * Additional file functions (eg. list_files)
+* 05 Jul 2021
+  * Now (optionally) writes Parquet files in chunks (user specified size) to avoid memory issues with pyarrow
+  * Default is to use pandas.to_parquet (with pyarrow), and to fall back on chunked writing if that fails
+  * Added multithreaded reading for DataVendorFlatFile
+* 04 Jul 2021
+  * Added extra support for reading/writing to S3 buckets of Parquet files
+* 02 Jul 2021
+  * Can download multiple CSVs in ZIP with time series data (DataVendorWeb) 
+* 29 Jun 2021
+  * Added downloads badge
+* 17 Jun 2021
+  * Can download different GDP releases from Bloomberg, without having to specify overrides
+* 03 Jun 2021
+  * Fix bug in ConfigManager
+* 29 May 2021
+  * Improved freeform queries for MarketDataRequest
+* 26 May 2021
+  * Added more flexible ticker calls (with a '_' prefix)
+* 23 May 2021
+  * Fixed various bugs with reading ECO_RELEASE_DT etc. dates from Bloomberg
+  * Fixed bugs when predefined ticker is defined differently in different categories  
+  * Can now write Parquet without date truncation errors (eg. ns to us)
+  * Reads/writes Parquet from S3
+* 22 May 2021
+  * Better reading/writing of files to disk
+* 21 May 2021
+  * Added revision periods to config tickers
+* 20 May 2021
+  * Added ability to query stored tickers by regular expressions
+  * Made field names/code consistent for tickers/vendor_tickers etc.
+* 17 May 2021
+  * Changed named conventions for CSV conf ticker files to be consistent with MarketDataRequest
+* 08 May 2021
+  * Added more ways to create a market data request
+* 07 May 2021
+  * Fixed freeform market data requests when contain different file formats
+* 06 May 2021
+  * Fixed bug when querying from files with dot in them for parquet
+* 04 May 2021
+  * Made fetching market data more flexible (can use a string for tickers which are not predefined)
+  * Added ability to call predefined tickers with a string
+* 29 Apr 2021
+  * Bug fix when getting empty ticker from Bloomberg
+* 22 Apr 2021
+  * Added 404 error for downloading from Dukascopy
+* 15 Apr 2021
+  * Constant overrides now persist
+* 13 Apr 2021
+  * Fix issue when conversion of date/time releases for certain Bloomberg economic events
+* 25 Mar 2021
+  * Fixed empty column download from data vendor downloading as object (rather than NaN)
+* 20 Mar 2021
+  * Improved Redis caching
+  * Can now fetch multiple MarketDataRequests
+* 11 Feb 2021
+  * Fixed timezone issues in Seasonality
+  * Add extra Dukascopy error checking/retry functionality/parameters
+  * Added Parquet to MarketDataRequest
+  * Started to write Numba implementations for join and align
+  * Added fields for downloading eg. FX vol data
+* 22 Jan 2021
+  * Fixed caching for tick
+* 14 Jan 2021
+  * Fixed ON FX option date expiry
+* 10 Jan 2021
+  * Added Dukascopy non-equities example
+* 08 Jan 2021
+  * Added extra calendar example
+* 05 Jan 2021
+  * MarketDataRequest accepts parsing of full month names
+* 28 Dec 2020
+  * Spun out Calendar into separate Python script
+* 26 Dec 2020
+  * Added missing holiday file
+  * Refactored Calendar (so is no longer dependent on Filter)
+* 24 Dec 2020
+  * Remove logger as field variable in IOEngine
+  * Fixed Calendar methods so can take single input
+* 19 Dec 2020
+  * Added functionality to download FX forwards based total indices from BBG
+  * Fixed downloading of forward points for NDFs
+  * Fixed missing timestamp issue with DukasCopy
+  * Adding holidays functionality and calculation of FX options expiries (and FX delivery dates)
+* 10 Dec 2020
+    * Added resample method on Calculations for tick data
+    * Fixed logger in DataVendorWeb
+    * Fixed setting no timezone method
+* 11 Nov 2020
+    * Added cumulative additive index returns 
+    * Removed log as field variable in DataVendorBBG
+    * Added 10am NYC cut for FX vol surface download
+* 02 Oct 2020
+    * Fix vol ticker mapping for 4M points
+    * Fix Bloomberg downloader for events
+* 30 Sep 2020
+    * Fix crypto downloaders (added tickers, fields etc. to CSV files)
+* 24 Sep 2020
+    * Refactoring of Calculations
+* 13 Sep 2020
+    * Removed multiprocessing_on_dill as dependency, which is no longer being used
+* 10 Sep 2020
+    * Adding Eikon as a market data source (daily, intraday and tick market data)
+* 25 Aug 2020
+    * Fixes for newer Pandas eg. 1.0.5
+    * Fixes for ALFRED downloading of economic data
+* 24 Aug 2020
+    * Removed .ix references (to work with newer Pandas)
 * 06 May 2020
     * Amended function to remove points outside FX hours to exclude 1 Jan every year
     * RetStats can now resample time series (removed kurtosis)