A Python package for downloading historical data published by the Australian Energy Market Operator (AEMO)
- Download Windows Application (GUI)
- Documentation
- Support NEMOSIS
- Get Updates, Ask Questions
- Using the Python Interface (API)
Choose the exe from the latest release
- Check out the wiki
- View worked examples:
- What data is available and data column definitions
- Watch a video
- Read our paper introducting NEMOSIS
Cite our paper in your publications that use data from NEMOSIS.
Join the NEMOSIS forum group.
pip install nemosis
Dynamic tables contain a datetime column that allows NEMOSIS to filter their content by a start and end time.
To learn more about each dynamic table visit the wiki.
You can view the dynamic tables available by printing the NEMOSIS default settings.
from nemosis import defaults
print(defaults.dynamic_tables)
#['DISPATCHLOAD', 'DUDETAILSUMMARY', 'DUDETAIL', 'DISPATCHCONSTRAINT', 'GENCONDATA', 'DISPATCH_UNIT_SCADA', 'DISPATCHPRICE', . . .Your workflow may determine how you use NEMOSIS. Because the GUI relies on data being stored as strings (rather than numeric types such as integers or floats), we suggest the following:
- If you are using NEMOSIS' API in your code, or using the same cache for the GUI and API, use
dynamic_data_compiler. This will allow your data to be handled by both the GUI and the API. Data read in via the API will be typed, i.e. datetime columns will be a datetime type, numeric columns will be integer/float, etc. See this section. - If you are using NEMOSIS to cache data in feather or parquet format for use with another application, use
cache_compiler. This will ensure that cached feather/parquet files are appropriately typed to make further external processing easier. It will also cache faster as it doesn't prepare a DataFrame for further analysis. See this section.
dynamic_data_compiler can be used to download and compile data from dynamic tables.
from nemosis import dynamic_data_compiler
start_time = '2017/01/01 00:00:00'
end_time = '2017/01/01 00:05:00'
table = 'DISPATCHPRICE'
raw_data_cache = 'C:/Users/your_data_storage'
price_data = dynamic_data_compiler(start_time, end_time, table, raw_data_cache)Using the default settings of dynamic_data_compiler will download CSV data from AEMO's NEMWeb portal and save it to the raw_data_cache directory. It will also create a feather file version of each CSV (feather files have a faster read time). Subsequent dynamic_data_compiler calls will check if any data in raw_data_cache matches the query and loads it. This means that subsequent dynamic_data_compiler will be faster so long as the cached data is available.
A number of options are available to configure filtering (i.e. what data NEMOSIS returns as a pandas DataFrame) and caching.
dynamic_data_compiler can be used to filter data before returning results.
To return only a subset of a particular table's columns, use the select_columns argument.
from nemosis import dynamic_data_compiler
price_data = dynamic_data_compiler(start_time, end_time, table, raw_data_cache,
select_columns=['REGIONID', 'SETTLEMENTDATE', 'RRP'])To see what columns a table has, you can inspect NEMOSIS' defaults.
from nemosis import defaults
print(defaults.table_columns['DISPATCHPRICE'])
# ['SETTLEMENTDATE', 'REGIONID', 'INTERVENTION', 'RRP', 'RAISE6SECRRP', 'RAISE60SECRRP', 'RAISE5MINRRP', . . .Columns can also be filtered by value. To do this, you need provide a column to be filtered (filter_cols) and a value or values to filter (filter_values) a corresponding column by. to filter by a column the column must be included as a filter column.
In the example below, the table will be filtered to only return rows where REGIONID == 'SA1'.
from nemosis import dynamic_data_compiler
price_data = dynamic_data_compiler(start_time, end_time, table, raw_data_cache, filter_cols=['REGIONID'], filter_values=(['SA1'],))Several filters can be applied simultaneously. A common filter is to extract pricing data excluding any physical intervention dispatch runs (INTERVENTION == 0 is the appropriate filter, see here). Below is an example of filtering to get data for Gladstone Unit 1 and Hornsdale Wind Farm 2 excluding any physical dispatch runs:
from nemosis import dynamic_data_compiler
unit_dispatch_data = dynamic_data_compiler(start_time, end_time, 'DISPATCHLOAD', raw_data_cache, filter_cols=['DUID', 'INTERVENTION'], filter_values=(['GSTONE1', 'HDWF2'], [0]))By default the options fformat='feather' and keep_csv=True are used.
If the option fformat='csv' is used then no feather files will be created, and all caching will be done using CSVs.
price_data = dynamic_data_compiler(start_time, end_time, table, raw_data_cache, fformat='csv')If you supply fformat='feather', the original AEMO CSVs will still be cached by default. To save disk space but still ensure your data will work with the API & GUI, use keep_csv=False in combination with fformat='feather' (which is the default option). This will delete the AEMO CSVs after the feather file is created.
price_data = dynamic_data_compiler(start_time, end_time, table, raw_data_cache, keep_csv=False)If the option fformat='parquet' is provided then no feather files will be created, and a parquet file will be used instead.
While feather might have faster read/write, parquet has excellent compression characteristics and good compatability with packages for handling large on-memory/cluster datasets (e.g. Dask). This helps with local storage (especially for Causer Pays data) and file size for version control.
This may be useful if you're using NEMOSIS to
build a data cache, but then process the cache using other packages or applications. It is particularly useful because cache_compiler will infer the data types of the columns before saving to parquet or feather, thereby eliminating the need to type convert data that is obtained using dynamic_data_compiler.
cache_compiler can be used to compile a cache of parquet or feather files. Parquet will likely be smaller, but feather can be read faster. cache_compiler will not run if it detects the appropriate files in the raw_data_cache directory. Otherwise, it will download CSVs, covert to the requested format and then delete the CSVs. It does not return any data, unlike dynamic_data_compiler.
The example below downloads parquet data into the cache.
from nemosis import cache_compiler
cache_compiler(start_time, end_time, table, raw_data_cache, fformat='parquet')By default NEMOSIS only includes a subset of an AEMO table's columns, the full set of columns are listed in the
MMS Data Model Reports,
or can be seen by inspecting the CSVs in the raw data cache. Users of the python interface can add additional
columns as shown below. If you using a feather or parquet based cache the rebuild option should be set to
true so the additional columns are added to the cache files when they are rebuilt. This method of adding additional
columns should also work with the cache_compiler function.
from nemosis import defaults, dynamic_data_compiler
defaults.table_columns['BIDPEROFFER_D'] += ['PASAAVAILABILITY']
start_time = '2017/01/01 00:00:00'
end_time = '2017/01/01 00:05:00'
table = 'BIDPEROFFER_D'
raw_data_cache = 'C:/Users/your_data_storage'
volume_bid_data = dynamic_data_compiler(start_time, end_time, table, raw_data_cache, rebuild=True)Static tables do not include a time column and cannot be filtered by start and end time.
To learn more about each static table visit the wiki.
You can view the static tables available by printing the tables in NEMOSIS' defaults:
from nemosis import defaults
print(defaults.static_tables)
# ['ELEMENTS_FCAS_4_SECOND', 'VARIABLES_FCAS_4_SECOND', 'Generators and Scheduled Loads', 'FCAS Providers']The static_table function can be used to access these tables
from nemosis import static_table
fcas_variables = static_table('VARIABLES_FCAS_4_SECOND', raw_data_cache)NEMOSIS uses the python logging module to print messages to the console. If desired, this can be disabled after imports, as shown below. This will disable log messages unless they are at least warnings.
import logging
from nemosis import dynamic_data_compiler
logging.getLogger("nemosis").setLevel(logging.WARNING)