Worked Examples

Getting Started: Generators and Scheduled Loads

A great place to start exploring the AEMO data sets is the list of Generators and Scheduled Loads, this data is from AEMO's list of registrations. Access it through nem-data by following the steps below, where a predefined session is used.

• Follow this link and download the latest release of the graphical user interface as an executable file.
• Download the session file for this example here
• Open the nem-data.exe file, windows will not recognise this file so you need to inform it that it is safe to run, click through any warnings
• Click the load session button and then select the example_zero.pkl file you just downloaded
• You will also need to create two new folders on you computer, one for the raw data cache and one to save the results in. Then provide the paths to these folder to nem-data in the text entry boxes provided.
• Start and end times are not given for the Generators and Scheduled Loads table as this is a static table and its contents does not change with respect to time
• Now you should be ready to run the query, click the 'Run queries' button. Any time you run a query that requires new data to be downloaded it may take some time while nem-data creates and formats the local data cache.

Now the data should be ready for you to view. Explore the generator set and try using generator's DUIDs to filter other queries like those outlined below.

Filtered Query of Dispatch Targets

This example outlines how to construct a query for dispatch targets of a single dispatch unit (Hallett 2 Wind Farm). The steps below describes how to create your own query from scratch.

• The query is given a name, this will be used when nem-data saves the results as a csv
• A start and end time are given
• The DISPATCH_LOAD table is selected as this contains the dispatch targets data
• The columns 'SETTLEMENTDATE', 'DUID' and 'INTERVENTION' are automatically selected by nem-data as they from the primary key for this table.
• Additionally we select the column 'TOTALCLEARED' to get the dispatch targets for each interval
• The filter for Capital Wind Farm is created by entering its DUID in to the filter entry box, pressing enter and then selecting it from the list below.
• The filter for the intervention flag is automatically opened by nem-data, but will be ignore unless values are added and selected.

The query can be run using the 'Run queries' button. It can also be saved and loaded using the buttons at the top of the window. The file to load this session can be found here.

SCADA Data for Solar Generators

This example outlines how to query for scada data and generator fuel types, additionally it demonstrates how to merge data sets.

• In the AEMO database generator information and unit scada data are stored separately, therefore in order create a data set of scada data for just solar generators we need to query two data sets.
• First all the scada data is pulled unfiltered, then all the fuel type data is pulled from the Generators and Scheduled Loads table but is filtered to just return 'Solar' generators.
• When an inner join is used to merge the data sets on the 'DUID' column all non solar generators are excluded from the result as they are not in the right hand table
• In this example query names not only provide file names for saving data, but are also referenced when merging data.

The query can be run using the 'Run queries' button. It can also be saved and loaded using the buttons at the top of the window. The file to load this session can be found here.

Custom tables | 4 second Data | Crashing the Query

nem-data is also equipped to host algorithms for producing custom data tables that synthesise publicly available AEMO data into something new and interesting. One example of this is the table FCAS_4s_SCADA_MAP. AEMO records 4 s load, generation and interconnector data for the purposes of the FCAS Causer Pays calculations. However the names in this data are from the Energy Management System (EMS) while most other tables use names from the Market Management System (MMS), AEMO does not provide publicly available mapping between these names. This is the job of the FCAS_4s_SCADA_MAP table, linking 4 s generation data and with other data like generator fuel types, this opens interesting possibility when unpicking 'sticky' ideas like generator 'reliability'. HOWEVER! 4 s data for all generators in the NEM is extremely memory intensive, this is where nem-data does not hold your hand, ask for too much data and it will happily crash your query with a memory error and not return any data. This particularly applies to the algorithm that creates the FCAS_4s_SCADA_MAP table.

Example: 4 s Data for Hallett 2 Wind Farm

Here we are going to pull the 4 s data for Hallett 2 Wind Farm from the same time period we looked at previously, its good to choose this time period because we can be confident that the wind farm is on and therefore that the matching algorithm. This is because the algorithim works by comparing MMS Scada data to the FCAS 4 s data.

• First create the query for the FCAS_4s_SCADA_MAP, filter this with the MARKETNAME (DUID) 'HALLWF2' and the start and end time shown below, here we contract the time window significantly as the matching algorithm has memory requirements higher memory requirements then just pulling the 4 s data. Also we won't select the 'ERROR' column as this is mostly for monitoring the accuracy of the matches.
• Then create the query to pull the raw FCAS_4s_DATA table, the memory requirements for just pulling the data are lower, so we can broaden the time window here.
• Lastly use an inner join of the two previous datasets to leave us just with the 4 s data for Hallett 2 Wind Farm.

The query can be run using the 'Run queries' button. It can also be saved and loaded using the buttons at the top of the window. The file to load this session can be found here.