How to use Data Subscription

Overview

The Data Model The definition of the interface to participants of data published by AEMO for gas or electricity. A database conforming to the Data Model can contain a local copy of all current participant-specific data recorded in the main database. The Data Model includes database tables, indexes, and primary keys. is updated in line with the release process for the wholesale system. The release cycle is normally every six months and usually there are changes to the Data Model, for example, new tables and new columns added to existing tables. After each new release of the Data Model, you use Data Subscription A web application in the Markets Portal used to subscribe to Data Interchange files and to obtain the latest MMS Data Model. These files are typically csv files containing public data and participant’s private data. AEMO recommends participant use of the Data Interchange suite of products, supplied by AEMO, to automate receiving and storage of the files to their systems. AEMO provides working sample software for FTP access (Participant Batcher) to MarketNet and working sample software for Data Interchange. to:

For help upgrading, see Upgrading a Data Interchange environment.

    AEMO recommends subscribing to all files and activating or deactivating them so you have greater flexibility to access historical data.

  1. Download and apply scripts to upgrade your existing database to the latest version of the Data Model.
  2. Check if there are any new files to subscribe to. For help, see Files not subscribed to.
  3. Check if there are any Legacy (redundant) files your are subscribed to. If yes, load the new versions of the files into your local DBMS at least once before unsubscribing. For an example of Legacy files, see Figure 24.
  4. Unsubscribe from Legacy files. For help, see.Unsubscribe from Legacy files.
  5. Download any upgrades to the Data Interchange software (pdrLoader and pdrBatcher). For help, see Release Schedules and Technical Specifications.

To receive the new release data, participants must keep their Data Model version in line with AEMO's. AEMO only supports the current and previous versions of software.

Managing CSV files

Activating and deactivating are different from subscribing and unsubscribing.

This section explains the difference between activation, deactivation, subscribing, and unsubscribing.

Subscribing

AEMO recommends subscribing to all files and activating or deactivating them so you have greater flexibility to access historical data.

Subscribing means subscribing to files in the Data Subscription web application allowing delivery of the files to your participant folder on AEMO Australian Energy Market Operator's Participant File Server The publishing point from AEMO systems to participant systems. Each participant is allocated an account and access to private and public areas. Participants are responsible for interfacing with the participant file server. where the pdrBatcher Software responsible for transferring files to and from AEMO's participant file server to the participant's local Data Interchange folders. It can be installed in any environment supporting Java. software can move them to your local folder.

The files remain on AEMO's Participant File Server for approximately six months.

Unsubscribing

Unsubscribing from files means they are not delivered to your participant folder on AEMO's Participant File Server.

AEMO systems keep an archive of all subscribed-to files for approximately six months. Unsubscribing from files in the Data Subscription web application means you cannot access six months of historical files in the future.

Activate

Once the files arrive at your site, you can activate them so the pdrLoader Software responsible for loading files to and from participant's local Data Interchange folders to the participant's DBMS. It can be installed in any environment supporting Java. loads them into your local Data Model database. If they are not activated, they remain in the folder at your local site.

Deactivate

Deactivating the files stops them loading into your local Data Model database. It does not stop the delivery of the files to your local folder.

If you deactivate a file without unsubscribing, then AEMO continues to keep an archive of the file so you can retrieve and load it in the future.

If you unsubscribe from a file, it is not possible to retrieve and load in the future.

Replication Manager

For help, see Replication Manager User Guide.

AEMO provides the Replication Manager software, a Windows-based application that runs locally, to activate or deactivate the loading of .csv Comma Separated Values. A file format for data using commas as delimiters. files into their local Data Model database.

 

Using the interface

Figure 22 Interface explanation

Explanation of fields

Table 31 Explanation of fields

Column Name

Description

This check box shows which items are selected. By default, no items are selected.

  • To select one file, click the check box beside the filename.
  • To select all files, click the check box in the header row.

>

Click the button to view detailed information about the data model tables included in the package.

It displays a table with the following information:

  • Data Model table

  • Report type

  • Report subtype

  • Report version

File ID

The File ID name is internal to EMMS and identifies the source of the data published in this file.

Files replaced by a new version are renamed by adding _LEGACY to the end of the previous name, for example: BIDMOVE_COMPLETE_LEGACY

This name is different to the Data Model Identifier name. To see the Data Model Identifier click the View Details option.

For more details, see Data Model Identifier in Description.

Package ID

Package name identifier

Description

Provides a description of the file contents. The description is also updated once a file becomes a legacy version.

The Data Model Identifier is not specifically displayed on this interface, but if it differs from the Identifier name, it is included in the Description field in italics, preceded by DM.

For example, if the Identifier is SETTLEMENTS_LEGACY and the Data Model Identifier is SETTLEMENTS then (DM.SETTLEMENTS) displays.

Status

Indicates the status of the file:

  • Active: a current version file.
  • Deprecated: a Legacy file that becomes redundant next time the file is updated.

Visibility

Indicates if the file is internal or external. Assignable indicates external files, participants can perform actions on these, for example, subscribe, unsubscribe, etc.. For internal files, you can only view the details.

Data Model

Indicates whether a table is a Data Model table or not. This column displays Y for Data Model tables and N for Non-data Model tables.

Subscribing and unsubscribing

Using the interface, you can subscribe and unsubscribe files and packages.

Based on the option you select from the drop-down options, you can either subscribe or unsubscribe files. For more information, see:

  • Subscribing files
  • Unsubscribing files

Filter or search by keywords or status

You can look for specific files or group of files by filtering the list:

  • Filter by Package: Enter the package name in the search field to see only files in a certain package.
  • Keywords: Takes a combination of free text
  • Status: Any: any status, Assignable: a current version file, Deprecated: a legacy file, Visible: no longer used.

For help, see Interface explanation.

Rules
  • You can only use one filter option at a time.
  • Searches are not case sensitive.

Populating a new DBMS

You use Baseline Data to populate a new Data Interchange A set of cooperating applications used to replicate data between AEMO's energy market systems and a participant's DBMS conforming to the MMS Data Model. environment. The data is in the same folder as the participant ID’s file server folders. AEMO updates the \MarketData\Baseline folder daily with current baseline data.

Historical data

Older files are moved to the Reports Archive and kept for 13 months. The data is in csv format and conforms to the Data Model. To obtain historical data, see http://www.nemweb.com.au/Reports/ARCHIVE/.

Data Model documentation

You can find detailed explanation of the packages, tables, and files in the following documentation:

  • Data Model Report: Describes all packages and tables.
  • Data Model table to file to report relationships: describes the relationships between packages, tables, and files.

    • For details about other Data Interchange documents, see Data Interchange Help.