This page is intended to help agent developers get started with the Power TAC server and agent framework. Please feel free to add your experiences and insights to the Discussion page (note that you may need to log in to edit that page).
There are currently two main components needed by broker developers: the simulation server and the sources for the sample broker. In order to use them, you will need a Java development environment (version 1.8 or higher), and Apache Maven (version 3.3.0 or higher). Mac users may want to refer to the MacSetup page. Release notes and download links for released versions are available on the Release History page. Note that these components are not packaged as "apps" for any particular operating system, and currently the only user interface is the web-based Visualizer in the server. If someone knows how to package (and automate the packaging) for systems such as Windows or Mac, please feel free to volunteer.
In addition to the Java and Maven tools, you will also need accurate network time synchronization on any machine that will run a broker against a remote server (such as the tournament servers). This requires a working NTP daemon. NTP setup is easy on Linux and Mac machines, but it can be less easy on Windows, especially if you do not have full administrative access. It is not enough to simply have your system clock synchronized when the machine boots; time offsets of over 50 msec can lead to messages being ignored, and clock drift on a heavily loaded machine during a 2-hour simulation session can cause your broker to be disconnected. One popular tool for NTP synchronization on Windows is the Meinberg package.
The simulation server is a Spring application distributed as a maven pom.xml file, along with a sample configuration file and instructions for setting up and running the server. When you run it, Maven will pull in all the server components and dependencies from the Maven Central repository. You may download version 1.4.1, used in the 2017 competition, or a reasonably stable development snapshot.
The sample broker is distributed in source form, along with instructions for building and running the broker. The Java version is a Spring application. In older versions, it was packaged as a separate core module and a set of decision services. This allows you to take advantage of newer versions simply by updating the maven configuration. The latest 2017 release supports a number of improvements including the new capacity transaction, and an accounting of total production and consumption in each timeslot. It also removes the broker core to a separate module, which should make it much easier to adapt existing brokers to new features. As always, the development snapshot is always the most up-to-date. We aim to keep it reasonably stable for the sake of broker developers.
If you choose to use the development snapshots, note that the sample broker on github is the HEAD of the development branch, while the server module will pull in the latest deployed snapshot, which is usually more stable than the tip of the development branches of all the components.
If you wish to modify or contribute to further development of the simulation server or the sample broker, you will need to set up a server source environment. Please start by reading through the Getting Started and Development Policies pages on the Power TAC github wiki. You may also want to browse through the Power TAC issue-tracking system on github.
In addition, you may want to download the Power TAC log-analysis tool.
- 1 Sample Broker
- 2 Power TAC simulation server
At the time of this writing, the only available implementation of a Power TAC sample broker is in Java. Others, possibly including Repast, R, and/or Python frameworks, are on the agenda but awaiting a volunteer to take responsibility.
Documentation for the Java version is available on the Minnesota Jenkins CI site. Documentation for the powertac-core module is also on the Jenkins CI server. This is where you will find details on all the message types that your broker can send and receive.
Broker developers have been using Windows, Mac, and various flavors of Linux successfully. The primary requirements are:
- A Java development kit (JDK) version 1.8 or higher. The Java JRE is not adequate.
- Apache Maven version 3.3.0 or higher. Maven provides a rich dependency-management infrastructure as well as goal-directed build automation. If you do not have a copy of maven 3.3.0 or higher on your system, you will need to install it.
- At least 4 Gb of memory is needed to run both the simulation server and a single broker using an IDE.
- To successfully run a broker against a remote server, both machines must be time-synchronized using ntp, the network time protocol, as described above. Server configurations of most operating systems run the ntp daemon by default, but many desktop setups do not.
The sample broker, like the Power TAC simulation server and the tournament manager, is a Spring application. It can be run from a command-line using Maven, or it can be run from a development environment like STS, the Spring Tool Suite. STS is Eclipse with a number of additions specific to Spring. Other IDEs can also be used if you prefer, but STS is what most of the server development team uses, with the result that there is help available within the Power TAC community.
Build and run the sample broker
The first thing to try after downloading the sample broker is to build and test it. You should do this first from the command-line using Maven as
mvn clean test
This will download a number of modules, then build the sample broker and run a few JUnit tests. If this does not work, then you most likely have a problem with either your Java or Maven setup.
The next step is to load the sample broker code into your IDE, then make sure you can build and run it there. In STS, you do this with
Import->Existing Maven project, then choose the directory where you unpacked the sample broker code. You should now be able to build and test the agent code with your IDE.
When you are ready to run your broker, details on the command-line arguments are given in the README file contained in the source distribution. You can, of course, use those same arguments inside your IDE by setting up "run configurations". This will allow you to set breakpoints and debug your broker. If you want to use breakpoints, you may want to make the server pause and wait when the broker hits a breakpoint. This can be done by (1) adding the property
server.brokerPauseAllowed = true to the server configuration file in server-distribution/config/server.properties, and (2) using the command-line argument
--interactive when starting the broker.
Finally, instructions for running your agent with the simulation server are given in the section on running the server. Note that the broker can be set up to run multiple "sessions" or games in sequence without being re-started. This allows around-the-clock tournament schedules without requiring grad students to stay up all night just to run their code.
Modify the sample broker
The sample broker (javadoc) is a Spring application based on three major modules.
- The common module in
org.powertac.commonis shared with the simulation server, and contains the elements of the Power TAC domain model along with some basic shared infrastructure such as repositories, xml-serialization code, and a configuration engine based on the Apache Commons configuration library. Source is available in the common repo on github.
- The broker core module in
org.powertac.samplebroker.corecontains the broker framework and the interface to the simulation server, including the broker-server communication protocol. Source is available in the broker-core repo on github.
- A set of example behaviors implemented as Spring service classes in
org.powertac.samplebroker, including the ContextManager, the PortfolioManager, and the MarketManager. These implement a basic set of agent behaviors, driven primarily by incoming message traffic from the server. They are intended to demonstrate recommended processes for handling messages, for per-timeslot activation, and for initialization and configuration. You are welcome to modify them or replace them with your own designs. Note that message handling is done with overloaded
handleMessage()methods that are dispatched from the core module using Java reflection, and configuration is based on an annotation scheme.
The recommended way to build your broker is to modify or replace the source code in the
org.powertac.samplebroker package, or to use this code as inspiration for your own code in your own packages. Although it is not absolutely necessary, we recommend that you build your broker as a maven project, or using some other tool that uses maven repositories, such as Gradle. This will eliminate the necessity to keep track of all the dependencies on Spring and Apache components.
org.powertac.samplebroker.core modules will be retrieved as needed by Maven. You can download sources from github for the common or broker core modules and import them into your IDE. Release downloads are available under the "Releases" tab, and the current development snapshot is available either by cloning or downloading a zip file near the lower-right corner of the page.
The broker agent is configured by adding name=value pairs to
src/main/resources/config/broker.properties. You can also specify a separate configuration file using the
-config option when you start the broker. Each property name is composed of the classname (without the org.powertac prefix) followed by the attribute name of a configurable value. Configurable values are instance variables or setter methods annotated in the source with
@ConfigurableValue along with type and a description. Configuration happens when a service calls
propertiesService.configureMe(this). See the configurable values and the
init() method in PortfolioManager for some examples. If you want to configure classes that are not in org.powertac, you can simply use the entire qualified classname.
All interaction with the simulation server is through xml-serialized jms messages. You can write code to receive them by writing a
handleMessage(msg) method to process each message type you want to see. All messages are instances of classes in packages org.powertac.common and org.powertac.common.msg in the common module. You may, of course, add additional message-handling modules simply by following the pattern in the existing MarketManagerService and PortfolioManagerService (see MarketManagerService.java for example). Any module that implements the
Initializable interface will have its
initialize(BrokerContext context) method called before each simulation session, and all the
handleMessage() methods will be registered to receive their respective message types. Outgoing messages are sent to the server by calling the
sendMessage(msg) method on the context. Javadocs are posted on our Jenkins CI server.
The server sends its per-timeslot messages out in batches, and the last one is guaranteed to be a TimeslotComplete message. When that message arrives, all modules that implement the
Activatable interface will have their
activate() methods called, in no particular order. If anyone feels it is important to control the order of activation, it would require a relatively easy modification in the core, along with some additional configuration. You may request such a change, report defects, or recommend other improvements, through the Power TAC issue tracker.
Brokers also need to interact with the Power TAC Tournament Manager, through a REST interface. This interaction is encapsulated in the
core.BrokerTournamentService module. The only purpose of this interaction is to acquire credentials and access details for simulation servers that run under control of the Tournament Manager. You should not need to modify this code.
If you want to add a new top-level module to the broker, it should be enough to simply annotate it with
@Service ahead of the class declaration, implement the
Initializable interface (provides per-session module initialization), and possibly implement the
Activatable interface (provides per-timeslot activation). MarketManagerService and PortfolioManagerService are annotated this way. This is all the information Spring needs to instantiate it and wire it in through dependency injection, as long as the module is in a package Spring knows about. That bit of configuration is in
src/main/resources/broker.xml. As shipped, all Service modules in
org.powertac or any subpackage are included by Spring. If you want to use your own packages, you simply need to add another component-scan clause listing the package name at the top of your own package hierarchy.
The sample code synchronizes access to variables between the
handleMessage() methods and the
activate() method in the service modules. This is necessary because the
handleMessage() methods run in threads managed by the JMS service, while the
activate() methods run in a thread that is driven by the simulation clock.
In a tournament situation, Broker agents are commonly started once for a series of games, which means that you have to be careful about how they re-initialize between games. Modules that implement the
Initializable interface will have their
initialize() methods called before the agent attempts to log in for the next game. All initialization code should be in
initialize() methods, and not in a constructor or elsewhere, otherwise your agent may not recover correctly between games.
When a broker logs in to the simulation server, it receives in the "broker-accept" message a number that's used as a prefix for the ID values in objects created and sent to the server during a game. This is important, because the server uses those object id values as keys in various maps. A consequence is that it does not work to keep objects (like tariff specifications, for example) over between games - if your id prefix is different in the next game, and it often is, your object ids can collide with objects from some other broker. The server will avoid this by rejecting these messages, so you need to either re-create those objects, or update their ids at the beginning of each game. You can get new id values using
IdGenerator.createId() after the login is complete. A good place to do that would be in
Power TAC simulation server
To test your broker, you will need it to interact with the server. The Power TAC simulation server (or just the "server") is a Spring application, designed to be installed and run using Maven. Note that the server release packages are simply maven configuration files with some documentation; all the actual code will be downloaded by maven the first time you try to run the server.
Running the server
The server runs the Power TAC simulation in two modes; detailed instructions are in the README file included with the server distribution package once you have downloaded and unpacked it:
- In "bootstrap" mode, the model runs for a period of time with only the default broker, and collects data on customer power usage, market prices, and weather. This data is saved to a file at the end of the bootstrap period. The resulting data file can be used to run multiple sim sessions.
- In "sim" mode, the model loads a bootstrap data file and restores the model to its state at the end of the bootstrap period, then allows brokers to log in and sends them the bootstrap data before starting a simulation. The simulation literally starts at the end of the bootstrap period, so for example the first timeslot is not number 0, but the number of timeslots in the bootstrap period. Note that not all of the timeslot-related data is carried over from the bootstrap sim - for example, you cannot see the orderbooks from timeslots that completed in the bootstrap period.
As described in the README file, you can run the server as a web-based app under the jetty webserver, or as a command-line app without a visual interface. In bootstrap mode, there is not much to see, and it runs too fast for the visualizer to keep up. However, you can run a bootstrap session using the visualizer to generate the bootstrap data file needed by the full simulator.
Using the visualizer (version 1)
To run the simulator/visualizer combination, you will need a machine with at least 4 Gb of memory and several Gb of free disk space. Two or more cores will be helpful, although the server, visualizer, and sample broker should not be compute-bound on any reasonably hefty desktop or laptop machine. When you start the server using
mvn -P web
it will (if necessary) download modules from the maven repository, then start an instance of the jetty webserver containing the server and visualizer. Once jetty is running, browse to http://localhost:8080/visualizer to get started. If you navigate to the "Settings" page, you will see a form that allows you to set up and run boot and sim sessions. The top of that page is shown here:
Common options allow you to select an alternate configuration file, and to specify a suffix for naming log files. If you leave them both blank, it will use the default configuration in
config/server.properties, and the logfiles will be named
log/powertac-boot-0.state for a boot session. Since a standard sim runs for about 2 hours, there is a sample configuration file
short-game.props provided that will finish in around 10 minutes. You are welcome to edit these files or copy and modify them as you see fit. Details on the various configuration options and what they do are provided in comments in
config/server.properties. Keep in mind that some values (like the competition.timeslotLength) may not have the expected effect if you change them, but values like interest rates and fees should have the expected effects.
To run a boot session, scroll down to the boot panel (shown below), fill in the name of the file you wish to have the bootstrap data saved to (for example,
boot.xml, and poke the "Run" button. A boot session normally takes less than 5 minutes to complete.
Once you have a bootstrap-data file available, you can run sim sessions. Simply scroll down to the sim panel (shown below), fill in the name of a bootstrap-data file (created by a boot session), add brokers, and start the session with the "Run" button. The server will load the bootstrap data, then wait for the brokers to log in before starting the simulation. Brokers are separate programs identified by their "usernames" -- this is the value of the
samplebroker.core.powerTacBroker.username property in
sample-broker/broker.properties. Note that the broker username is case-sensitive; in other words, the username in the broker.properties file must exactly match the name filled in as the broker name in the visualizer dialog, including capitalization. The username may not contain embedded spaces, and the only acceptable characters are alphanumeric and underscore.
The broker is started with
mvn exec:exec. Additional options are explained in the README file.
If you want to run a broker on a separate machine, you will have to choose a JMS URL that is visible on the network. The default url is
tcp://localhost:61616; if you want to use an external URL and you are running the server on a machine "tardis.powertac.org", the url would be
tcp://tardis.powertac.org:61616. This value would also have to be set in the broker's broker.properties file for all participating brokers.
Once all the brokers have logged in and the boot data has been loaded, the Game Status indicator at the top of the page will change to "running" and you can poke the Game View link to see the visualizer display. In the visualizer display, the wrench icon brings you back to the competition control page.
Using the visualizer (version 2)
You can run the new version of the visualizer by issuing the Maven command
mvn -P web2
This will start up a server on http://localhost:8080. It takes a little while for it to start up, so please be patient.
Like the version 1 visualizer, it allows you to run boot games, and subsequently -- given the record of a boot game -- to run simulations with brokers. Unlike the version 1 visualizer, you can replay earlier games (either ones that you've run within the visualizer, or using the state logs from external sources). You can even provide an URL to a compressed log bundle as produced during the annual Tournament, e.g. one of the entries at http://ts.powertac.org/log_archive/finals_2016_06/ , and replay that directly.
The GUI to set up and start/stop games, of all varieties, is in the Games tab of the application, and ought to be more or less self-explanatory. (You don't see all input fields at once, they become visible as you enter information).
Once a game is running you can monitor its progress in the Graphs tab. Note that this page is a bit tall, scroll down to see further plots. One thing to keep in mind, when running sims, is that the game doesn't actually start until all the brokers you have added to the game have connected. Until that time, the game status will remain "WAITING".
Each of the plots has a small button in its top-right corner, which allows you to save the contents of the plot as a separate image file.
In the Files tab you'll find a list of materials, like config files, state- and trace-logs, which you can download. In this same tab you can also upload material to be used in running or replaying games.
All of the files, as well as the logs generated by sims/boots and the visualizer itself, are under the /files directory. If you drop in material, e.g. a configuration .properties file, into the appropriate location there, the visualizer should detect them and make them available via the GUI the next time you initialize a game.
Using the command-line interface
For many testing purposes, it makes more sense to run the server from the command line. Detailed instructions are given in the README file.
Understanding the log files
As the server runs, it dumps data to two log files, known as the trace log and the state log. See the Logfile Analysis page for details on interpreting this data.
Running your broker in a competition
Once you are ready to participate in a competition, you will need to register, and set up your broker(s) with the server-supplied credentials. These credentials are specific to a particular tournament. Detailed instructions are currently on the github wiki.