# Benchmark structure

The Automata wiki is build using the Pmwiki wiki software which allows wiki pages to be organized into groups of related pages. Each benchmark is organized as a wiki group of a set of related pages and has to follow a simple prescribed structure. Below we will describe this structure using the BenchmarkPassport as an example.

A benchmark can be about a specific case study, but it can also be specific collection of models. For example you have a set of models you used in a set of publications. You can then make a benchmark for this set of models. The benchmark then describes this set of models where you refer in your description to the literature.

To create a new benchmark you must carefully follow the instructions as described in the Help section, where you can also read about all technical wiki editing details.

Each benchmark must have :

• a Description page
• a Bibliography page which list the literature you are referring to in the benchmark.
• a data page per basic model type
• with the page named after the basic model type : <modelType>
• example: Mealy
• which using wiki syntax lists the models of that type the benchmark has
• the following basic model types are supported: DFA,NFA,Interface,Moore,Mealy,Register
• some model types are not in the benchmark, so then the benchmark has no page for that type
• the models will be automatically added to the automatically generated distribution zip files on the Downloads page
• every 5 minutes a script check if there are changes to the model data, and will regenerate any affected zip file
• example: Register model in BenchmarkPassport
• a SideBar page which list above pages
• each wiki group can define its own SideBar page
• when a wiki page is shown then the Sidebar of the wiki page's group is shown left to the wiki page
• the SideBar is itself a wiki page which can therefore easily be edited
• when viewing a page in a wikigroup its SideBar page can be easily edited with the "edit_menu" link located at the top of the page, however only when logged in

Optionally a benchmark can have:

• an Extra data page per basic model type of less important models
• with the page named: Extra<modelType>
• example: ExtraMealy
• which you don't want to be linked in the Sidebar page because they are not informative for the benchmark
• but the extra models will, as the other models, be added to the nightly generated zip files in the Downloads page
• for example the extra models can be derived models from a main benchmark model
• example: we flattened the Register model in the BenchmarkPassport for several data ranges giving us ExtraMealy models
• any other page you want to add to the benchmark
• for example you could describe a learning experiment with some learner tool on the benchmark