print · login   

Benchmark structure

The Automata wiki is build using the Pmwiki wiki software which allows wiki pages to be organized into groups of related pages. Each benchmark is organized as a wiki group of a set of related pages and has to follow a simple prescribed structure. Below we will describe this structure using the BenchmarkPassport? as an example.

A benchmark can be about a specific case study, but it can also be specific collection of models. For example you have a set of models you used in a set of publications. You can then make a benchmark for this set of models. The benchmark then describes this set of models where you refer in your description to the literature.

To create a new benchmark you must carefully follow the instructions as described in the Help section, where you can also read about all technical wiki editing details.

Each benchmark must have :

  • a Description? page
  • a Bibliography? page which list the literature you are referring to in the benchmark.
  • a data page per basic model type
    • with the page named after the basic model type : <modelType>
    • example: Mealy
    • which using wiki syntax lists the models of that type the benchmark has
    • the following basic model types are supported: DFA,NFA,Interface,Moore,Mealy,Register
    • some model types are not in the benchmark, so then the benchmark has no page for that type
    • the models will be automatically added to the automatically generated distribution zip files on the Downloads page
    • every 5 minutes a script check if there are changes to the model data, and will regenerate any affected zip file
    • example: Register? model in BenchmarkPassport?
  • a SideBar? page which list above pages
    • each wiki group can define its own SideBar page
    • when a wiki page is shown then the Sidebar of the wiki page's group is shown left to the wiki page
    • the SideBar is itself a wiki page which can therefore easily be edited
    • when viewing a page in a wikigroup its SideBar page can be easily edited with the "edit_menu" link located at the top of the page, however only when logged in

Optionally a benchmark can have:

  • an Extra data page per basic model type of less important models
    • with the page named: Extra<modelType>
    • example: ExtraMealy
    • which you don't want to be linked in the Sidebar page because they are not informative for the benchmark
    • but the extra models will, as the other models, be added to the nightly generated zip files in the Downloads page
    • for example the extra models can be derived models from a main benchmark model
    • example: we flattened the Register? model in the BenchmarkPassport? for several data ranges giving us ExtraMealy? models
  • any other page you want to add to the benchmark
    • for example you could describe a learning experiment with some learner tool on the benchmark