Differences between revisions 2 and 3
Revision 2 as of 2010-08-05 15:14:01
Size: 1276
Editor: JendrikSeipp
Comment:
Revision 3 as of 2010-08-05 15:56:59
Size: 2323
Editor: JendrikSeipp
Comment:
Deletions are marked like this. Additions are marked like this.
Line 6: Line 6:
An experiment is conducted in three stages: Generation of experiments, fetching of results and production of reports. Each stage has its own generic main module: `experiments.py`, `resultfetcher.py`, `reports.py`. These modules provide useful classes and methods and can be imported by scripts that actually define concrete actions. For the fast downward planning system the example scripts that use these modules are `downward-experiments.py`, `downward-resultfetcher.py`, `downward-reports.py`. The first one can be seen as a reference example for own experiments, the other two can be used as they are from the commandline. Passing `-h` on the commandline gives you an overview of each script's commands. An experiment is conducted in three stages: Generation of experiments, fetching of results and production of reports. Each stage has its own generic main module: `experiments.py`, `resultfetcher.py` and `reports.py`. These modules provide useful classes and methods and can be imported by scripts that actually define concrete actions. For the fast downward planning system the example scripts that use these modules are `downward-experiments.py`, `downward-resultfetcher.py` and `downward-reports.py`. The first one can be seen as a reference example for own experiments, the other two can be used as they are from the commandline. Passing `-h` on the commandline gives you an overview of each script's commands.
Line 22: Line 22:
Traverses the directory tree under "test-exp" and parses Traverses the directory tree under "test-exp" and parses each run's experiment files. The results are written into a new directory structure under "test-exp-eval". In the process each run's properties file is read and its "id" determines the run's destination directory in the new directory tree. By default only the properties file is copied and the parsed values are added to it. To copy all files you can pass the "-c" option.
Line 29: Line 29:
Reads all properties files found under "test-exp-eval" and generates a big dataset from them. This dataset is serialized into the "test-exp-eval" directory for faster future reports. If you want to reload the information directly from the properties files, pass the "--reload" parameter.

The dataset is then used to generate a report. By default this report contains absolute numbers, writes a Latex file and analyzes all numeric attributes found in the dataset. You can however choose only a subset of attributes and filter by configurations or suites, too. A detailed description of the available parameters can be obtained by invoking `downward-reports.py -h`.

Back to HomePage.

Experiment scripts

In the directory "new-scripts" you find some scripts that facilitate conducting experiments. An experiment is conducted in three stages: Generation of experiments, fetching of results and production of reports. Each stage has its own generic main module: experiments.py, resultfetcher.py and reports.py. These modules provide useful classes and methods and can be imported by scripts that actually define concrete actions. For the fast downward planning system the example scripts that use these modules are downward-experiments.py, downward-resultfetcher.py and downward-reports.py. The first one can be seen as a reference example for own experiments, the other two can be used as they are from the commandline. Passing -h on the commandline gives you an overview of each script's commands.

Generate an experiment

./downward-experiments.py test-exp -c cea -s TEST

Generates a simple planning experiment with the configuration cea and the suite TEST in the directory "test-exp".

Fetch and parse results

./downward-resultfetcher.py test-exp

Traverses the directory tree under "test-exp" and parses each run's experiment files. The results are written into a new directory structure under "test-exp-eval". In the process each run's properties file is read and its "id" determines the run's destination directory in the new directory tree. By default only the properties file is copied and the parsed values are added to it. To copy all files you can pass the "-c" option.

Make reports

./downward-reports.py test-exp-eval

Reads all properties files found under "test-exp-eval" and generates a big dataset from them. This dataset is serialized into the "test-exp-eval" directory for faster future reports. If you want to reload the information directly from the properties files, pass the "--reload" parameter.

The dataset is then used to generate a report. By default this report contains absolute numbers, writes a Latex file and analyzes all numeric attributes found in the dataset. You can however choose only a subset of attributes and filter by configurations or suites, too. A detailed description of the available parameters can be obtained by invoking downward-reports.py -h.