|
Size: 10037
Comment:
|
← Revision 43 as of 2023-10-12 12:31:57 ⇥
Size: 0
Comment: Information was moved to README.md in repository (or considered irrelevant here)
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| Back to HomePage. = Experiment scripts = In the directory "new-scripts" you find some scripts that facilitate conducting experiments. An experiment is conducted in three stages: Generation of experiments, fetching of results and production of reports. Each stage has its own generic main module: `experiments.py`, `resultfetcher.py` and `reports.py`. These modules provide useful classes and methods and can be imported by scripts that actually define concrete actions. For the Fast Downward planning system the example scripts that use these modules are `downward_experiments.py`, `downward-resultfetcher.py` and `downward-reports.py`. Together they can be used to conduct Fast Downward experiments. '''Passing `-h` on the command line gives you an overview of each script's commands. Be sure to check out the available options as they may be more up to date than the instructions here.''' == Conducting a local Fast Downward experiment for the impatient == {{{#!highlight bash # Preprocessing ./downward_experiments.py expname -s TEST --preprocess ./expname-p/run ./resultfetcher.py expname-p # Build and run the experiment ./downward_experiments.py expname -c downward_configs.py:yY -s TEST ./expname/run # Make reports ./downward-resultfetcher.py expname ./downward-reports.py expname-eval }}} Below you find a detailed description of the above steps. == Generate an experiment == {{{ ./downward_experiments.py expname -c downward_configs.py:yY -s TEST }}} Generates a planning experiment for the suite TEST in the directory "expname". The planner will use the configuration string `yY` found in the file `downward_configs.py`. When you invoke the script you can specify on the command line whether you want the experiment to be run locally or on the gkigrid. You can also directly set the timeout, memory limit, number of processes, etc. Before you can execute that command however you have to run it once with the `--preprocess` parameter. This will generate a preprocessing experiment. After you have run this preprocessing experiment and fetched the results with `./resultfetcher.py` you can generate the search experiment. Local experiments can be started by running {{{ ./expname/run }}} Gkigrid experiments are submitted to the queue by running {{{ qsub expname/expname.q }}} == Fetch and parse results == {{{ ./downward-resultfetcher.py expname }}} Traverses the directory tree under "expname" and parses each run's experiment files. The results are written into a new directory structure under "expname-eval". In the process each run's properties file is read and its "id" determines the run's destination directory in the new directory tree. By default only the properties file is copied and the parsed values are added to it. To copy all files you can pass the "--copy-all" option. === Partial fetching === If you don't need the results of the preprocessing or search phase, you can pass the options `--no-preprocess` and `--no-search` to `downward-resultfetcher.py`. This obviously speeds up parsing, too. === Combine results of multiple experiments === It is possible to combine the results of multiple experiments by running the above command on all the experiment directories while specifying the target evaluation directory. If you don't specify the evaluation directory it defaults to "exp-name-eval". An example would be {{{ ./downward-resultfetcher.py exp1 --dest my-eval-dir ./downward-resultfetcher.py exp2 --dest my-eval-dir }}} == Make reports == {{{ ./downward-reports.py expname-eval }}} Reads all properties files found under "test-exp-eval" and generates a big dataset from them. This dataset is serialized into the "expname-eval" directory for faster future reports. If you want to reload the information directly from the properties files, pass the "--reload" parameter. The dataset is then used to generate a report. By default this report contains absolute numbers, writes a Latex file and analyzes all numeric attributes found in the dataset. You can however choose only a subset of attributes and filter by configurations or suites, too. A detailed description of the available parameters can be obtained by invoking `downward-reports.py -h`. === Show significant changes only === You can also compare '''two''' configs/revisions and only show the rows that have changed significantly. To do so select a relative report (`-r rel`) and specify a percent number (`--change number`). The report will then only contain those rows for which the two values in the row have changed by more than `number` percent. Here is an example: {{{#!highlight python ./downward-reports.py expname-eval/ -a expanded -r rel --change 5 }}} This will only show the rows where the number of expansions has changed by more than five percent. === Making a problem suite from results === The `downward-reports.py` also gives you the possibility to create a new problem suite based on the results of an experiment. To select a subset of problems you can specify filters for the set of the runs. E.g. to get a list of problems that had more than 1000 states expanded in the `ou` config, you could issue the following command: {{{#!highlight bash ./downward-reports.py expname-eval --filter config:eq:ou expanded:gt:1000 }}} (Remember to pass the correct name for config, it might not be just its nickname) As you can see the format of a filter is '''<attribute_name>:<operator from the [[http://docs.python.org/library/operator.html|operator]] module>:<value>'''. If the expression '''operator(run[attribute], value)''' evaluates to True, the run's planning problem is '''not''' removed from the result list. == Comparing different revisions == If you want to compare different revisions of Fast Downward, you can write your own script that imports python module `downward_experiments.py`. This module provides an easy way to select and compare specific revisions of the three subsystems (translate, preprocess and search). It does so by using the `experiments.py` module. The usage is pretty simple. As an example we will look at the code that has been used to get some information about [[http://issues.fast-downward.org/issue69|issue69]] from the issue tracker (The code resides in issue69.py): {{{#!highlight python from downward_experiments import * combinations = [ (TranslatorSvnCheckout(), PreprocessorSvnCheckout(), PlannerSvnCheckout(rev=3612)), (TranslatorSvnCheckout(), PreprocessorSvnCheckout(), PlannerSvnCheckout(rev=3613)), (TranslatorSvnCheckout(), PreprocessorSvnCheckout(), PlannerSvnCheckout(rev='HEAD')), ] build_experiment(combinations) }}} This code builds an experiment that compares three revisions of the search component; rev 3612, rev 3613 and the latest (HEAD) revision. As you can see, the translation and preprocessing components have been assigned no explicit revision. This can be done since all different Checkouts default to the HEAD or tip revision. The different Checkout classes also have another keyword parameter called `repo` that can be used when you don't want to checkout a subsystem from trunk. One combination of three checkouts results in one run of the fast-downward system (translate -> preprocess -> search) for each problem and configuration. Obviously you should checkout different revisions of the subsystems you want to compare and let the other subsystems have the same revisions in all runs. As another example, if you want to compare your modified translator in your own branch with the one from trunk, you could do: {{{#!highlight python combinations = [ (TranslatorSvnCheckout(repo='svn+ssh://downward/branches/my-own-translator/downward/translate', rev=1234), PreprocessorSvnCheckout(), PlannerSvnCheckout()), (TranslatorSvnCheckout(), PreprocessorSvnCheckout(), PlannerSvnCheckout()), ] }}} When running your script, you'll be prompted to specify the suites and configurations. You have the same options here as for the `downward_experiments.py` script. === Example: Issue 7 === You don't have to supply new Checkout instances for each combination. See the comparison experiment for [[http://issues.fast-downward.org/issue7|issue7]] as an example. This code compares three different revisions of the translator in an external branch with the checked-in translator from trunk. The revisions for the preprocessor and the search component remain the same for all combinations. {{{#!highlight python from downward_experiments import * branch = 'svn+ssh://downward-svn/branches/translate-andrew/downward/translate' preprocessor = PreprocessorSvnCheckout() planner = PlannerSvnCheckout(rev=3842) combinations = [ (TranslatorSvnCheckout(repo=branch, rev=3827), preprocessor, planner), (TranslatorSvnCheckout(repo=branch, rev=3829), preprocessor, planner), (TranslatorSvnCheckout(repo=branch, rev=3840), preprocessor, planner), (TranslatorSvnCheckout(rev=4283), preprocessor, planner), ] build_experiment(combinations) }}} === Example: Checking impacts of your changes === If you want to check the impacts of your changes in your working copy with the latest checked-in revision you could pass the following combinations to the function `build_experiment()` in `downward_experiments.py`. {{{#!highlight python from downward_experiments import * combinations = [ (TranslatorHgCheckout(rev='tip'), PreprocessorHgCheckout(rev='tip'), PlannerHgCheckout(rev='tip')), (TranslatorHgCheckout(rev='WORK'), PreprocessorHgCheckout(rev='WORK'), PlannerHgCheckout(rev='WORK')), ] build_experiment(combinations) }}} This command creates an experiment that compares the subsystems in the working copy with the versions of the subsystems in the tip revision. You can also see that you can set `rev` to `'WORK'` to use the working copy. |