irace/0000755000176200001440000000000014325252713011341 5ustar liggesusersirace/NAMESPACE0000644000176200001440000000273714325242447012574 0ustar liggesusers# Generated by roxygen2: do not edit by hand export(CommandArgsParser) export(ablation) export(ablation_cmdline) export(buildCommandLine) export(checkIraceScenario) export(checkParameters) export(checkScenario) export(cmdline_usage) export(configurations.print) export(configurations.print.command) export(defaultScenario) export(getConfigurationById) export(getConfigurationByIteration) export(getFinalElites) export(irace) export(irace.cmdline) export(irace.license) export(irace.main) export(irace.version) export(path_rel2abs) export(plotAblation) export(printParameters) export(printScenario) export(psRace) export(readConfigurationsFile) export(readParameters) export(readScenario) export(read_logfile) export(read_pcs_file) export(removeConfigurationsMetaData) export(scenario.update.paths) export(scenario_update_paths) export(target.evaluator.default) export(target.runner.default) export(testConfigurations) export(testing_fromfile) export(testing_fromlog) import(compiler) import(stats) import(utils) importFrom(R6,R6Class) importFrom(grDevices,dev.new) importFrom(grDevices,dev.off) importFrom(grDevices,pdf) importFrom(graphics,abline) importFrom(graphics,axis) importFrom(graphics,barplot) importFrom(graphics,boxplot) importFrom(graphics,bxp) importFrom(graphics,grid) importFrom(graphics,lines) importFrom(graphics,matplot) importFrom(graphics,mtext) importFrom(graphics,par) importFrom(graphics,plot) importFrom(graphics,points) importFrom(graphics,strwidth) importFrom(graphics,text) irace/.Rinstignore0000644000176200001440000000010714157104332013637 0ustar liggesusersvignettes/section/irace-options.tex ^GenericWrapper4AC \.github _snaps irace/README.md0000644000176200001440000003350214325211523012615 0ustar liggesusers**irace**: Iterated Racing for Automatic Algorithm Configuration ================================================================ [![CRAN Status](https://www.r-pkg.org/badges/version-last-release/irace)](https://cran.r-project.org/package=irace) [![CRAN Downloads](https://cranlogs.r-pkg.org/badges/grand-total/irace)](https://CRAN.R-project.org/package=irace) [![R build status](https://github.com/MLopez-Ibanez/irace/workflows/R-CMD-check/badge.svg)](https://github.com/MLopez-Ibanez/irace/actions) [![Codecov test coverage](https://codecov.io/gh/MLopez-Ibanez/irace/branch/master/graph/badge.svg)](https://app.codecov.io/gh/MLopez-Ibanez/irace?branch=master) [ [**Homepage**](https://mlopez-ibanez.github.io/irace/) ] [ [**User Guide (PDF)**](https://cran.r-project.org/package=irace/vignettes/irace-package.pdf) ] **Maintainers:** [Manuel López-Ibáñez](https://lopez-ibanez.eu/), Leslie Pérez Cáceres **Creators:** [Manuel López-Ibáñez](https://lopez-ibanez.eu/), Jérémie Dubois-Lacoste **Contributors:** Jérémie Dubois-Lacoste, Thomas Stützle, Mauro Birattari, Eric Yuan and Prasanna Balaprakash. **Contact:** --------------------------------------- Introduction ============ The **irace** package implements the Iterated Race method, which is a generalization of the Iterated F-race method for the automatic configuration of optimization algorithms, that is, the tuning of their parameters by finding the most appropriate settings given a set of instances of an optimization problem. It builds upon the race package by Birattari and it is implemented in R. **Keywords:** automatic configuration, offline tuning, parameter tuning, racing, F-race. **Relevant literature:** 1. M. López-Ibáñez, J. Dubois-Lacoste, L. Pérez Cáceres, T. Stützle, and M. Birattari. [The irace package: Iterated Racing for Automatic Algorithm Configuration](http://dx.doi.org/10.1016/j.orp.2016.09.002). *Operations Research Perspectives*, 3:43–58, 2016.
[ [bibtex](https://lopez-ibanez.eu/LopezIbanez_bib.html#LopDubPerStuBir2016irace) | doi: [10.1016/j.orp.2016.09.002](http://dx.doi.org/10.1016/j.orp.2016.09.002) ] 2. Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Thomas Stützle, and Mauro Birattari. [The irace package, Iterated Race for Automatic Algorithm Configuration](https://iridia.ulb.ac.be/IridiaTrSeries/link/IridiaTr2011-004.pdf). Technical Report TR/IRIDIA/2011-004, IRIDIA, Université libre de Bruxelles, Belgium, 2011.
[ [bibtex](https://iridia-ulb.github.io/references/index.html#LopDubStu2011irace) | [PDF](https://iridia.ulb.ac.be/IridiaTrSeries/link/IridiaTr2011-004.pdf) ] 3. Thomas Stützle and Manuel López-Ibáñez. [Tutorial: Automated algorithm configuration and design](https://doi.org/10.1145/3449726.3461404). GECCO '21: Proceedings of the Genetic and Evolutionary Computation Conference Companion, July 2021. [doi:10.1145/3449726.3461404](https://doi.org/10.1145/3449726.3461404) Requisites ---------- * R () is required for running irace, but you don't need to know the R language to use it. User guide ---------- A complete [user guide](https://cran.r-project.org/package=irace/vignettes/irace-package.pdf) comes with the package. You can access it online or, after installing the irace package, invoking from R the following command: ```R R> vignette("irace-package") ``` The following is a quick-start guide. The user guide gives more detailed instructions. Quick Start =========== 1. Install R (with your favourite package manager, and see more details below). 2. Install irace. This command works on CMD and Powershell with R added to PATH (see detailed instructions below). ```bash $ Rscript -e "install.packages('irace', repos='https://cloud.r-project.org')" ``` 3. Add irace to path. For windows user, this step is unfortunately more involved, so please see more detailed instructions below. ```bash $ export PATH="$(Rscript -e "cat(paste0(system.file(package='irace', 'bin', mustWork=TRUE), ':'))" 2> /dev/null)${PATH}" ``` Consider adding this line to your `~/.bashrc`, `~/.zshrc`, or `~/.profile` for it to persist between sessions. 4. You can open the user guide with the following command. This command works on CMD and Powershell with R added to PATH (see detailed instructions below). ```bash $ Rscript -e "vignette('irace-package')" ``` Installing R ============ The official instructions are available at . We give below a quick R installation guide that will work in most cases. GNU/Linux --------- You should install R from your package manager. On a Debian/Ubuntu system it will be something like: $ sudo apt-get install r-base Once R is installed, you can launch R from the Terminal and from the R prompt install the irace package. See instructions below. OS X ---- You can install R directly from a CRAN mirror (). Alternatively, if you use homebrew, you can just do ``` $ brew install --cask r ``` (Using `brew install r` is not recommended because that will build R from source and you will not be able to use any CRAN binary, possibly resulting in annoying build failiures). Once R is installed, you can launch R from the Terminal (or from your Applications), and from the R prompt install the irace package. See instructions below. Windows ------- You can install R from a CRAN mirror (). Once R is installed, you can launch the R console and install the irace package from it. See instructions below. In addition to using the R console, it might be very useful to add R to PATH so you can run most of the GUN/Linux shell commands without modification in CMD or Powershell. Usually, R is installed in `C:\Program Files\R\R-4.1.3` (the version number depends on your installation). You should add the following line [to PATH](https://www.architectryan.com/2018/03/17/add-to-the-path-on-windows-10/) (if you want to use the 64-bit version) ``` C:\Program Files\R\R-4.1.3\bin\x64 ``` Or, if you are on a 32-bit version ``` C:\Program Files\R\R-4.1.3\bin\i386 ``` Installing the irace package ============================ There are two methods for installing the [irace](https://mlopez-ibanez.github.io/irace/) R package on your computer: 1. Install within R (automatic download): ```R $ R R> install.packages("irace") ``` select a mirror close to you, and test the installation with ```R R> library(irace) R> CTRL+d ``` 2. Manually [download the package from CRAN](https://cran.r-project.org/package=irace) and invoke at the command-line: ```bash $ R CMD INSTALL ``` where `` is one of the three versions available: `.tar.gz` (Unix/BSD/GNU/Linux), `.tgz` (MacOS X), or `.zip` (Windows). If the package fails to install because of insufficient permissions, you need to force a *local installation* by doing: ```bash $ mkdir ~/R $ R CMD INSTALL --library=~/R irace.tar.gz $ export R_LIBS=~/R:${R_LIBS} ``` Once installed, test that it is working by doing: ```R $ R R> library(irace) R> cat(system.file(package="irace", "bin", mustWork=TRUE), "\n") /home/user/R/irace/bin ``` The last command tells you the installation directory of `irace`. GNU/Linux and OS X ------------------ Save the installation directory of `irace` to a variable, and add it to your `.bash_profile`, `.bashrc` or `.profile`: ```bash export IRACE_HOME=/home/user/R/irace/bin/ # Path given by system.file(package="irace", "bin", mustWork=TRUE) export PATH=${IRACE_HOME}:$PATH # export R_LIBS=~/R:${R_LIBS} # Only if local installation was forced ``` After adding this and opening a new terminal, you should be able to invoke `irace` as follows: ```bash $ irace --help ``` Windows ------- You can find out where the irace binary is installed by running the following in Powershell or CMD: ```Powershell C:\> Rscript -e "cat(gsub('/', '\\\\', system.file(package='irace', 'bin', 'x64', mustWork=TRUE)))" ``` It will output a path, such as `C:\Program Files\R\R-4.1.3\library\irace\bin\x64` (replace `x64` with `i386` if you are on a 32-bit system), which can you [add to PATH](https://www.architectryan.com/2018/03/17/add-to-the-path-on-windows-10/). Then running the following should work: ```Powershell C:\> irace --help ``` You can also launch irace by opening the R console and executing: ```R R> library(irace) R> irace.cmdline("--help") ``` GitHub (Development version) ---------------------------- If you wish to try the development version, you can install it by executing the following commands within the R console: ```R R> install.packages("devtools") R> devtools::install_github("MLopez-Ibanez/irace") ``` Python ------ You can use the irace R package from Python using `rpy2`. An example on how to do this is the implementation of [iracepy](https://github.com/auto-optimization/iracepy). Usage ===== 1. Create a directory for storing the tuning scenario setup ```bash $ mkdir ~/tuning $ cd ~/tuning ``` 2. Initialize your tuning directory with template config files ```bash $ $IRACE_HOME/bin/irace --init ``` 3. Modify the generated files following the instructions found within each file. In particular, * The scripts `target-runner` and `target-evaluator` (if you need it at all) should be executable. The output of `target-runner` (or `target-evaluator` if you use a separate evaluation step) is minimized by default. If you wish to maximize it, just multiply the value by `-1` within the script. * In `scenario.txt`, uncomment and assign only the parameters for which you need a value different than the default one. There are examples in `$IRACE_HOME/examples/`. 4. Put the instances in `~/tuning/Instances/`. In addition, you can create a file that specifies which instances from that directory should be run and which instance-specific parameters to use. See `scenario.txt` and `instances-list.txt` for examples. The command irace will not attempt to create the execution directory (`execDir`), so it must exist before calling irace. The default `execDir` is the current directory. 5. Calling the command: ```bash $ cd ~/tuning/ && $IRACE_HOME/bin/irace ``` performs one run of Iterated Race. See the output of `irace --help` for additional irace parameters. Command-line parameters override the scenario setup specified in the `scenario.txt` file. Many tuning runs in parallel ---------------------------- For executing several repetitions of irace in parallel, call the program ```bash $ cd ~/tuning/ && $IRACE_HOME/bin/parallel-irace N ``` where N is the number of repetitions. By default, the execution directory of each run of irace will be set to `./execdir-dd`, where `dd` is a number padded with zeroes. **Be careful**, `parallel-irace` will create these directories from scratch, deleting them first if they already exist. Check the help of `parallel-irace` by running it without parameters. Parallelize one tuning ---------------------- A single run of irace can be done much faster by executing the calls to `targetRunner` (the runs of the algorithm being tuned) in parallel. See the [user guide](https://cran.r-project.org/package=irace/vignettes/irace-package.pdf) for the details. License ======= **irace** is Copyright (C) 2010-2020 Manuel López-Ibáñez, Jérémie Dubois-Lacoste and Leslie Pérez-Cáceres. This program is free software (software libre); you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the [GNU General Public License](https://cran.r-project.org/web/licenses/GPL-3) for more details. **IMPORTANT NOTE:** Please be aware that the fact that this program is released as Free Software does not excuse you from scientific propriety, which obligates you to give appropriate credit! If you write a scientific paper describing research that made substantive use of this program, it is your obligation as a scientist to (a) mention the fashion in which this software was used in the Methods section; (b) mention the algorithm in the References section. The appropriate citation is: * Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, Thomas Stützle, and Mauro Birattari. [**The irace package: Iterated Racing for Automatic Algorithm Configuration.**](http://dx.doi.org/10.1016/j.orp.2016.09.002) _Operations Research Perspectives_, 2016. doi: 10.1016/j.orp.2016.09.002 The **irace** package uses code under the GPL from the [race package](https://CRAN.R-project.org/package=race) is Copyright (C) 2003 Mauro Birattari. Building an irace standalone container ====================================== Thanks to [Singularity](https://sylabs.io/singularity/), you can build a standalone container of `irace` using the file `irace.sindef` which is available in the directory `inst/` in the source tarball and github repository or, after installing the irace R package, in the installation directory given by the R expression `system.file(package="irace")`. After installing SingularityCE, the container may be build using: sudo singularity build irace.sindef irace.sif and run with: singularity run irace.sif Frequently Asked Questions ========================== The [user guide](https://cran.r-project.org/package=irace/vignettes/irace-package.pdf) contains a list of frequently asked questions. irace/man/0000755000176200001440000000000014321267520012112 5ustar liggesusersirace/man/printParameters.Rd0000644000176200001440000000224014260577373015573 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/readParameters.R \name{printParameters} \alias{printParameters} \title{Print parameter space in the textual format accepted by irace.} \usage{ printParameters(params, digits = 15L) } \arguments{ \item{params}{(\code{list()}) Parameter object stored in \code{irace.Rdata} or read with \code{irace::readParameters()}.} \item{digits}{(\code{integer()}) The desired number of digits after the decimal point for real-valued parameters. Default is 15, but it should be the value in \code{scenario$digits}.} } \description{ FIXME: Dependent parameter bounds are not supported yet. } \examples{ parameters.table <- ' # name switch type values [conditions (using R syntax)] algorithm "--" c (as,mmas,eas,ras,acs) localsearch "--localsearch " c (0, 1, 2, 3) ants "--ants " i,log (5, 100) q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" nnls "--nnls " i (5, 50) | localsearch \%in\% c(1,2,3) ' parameters <- readParameters(text=parameters.table) printParameters(parameters) } irace/man/removeConfigurationsMetaData.Rd0000644000176200001440000000205114240241322020200 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/utils.R \name{removeConfigurationsMetaData} \alias{removeConfigurationsMetaData} \title{removeConfigurationsMetaData} \usage{ removeConfigurationsMetaData(configurations) } \arguments{ \item{configurations}{(\code{data.frame}) \cr Parameter configurations of the target algorithm (one per row).} } \value{ The same matrix without the "metadata". } \description{ Remove the columns with "metadata" of a matrix containing some configuration configurations. These "metadata" are used internaly by \pkg{irace}. This function can be used e.g. before printing the configurations, to output only the values for the parameters of the configuration without data possibly useless to the user. } \seealso{ \code{\link[=configurations.print.command]{configurations.print.command()}} to print the configurations as command lines. \code{\link[=configurations.print]{configurations.print()}} to print the configurations as a data frame. } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } irace/man/configurations.print.Rd0000644000176200001440000000151014240241322016553 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/utils.R \name{configurations.print} \alias{configurations.print} \title{Print configurations as a data frame} \usage{ configurations.print(configurations, metadata = FALSE) } \arguments{ \item{configurations}{(\code{data.frame}) \cr Parameter configurations of the target algorithm (one per row).} \item{metadata}{A Boolean specifying whether to print the metadata or not. The metadata are data for the configurations (additionally to the value of each parameter) used by \pkg{irace}.} } \value{ None. } \description{ Print configurations as a data frame } \seealso{ \code{\link[=configurations.print.command]{configurations.print.command()}} to print the configurations as command-line strings. } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } irace/man/configurations.print.command.Rd0000644000176200001440000000160714240241322020177 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/utils.R \name{configurations.print.command} \alias{configurations.print.command} \title{Print configurations as command-line strings.} \usage{ configurations.print.command(configurations, parameters) } \arguments{ \item{configurations}{(\code{data.frame}) \cr Parameter configurations of the target algorithm (one per row).} \item{parameters}{(\code{list()}) \cr Data structure containing the parameter space definition. The data structure has to similar to the one returned by the function \code{\link{readParameters}}.} } \value{ None. } \description{ Prints configurations after converting them into a representation for the command-line. } \seealso{ \code{\link[=configurations.print]{configurations.print()}} to print the configurations as a data frame. } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } irace/man/checkScenario.Rd0000644000176200001440000000315214240244375015146 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/readConfiguration.R \name{checkScenario} \alias{checkScenario} \title{Check and correct the given scenario} \usage{ checkScenario(scenario = defaultScenario()) } \arguments{ \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} settings. The data structure has to be the one returned by the function \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} } \value{ The scenario received as a parameter, possibly corrected. Unset scenario settings are set to their default values. } \description{ Checks for errors a (possibly incomplete) scenario setup of \pkg{irace} and transforms it into a valid scenario. } \details{ This function checks that the directories and the file names provided and required by the \pkg{irace} exist. It also checks that the settings are of the proper type, e.g. that settings expected to be integers are really integers. Finally, it also checks that there is no inconsistency between settings. If an error is found that prevents \pkg{irace} from running properly, it will stop with an error. } \seealso{ \describe{ \item{\code{\link[=readScenario]{readScenario()}}}{for reading a configuration scenario from a file.} \item{\code{\link[=printScenario]{printScenario()}}}{prints the given scenario.} \item{\code{\link[=defaultScenario]{defaultScenario()}}}{returns the default scenario settings of \pkg{irace}.} \item{\code{\link[=checkScenario]{checkScenario()}}}{to check that the scenario is valid.} } } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } irace/man/readConfigurationsFile.Rd0000644000176200001440000000347514242735410017040 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/readConfiguration.R \name{readConfigurationsFile} \alias{readConfigurationsFile} \title{Read parameter configurations from a file} \usage{ readConfigurationsFile(filename, parameters, debugLevel = 0, text) } \arguments{ \item{filename}{(\code{character(1)}) \cr Filename from which the configurations should be read. The contents should be readable by \code{read.table( , header=TRUE)}.} \item{parameters}{(\code{list()}) \cr Data structure containing the parameter space definition. The data structure has to similar to the one returned by the function \code{\link{readParameters}}.} \item{debugLevel}{(\code{integer(1)}) \cr Larger values produce more verbose output.} \item{text}{(\code{character(1)}) \cr If \code{file} is not supplied and this is, then parameters are read from the value of \code{text} via a text connection.} } \value{ A data frame containing the obtained configurations. Each row of the data frame is a candidate configuration, the columns correspond to the parameter names in \code{parameters}. } \description{ Reads a set of target-algorithm configurations from a file and puts them in \pkg{irace} format. The configurations are checked to match the parameters description provided. } \details{ Example of an input file:\preformatted{# This is a comment line ... 0.5 "value_1" ... 1.0 "value_2" ... 1.2 "value_3" ... ... ... } The order of the columns does not necessarily have to be the same as in the file containing the definition of the parameters. } \seealso{ \code{\link[=readParameters]{readParameters()}} to obtain a valid parameter structure from a parameters file. } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } irace/man/CommandArgsParser.Rd0000644000176200001440000000456114233555030015755 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/argparser.R \name{CommandArgsParser} \alias{CommandArgsParser} \alias{cmdline_usage} \title{R6 Class for parsing command-line arguments} \usage{ cmdline_usage(cmdline_args) } \arguments{ \item{cmdline_args}{Definition of the command-line arguments.} } \description{ R6 Class for parsing command-line arguments R6 Class for parsing command-line arguments \code{cmdline_usage()} prints the output of \code{--help} } \section{Methods}{ \subsection{Public methods}{ \itemize{ \item \href{#method-new}{\code{CommandArgsParser$new()}} \item \href{#method-readCmdLineParameter}{\code{CommandArgsParser$readCmdLineParameter()}} \item \href{#method-readArg}{\code{CommandArgsParser$readArg()}} \item \href{#method-readAll}{\code{CommandArgsParser$readAll()}} \item \href{#method-cmdline_usage}{\code{CommandArgsParser$cmdline_usage()}} } } \if{html}{\out{
}} \if{html}{\out{}} \if{latex}{\out{\hypertarget{method-new}{}}} \subsection{Method \code{new()}}{ \subsection{Usage}{ \if{html}{\out{
}}\preformatted{CommandArgsParser$new(argv, argsdef)}\if{html}{\out{
}} } } \if{html}{\out{
}} \if{html}{\out{}} \if{latex}{\out{\hypertarget{method-readCmdLineParameter}{}}} \subsection{Method \code{readCmdLineParameter()}}{ \subsection{Usage}{ \if{html}{\out{
}}\preformatted{CommandArgsParser$readCmdLineParameter(paramName, default = NULL)}\if{html}{\out{
}} } } \if{html}{\out{
}} \if{html}{\out{}} \if{latex}{\out{\hypertarget{method-readArg}{}}} \subsection{Method \code{readArg()}}{ \subsection{Usage}{ \if{html}{\out{
}}\preformatted{CommandArgsParser$readArg(short = "", long = "")}\if{html}{\out{
}} } } \if{html}{\out{
}} \if{html}{\out{}} \if{latex}{\out{\hypertarget{method-readAll}{}}} \subsection{Method \code{readAll()}}{ \subsection{Usage}{ \if{html}{\out{
}}\preformatted{CommandArgsParser$readAll()}\if{html}{\out{
}} } } \if{html}{\out{
}} \if{html}{\out{}} \if{latex}{\out{\hypertarget{method-cmdline_usage}{}}} \subsection{Method \code{cmdline_usage()}}{ \subsection{Usage}{ \if{html}{\out{
}}\preformatted{CommandArgsParser$cmdline_usage()}\if{html}{\out{
}} } } } irace/man/defaultScenario.Rd0000644000176200001440000003060014321267520015510 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/readConfiguration.R \name{defaultScenario} \alias{defaultScenario} \title{Default scenario settings} \usage{ defaultScenario(scenario = list(), params_def = .irace.params.def) } \arguments{ \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} settings. The data structure has to be the one returned by the function \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} \item{params_def}{(\code{data.frame()}) \cr Definition of the options accepted by the scenario. This should only be modified by packages that wish to extend \pkg{irace}.} } \value{ A list indexed by the \pkg{irace} parameter names, containing the default values for each parameter, except for those already present in the scenario passed as argument. The scenario list contains the following elements: \itemize{ \item General options: \describe{ \item{\code{scenarioFile}}{Path of the file that describes the configuration scenario setup and other irace settings. (Default: \code{"./scenario.txt"})} \item{\code{execDir}}{Directory where the programs will be run. (Default: \code{"./"})} \item{\code{logFile}}{File to save tuning results as an R dataset, either absolute path or relative to execDir. (Default: \code{"./irace.Rdata"})} \item{\code{quiet}}{Reduce the output generated by irace to a minimum. (Default: \code{0})} \item{\code{debugLevel}}{Debug level of the output of \code{irace}. Set this to 0 to silence all debug messages. Higher values provide more verbose debug messages. (Default: \code{0})} \item{\code{seed}}{Seed of the random number generator (by default, generate a random seed). (Default: \code{NA})} \item{\code{repairConfiguration}}{User-defined R function that takes a configuration generated by irace and repairs it. (Default: \code{""})} \item{\code{postselection}}{Percentage of the configuration budget used to perform a postselection race of the best configurations of each iteration after the execution of irace. (Default: \code{0})} \item{\code{aclib}}{Enable/disable AClib mode. This option enables compatibility with GenericWrapper4AC as targetRunner script. (Default: \code{0})} } \item Elitist \code{irace}: \describe{ \item{\code{elitist}}{Enable/disable elitist irace. (Default: \code{1})} \item{\code{elitistNewInstances}}{Number of instances added to the execution list before previous instances in elitist irace. (Default: \code{1})} \item{\code{elitistLimit}}{In elitist irace, maximum number per race of elimination tests that do not eliminate a configuration. Use 0 for no limit. (Default: \code{2})} } \item Internal \code{irace} options: \describe{ \item{\code{sampleInstances}}{Randomly sample the training instances or use them in the order given. (Default: \code{1})} \item{\code{softRestart}}{Enable/disable the soft restart strategy that avoids premature convergence of the probabilistic model. (Default: \code{1})} \item{\code{softRestartThreshold}}{Soft restart threshold value for numerical parameters. If \code{NA}, \code{NULL} or \code{""}, it is computed as \code{10^-digits}. (Default: \code{""})} \item{\code{nbIterations}}{Maximum number of iterations. (Default: \code{0})} \item{\code{nbExperimentsPerIteration}}{Number of runs of the target algorithm per iteration. (Default: \code{0})} \item{\code{minNbSurvival}}{Minimum number of configurations needed to continue the execution of each race (iteration). (Default: \code{0})} \item{\code{nbConfigurations}}{Number of configurations to be sampled and evaluated at each iteration. (Default: \code{0})} \item{\code{mu}}{Parameter used to define the number of configurations sampled and evaluated at each iteration. (Default: \code{5})} } \item Target algorithm parameters: \describe{ \item{\code{parameterFile}}{File that contains the description of the parameters of the target algorithm. (Default: \code{"./parameters.txt"})} \item{\code{forbiddenExps}}{Vector of R logical expressions that cannot evaluate to \code{TRUE} for any evaluated configuration. (Default: \code{""})} \item{\code{forbiddenFile}}{File that contains a list of logical expressions that cannot be \code{TRUE} for any evaluated configuration. If empty or \code{NULL}, do not use forbidden expressions. (Default: \code{""})} \item{\code{digits}}{Maximum number of decimal places that are significant for numerical (real) parameters. (Default: \code{4})} } \item Target algorithm execution: \describe{ \item{\code{targetRunner}}{Executable called for each configuration that executes the target algorithm to be tuned. See the templates and examples provided. (Default: \code{"./target-runner"})} \item{\code{targetRunnerLauncher}}{Executable that will be used to launch the target runner, when \code{targetRunner} cannot be executed directly (.e.g, a Python script in Windows). (Default: \code{""})} \item{\code{targetRunnerLauncherArgs}}{Command-line arguments provided to \code{targetRunnerLauncher}. The substrings \code{\{targetRunner\}} and \code{\{targetRunnerArgs\}} will be replaced by the value of the option \code{targetRunner} and by the arguments usually passed when calling \code{targetRunner}, respectively. Example: \code{"-m {targetRunner} --args {targetRunnerArgs}"}. (Default: \code{"{targetRunner} {targetRunnerArgs}"})} \item{\code{targetRunnerRetries}}{Number of times to retry a call to \code{targetRunner} if the call failed. (Default: \code{0})} \item{\code{targetRunnerData}}{Optional data passed to \code{targetRunner}. This is ignored by the default \code{targetRunner} function, but it may be used by custom \code{targetRunner} functions to pass persistent data around. (Default: \code{""})} \item{\code{targetRunnerParallel}}{Optional R function to provide custom parallelization of \code{targetRunner}. (Default: \code{""})} \item{\code{targetEvaluator}}{Optional script or R function that provides a numeric value for each configuration. See templates/target-evaluator.tmpl (Default: \code{""})} \item{\code{deterministic}}{If the target algorithm is deterministic, configurations will be evaluated only once per instance. (Default: \code{0})} \item{\code{parallel}}{Number of calls to \code{targetRunner} to execute in parallel. Values \code{0} or \code{1} mean no parallelization. (Default: \code{0})} \item{\code{loadBalancing}}{Enable/disable load-balancing when executing experiments in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster. (Default: \code{1})} \item{\code{mpi}}{Enable/disable MPI. Use \code{Rmpi} to execute \code{targetRunner} in parallel (parameter \code{parallel} is the number of slaves). (Default: \code{0})} \item{\code{batchmode}}{Specify how irace waits for jobs to finish when \code{targetRunner} submits jobs to a batch cluster: sge, pbs, torque, slurm or htcondor. \code{targetRunner} must submit jobs to the cluster using, for example, \code{qsub}. (Default: \code{0})} } \item Initial configurations: \describe{ \item{\code{initConfigurations}}{Data frame describing initial configurations (usually read from a file using \code{readConfigurations}). (Default: \code{""})} \item{\code{configurationsFile}}{File that contains a table of initial configurations. If empty or \code{NULL}, all initial configurations are randomly generated. (Default: \code{""})} } \item Training instances: \describe{ \item{\code{instances}}{Character vector of the instances to be used in the \code{targetRunner}. (Default: \code{""})} \item{\code{trainInstancesDir}}{Directory where training instances are located; either absolute path or relative to current directory. If no \code{trainInstancesFiles} is provided, all the files in \code{trainInstancesDir} will be listed as instances. (Default: \code{"./Instances"})} \item{\code{trainInstancesFile}}{File that contains a list of training instances and optionally additional parameters for them. If \code{trainInstancesDir} is provided, \code{irace} will search for the files in this folder. (Default: \code{""})} } \item Tuning budget: \describe{ \item{\code{maxExperiments}}{Maximum number of runs (invocations of \code{targetRunner}) that will be performed. It determines the maximum budget of experiments for the tuning. (Default: \code{0})} \item{\code{maxTime}}{Maximum total execution time in seconds for the executions of \code{targetRunner}. \code{targetRunner} must return two values: cost and time. (Default: \code{0})} \item{\code{budgetEstimation}}{Fraction (smaller than 1) of the budget used to estimate the mean computation time of a configuration. Only used when \code{maxTime} > 0 (Default: \code{0.02})} \item{\code{minMeasurableTime}}{Minimum time unit that is still (significantly) measureable. (Default: \code{0.01})} } \item Statistical test: \describe{ \item{\code{testType}}{Statistical test used for elimination. The default value selects \code{t-test} if \code{capping} is enabled or \code{F-test}, otherwise. Valid values are: F-test (Friedman test), t-test (pairwise t-tests with no correction), t-test-bonferroni (t-test with Bonferroni's correction for multiple comparisons), t-test-holm (t-test with Holm's correction for multiple comparisons). (Default: \code{""})} \item{\code{firstTest}}{Number of instances evaluated before the first elimination test. It must be a multiple of \code{eachTest}. (Default: \code{5})} \item{\code{eachTest}}{Number of instances evaluated between elimination tests. (Default: \code{1})} \item{\code{confidence}}{Confidence level for the elimination test. (Default: \code{0.95})} } \item Adaptive capping: \describe{ \item{\code{capping}}{Enable the use of adaptive capping, a technique designed for minimizing the computation time of configurations. This is only available when \code{elitist} is active. (Default: \code{0})} \item{\code{cappingType}}{Measure used to obtain the execution bound from the performance of the elite configurations.\itemize{\item median: Median performance of the elite configurations.\item mean: Mean performance of the elite configurations.\item best: Best performance of the elite configurations.\item worst: Worst performance of the elite configurations.} (Default: \code{"median"})} \item{\code{boundType}}{Method to calculate the mean performance of elite configurations.\itemize{\item candidate: Mean execution times across the executed instances and the current one.\item instance: Execution time of the current instance.} (Default: \code{"candidate"})} \item{\code{boundMax}}{Maximum execution bound for \code{targetRunner}. It must be specified when capping is enabled. (Default: \code{0})} \item{\code{boundDigits}}{Precision used for calculating the execution time. It must be specified when capping is enabled. (Default: \code{0})} \item{\code{boundPar}}{Penalization constant for timed out executions (executions that reach \code{boundMax} execution time). (Default: \code{1})} \item{\code{boundAsTimeout}}{Replace the configuration cost of bounded executions with \code{boundMax}. (Default: \code{1})} } \item Recovery: \describe{ \item{\code{recoveryFile}}{Previously saved log file to recover the execution of \code{irace}, either absolute path or relative to the current directory. If empty or \code{NULL}, recovery is not performed. (Default: \code{""})} } \item Testing: \describe{ \item{\code{testInstancesDir}}{Directory where testing instances are located, either absolute or relative to current directory. (Default: \code{""})} \item{\code{testInstancesFile}}{File containing a list of test instances and optionally additional parameters for them. (Default: \code{""})} \item{\code{testInstances}}{Character vector of the instances to be used in the \code{targetRunner} when executing the testing. (Default: \code{""})} \item{\code{testNbElites}}{Number of elite configurations returned by irace that will be tested if test instances are provided. (Default: \code{1})} \item{\code{testIterationElites}}{Enable/disable testing the elite configurations found at each iteration. (Default: \code{0})} } } } \description{ Return scenario object with default values. } \seealso{ \describe{ \item{\code{\link[=readScenario]{readScenario()}}}{for reading a configuration scenario from a file.} \item{\code{\link[=printScenario]{printScenario()}}}{prints the given scenario.} \item{\code{\link[=defaultScenario]{defaultScenario()}}}{returns the default scenario settings of \pkg{irace}.} \item{\code{\link[=checkScenario]{checkScenario()}}}{to check that the scenario is valid.} } } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } irace/man/checkIraceScenario.Rd0000644000176200001440000000314214252663436016117 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/main.R \name{checkIraceScenario} \alias{checkIraceScenario} \title{Test that the given irace scenario can be run.} \usage{ checkIraceScenario(scenario, parameters) } \arguments{ \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} settings. The data structure has to be the one returned by the function \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} \item{parameters}{(\code{list()}) \cr Data structure containing the parameter space definition. The data structure has to similar to the one returned by the function \code{\link{readParameters}}.} } \value{ returns \code{TRUE} if successful and gives an error and returns \code{FALSE} otherwise. } \description{ Test that the given irace scenario can be run by checking the scenario settings provided and trying to run the target-algorithm. } \details{ If the \code{parameters} argument is missing, then the parameters will be read from the file \code{parameterFile} given by \code{scenario}. If \code{parameters} is provided, then \code{parameterFile} will not be read. This function will try to execute the target-algorithm. } \seealso{ \describe{ \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} \item{\code{\link{printScenario}}}{prints the given scenario.} \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} } } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } irace/man/readParameters.Rd0000644000176200001440000001011414315062156015336 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/readParameters.R \name{readParameters} \alias{readParameters} \title{Reads the parameters to be tuned by \pkg{irace} from a file or from a character string.} \usage{ readParameters(file, digits = 4, debugLevel = 0, text) } \arguments{ \item{file}{(\code{character(1)}) \cr Filename containing the definitions of the parameters to be tuned.} \item{digits}{The number of decimal places to be considered for the real parameters.} \item{debugLevel}{(\code{integer(1)}) \cr Larger values produce more verbose output.} \item{text}{(\code{character(1)}) \cr If \code{file} is not supplied and this is, then parameters are read from the value of \code{text} via a text connection.} } \value{ A list containing the definitions of the parameters read. The list is structured as follows: \describe{ \item{\code{names}}{Vector that contains the names of the parameters.} \item{\code{types}}{Vector that contains the type of each parameter 'i', 'c', 'r', 'o'. Numerical parameters can be sampled in a log-scale with 'i,log' and 'r,log' (no spaces).} \item{\code{switches}}{Vector that contains the switches to be used for the parameters on the command line.} \item{\code{domain}}{List of vectors, where each vector may contain two values (minimum, maximum) for real and integer parameters, or possibly more for categorical parameters.} \item{\code{conditions}}{List of R logical expressions, with variables corresponding to parameter names.} \item{\code{isFixed}}{Logical vector that specifies which parameter is fixed and, thus, it does not need to be tuned.} \item{\code{nbParameters}}{An integer, the total number of parameters.} \item{\code{nbFixed}}{An integer, the number of parameters with a fixed value.} \item{\code{nbVariable}}{Number of variable (to be tuned) parameters.} \item{\code{depends}}{List of character vectors, each vector specifies which parameters depend on this one.} \item{\code{isDependent}}{Logical vector that specifies which parameter has a dependent domain.} } } \description{ Reads the parameters to be tuned by \pkg{irace} from a file or from a character string. } \details{ Either \code{file} or \code{text} must be given. If \code{file} is given, the parameters are read from the file \code{file}. If \code{text} is given instead, the parameters are read directly from the \code{text} character string. In both cases, the parameters must be given (in \code{text} or in the file whose name is \code{file}) in the expected form. See the documentation for details. If none of these parameters is given, \pkg{irace} will stop with an error. A fixed parameter is a parameter that should not be sampled but instead should be always set to the only value of its domain. In this function we set isFixed to TRUE only if the parameter is a categorical and has only one possible value. If it is an integer and the minimum and maximum are equal, or it is a real and the minimum and maximum values satisfy \code{round(minimum, digits) == round(maximum, digits)}, then the parameter description is rejected as invalid to identify potential user errors. } \examples{ ## Read the parameters directly from text parameters.table <- ' # name switch type values [conditions (using R syntax)] algorithm "--" c (as,mmas,eas,ras,acs) localsearch "--localsearch " c (0, 1, 2, 3) alpha "--alpha " r (0.00, 5.00) beta "--beta " r (0.00, 10.00) rho "--rho " r (0.01, 1.00) ants "--ants " i,log (5, 100) q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" rasrank "--rasranks " i (1, "min(ants, 10)") | algorithm == "ras" elitistants "--elitistants " i (1, ants) | algorithm == "eas" nnls "--nnls " i (5, 50) | localsearch \%in\% c(1,2,3) dlb "--dlb " c (0, 1) | localsearch \%in\% c(1,2,3) ' parameters <- readParameters(text=parameters.table) str(parameters) } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } irace/man/path_rel2abs.Rd0000644000176200001440000000122414233311226014741 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/path_rel2abs.R \name{path_rel2abs} \alias{path_rel2abs} \title{Converts a relative path to an absolute path. It tries really hard to create canonical paths.} \usage{ path_rel2abs(path, cwd = getwd()) } \arguments{ \item{path}{(\code{character(1)}) Character string representing a relative path.} \item{cwd}{(\code{character(1)}) Current working directory.} } \value{ (\code{character(1)}) Character string representing the absolute path } \description{ Converts a relative path to an absolute path. It tries really hard to create canonical paths. } \examples{ path_rel2abs("..") } irace/man/buildCommandLine.Rd0000644000176200001440000000320214157104332015602 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/race-wrapper.R \name{buildCommandLine} \alias{buildCommandLine} \title{Generate a command-line representation of a configuration} \usage{ buildCommandLine(values, switches) } \arguments{ \item{values}{A vector containing the value of each parameter for the candidate configuration.} \item{switches}{A vector containing the switches of each paramter (in an order that corresponds to the values vector).} } \value{ A string concatenating each element of \code{switches} and \code{values} for all parameters with a space between each pair of parameters (but none between the switches and the corresponding values). } \description{ \code{buildCommandLine} receives two vectors, one containing the values of the parameters, the other containing the switches of the parameters. It builds a string with the switches and the values that can be used as a command line to call the program to be tuned, thus generating one candidate configuration. } \examples{ switches <- c("--switch1 ", "--switch2 ") values <- c("value_1", "value_2") buildCommandLine (values, switches) ## Build a command-line from the results produced by a previous run of irace. # First, load the data produced by irace. irace.logfile <- file.path(system.file(package="irace"), "exdata", "irace-acotsp.Rdata") load(irace.logfile) allConfigurations <- iraceResults$allConfigurations parameters <- iraceResults$parameters apply(allConfigurations[1:10, unlist(parameters$names)], 1, buildCommandLine, unlist(parameters$switches)) } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } irace/man/scenario_update_paths.Rd0000644000176200001440000000262214321266642016753 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/utils.R \name{scenario_update_paths} \alias{scenario_update_paths} \alias{scenario.update.paths} \title{Update filesystem paths of a scenario consistently.} \usage{ scenario_update_paths(scenario, from, to, fixed = TRUE) scenario.update.paths(scenario, from, to, fixed = TRUE) } \arguments{ \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} settings. The data structure has to be the one returned by the function \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} \item{from}{character string containing a regular expression (or character string for \code{fixed = TRUE}) to be matched.} \item{to}{the replacement string.character string. For \code{fixed = FALSE} this can include backreferences \code{"\\1"} to \code{"\\9"} to parenthesized subexpressions of \code{from}.} \item{fixed}{logical. If \code{TRUE}, \code{from} is a string to be matched as is.} } \value{ The updated scenario } \description{ This function should be used to change the filesystem paths stored in a scenario object. Useful when moving a scenario from one computer to another. } \examples{ \dontrun{ scenario <- readScenario(filename = "scenario.txt") scenario <- scenario_update_paths(scenario, from = "/home/manuel/", to = "/home/leslie") } } \seealso{ \code{\link[base:grep]{base::grep()}} } irace/man/irace.license.Rd0000644000176200001440000000054214157104332015104 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/main.R \docType{data} \name{irace.license} \alias{irace.license} \title{irace.license} \format{ An object of class \code{character} of length 1. } \usage{ irace.license } \description{ A character string containing the license information of \pkg{irace}. } \keyword{datasets} irace/man/getConfigurationByIteration.Rd0000644000176200001440000000226614233311226020063 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/parameterAnalysis.R \name{getConfigurationByIteration} \alias{getConfigurationByIteration} \title{Returns the configurations by the iteration in which they were executed.} \usage{ getConfigurationByIteration(iraceResults, iterations, drop.metadata = FALSE) } \arguments{ \item{iraceResults}{(\code{list()}|\code{character(1)}) \code{iraceResults} object created by \pkg{irace} and typically saved in the log file \code{irace.Rdata}. If a character string is given, then it is interpreted as the path to the log file from which the \code{iraceResults} object will be loaded.} \item{iterations}{The iteration number or a vector of iteration numbers from where the configurations should be obtained.} \item{drop.metadata}{(\code{FALSE}) Remove metadata, such the configuration ID and the ID of the parent, from the returned configurations. See \code{\link{removeConfigurationsMetaData}}.} } \value{ A data frame containing the elite configurations required. } \description{ Returns the configurations by the iteration in which they were executed. } \author{ Manuel López-Ibáñez and Leslie Pérez Cáceres } \concept{analysis} irace/man/readScenario.Rd0000644000176200001440000000316014250104555014777 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/readConfiguration.R \name{readScenario} \alias{readScenario} \title{Reads from a file the scenario settings to be used by \pkg{irace}.} \usage{ readScenario(filename = "", scenario = list(), params_def = .irace.params.def) } \arguments{ \item{filename}{(\code{character(1)}) \cr Filename from which the scenario will be read. If empty, the default \code{scenarioFile} is used. An example scenario file is provided in \verb{system.file(``package="irace",} \verb{"templates/scenario.txt.tmpl")}.} \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} settings. The data structure has to be the one returned by the function \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}. This is an initial scenario that is overwritten} \item{params_def}{(\code{data.frame()}) \cr Definition of the options accepted by the scenario. This should only be modified by packages that wish to extend \pkg{irace}.} } \value{ The scenario list read from the file. The scenario settings not present in the file are not present in the list, i.e., they are \code{NULL}. } \description{ Reads from a file the scenario settings to be used by \pkg{irace}. } \seealso{ \describe{ \item{\code{\link[=printScenario]{printScenario()}}}{prints the given scenario.} \item{\code{\link[=defaultScenario]{defaultScenario()}}}{returns the default scenario settings of \pkg{irace}.} \item{\code{\link[=checkScenario]{checkScenario()}}}{to check that the scenario is valid.} } } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } irace/man/getFinalElites.Rd0000644000176200001440000000251214233311226015273 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/parameterAnalysis.R \name{getFinalElites} \alias{getFinalElites} \title{Return the elite configurations of the final iteration.} \usage{ getFinalElites(iraceResults, n = 0L, drop.metadata = FALSE) } \arguments{ \item{iraceResults}{(\code{list()}|\code{character(1)}) \code{iraceResults} object created by \pkg{irace} and typically saved in the log file \code{irace.Rdata}. If a character string is given, then it is interpreted as the path to the log file from which the \code{iraceResults} object will be loaded.} \item{n}{Number of elite configurations to return, if \code{n} is larger than the number of configurations, then only the existing ones are returned. The default (\code{n=0}) returns all of them.} \item{drop.metadata}{Remove metadata, such the configuration ID and the ID of the parent, from the returned configurations. See \code{\link{removeConfigurationsMetaData}}.} } \value{ A data frame containing the elite configurations required. } \description{ Return the elite configurations of the final iteration. } \examples{ log_file <- system.file("exdata/irace-acotsp.Rdata", package="irace", mustWork=TRUE) print(removeConfigurationsMetaData(getFinalElites(log_file, n=1))) } \author{ Manuel López-Ibáñez and Leslie Pérez Cáceres } \concept{analysis} irace/man/getConfigurationById.Rd0000644000176200001440000000210714232037511016454 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/parameterAnalysis.R \name{getConfigurationById} \alias{getConfigurationById} \title{Returns the configurations selected by ID.} \usage{ getConfigurationById(iraceResults, ids, drop.metadata = FALSE) } \arguments{ \item{iraceResults}{(\code{list()}|\code{character(1)}) \code{iraceResults} object created by \pkg{irace} and typically saved in the log file \code{irace.Rdata}. If a character string is given, then it is interpreted as the path to the log file from which the \code{iraceResults} object will be loaded.} \item{ids}{The id or a vector of ids of the candidates configurations to obtain.} \item{drop.metadata}{Remove metadata, such the configuration ID and the ID of the parent, from the returned configurations. See \code{\link[=removeConfigurationsMetaData]{removeConfigurationsMetaData()}}.} } \value{ A data frame containing the elite configurations required. } \description{ Returns the configurations selected by ID. } \author{ Manuel López-Ibáñez and Leslie Pérez Cáceres } \concept{analysis} irace/man/read_logfile.Rd0000644000176200001440000000110314232020734015003 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/utils.R \name{read_logfile} \alias{read_logfile} \title{Read the log file produced by irace (\code{irace.Rdata}).} \usage{ read_logfile(filename, name = "iraceResults") } \arguments{ \item{filename}{Filename that contains the log file saved by irace. Example: \code{irace.Rdata}.} \item{name}{Optional argument that allows overriding the default name of the object in the file.} } \value{ (\code{list()}) } \description{ Read the log file produced by irace (\code{irace.Rdata}). } \concept{analysis} irace/man/irace.main.Rd0000644000176200001440000001071414252661514014416 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/main.R \name{irace.main} \alias{irace.main} \title{Higher-level interface to launch irace.} \usage{ irace.main(scenario, output.width = 9999L) } \arguments{ \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} settings. The data structure has to be the one returned by the function \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} \item{output.width}{(\code{integer(1)}) The width used for the screen output.} } \value{ (\code{invisible(data.frame)}) A data frame with the set of best algorithm configurations found by \pkg{irace}. The data frame has the following columns: \itemize{ \item \code{.ID.} : Internal id of the candidate configuration. \item \verb{Parameter names} : One column per parameter name in \code{parameters}. \item \code{.PARENT.} : Internal id of the parent candidate configuration. } Additionally, this function saves an R data file containing an object called \code{iraceResults}. The path of the file is indicated in \code{scenario$logFile}. The \code{iraceResults} object is a list with the following structure: \describe{ \item{\code{scenario}}{The scenario R object containing the \pkg{irace} options used for the execution. See \code{\link{defaultScenario}} for more information.} \item{\code{parameters}}{The parameters R object containing the description of the target algorithm parameters. See \code{\link{readParameters}}.} \item{\code{allConfigurations}}{The target algorithm configurations generated by \pkg{irace}. This object is a data frame, each row is a candidate configuration, the first column (\code{.ID.}) indicates the internal identifier of the configuration, the following columns correspond to the parameter values, each column named as the parameter name specified in the parameter object. The final column (\code{.PARENT.}) is the identifier of the configuration from which model the actual configuration was sampled.} \item{\code{allElites}}{A list that contains one element per iteration, each element contains the internal identifier of the elite candidate configurations of the corresponding iteration (identifiers correspond to \code{allConfigurations$.ID.}).} \item{\code{iterationElites}}{A vector containing the best candidate configuration internal identifier of each iteration. The best configuration found corresponds to the last one of this vector.} \item{\code{experiments}}{A matrix with configurations as columns and instances as rows. Column names correspond to the internal identifier of the configuration (\code{allConfigurations$.ID.}).} \item{\code{experimentLog}}{A matrix with columns \code{iteration}, \code{instance}, \code{configuration}, \code{time}. This matrix contains the log of all the experiments that \pkg{irace} performs during its execution. The instance column refers to the index of the \code{scenario$instancesList} data frame. Time is saved ONLY when reported by the \code{targetRunner}.} \item{\code{softRestart}}{A logical vector that indicates if a soft restart was performed on each iteration. If \code{FALSE}, then no soft restart was performed.} \item{\code{state}}{A list that contains the state of \pkg{irace}, the recovery is done using the information contained in this object.} \item{\code{testing}}{A list that contains the testing results. The elements of this list are: \code{experiments} a matrix with the testing experiments of the selected configurations in the same format as the explained above and \code{seeds} a vector with the seeds used to execute each experiment.} } } \description{ Higher-level interface to launch irace. } \details{ This function checks the correctness of the scenario, reads the parameter space from \code{scenario$parameterFile}, invokes \code{\link[=irace]{irace()}}, prints its results in various formatted ways, (optionally) calls \code{\link[=psRace]{psRace()}} and, finally, evaluates the best configurations on the test instances (if provided). If you want a lower-level interface that just runs irace, please see function \code{\link[=irace]{irace()}}. } \seealso{ \code{\link[=irace.cmdline]{irace.cmdline()}} a higher-level command-line interface to \code{\link[=irace]{irace()}} \code{\link[=readScenario]{readScenario()}} to read the scenario setup from a file. \code{\link[=defaultScenario]{defaultScenario()}} to provide a default scenario for \pkg{irace}. } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } \concept{running} irace/man/irace.cmdline.Rd0000644000176200001440000003630214321267520015102 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/main.R \name{irace.cmdline} \alias{irace.cmdline} \title{Launch \code{irace} with command-line options.} \usage{ irace.cmdline(argv = commandArgs(trailingOnly = TRUE)) } \arguments{ \item{argv}{(\code{character()}) \cr The arguments provided on the R command line as a character vector, e.g., \code{c("--scenario", "scenario.txt", "-p", "parameters.txt")}. Using the default value (not providing the parameter) is the easiest way to call \code{irace.cmdline}.} } \value{ (\code{invisible(data.frame)}) A data frame with the set of best algorithm configurations found by \pkg{irace}. The data frame has the following columns: \itemize{ \item \code{.ID.} : Internal id of the candidate configuration. \item \verb{Parameter names} : One column per parameter name in \code{parameters}. \item \code{.PARENT.} : Internal id of the parent candidate configuration. } Additionally, this function saves an R data file containing an object called \code{iraceResults}. The path of the file is indicated in \code{scenario$logFile}. The \code{iraceResults} object is a list with the following structure: \describe{ \item{\code{scenario}}{The scenario R object containing the \pkg{irace} options used for the execution. See \code{\link{defaultScenario}} for more information.} \item{\code{parameters}}{The parameters R object containing the description of the target algorithm parameters. See \code{\link{readParameters}}.} \item{\code{allConfigurations}}{The target algorithm configurations generated by \pkg{irace}. This object is a data frame, each row is a candidate configuration, the first column (\code{.ID.}) indicates the internal identifier of the configuration, the following columns correspond to the parameter values, each column named as the parameter name specified in the parameter object. The final column (\code{.PARENT.}) is the identifier of the configuration from which model the actual configuration was sampled.} \item{\code{allElites}}{A list that contains one element per iteration, each element contains the internal identifier of the elite candidate configurations of the corresponding iteration (identifiers correspond to \code{allConfigurations$.ID.}).} \item{\code{iterationElites}}{A vector containing the best candidate configuration internal identifier of each iteration. The best configuration found corresponds to the last one of this vector.} \item{\code{experiments}}{A matrix with configurations as columns and instances as rows. Column names correspond to the internal identifier of the configuration (\code{allConfigurations$.ID.}).} \item{\code{experimentLog}}{A matrix with columns \code{iteration}, \code{instance}, \code{configuration}, \code{time}. This matrix contains the log of all the experiments that \pkg{irace} performs during its execution. The instance column refers to the index of the \code{scenario$instancesList} data frame. Time is saved ONLY when reported by the \code{targetRunner}.} \item{\code{softRestart}}{A logical vector that indicates if a soft restart was performed on each iteration. If \code{FALSE}, then no soft restart was performed.} \item{\code{state}}{A list that contains the state of \pkg{irace}, the recovery is done using the information contained in this object.} \item{\code{testing}}{A list that contains the testing results. The elements of this list are: \code{experiments} a matrix with the testing experiments of the selected configurations in the same format as the explained above and \code{seeds} a vector with the seeds used to execute each experiment.} } } \description{ Calls \code{\link[=irace.main]{irace.main()}} using command-line options, maybe parsed from the command line used to invoke R. } \details{ The function reads the parameters given on the command line used to invoke R, finds the name of the scenario file, initializes the scenario from the file (with the function \code{\link{readScenario}}) and possibly from parameters passed in the command line. It finally starts \pkg{irace} by calling \code{\link{irace.main}}. List of command-line options:\preformatted{-h,--help Show this help. -v,--version Show irace package version. -c,--check Check scenario. -i,--init Initialize the working directory with template config files. --only-test Only test the configurations given in the file passed as argument. -s,--scenario File that describes the configuration scenario setup and other irace settings. Default: ./scenario.txt. --exec-dir Directory where the programs will be run. Default: ./. -p,--parameter-file File that contains the description of the parameters of the target algorithm. Default: ./parameters.txt. --forbidden-file File that contains a list of logical expressions that cannot be TRUE for any evaluated configuration. If empty or NULL, do not use forbidden expressions. --configurations-file File that contains a table of initial configurations. If empty or NULL, all initial configurations are randomly generated. -l,--log-file File to save tuning results as an R dataset, either absolute path or relative to execDir. Default: ./irace.Rdata. --recovery-file Previously saved log file to recover the execution of irace, either absolute path or relative to the current directory. If empty or NULL, recovery is not performed. --train-instances-dir Directory where training instances are located; either absolute path or relative to current directory. If no trainInstancesFiles is provided, all the files in trainInstancesDir will be listed as instances. Default: ./Instances. --train-instances-file File that contains a list of training instances and optionally additional parameters for them. If trainInstancesDir is provided, irace will search for the files in this folder. --sample-instances Randomly sample the training instances or use them in the order given. Default: 1. --test-instances-dir Directory where testing instances are located, either absolute or relative to current directory. --test-instances-file File containing a list of test instances and optionally additional parameters for them. --test-num-elites Number of elite configurations returned by irace that will be tested if test instances are provided. Default: 1. --test-iteration-elites Enable/disable testing the elite configurations found at each iteration. Default: 0. --test-type Statistical test used for elimination. The default value selects t-test if capping is enabled or F-test, otherwise. Valid values are: F-test (Friedman test), t-test (pairwise t-tests with no correction), t-test-bonferroni (t-test with Bonferroni's correction for multiple comparisons), t-test-holm (t-test with Holm's correction for multiple comparisons). --first-test Number of instances evaluated before the first elimination test. It must be a multiple of eachTest. Default: 5. --each-test Number of instances evaluated between elimination tests. Default: 1. --target-runner Executable called for each configuration that executes the target algorithm to be tuned. See the templates and examples provided. Default: ./target-runner. --target-runner-launcher Executable that will be used to launch the target runner, when targetRunner cannot be executed directly (.e.g, a Python script in Windows). --target-runner-args Command-line arguments provided to targetRunnerLauncher. The substrings \\\{targetRunner\\\} and \\\{targetRunnerArgs\\\} will be replaced by the value of the option targetRunner and by the arguments usually passed when calling targetRunner, respectively. Example: "-m \{targetRunner --args \{targetRunnerArgs\}"\}. Default: \{targetRunner\} \{targetRunnerArgs\}. --target-runner-retries Number of times to retry a call to targetRunner if the call failed. Default: 0. --target-evaluator Optional script or R function that provides a numeric value for each configuration. See templates/target-evaluator.tmpl --deterministic If the target algorithm is deterministic, configurations will be evaluated only once per instance. Default: 0. --max-experiments Maximum number of runs (invocations of targetRunner) that will be performed. It determines the maximum budget of experiments for the tuning. Default: 0. --max-time Maximum total execution time in seconds for the executions of targetRunner. targetRunner must return two values: cost and time. Default: 0. --budget-estimation Fraction (smaller than 1) of the budget used to estimate the mean computation time of a configuration. Only used when maxTime > 0 Default: 0.02. --min-measurable-time Minimum time unit that is still (significantly) measureable. Default: 0.01. --parallel Number of calls to targetRunner to execute in parallel. Values 0 or 1 mean no parallelization. Default: 0. --load-balancing Enable/disable load-balancing when executing experiments in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster. Default: 1. --mpi Enable/disable MPI. Use Rmpi to execute targetRunner in parallel (parameter parallel is the number of slaves). Default: 0. --batchmode Specify how irace waits for jobs to finish when targetRunner submits jobs to a batch cluster: sge, pbs, torque, slurm or htcondor. targetRunner must submit jobs to the cluster using, for example, qsub. Default: 0. --digits Maximum number of decimal places that are significant for numerical (real) parameters. Default: 4. -q,--quiet Reduce the output generated by irace to a minimum. Default: 0. --debug-level Debug level of the output of irace. Set this to 0 to silence all debug messages. Higher values provide more verbose debug messages. Default: 0. --seed Seed of the random number generator (by default, generate a random seed). --soft-restart Enable/disable the soft restart strategy that avoids premature convergence of the probabilistic model. Default: 1. --soft-restart-threshold Soft restart threshold value for numerical parameters. If NA, NULL or "", it is computed as 10^-digits. -e,--elitist Enable/disable elitist irace. Default: 1. --elitist-new-instances Number of instances added to the execution list before previous instances in elitist irace. Default: 1. --elitist-limit In elitist irace, maximum number per race of elimination tests that do not eliminate a configuration. Use 0 for no limit. Default: 2. --capping Enable the use of adaptive capping, a technique designed for minimizing the computation time of configurations. This is only available when elitist is active. Default: 0. --capping-type Measure used to obtain the execution bound from the performance of the elite configurations: median, mean, worst, best. Default: median. --bound-type Method to calculate the mean performance of elite configurations: candidate or instance. Default: candidate. --bound-max Maximum execution bound for targetRunner. It must be specified when capping is enabled. Default: 0. --bound-digits Precision used for calculating the execution time. It must be specified when capping is enabled. Default: 0. --bound-par Penalization constant for timed out executions (executions that reach boundMax execution time). Default: 1. --bound-as-timeout Replace the configuration cost of bounded executions with boundMax. Default: 1. --postselection Percentage of the configuration budget used to perform a postselection race of the best configurations of each iteration after the execution of irace. Default: 0. --aclib Enable/disable AClib mode. This option enables compatibility with GenericWrapper4AC as targetRunner script. Default: 0. --iterations Maximum number of iterations. Default: 0. --experiments-per-iteration Number of runs of the target algorithm per iteration. Default: 0. --min-survival Minimum number of configurations needed to continue the execution of each race (iteration). Default: 0. --num-configurations Number of configurations to be sampled and evaluated at each iteration. Default: 0. --mu Parameter used to define the number of configurations sampled and evaluated at each iteration. Default: 5. --confidence Confidence level for the elimination test. Default: 0.95. } } \examples{ irace.cmdline("--version") } \seealso{ \code{\link[=irace.main]{irace.main()}} to start \pkg{irace} with a given scenario. } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } \concept{running} irace/man/testing_fromlog.Rd0000644000176200001440000000416314157156064015616 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/main.R \name{testing_fromlog} \alias{testing_fromlog} \title{Test configurations given in \code{.Rdata} file} \usage{ testing_fromlog( logFile, testNbElites, testIterationElites, testInstancesDir, testInstancesFile, testInstances ) } \arguments{ \item{logFile}{Path to the \code{.Rdata} file produced by \pkg{irace}.} \item{testNbElites}{Number of (final) elite configurations to test. Overrides the value found in \code{logFile}.} \item{testIterationElites}{(\code{logical(1)}) If \code{FALSE}, only the final \code{testNbElites} configurations are tested; otherwise, also test the best configurations of each iteration. Overrides the value found in \code{logFile}.} \item{testInstancesDir}{Directory where testing instances are located, either absolute or relative to current directory.} \item{testInstancesFile}{File containing a list of test instances and optionally additional parameters for them.} \item{testInstances}{Character vector of the instances to be used in the \code{targetRunner} when executing the testing.} } \value{ Boolean. \code{TRUE} if the testing ended successfully otherwise, \code{FALSE}. } \description{ \code{testing_fromlog} executes the testing of the target algorithm configurations found by an \pkg{irace} execution. } \details{ The function \code{testing_fromlog} loads the \code{logFile} and obtains the testing setup and configurations to be tested. Within the \code{logFile}, the variable \code{scenario$testNbElites} specifies how many final elite configurations to test and \code{scenario$testIterationElites} indicates whether test the best configuration of each iteration. The values may be overridden by setting the corresponding arguments in this function. The set of testing instances must appear in \code{scenario[["testInstances"]]}. } \seealso{ \code{\link[=defaultScenario]{defaultScenario()}} to provide a default scenario for \pkg{irace}. \code{\link[=testing_fromfile]{testing_fromfile()}} provides a different interface for testing. } \author{ Manuel López-Ibáñez and Leslie Pérez Cáceres } \concept{running} irace/man/ablation_cmdline.Rd0000644000176200001440000000706614321271075015676 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/ablation.R \name{ablation_cmdline} \alias{ablation_cmdline} \title{Launch ablation with command-line options.} \usage{ ablation_cmdline(argv = commandArgs(trailingOnly = TRUE)) } \arguments{ \item{argv}{(\code{character()}) \cr The arguments provided on the R command line as a character vector, e.g., \code{c("-i", "irace.Rdata", "--src", 1)}.} } \value{ A list containing the following elements: \describe{ \item{configurations}{Configurations tested in the ablation.} \item{instances}{A matrix with the instances used in the experiments. First column has the instances IDs from \code{iraceResults$scenario$instances}, second column the seed assigned to the instance.} \item{experiments}{A matrix with the results of the experiments (columns are configurations, rows are instances).} \item{scenario}{Scenario object with the settings used for the experiments.} \item{trajectory}{IDs of the best configurations at each step of the ablation.} \item{best}{Best configuration found in the experiments.} } } \description{ Launch \code{\link[=ablation]{ablation()}} with the same command-line options as the command-line executable (\code{ablation.exe} in Windows). } \details{ The function reads the parameters given on the command line used to invoke R, launches \code{\link[=ablation]{ablation()}} and possibly \code{\link[=plotAblation]{plotAblation()}}. List of command-line options:\preformatted{-l,--log-file Path to the (.Rdata) file created by irace from which the "iraceResults" object will be loaded. -S,--src Source configuration ID. Default: 1. -T,--target Target configuration ID. By default the best configuration found by irace. -P,--params Specific parameter names to be used for the ablation (separated with commas). By default use all -t,--type Type of ablation to perform: "full" will execute each configuration on all "--n-instances" to determine the best-performing one; "racing" will apply racing to find the best configurations. Default: full. -n,--n-instances Number of instances used in "full" ablation will be n_instances * scenario$firstTest. Default: 1. --seed Integer value to use as seed for the random number generation. Default: 1234567. -o,--output-file Log file to save the ablation log. If "", the results are not saved to a file. Default: log-ablation.Rdata. -p,--plot Output filename (.pdf) for the plot. If not given, no plot is created. -O,--plot-type Type of plot. Supported values are "mean" and "boxplot". Default: mean. --old-path Old path found in the log-file (.Rdata) given as input to be replaced by --new-path. --new-path New path to replace the path found in the log-file (.Rdata) given as input. -e,--exec-dir Directory where the target runner will be run. -s,--scenario Scenario file to override the scenario given in the log-file (.Rdata) --parallel Number of calls to targetRunner to execute in parallel. Values 0 or 1 mean no parallelization. } } \examples{ # ablation_cmdline("--help") } \author{ Manuel López-Ibáñez } \concept{running} irace/man/testConfigurations.Rd0000644000176200001440000000247314240244046016277 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/testing.R \name{testConfigurations} \alias{testConfigurations} \title{Execute the given configurations on the testing instances specified in the scenario} \usage{ testConfigurations(configurations, scenario, parameters) } \arguments{ \item{configurations}{(\code{data.frame}) \cr Parameter configurations of the target algorithm (one per row).} \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} settings. The data structure has to be the one returned by the function \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} \item{parameters}{(\code{list()}) \cr Data structure containing the parameter space definition. The data structure has to similar to the one returned by the function \code{\link{readParameters}}.} } \value{ A list with the following elements: \describe{ \item{\code{experiments}}{Experiments results.} \item{\code{seeds}}{Array of the instance seeds used in the experiments.} } } \description{ Execute the given configurations on the testing instances specified in the scenario } \details{ A test instance set must be provided through \code{scenario[["testInstances"]]}. } \seealso{ \code{\link[=testing_fromlog]{testing_fromlog()}} } \author{ Manuel López-Ibáñez } irace/man/ablation.Rd0000644000176200001440000000637514321271075014205 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/ablation.R \name{ablation} \alias{ablation} \title{Performs ablation between two configurations (from source to target).} \usage{ ablation( iraceResults, src = 1L, target = NULL, ab.params = NULL, type = c("full", "racing"), n_instances = 1L, seed = 1234567, ablationLogFile = "log-ablation.Rdata", ... ) } \arguments{ \item{iraceResults}{(\code{list()}|\code{character(1)}) \code{iraceResults} object created by \pkg{irace} and typically saved in the log file \code{irace.Rdata}. If a character string is given, then it is interpreted as the path to the log file from which the \code{iraceResults} object will be loaded.} \item{src, target}{Source and target configuration IDs. By default, the first configuration ever evaluated (ID 1) is used as \code{src} and the best configuration found by irace is used as target.} \item{ab.params}{Specific parameter names to be used for the ablation. They must be in \code{parameters$names}. By default, use all parameters.} \item{type}{Type of ablation to perform: \code{"full"} will execute each configuration on all \code{n_instances} to determine the best-performing one; \code{"racing"} will apply racing to find the best configurations.} \item{n_instances}{(\code{integer(1)}) Number of instances used in \code{"full"} ablation will be \code{n_instances * scenario$firstTest}.} \item{seed}{(\code{integer(1)}) Integer value to use as seed for the random number generation.} \item{ablationLogFile}{(\code{character(1)}) Log file to save the ablation log. If \code{NULL}, the results are not saved to a file.} \item{...}{Further arguments to override scenario settings, e.g., \code{debugLevel}, \code{parallel}, etc.} } \value{ A list containing the following elements: \describe{ \item{configurations}{Configurations tested in the ablation.} \item{instances}{A matrix with the instances used in the experiments. First column has the instances IDs from \code{iraceResults$scenario$instances}, second column the seed assigned to the instance.} \item{experiments}{A matrix with the results of the experiments (columns are configurations, rows are instances).} \item{scenario}{Scenario object with the settings used for the experiments.} \item{trajectory}{IDs of the best configurations at each step of the ablation.} \item{best}{Best configuration found in the experiments.} } } \description{ Ablation is a method for analyzing the differences between two configurations. } \examples{ \donttest{ logfile <- system.file(package="irace", "exdata", "sann.rda") # Execute ablation between the first and the best configuration found by irace. ablog <- ablation(logfile, ablationLogFile = NULL) plotAblation(ablog) # Execute ablation between two selected configurations, and selecting only a # subset of parameters, directly reading the setup from the irace log file. ablog <- ablation(logfile, src = 1, target = 10, ab.params = c("temp"), ablationLogFile = NULL) plotAblation(ablog) } } \references{ C. Fawcett and H. H. Hoos. Analysing differences between algorithm configurations through ablation. Journal of Heuristics, 22(4):431–458, 2016. } \seealso{ \code{\link[=plotAblation]{plotAblation()}} } \author{ Leslie Pérez Cáceres and Manuel López-Ibáñez } irace/man/psRace.Rd0000644000176200001440000000516314252663436013634 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/parameterExploration.R \name{psRace} \alias{psRace} \title{psRace} \usage{ psRace( iraceLogFile = NULL, iraceResults = NULL, conf.ids = NULL, postselection = NULL, max.experiments = NULL, elites = FALSE, seed = 1234567 ) } \arguments{ \item{iraceLogFile}{NULL Log file created by \pkg{irace}, this file must contain the \code{iraceResults} object.} \item{iraceResults}{NULL Object created by \pkg{irace} and saved in \code{scenario$logFile}.} \item{conf.ids}{NULL IDs of the configurations in iraceResults$allConfigurations to be used for ablation. If NULL, the elites argument will be used.} \item{postselection}{NULL Percentage of the maxExperiments provided in the scenario to be used in the race.} \item{max.experiments}{NULL Number of experiments available for the race. If NULL budget for the race is set by the parameter scenario$postselection, which defines the percentage of the total budget of \pkg{irace} (iraceResults$scenario$maxExperiments or iraceResults$scenario$maxTime/iraceResults$state$timeEstimate) to use for the postselection.} \item{elites}{FALSE Flag for selecting configurations. If FALSE, the best configurations of each iteration are used for the race. If TRUE, the elite configurtions of each iteration are used for the race.} \item{seed}{1234567 Numerical value to use as seed for the random number generation.} } \value{ If iraceLogFile is NULL, it returns a list with the following elements: \describe{ \item{configurations}{Configurations used in the race.} \item{instances}{A matrix with the instances used in the experiments. First column has the instances ids from iraceResults$scenario$instances, second column the seed assigned to the instance.} \item{maxExperiments}{Maximum number of experiments set for the race.} \item{experiments}{A matrix with the results of the experiments (columns are configurations, rows are instances).} \item{elites}{Best configurations found in the experiments.} } If \code{iraceLogFile} is provided this list object will be saved in \code{iraceResults$psrace.log}. } \description{ \code{psRace} performs a postselection race a set of configurations. } \examples{ \dontrun{ # Execute the postselection automatically after irace scenario <- readScenario(filename="scenario.txt") parameters <- readParameters("parameters.txt") # Use 10\% of the total budget scenario$postselection <- 0.1 irace(scenario=scenario, parameters=parameters) # Execute the postselection after the execution of \pkg{irace}. psRace(iraceLogFile="irace.Rdata", max.experiments=120) } } \author{ Leslie Pérez Cáceres } irace/man/target.evaluator.default.Rd0000644000176200001440000000533314240244046017315 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/race-wrapper.R \name{target.evaluator.default} \alias{target.evaluator.default} \title{target.evaluator.default} \usage{ target.evaluator.default( experiment, num.configurations, all.conf.id, scenario, target.runner.call ) } \arguments{ \item{experiment}{A list describing the experiment. It contains at least: \describe{ \item{\code{id.configuration}}{An alphanumeric string that uniquely identifies a configuration;} \item{\code{id.instance}}{An alphanumeric string that uniquely identifies an instance;} \item{\code{seed}}{Seed for the random number generator to be used for this evaluation, ignore the seed for deterministic algorithms;} \item{\code{instance}}{String giving the instance to be used for this evaluation;} \item{\code{bound}}{(only when \code{capping} is enabled) Time bound for the execution;} \item{\code{configuration}}{1-row data frame with a column per parameter name;} \item{\code{switches}}{Vector of parameter switches (labels) in the order of parameters used in \code{configuration}.} }} \item{num.configurations}{Number of configurations alive in the race.} \item{all.conf.id}{Vector of configuration IDs of the alive configurations.} \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} settings. The data structure has to be the one returned by the function \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} \item{target.runner.call}{String describing the call to \code{targetRunner} that corresponds to this call to \code{targetEvaluator}. This is used for providing extra information to the user, for example, in case \code{targetEvaluator} fails.} } \value{ The function \code{targetEvaluator} must return a list with one element \code{"cost"}, the numerical value corresponding to the cost measure of the given configuration on the given instance. The return list may also contain the following optional elements that are used by \pkg{irace} for reporting errors in \code{targetEvaluator}: \describe{ \item{\code{error}}{is a string used to report an error;} \item{\code{outputRaw}}{is a string used to report the raw output of calls to an external program or function;} \item{\code{call}}{is a string used to report how \code{targetRunner} called an external program or function.} } } \description{ \code{target.evaluator.default} is the default \code{targetEvaluator} function that is invoked if \code{targetEvaluator} is a string (by default \code{targetEvaluator} is \code{NULL} and this function is not invoked). You can use it as an advanced example of how to create your own \code{targetEvaluator} function. } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } irace/man/irace-package.Rd0000644000176200001440000001370614233311226015057 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/irace-package.R \docType{package} \name{irace-package} \alias{irace-package} \title{The irace package: \packageTitle{irace}} \description{ \packageDescription{irace} } \details{ License: GPL (>= 2) } \examples{ ####################################################################### # This example illustrates how to tune the parameters of the simulated # annealing algorithm (SANN) provided by the optim() function in the # R base package. The goal in this example is to optimize instances of # the following family: # f(x) = lambda * f_rastrigin(x) + (1 - lambda) * f_rosenbrock(x) # where lambda follows a normal distribution whose mean is 0.9 and # standard deviation is 0.02. f_rastrigin and f_rosenbrock are the # well-known Rastrigin and Rosenbrock benchmark functions (taken from # the cmaes package). In this scenario, different instances are given # by different values of lambda. ####################################################################### ## First we provide an implementation of the functions to be optimized: f_rosenbrock <- function (x) { d <- length(x) z <- x + 1 hz <- z[1:(d - 1)] tz <- z[2:d] s <- sum(100 * (hz^2 - tz)^2 + (hz - 1)^2) return(s) } f_rastrigin <- function (x) { sum(x * x - 10 * cos(2 * pi * x) + 10) } ## We generate 20 instances (in this case, weights): weights <- rnorm(20, mean = 0.9, sd = 0.02) ## On this set of instances, we are interested in optimizing two ## parameters of the SANN algorithm: tmax and temp. We setup the ## parameter space as follows: parameters_table <- ' tmax "" i,log (1, 5000) temp "" r (0, 100) ' ## We use the irace function readParameters to read this table: parameters <- readParameters(text = parameters_table) ## Next, we define the function that will evaluate each candidate ## configuration on a single instance. For simplicity, we restrict to ## three-dimensional functions and we set the maximum number of ## iterations of SANN to 1000. target_runner <- function(experiment, scenario) { instance <- experiment$instance configuration <- experiment$configuration D <- 3 par <- runif(D, min=-1, max=1) fn <- function(x) { weight <- instance return(weight * f_rastrigin(x) + (1 - weight) * f_rosenbrock(x)) } res <- stats::optim(par,fn, method="SANN", control=list(maxit=1000 , tmax = as.numeric(configuration[["tmax"]]) , temp = as.numeric(configuration[["temp"]]) )) ## New output interface in irace 2.0. This list may also contain: ## - 'time' if irace is called with 'maxTime' ## - 'error' is a string used to report an error ## - 'outputRaw' is a string used to report the raw output of calls to ## an external program or function. ## - 'call' is a string used to report how target_runner called the ## external program or function. return(list(cost = res$value)) } ## We define a configuration scenario by setting targetRunner to the ## function define above, instances to the first 10 random weights, and ## a maximum budget of 'maxExperiments' calls to targetRunner. scenario <- list(targetRunner = target_runner, instances = weights[1:10], maxExperiments = 500, # Do not create a logFile logFile = "") ## We check that the scenario is valid. This will also try to execute ## target_runner. checkIraceScenario(scenario, parameters = parameters) \donttest{ ## We are now ready to launch irace. We do it by means of the irace ## function. The function will print information about its ## progress. This may require a few minutes, so it is not run by default. tuned_confs <- irace(scenario = scenario, parameters = parameters) ## We can print the best configurations found by irace as follows: configurations.print(tuned_confs) ## We can evaluate the quality of the best configuration found by ## irace versus the default configuration of the SANN algorithm on ## the other 10 instances previously generated. ## To do so, first we apply the default configuration of the SANN ## algorithm to these instances: test <- function(configuration) { res <- lapply(weights[11:20], function(x) target_runner( experiment = list(instance = x, configuration = configuration), scenario = scenario)) return (sapply(res, getElement, name = "cost")) } default <- test(data.frame(tmax=10, temp=10)) ## We extract and apply the winning configuration found by irace ## to these instances: tuned <- test(removeConfigurationsMetaData(tuned_confs[1,])) ## Finally, we can compare using a boxplot the quality obtained with the ## default parametrization of SANN and the quality obtained with the ## best configuration found by irace. boxplot(list(default = default, tuned = tuned)) } } \references{ Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, Thomas Stützle, and Mauro Birattari. The irace package: Iterated Racing for Automatic Algorithm Configuration. \emph{Operations Research Perspectives}, 2016. \doi{10.1016/j.orp.2016.09.002} Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Thomas Stützle, and Mauro Birattari. \emph{The irace package, Iterated Race for Automatic Algorithm Configuration}. Technical Report TR/IRIDIA/2011-004, IRIDIA, Université Libre de Bruxelles, Belgium, 2011. Manuel López-Ibáñez and Thomas Stützle. The Automatic Design of Multi-Objective Ant Colony Optimization Algorithms. \emph{IEEE Transactions on Evolutionary Computation}, 2012. } \seealso{ \code{\link{irace.main}} to start \pkg{irace} with a given scenario. } \author{ Maintainers: Manuel López-Ibáñez and Leslie Pérez Cáceres \email{irace-package@googlegroups.com} } \keyword{automatic} \keyword{configuration} \keyword{optimize} \keyword{package} \keyword{tuning} irace/man/irace.version.Rd0000644000176200001440000000053214204134677015156 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/version.R \docType{data} \name{irace.version} \alias{irace.version} \title{irace.version} \format{ An object of class \code{character} of length 1. } \usage{ irace.version } \description{ A character string containing the version of \code{irace}. } \keyword{datasets} irace/man/printScenario.Rd0000644000176200001440000000173014240244046015220 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/readConfiguration.R \name{printScenario} \alias{printScenario} \title{Prints the given scenario} \usage{ printScenario(scenario) } \arguments{ \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} settings. The data structure has to be the one returned by the function \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} } \description{ Prints the given scenario } \seealso{ \describe{ \item{\code{\link[=readScenario]{readScenario()}}}{for reading a configuration scenario from a file.} \item{\code{\link[=printScenario]{printScenario()}}}{prints the given scenario.} \item{\code{\link[=defaultScenario]{defaultScenario()}}}{returns the default scenario settings of \pkg{irace}.} \item{\code{\link[=checkScenario]{checkScenario()}}}{to check that the scenario is valid.} } } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } irace/man/target.runner.default.Rd0000644000176200001440000000447014252053064016626 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/race-wrapper.R \name{target.runner.default} \alias{target.runner.default} \title{Default \code{targetRunner} function.} \usage{ target.runner.default(experiment, scenario) } \arguments{ \item{experiment}{A list describing the experiment. It contains at least: \describe{ \item{\code{id.configuration}}{An alphanumeric string that uniquely identifies a configuration;} \item{\code{id.instance}}{An alphanumeric string that uniquely identifies an instance;} \item{\code{seed}}{Seed for the random number generator to be used for this evaluation, ignore the seed for deterministic algorithms;} \item{\code{instance}}{String giving the instance to be used for this evaluation;} \item{\code{bound}}{(only when \code{capping} is enabled) Time bound for the execution;} \item{\code{configuration}}{1-row data frame with a column per parameter name;} \item{\code{switches}}{Vector of parameter switches (labels) in the order of parameters used in \code{configuration}.} }} \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} settings. The data structure has to be the one returned by the function \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} } \value{ If \code{targetEvaluator} is \code{NULL}, then the \code{targetRunner} function must return a list with at least one element \code{"cost"}, the numerical value corresponding to the evaluation of the given configuration on the given instance. If the scenario option \code{maxTime} is non-zero or if \code{capping} is enabled then the list must contain at least another element \code{"time"} that reports the execution time for this call to \code{targetRunner}. The return list may also contain the following optional elements that are used by \pkg{irace} for reporting errors in \code{targetRunner}: \describe{ \item{\code{error}}{is a string used to report an error;} \item{\code{outputRaw}}{is a string used to report the raw output of calls to an external program or function;} \item{\code{call}}{is a string used to report how \code{targetRunner} called an external program or function.} } } \description{ Use it as an advanced example of how to create your own \code{targetRunner} function. } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } irace/man/plotAblation.Rd0000644000176200001440000000254314321266642015041 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/ablation.R \name{plotAblation} \alias{plotAblation} \title{Create plot from an ablation log} \usage{ plotAblation( ablog, pdf.file = NULL, pdf.width = 20, type = c("mean", "boxplot"), mar = par("mar"), ylab = "Mean configuration cost", ylim = NULL, ... ) } \arguments{ \item{ablog}{(\code{list()}|\code{character(1)}) Ablation log object returned by \code{\link[=ablation]{ablation()}}. Alternatively, the path to an \code{.Rdata} file, e.g., \code{"log-ablation.Rdata"}, from which the object will be loaded.} \item{pdf.file}{Output filename.} \item{pdf.width}{Width provided to create the pdf file.} \item{type}{Type of plot. Supported values are \code{"mean"} and \code{"boxplot"}.} \item{mar}{Vector with the margins for the ablation plot.} \item{ylab}{Label of y-axis.} \item{ylim}{Numeric vector of length 2, giving the y coordinates ranges.} \item{...}{Further graphical parameters may also be supplied as arguments. See \code{\link[graphics:plot.default]{graphics::plot.default()}}.} } \description{ Create plot from an ablation log } \examples{ logfile <- file.path(system.file(package="irace"), "exdata", "log-ablation.Rdata") plotAblation(ablog = logfile) } \seealso{ \code{\link[=ablation]{ablation()}} } \author{ Leslie Pérez Cáceres and Manuel López-Ibáñez } irace/man/read_pcs_file.Rd0000644000176200001440000000532714310634160015163 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/readParameters.R \name{read_pcs_file} \alias{read_pcs_file} \title{Read parameters in PCS (AClib) format and write them in irace format.} \usage{ read_pcs_file(file, digits = 4, debugLevel = 0, text) } \arguments{ \item{file}{(\code{character(1)}) \cr Filename containing the definitions of the parameters to be tuned.} \item{digits}{The number of decimal places to be considered for the real parameters.} \item{debugLevel}{(\code{integer(1)}) \cr Larger values produce more verbose output.} \item{text}{(\code{character(1)}) \cr If \code{file} is not supplied and this is, then parameters are read from the value of \code{text} via a text connection.} } \value{ A string representing the parameters in irace format. } \description{ Read parameters in PCS (AClib) format and write them in irace format. } \details{ Either \code{file} or \code{text} must be given. If \code{file} is given, the parameters are read from the file \code{file}. If \code{text} is given instead, the parameters are read directly from the \code{text} character string. In both cases, the parameters must be given (in \code{text} or in the file whose name is \code{file}) in the expected form. See the documentation for details. If none of these parameters is given, \pkg{irace} will stop with an error. \strong{FIXME:} Forbidden configurations, default configuration and transformations ("log") are currently ignored. See \url{https://github.com/MLopez-Ibanez/irace/issues/31} } \examples{ ## Read the parameters directly from text pcs_table <- ' # name domain algorithm {as,mmas,eas,ras,acs}[as] localsearch {0, 1, 2, 3}[0] alpha [0.00, 5.00][1] beta [0.00, 10.00][1] rho [0.01, 1.00][0.95] ants [5, 100][10]i q0 [0.0, 1.0][0] rasrank [1, 100][1]i elitistants [1, 750][1]i nnls [5, 50][5]i dlb {0, 1}[1] Conditionals: q0 | algorithm in {acs} rasrank | algorithm in {ras} elitistants | algorithm in {eas} nnls | localsearch in {1,2,3} dlb | localsearch in {1,2,3} ' parameters_table <- read_pcs_file(text=pcs_table) cat(parameters_table) parameters <- readParameters(text=parameters_table) str(parameters) } \references{ Frank Hutter, Manuel López-Ibáñez, Chris Fawcett, Marius Thomas Lindauer, Holger H. Hoos, Kevin Leyton-Brown, and Thomas Stützle. \strong{AClib: A Benchmark Library for Algorithm Configuration}. In P. M. Pardalos, M. G. C. Resende, C. Vogiatzis, and J. L. Walteros, editors, \emph{Learning and Intelligent Optimization, 8th International Conference, LION 8}, volume 8426 of Lecture Notes in Computer Science, pages 36–40. Springer, Heidelberg, 2014. } \author{ Manuel López-Ibáñez } irace/man/checkParameters.Rd0000644000176200001440000000100614157104332015475 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/readParameters.R \name{checkParameters} \alias{checkParameters} \title{checkParameters} \usage{ checkParameters(parameters) } \arguments{ \item{parameters}{(\code{list()}) \cr Data structure containing the parameter space definition. The data structure has to similar to the one returned by the function \code{\link{readParameters}}.} } \description{ FIXME: This is incomplete, for now we only repair inputs from previous irace versions. } irace/man/irace.Rd0000644000176200001440000001203614252663436013477 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/irace.R \name{irace} \alias{irace} \title{irace} \usage{ irace(scenario, parameters) } \arguments{ \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} settings. The data structure has to be the one returned by the function \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} \item{parameters}{(\code{list()}) \cr Data structure containing the parameter space definition. The data structure has to similar to the one returned by the function \code{\link{readParameters}}.} } \value{ (\code{data.frame}) A data frame with the set of best algorithm configurations found by \pkg{irace}. The data frame has the following columns: \itemize{ \item \code{.ID.} : Internal id of the candidate configuration. \item \verb{Parameter names} : One column per parameter name in \code{parameters}. \item \code{.PARENT.} : Internal id of the parent candidate configuration. } Additionally, this function saves an R data file containing an object called \code{iraceResults}. The path of the file is indicated in \code{scenario$logFile}. The \code{iraceResults} object is a list with the following structure: \describe{ \item{\code{scenario}}{The scenario R object containing the \pkg{irace} options used for the execution. See \code{\link{defaultScenario}} for more information.} \item{\code{parameters}}{The parameters R object containing the description of the target algorithm parameters. See \code{\link{readParameters}}.} \item{\code{allConfigurations}}{The target algorithm configurations generated by \pkg{irace}. This object is a data frame, each row is a candidate configuration, the first column (\code{.ID.}) indicates the internal identifier of the configuration, the following columns correspond to the parameter values, each column named as the parameter name specified in the parameter object. The final column (\code{.PARENT.}) is the identifier of the configuration from which model the actual configuration was sampled.} \item{\code{allElites}}{A list that contains one element per iteration, each element contains the internal identifier of the elite candidate configurations of the corresponding iteration (identifiers correspond to \code{allConfigurations$.ID.}).} \item{\code{iterationElites}}{A vector containing the best candidate configuration internal identifier of each iteration. The best configuration found corresponds to the last one of this vector.} \item{\code{experiments}}{A matrix with configurations as columns and instances as rows. Column names correspond to the internal identifier of the configuration (\code{allConfigurations$.ID.}).} \item{\code{experimentLog}}{A matrix with columns \code{iteration}, \code{instance}, \code{configuration}, \code{time}. This matrix contains the log of all the experiments that \pkg{irace} performs during its execution. The instance column refers to the index of the \code{scenario$instancesList} data frame. Time is saved ONLY when reported by the \code{targetRunner}.} \item{\code{softRestart}}{A logical vector that indicates if a soft restart was performed on each iteration. If \code{FALSE}, then no soft restart was performed.} \item{\code{state}}{A list that contains the state of \pkg{irace}, the recovery is done using the information contained in this object.} \item{\code{testing}}{A list that contains the testing results. The elements of this list are: \code{experiments} a matrix with the testing experiments of the selected configurations in the same format as the explained above and \code{seeds} a vector with the seeds used to execute each experiment.} } } \description{ \code{irace} implements iterated Race. It receives some parameters to be tuned and returns the best configurations found, namely, the elite configurations obtained from the last iterations (and sorted by rank). } \details{ The function \code{irace} executes the tuning procedure using the information provided in \code{scenario} and \code{parameters}. Initially it checks the correctness of \code{scenario} and recovers a previous execution if \code{scenario$recoveryFile} is set. A R data file log of the execution is created in \code{scenario$logFile}. } \examples{ \dontrun{ parameters <- readParameters("parameters.txt") scenario <- readScenario(filename = "scenario.txt") irace(scenario = scenario, parameters = parameters) } } \seealso{ \describe{ \item{\code{\link[=irace.main]{irace.main()}}}{a higher-level interface to \code{irace}.} \item{\code{\link[=irace.cmdline]{irace.cmdline()}}}{a command-line interface to \code{irace}.} \item{\code{\link[=readScenario]{readScenario()}}}{for reading a configuration scenario from a file.} \item{\code{\link[=readParameters]{readParameters()}}}{read the target algorithm parameters from a file.} \item{\code{\link[=defaultScenario]{defaultScenario()}}}{returns the default scenario settings of \pkg{irace}.} \item{\code{\link[=checkScenario]{checkScenario()}}}{to check that the scenario is valid.} } } \author{ Manuel López-Ibáñez and Jérémie Dubois-Lacoste } \concept{running} irace/man/testing_fromfile.Rd0000644000176200001440000000216114260577373015755 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/main.R \name{testing_fromfile} \alias{testing_fromfile} \title{Test configurations given an explicit table of configurations and a scenario file} \usage{ testing_fromfile(filename, scenario) } \arguments{ \item{filename}{Path to a file containing configurations: one configuration per line, one parameter per column, parameter names in header.} \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} settings. The data structure has to be the one returned by the function \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} } \value{ iraceResults } \description{ Executes the testing of an explicit list of configurations given in \code{filename} (same format as in \code{\link[=readConfigurationsFile]{readConfigurationsFile()}}). A \code{logFile} is created unless disabled in \code{scenario}. This may overwrite an existing one! } \seealso{ \code{\link[=testing_fromlog]{testing_fromlog()}} provides a different interface for testing. } \author{ Manuel López-Ibáñez } \concept{running} irace/DESCRIPTION0000644000176200001440000000415014325252712013046 0ustar liggesusersPackage: irace Type: Package Title: Iterated Racing for Automatic Algorithm Configuration Description: Iterated race is an extension of the Iterated F-race method for the automatic configuration of optimization algorithms, that is, (offline) tuning their parameters by finding the most appropriate settings given a set of instances of an optimization problem. M. López-Ibáñez, J. Dubois-Lacoste, L. Pérez Cáceres, T. Stützle, and M. Birattari (2016) . Version: 3.5 Authors@R: c(person("Manuel", "López-Ibáñez", role = c("aut", "cre"), email = "manuel.lopez-ibanez@manchester.ac.uk", comment = c(ORCID = "0000-0001-9974-1295")), person("Jérémie", "Dubois-Lacoste", role = "aut"), person("Leslie", "Pérez Cáceres", role = "aut"), person("Thomas", "Stützle", role = "aut"), person("Mauro", "Birattari", role = "aut"), person("Eric", "Yuan", role = "ctb"), person("Prasanna", "Balaprakash", role = "ctb"), person("Nguyen", "Dang", role = "ctb", email = "nttd@st-andrews.ac.uk")) Depends: R (>= 3.2.0) Imports: stats, utils, compiler, R6 Suggests: Rmpi (>= 0.6.0), parallel, knitr, testthat, withr, mlr (>= 2.15.0), ParamHelpers, devtools, covr VignetteBuilder: knitr License: GPL (>= 2) URL: https://mlopez-ibanez.github.io/irace/, https://github.com/MLopez-Ibanez/irace BugReports: https://github.com/MLopez-Ibanez/irace/issues ByteCompile: yes Encoding: UTF-8 RoxygenNote: 7.1.1 SystemRequirements: GNU make NeedsCompilation: yes Packaged: 2022-10-23 13:44:18 UTC; manu Author: Manuel López-Ibáñez [aut, cre] (), Jérémie Dubois-Lacoste [aut], Leslie Pérez Cáceres [aut], Thomas Stützle [aut], Mauro Birattari [aut], Eric Yuan [ctb], Prasanna Balaprakash [ctb], Nguyen Dang [ctb] Maintainer: Manuel López-Ibáñez Repository: CRAN Date/Publication: 2022-10-23 14:55:06 UTC irace/build/0000755000176200001440000000000014325242462012440 5ustar liggesusersirace/build/vignette.rds0000644000176200001440000000033514325242462015000 0ustar liggesusers} @G,,vQu)Le5[O^0|{LS+dc2,p(-#2:#bx vjoyQ1ϔX)jnzbSJ `T`d?ӸY2x Vx2j:0VlnUu6=h{F,n,K4뾟ؤصirace/build/partial.rdb0000644000176200001440000001611414325242457014574 0ustar liggesusers][wƑRH]e)v6iYC{81%Q$%ӡDyD%hc#*Me9y?>9o?cO~@]L,:YVp.ٻ'7ٯ\n)Xn3:5T#mb[[Mj".h)ˍ [JoXZqcFc6m0/R*iSS ݠ8~~gTCl\z$5UKfHMMN!:; a?h_߬۫/+$]*N~}GonhzssEt[aۊf;@ib5ܱZgvh߄ߡqUzQF{ llcCT߄lVxA=o(! 8 vtWLq h E@X..-?h"P@;3vm?#X>Td5֟ңX>{?&P2C千h`'f晿fG11HO`9"[[S ]qbA"$U@'Wžs|E 3O$>,%|R6wmt YSte8̙C]5h~j25v8G2ťz W/#-b(UЅj+m! /V]tsʜ8Ұlu6Uh4-[w[md ٵ\ K$b\d#B#M!63z!#@J8 @J9ǝF;wYI7%,mSp[@soWC-V̀*]mY UN_9 ~?XYUdy+}q s$Pr)o5nvM0Ifs.;$=Ҷ(mulݗor2i) n!iԁ?>zqT7h ~wQ$kO١fWOyHuKwfu -"s>"KORf dHnO}d/XSL ػL*Nsfr*}TNnKb%յ10 +#4ѬGΙ0Crjd]p hZ8cĚo27#Jh9.B]E7"4ړ^Sv4|{pƫK+xagbY%Hs*WS&-@-1/i]!-k.{'߂.케,}fcG1MQDx x!u[EϐFTd343Gװ|- ]#UR&2#MK1TzC(6rz o" '\mecß!}ì_F?"dWND ^zw\X (2I !Dt5n+ *L)f-6VA}<攠N2P-|^c.@e4! QN $Oܼ&Uj *TNYʷ}w}4[7wXPC#C%aڦn6<P? s(5=4a7YDjJoڦtP S|Ӹ-/P=rsvt/8| 7w`g2Q" l(}Ļ+6sq;%+ cޏ}Uߠ7[3#E@/bbj>""6\r *Gr×G@xϿgL{*CVy蠛.,[ih4#H]l̘A"'n[K۝"䙱$VB8@5'A'u:=Vqo6FrǮ%潭r=o̦voyX L*cv]bj> $Lynl kyu ]m7Lߥ |-YMo{Cag|ombvu6&X;a=&㰙+fXIy4I^01R+޶;Pv6mp,>ݔWjPS;ԜYr?Kwà,z9˔|jH~Ρ>Mm۲dL2fr&XJпp a}J{SU6‡Vz"3'I L%%C3CBɆc[M[i~|y1"Rg A=8qP$Dfx×~_G@ ݰyy6X l-4( ɂ:]HWqم0#G%)cϏЩz#'5|OZ/wT҂ A& wRffԬ[Hvy*)(b5UWzIlot}Ё^BݔljDth!ho!Te,SÞVJ||O'RU!AUa/p?D@ pɲw,w(İtC:Z+UN}2NMOuAh@ nL0wqM0qQ[T}g|4$U|T nQ`/;WDk%od!c$)Ix(|/^+.ʫw1]]T]G,a>+P#-qPxNx6q,~>> [-/AilLa\o/͋]z]M,'zN#?5B\yW-4 u TXJ7. lM?6̚ K1t2}d6/*$SRrWC3\czY/ihvϋ%0f1[^DX, y'kk2M>Ai̇lw CuI!2㊷!Џ);6}[] 4ZyAHb bb a?=We>b-Y Hr;ޢӡ?-ώ) nh;\cN+F%l {dePF"f߻߻k|Ca/`N"=e=Pc cѽ{4@3 悀BBGX1:C: `O#4;tdMg(jwah6OTXة@KXNJ[ɿ¦}\̡gH7!q׈2p>kz#/HM l2ɺްw ZVWVVȆ''+Y kX^K-ٗVZbo3;]7[P=_y\f_+ev *ױzfB8Pcu'jߏP?ɥ0#P#W=W5|WdKT wVtٱS?ГX>X2tALH_rjiQOQӀfj?rG?t S# a)D|[u4cZF:ϫ)NbsTE+XN`d5Rx(9|}.-GΨ'PRzo0<Foe;^|fck|b }iYM6mqՖO9%@`Lf`/݈iwX~?4VxeDM-ˎQts:QaUݧѠ$ycX>Y~@7-[iyiv,H 4Bxe/ a=a}3+7)QLo8D vУX>\ ǰhh@ʨ[&p)ħXGC|=i.z+RXbꕽnvsϰ#>}&:P'|z4nD⫏m??Gs'mE-xHr _ }z(NGYUSSm*c5jB^P(-Pg*T;p]C16 E/Gӫ09Pa߰QGXh/pݰת_7V9ݺ㭹ұ0A)X3m_6;A*tEm U_~ٜ.ǎꉚVJ0 k]:0#tb߿fh+JPfTа |@cj6(!f.&ykÍ6<+T([G`. iajTSe /F*+em*FKY]9B50} >2zǹ\ "5}]Π¦)qUpQ j uUJn4zγ0oCZnlw _;UE dҕ#5ɚԻFN.<j싔vU[ AIQN4)J'4}mӹ&KeE?99-ʞZwBX;##/ίEbotLtۂ6Ux?}^rrʼnQou"/!k?^FQٶϳr{zkcuf6H!"ٹ<r8a8Y಼R SR0q *-e(ޗ݃/39P Z2= 88<0ثgqY l }GcS8 dCD euirace/tests/0000755000176200001440000000000014325242462012503 5ustar liggesusersirace/tests/testthat/0000755000176200001440000000000014325252712014342 5ustar liggesusersirace/tests/testthat/test-bad_scenario.R0000644000176200001440000000042514252665236020064 0ustar liggesuserscontext("bug") skip_on_cran() withr::with_output_sink("test-bad_scenario.Rout", { expect_error(irace.cmdline("--scenario bad_scenario.txt"), "unknown variables") expect_error(irace.cmdline("--scenario scenario.txt --unknown"), "Unknown command-line options: --unknown") }) irace/tests/testthat/scenario.txt0000644000176200001440000000004114252665125016706 0ustar liggesusersparameterFile = "parameters.txt" irace/tests/testthat/test-maxTime.R0000644000176200001440000000350414157104332017045 0ustar liggesuserscontext("irace") withr::with_output_sink("test-maxTime.Rout", { target.runner <- function(experiment, scenario) { configuration <- experiment$configuration tmax <- as.numeric(configuration[["tmax"]]) temp <- as.numeric(configuration[["temp"]]) time <- max(1, abs(rnorm(1, mean=(tmax+temp)/10))) return(list(cost = time, time = time, call = toString(experiment))) } time.irace <- function(...) { args <- list(...) weights <- rnorm(200, mean = 0.9, sd = 0.02) test_weights <- rnorm(2, mean = 0.9, sd = 0.02) parameters <- readParameters(text = ' tmax "" i (1, 50) temp "" r (0, 10) ') scenario <- list(targetRunner = target.runner, instances = weights, testInstances = test_weights, seed = 1234567) scenario <- modifyList(scenario, args) scenario <- checkScenario (scenario) irace:::checkTargetFiles(scenario = scenario, parameters = parameters) confs <- irace(scenario = scenario, parameters = parameters) final_ids <- as.character(sort(confs$.ID.[1:scenario$testNbElites])) expect_gt(nrow(confs), 0L) testing_fromlog(scenario$logFile) load(scenario$logFile) if (scenario$testIterationElites) { # FIXME: We could test here that the correct configurations are tested. expect_gte(ncol(iraceResults$testing$experiments), scenario$testNbElites) } else { test_ids <- sort(colnames(iraceResults$testing$experiments)) expect_equivalent(final_ids, test_ids) } return(confs) } test_that("maxTime=500 testNbElites=2 testIterationElites=FALSE", { generate.set.seed() time.irace(maxTime = 500, testNbElites=2) }) test_that("maxTime=1111 testNbElites=3 testIterationElites=TRUE", { skip_on_cran() generate.set.seed() time.irace(maxTime = 1111, testNbElites=3, testIterationElites=TRUE) }) }) # withr::with_output_sink() irace/tests/testthat/test-targetRunnerParallel.R0000644000176200001440000000422514157104332021577 0ustar liggesuserscontext("targetRunnerParallel") withr::with_output_sink("test-targetRunnerParallel.Rout", { test_that("targetRunnerParallel", { setClasses <- function(x, classes) { class(x) = classes x } # test that targetRunnerParallel works with arbitrary instance objects. targetRunnerParallel = function(experiment, exec.target.runner, scenario, target.runner) { # get our param settings that irace should try cands = lapply(experiment, "[[", "configuration") # the instance is always the same for all different param setting theinst = experiment[[1L]]$instance # we check that we have instances of correct class expect_is(theinst, "foo") # fabricate some random fitness vals ys = rnorm(length(cands)) ys = lapply(ys, function(y) list(cost = y, time = NA_real_)) return(ys) } n.inst = 7L instances = replicate(n.inst, setClasses(list(x=123), classes = "foo"), simplify = FALSE) parameters = readParameters(text=' x1 "" r (0,1) ') log.file = tempfile() tuner.config = list(maxExperiments = 40L, nbIterations = 1L, minNbSurvival = 1, targetRunnerParallel = targetRunnerParallel, instances = instances, logFile = log.file) confs <- irace(scenario = tuner.config, parameters = parameters) expect_gt(nrow(confs), 0L) }) test_that("targetRunnerData", { targetRunnerParallel = function(experiments, exec.target.runner, scenario, target.runner) { cat("a = ", scenario$targetRunnerData$a, ", b = ", scenario$targetRunnerData$b, "\n", sep = "") # get our param settings that irace should try cands = lapply(experiments, "[[", "configuration") # fabricate some random fitness vals ys = rnorm(length(cands)) ys = lapply(ys, function(y) list(cost = y, time = NA_real_)) return(ys) } parameters = readParameters(text=' x "x" i (1,2) ') expect_output( irace(scenario = list(targetRunnerParallel = targetRunnerParallel, instances = lapply(1:5, function(i) 10), targetRunnerData = list(a=1, b=2), maxExperiments = 500), parameters = parameters), "a = 1, b = 2") }) }) # with_output_sink() irace/tests/testthat/test-raceconfs.R0000644000176200001440000000224614157104332017406 0ustar liggesuserscontext("raceconfs") withr::with_output_sink("test-raceconfs.Rout", { parameters.txt <- ' launch_method "--launch_method=" c (L-BFGS-B,SLSQP) visit_method "--visit_method=" c (DA, DE) global_method "--global_method=" c (CMAES,DE) ' confs.txt <- ' launch_method visit_method global_method L-BFGS-B DA CMAES SLSQP DA CMAES L-BFGS-B DE CMAES SLSQP DE CMAES L-BFGS-B DA DE SLSQP DA DE L-BFGS-B DE DE SLSQP DE DE ' target.runner <- function(experiment, scenario) return(list(cost = 100, call = toString(experiment))) withr::with_options(list(warning=2), { parameters <- irace:::readParameters(text=parameters.txt) initconfs <- irace:::readConfigurationsFile(text=confs.txt, parameters=parameters) scenario <- list(targetRunner = target.runner, instances=1:10, nbConfigurations=8, maxExperiments = 96, logFile = "", initConfigurations=initconfs) scenario <- checkScenario (scenario) confs <- irace(scenario = scenario, parameters = parameters) }) }) irace/tests/testthat/test-similar.R0000644000176200001440000000244314157104332017102 0ustar liggesuserscontext("similarConfigurations") test_that("similarConfigurations", { parameters <- irace:::readParameters(text = ' n1 "" r (0,1) n2 "" r (0,1) n3 "" r (0,1) c1 "" c ("a","b") c2 "" c ("a","b") c3 "" c ("a","b") ') confs <- read.table(header = TRUE, stringsAsFactors = FALSE, text = ' .ID. n1 n2 n3 c1 c2 c3 1 0.5 0.5 0.5 "a" "a" "a" 2 NA 0.5 0.5 "a" "a" "a" 3 0.5 0.5 0.5 "a" "a" "b" 4 0.5 0.501 0.5 "a" "a" "a" 5 0.5 0.5 0.499 "a" "a" "a" 6 0.5 0.5 0.5 "a" "a" "a" 7 0.5 0.5 0.5 "a" NA "a" 8 0.5 0.1 0.5 "a" "a" "a" 9 0.5 0.49 0.5 "a" "a" "a" 10 0.5 0.5 0.5 "a" "a" NA 11 0.5 0.5 0.5 "a" "a" NA 12 NA 0.5 0.5 "a" "a" "a" ') expect_identical( irace:::similarConfigurations(confs[1:10,], parameters, threshold = 0.001), as.integer(c(1,6))) expect_identical( irace:::similarConfigurations(confs[1:10,], parameters, threshold = 0.01), as.integer(c(1,4,5,6))) expect_identical( irace:::similarConfigurations(confs[1:10,], parameters, threshold = 0.1), as.integer(c(1,4,5,6,9))) expect_identical( # FIXME: The output should be already sorted sort(irace:::similarConfigurations(confs, parameters, threshold = 0.001)), as.integer(c(1,2,6,10,11,12))) }) irace/tests/testthat/helper-common.R0000644000176200001440000000061414157104332017230 0ustar liggesusers# This file is loaded automatically by testthat. generate.set.seed <- function() { seed <- sample(2^30, 1) cat("Seed: ", seed, "\n") set.seed(seed) } test_irace_detectCores <- function() { if (identical(Sys.getenv("NOT_CRAN"), "true")) { x <- Sys.getenv("_R_CHECK_LIMIT_CORES_", "") if (nzchar(x) && x == "TRUE") return(2L) return(parallel::detectCores()) } return(1L) } irace/tests/testthat/saved_maxim_bug2.rds0000644000176200001440000001547714157104332020303 0ustar liggesusers] xU@ d`IWR]u{ \@@dGIn;aQDT긃"3:O EDqfWTݺmӭ>}Xշ*sqZbic),*)LvWS^;[,.oxE^zIJJj3%JT-n|&컴c ##8]2ɝ::77szbTCzWƞٸ'KeU3*}C̵e9soC}g#8Gp~3o*fU^r|v gѱ2}^D0gPe~:ơ6e/ޔ̅{0d˯TYW_<>5mwi:3OĨ݊T>6S~0 >K qi3(ks'+':RxI] ^_o3_}Mkw>΃z  #WBW<3دgd};qXT |Cs> q 4M<#/}>҅}'Թ5q\=z]oˍǀ!qS2 U>ga dS`}a߫ė1#8盏qLD !nT0/Q>t>WS?Yy\ 'p5yWnTuq!uhx?Mda6 O^[g#@L\݀*qn~_O[Li <1ay;JE{ߚ6?ҖYc[Я pA#E8*LA|埔< I^4wѐq_;.j_ᶔ!nsx<(GY̯4/QŝkrǨ>q4GT:<*~7Gk8?@/( e>Kv*_l?o󊯨W3a>y R>4TTe+lfT rrWv-)JSO|/u#ʂD׹5؍ Gk̼c-i~oԎG|ษUrVs#ߵےES9/{z-*ϥP{S/1}au=kf9 78D 5ՙqS7ȍ9鸨Am;?̚TcbIh&< ~ PuK'&y71߆~!SE<]AX+RA_<v+D_7Bio+.2sQ{|pqG݇swoFnMŴ>_8d gy.-uԚ9N<EòSJrߤOZ~^zXc-xʐgIZMy߀]}Ft9[:㸟=k>BN/p?C`'~z <0룝 ̇ YZqQ3򇃯'sy9)6š.a;?r䡦c 3<3TY('\f@1<_;nߌVs>pZJg>R-8J~y)};_|l98no?wMMە<|=@NgDM4Ty -u>-- |OiG\Y;'ʧzܘS|@s='<\#8^r~>;|^eօ:8!y7>>iţ. -跇w|-*s9A\e[|j[K絟ʻsכ.1 >YvSNC * Wqjp(>eTGu9΋H߀&YyI37E?D\m?$KۭN_ߧk&_6 ^5&lJհNGsˋk`_ʚO3vo_ ?n?Nvsx6U/7K+pD|I8Ë>#wEA/**.;\*>8lN?j+A>z {|B;"J䓋~fR|An7@y [^77 ̨/˰n/Lm0]լ _Ŀ[$<~$X'nsu#>4ęj+7wq>by^o~}㐗& 50 VRG|z/|:{“LMu/*x]{N|rYUx CAߞ'lߐ5;뉢y}Z7p\DijY̓%ѫ|oRFm(Y:pzdI;U+WPRuɼ h7nƣxvnM%޴QdN~|7Σy;~At cD :1At cu 1:[pnícu 1<:Gãcxt 1<:W^ëcxu 1:W>çct 1|:O>çcu 1:_~ïcu 1:F@#ct1:FQcT:FQcT:FQa*nvKحzحح-CS04) Mah CSC# 04C# 04C# < < < < < м м м м м | | | | | - - - - - U2JV*Z%Cdh 1.!K¸0.!K¸0.!K¸0.!K¸0.!K¸0.!K¸0.!K¸0.!K¸0.!K¸0.!K¸0.!K¸0.!K%ˏ,ȿ#;enuxCk)}+qe Kۍ۹{W5%lH}𾯯w$\s}F^18o\35g-urak &tGǟw/8!c~YS^8|-_Z8MF:oܘoRrY/,d q{/sGw|~hￎo_KHvtՁcNz`ӵ l5h7 ~OU]sfL5EWvhSӆ} bNE9aШONki]{ [󗒂?|מ3=?s郧_:gSodĮCm?kwz+z("sOZ~3nך,>y7 -lmݷg|nYO]xؙ킵 u².>vA:8vƳ?36S3?Ǧ-/7l3dL5gus8?ߨ}#_/L}glD톹7vv]s]=f|(W{'7{ڝ>j Tg~؁(|Cz=>&9{ʫ%KbqC3ro3ni;nMuûvU9C/ukXzT= =}@m{>dcЭiF |`Yk#}KVzzU[XŒv>Mњ~.JBY!+i]j"5SAivB}K&oKVY}C* Ǣ`u8N͟~W%Pa/0od^ҞL5DѺ T8 Nj œӮN':0> (?=3DU(3?vڇ۵CQT%xx(Mks(MC[.ݖ_P篇ҴVRSD$rڽoSC!5?hN?Q-Ek9hϋɑyj܄R*&gP<^0'J5scCiz|0 B hϒh`"uOg?5.fFret\D&$MLuYθP}IT-WH@aC2,'^YV?օ RDy(69CHCF;kcrC{s'ha??~蒌~v"F>>OG /3.A?};v1ӈ~C:lvpFB5AbT L4&C*0y`~>JrPʙhY E${ JοҜ"<r\cሖuU+)a䰖t~j#Cp}(ث4Y|3Ya]GbuDT뢛Ju 0|ëTjֆ391RmU멋ih=\և\Rnj E4-IU3:k`Sdj.'Q{!cܩ>=zRC1F 2;FO;զEp2t,#Bz"}٩*b0&IsbZ] j*w -M?8zu1tϼ pC꧰>lY1zqM0W67hxu!Zkhi8ٸ3M[4yhr}id*jғb`M$\0IG. 覮Lj8?QM&IO[cx~5g#tQ--=zmjo*R4^ &UFpt`C h|"``D]3"A5PJ;i8  [&5Zd.P+V7~餳g% FC刉irace/tests/testthat/setup.R0000644000176200001440000000027414157104332015625 0ustar liggesusersold_opts = options( warnPartialMatchArgs = TRUE, warnPartialMatchAttr = TRUE, warnPartialMatchDollar = TRUE ) old_opts = lapply(old_opts, function(x) if (is.null(x)) FALSE else x) irace/tests/testthat/bug_large_new_instances-confs.txt0000644000176200001440000000050314157104332023053 0ustar liggesusersp pe pm rhoe alpha beta dest nodes prob 181 0.185 0.260 0.855 0.128 0.958 0.649 139 0.158 123 0.149 0.297 0.724 0.115 0.785 0.520 125 0.031 160 0.130 0.277 0.748 0.109 0.810 0.535 92 0.077 143 0.205 0.330 0.742 0.167 0.786 0.658 105 0.130 131 0.118 0.242 0.742 0.165 0.929 0.663 142 0.084 irace/tests/testthat/test-readconfs.R0000644000176200001440000000130614157104332017403 0ustar liggesuserscontext("Test read configurations") withr::with_output_sink("test-readconfs.Rout", { test_that("checkDuplicates", { params <- irace:::readParameters("parameters.txt") expect_error(irace:::readConfigurationsFile(text=' param1 param2 mode real mutation 5 NA "x2" 4.0 "low" 1 NA "x2" 4.0 "low" 5 NA "x2" 4.0 "low" NA NA "x3" 4.5 "low" ', parameters = params), "Duplicated") expect_error(irace:::readConfigurationsFile(text=' param1 param2 mode real mutation 5 NA "x2" 4.0 "low" 1 NA "x2" 4.0 "low" 5 NA "x2" 4.0 "low" 1 NA "x2" 4.0 "low" ', parameters = params), "Duplicated") }) }) irace/tests/testthat/teardown.R0000644000176200001440000000002214157104332016277 0ustar liggesusersoptions(old_opts) irace/tests/testthat/forbidden.txt0000644000176200001440000000117514157104332017040 0ustar liggesusers## Template for specifying forbidden parameter configurations in irace. ## ## This filename must be specified via the --forbidden-file command-line option ## (or forbiddenFile in scenario.txt). ## ## The format is one constraint per line. Each constraint is a logical ## expression (in R syntax). If a parameter configuration ## is generated that makes the logical expression evaluate to TRUE, ## then the configuration is discarded. ## ## Examples of valid logical operators are: == != >= <= > < & | ! %in% param1 < 5 & mode == "x1" (param2 > 6 & mode == "x1") | (param2 <= 6 & mode == "x3") real < 4 & mode %in% c("x2", "x3") irace/tests/testthat/bad_scenario.txt0000644000176200001440000000004214203753710017507 0ustar liggesusersparametersFile = "parameters.txt" irace/tests/testthat/dependencies.txt0000644000176200001440000000057114157104332017531 0ustar liggesusersparam1 "--param1 " i (1, "real") param2 "--param2 " i ("real", 10) mode "--" c ("x1" ,"x2", "x3") real "--paramreal=" r (1.5, 7.5) mutation "--mutation=" o ("none", "very low", "low", "medium", "high", "very high", "all") #unused "-u " c (1, 2, 10, 20) irace/tests/testthat/test-path.R0000644000176200001440000001700514157104332016376 0ustar liggesuserscontext("path_rel2abs") test_that("test.path_rel2abs", { # Try to set wd; otherwise fail silently. old.cwd <- getwd() skip_if(is.null(old.cwd)) on.exit(setwd(old.cwd), add = TRUE) tryCatch(setwd("/tmp"), error = function(e) { skip(e) }) testcases <- read.table(text=' "." "/tmp" "/tmp" ".." "/tmp" "/" "../" "/tmp" "/" "../." "/tmp" "/" "../.." "/tmp" "/" "../../" "/tmp" "/" "../../x.r" "/tmp" "/x.r" "../leslie/" "/tmp" "/leslie" "../leslie/x.r" "/tmp" "/leslie/x.r" "../x.r" "/tmp" "/x.r" "..irace" "/tmp" "/tmp/..irace" "./" "/tmp" "/tmp" "./." "/tmp" "/tmp" "./" "/tmp/" "/tmp" "./." "/tmp/" "/tmp" "././x.r" "/tmp" "/tmp/x.r" "./irace/../x.r" "/tmp" "/tmp/x.r" "./x.r" "/tmp" "/tmp/x.r" ".x.R" "/tmp" "/tmp/.x.R" "/./x.r" "/tmp" "/x.r" "/home" "/tmp" "/home" "/home/leslie/././x.r" "/tmp" "/home/leslie/x.r" "/home/leslie/~/x.r" "/tmp" "/home/leslie/~/x.r" "/~/x.r" "/tmp" "/~/x.r" "e:/home/leslie/x.r" "/tmp" "e:/home/leslie/x.r" "leslie/leslie/../../irace" "/tmp" "/tmp/irace" "x.r" "/tmp" "/tmp/x.r" "~/irace/../x.r" "/tmp" "~/x.r" "~/x.r" "/tmp" "~/x.r" "../../../data" "./" "/data" "../../../data" "/tmp/a/b/c/" "/tmp/data" "..//a" ".//" "/a" "R" "/tmp/" "Sys.which" ', stringsAsFactors=FALSE) for(i in 1:nrow(testcases)) { orig <- testcases[i,1] cwd <- testcases[i,2] res <- irace:::path_rel2abs(testcases[i,1], cwd) if (testcases[i,3] == "Sys.which") { exp <- normalizePath(Sys.which(testcases[i,1]), winslash = "/", mustWork = NA) } else { exp <- gsub("\\", "/", path.expand(testcases[i,3]), fixed = TRUE) } if (res == exp) { # cat("[OK] (", orig, ", ", cwd, ") -> ", res, "\n", sep="") } else { cat("[FAILED] (", orig, ", ", cwd, ") -> ", res, " but expected: ", exp, "\n") } expect_match(res, exp, fixed = TRUE) } }) test_that("test.path_rel2abs for windows", { testcases <- read.table(text=' . N:\\\\tmp N:/tmp .. N:\\\\tmp N:/ ..\\\\ N:\\\\tmp N:/ ..\\\\. N:\\\\tmp N:/ ..\\\\.. N:\\\\tmp N:/ ..\\\\..\\\\ N:\\\\tmp N:/ ..\\\\..\\\\x.r N:\\\\tmp N:/x.r ..\\\\leslie\\\\ N:\\\\tmp N:/leslie ..\\\\leslie\\\\x.r N:\\\\tmp N:/leslie/x.r ..\\\\x.r N:\\\\tmp N:/x.r ..irace N:\\\\tmp N:/tmp/..irace .\\\\ N:\\\\tmp N:/tmp .\\\\. N:\\\\tmp N:/tmp .\\\\ N:\\\\tmp\\\\ N:/tmp .\\\\. N:\\\\tmp\\\\ N:/tmp .\\\\.\\\\x.r N:\\\\tmp N:/tmp/x.r .\\\\irace\\\\..\\\\x.r N:\\\\tmp N:/tmp/x.r .\\\\x.r N:\\\\tmp N:/tmp/x.r .x.R N:\\\\tmp N:/tmp/.x.R . N:\\tmp N:/tmp .. N:\\tmp N:/ ..\\ N:\\tmp N:/ ..\\. N:\\tmp N:/ ..\\.. N:\\tmp N:/ ..\\..\\ N:\\tmp N:/ ..\\..\\x.r N:\\tmp N:/x.r ..\\leslie\\ N:\\tmp N:/leslie ..\\leslie\\x.r N:\\tmp N:/leslie/x.r ..\\x.r N:\\tmp N:/x.r ..irace N:\\tmp N:/tmp/..irace .\\ N:\\tmp N:/tmp .\\. N:\\tmp N:/tmp .\\ N:\\tmp\\ N:/tmp .\\. N:\\tmp\\ N:/tmp .\\.\\x.r N:\\tmp N:/tmp/x.r .\\irace\\..\\x.r N:\\tmp N:/tmp/x.r .\\x.r N:\\tmp N:/tmp/x.r .x.R N:\\tmp N:/tmp/.x.R . N: N:/ .. N: N:/ ..\\\\ N: N:/ ..\\\\. N: N:/ ..\\\\.. N: N:/ ..\\\\..\\\\ N: N:/ ..\\\\..\\\\x.r N: N:/x.r ..\\\\leslie\\\\ N: N:/leslie ..\\\\leslie\\\\x.r N: N:/leslie/x.r ..\\\\x.r N: N:/x.r ..\\ N: N:/ ..\\. N: N:/ ..\\.. N: N:/ ..\\..\\ N: N:/ ..\\..\\x.r N: N:/x.r ..\\leslie\\ N: N:/leslie ..\\leslie\\x.r N: N:/leslie/x.r ..\\x.r N: N:/x.r ..irace N: N:/..irace .\\\\ N: N:/ .\\\\. N: N:/ .\\\\ N:\\\\ N:/ .\\\\. N:\\\\ N:/ .\\\\.\\\\x.r N: N:/x.r .\\\\irace\\\\..\\\\x.r N: N:/x.r .\\\\x.r N: N:/x.r .\\ N: N:/ .\\. N: N:/ .\\ N:\\ N:/ .\\. N:\\ N:/ .\\.\\x.r N: N:/x.r .\\irace\\..\\x.r N: N:/x.r .\\x.r N: N:/x.r .x.R N: N:/.x.R . N:/tmp N:/tmp .. N:/tmp N:/ ../ N:/tmp N:/ ../. N:/tmp N:/ ../.. N:/tmp N:/ ../../ N:/tmp N:/ ../../x.r N:/tmp N:/x.r ../leslie/ N:/tmp N:/leslie ../leslie/x.r N:/tmp N:/leslie/x.r ../x.r N:/tmp N:/x.r ..irace N:/tmp N:/tmp/..irace ./ N:/tmp N:/tmp ./. N:/tmp N:/tmp ./ N:/tmp/ N:/tmp ./. N:/tmp/ N:/tmp ././x.r N:/tmp N:/tmp/x.r ./irace/../x.r N:/tmp N:/tmp/x.r ./x.r N:/tmp N:/tmp/x.r .x.R N:/tmp N:/tmp/.x.R D:/./x.r N:/tmp D:/x.r D:\\\\.\\\\x.r N:/tmp D:/x.r D:\\.\\x.r N:/tmp D:/x.r D: N:/tmp D:/ D:\\\\ N:/tmp D:/ D:/ N:/tmp D:/ D:/leslie/././x.r N:/tmp D:/leslie/x.r D:/leslie/~/x.r N:/tmp D:/leslie/~/x.r e:/home/leslie/x.r /tmp e:/home/leslie/x.r leslie/leslie/../../irace N:/tmp N:/tmp/irace x.r N:/tmp N:/tmp/x.r ~/irace/../x.r N:/tmp ~/x.r ~/x.r N:/tmp ~/x.r "R" "/tmp/" "Sys.which" ', stringsAsFactors=FALSE) for(i in 1:nrow(testcases)) { orig <- testcases[i,1] cwd <- testcases[i,2] res <- path_rel2abs(testcases[i,1], cwd) if (testcases[i,3] == "Sys.which") { exp <- normalizePath(Sys.which(testcases[i,1]), winslash = "/", mustWork = NA) } else { exp <- gsub("\\", "/", path.expand(testcases[i,3]), fixed = TRUE) } if (res == exp) { #cat("[OK] ", i, ": path_rel2abs(\"", orig, "\", \"", cwd, "\") -> ", res, "\n", sep="") } else { cat("[FAILED] ", i, ": path_rel2abs(\"", orig, "\", \"", cwd, "\") -> ", res, " but expected: ", exp, "\n") } expect_match(res, exp, fixed = TRUE) } }) irace/tests/testthat/test-sann-irace.R0000644000176200001440000000631314315062057017465 0ustar liggesuserscontext("irace") withr::with_output_sink("test-sann-irace.Rout", { ## Functions ########################################################## f_rosenbrock <- function (x) { d <- length(x) z <- x + 1 hz <- z[1:(d - 1)] tz <- z[2:d] s <- sum(100 * (hz^2 - tz)^2 + (hz - 1)^2) return(s) } f_rastrigin <- function (x) { sum(x * x - 10 * cos(2 * pi * x) + 10) } ## target runner ########################################################### target.runner <- function(experiment, scenario) { debugLevel <- scenario$debugLevel configuration.id <- experiment$id.configuration instance.id <- experiment$id.instance seed <- experiment$seed configuration <- experiment$configuration instance <- experiment$instance D <- 3 par <- runif(D, min = -1, max = 1) fn <- function(x) { weight <- instance return(weight * f_rastrigin(x) + (1 - weight) * f_rosenbrock(x)) } tmax = 1 + as.numeric(configuration[["tmax"]]) temp = 11.0 + as.numeric(configuration[["temp"]]) stopifnot(tmax > 0) stopifnot(temp > 0) res <- optim(par, fn, method = "SANN", control = list(maxit = 10, tmax = tmax, temp = temp)) result <- list(cost = res$value, call = toString(experiment)) return(result) } ## target runner ########################################################### target.runner.reject <- function(experiment, scenario) { if (runif(1) <= 0.05) return (list(cost = -Inf, call = toString(experiment))) return (target.runner(experiment, scenario)) } ## Run function ######################################################## sann.irace <- function(log.param=FALSE, ...) { args <- list(...) # tmax and temp must be > 0 if (log.param) parameters.table <- ' tmax "" i,log (1, 5000) temp "" r,log (1, 100) ' else parameters.table <- ' tmax "" i (1, 5000) temp "" r (1, 100) ' parameters <- readParameters(text = parameters.table) scenario <- list(targetRunner = target.runner, maxExperiments = 1000, seed = 1234567) scenario <- modifyList(scenario, args) scenario <- checkScenario (scenario) confs <- irace(scenario = scenario, parameters = parameters) best.conf <- getFinalElites(scenario$logFile, n = 1, drop.metadata = TRUE) expect_identical(removeConfigurationsMetaData(confs[1, , drop = FALSE]), best.conf) } test_that("parallel", { skip_on_cran() # Reproducible results generate.set.seed() weights <- rnorm(200, mean = 0.9, sd = 0.02) sann.irace(instances = weights, parallel = test_irace_detectCores()) }) test_that("parallel reject", { # Reproducible results generate.set.seed() weights <- rnorm(200, mean = 0.9, sd = 0.02) sann.irace(instances = weights, parallel = test_irace_detectCores(), targetRunner = target.runner.reject) }) test_that("deterministic", { skip_on_cran() # Reproducible results generate.set.seed() weights <- rnorm(200, mean = 0.9, sd = 0.02) sann.irace(deterministic = TRUE, instances = weights[1:7]) }) test_that("log", { skip_on_cran() # Reproducible results generate.set.seed() weights <- rnorm(200, mean = 0.9, sd = 0.02) sann.irace(log.param=TRUE, instances = weights) }) }) # withr::with_output_sink() irace/tests/testthat/test-bugs.R0000644000176200001440000001315714157104332016406 0ustar liggesuserscontext("bugs") withr::with_output_sink("test-bugs.Rout", { run_irace_from_rds <- function(rds_filename) { # FIXME: For some unknown reason, this test doesn't work in R 3.6 skip_if(grepl("^3.6", getRversion())) skip_if(grepl("^4", getRversion())) scenario <- readRDS(rds_filename)$scenario parameters <- readRDS(rds_filename)$parameters scenario$targetRunner <- function(experiment, scenario, filename = rds_filename) { saved <- readRDS(filename) row <- which(saved$instancesList[, "instance"] == experiment[["id.instance"]] & saved$instancesList[, "seed"] == experiment[["seed"]]) cost <- saved$experiments[row, experiment[["id.configuration"]] ] if (length(cost) == 0 || is.na(cost)) { print(row) print(experiment) } expect_false(is.na(cost)) return(list(cost = cost)) } confs <- irace(scenario = scenario, parameters = parameters) expect_gt(nrow(confs), 0L) } test_that("bug_large_new_instances", { skip_on_cran() # FIXME: For some unknown reason, this test doesn't work in R 3.6 skip_if(grepl("^3.6", getRversion())) skip_if(grepl("^4", getRversion())) # FIXME: Convert this test to use run_irace_from_rds() load("bug_large_new_instances.Rdata", verbose = TRUE) scenario$targetRunner <- function(experiment, scenario) { saved_instances_list <- dynGet("saved_instances_list", inherits = TRUE) saved_experiments <- dynGet("saved_experiments", inherits = TRUE) row <- which(saved_instances_list[, "instance"] == experiment[["id.instance"]] & saved_instances_list[, "seed"] == experiment[["seed"]]) cost <- saved_experiments[row, experiment[["id.configuration"]] ] if (length(cost) == 0 || is.na(cost)) { print(row) print(experiment) } expect_false(is.na(cost)) return(list(cost = cost)) } confs <- irace(scenario = scenario, parameters = parameters) expect_gt(nrow(confs), 0L) }) test_that("target.runner as string", { target.runner.local <- function(experiment, scenario) return(list(cost=1L)) expect_true(irace:::is.function.name("target.runner.local")) # Test that a function can be given as a string. scenario <- list(targetRunner = "target.runner.local", instances = 1:10, maxExperiments = 1000) scenario <- checkScenario (scenario) expect_equal(scenario$targetRunner, target.runner.local) expect_is(scenario$targetRunner, "function") }) target.runner.global <- function(experiment, scenario) return(list(cost=1L)) test_that("target.runner as string (global)", { expect_true(irace:::is.function.name("target.runner.global")) # Test that a function can be given as a string. scenario <- list(targetRunner = "target.runner.global", instances = 1:10, maxExperiments = 1000) scenario <- checkScenario (scenario) expect_equal(scenario$targetRunner, target.runner.global) expect_is(scenario$targetRunner, "function") }) test_that("ordered assert", { library(irace) parameters <- readParameters(text=' x "" o (a,b,c,d) ') confs <- irace:::sampleUniform(parameters, 1, 0) confs$.ID. <- 1 model <- irace:::initialiseModel(parameters, confs, 0) confs <- irace:::sampleModel(parameters, confs, model, 1, 0) expect_true(confs$x %in% parameters$domain$x) }) test_that("maxim_bug", { skip_on_cran() run_irace_from_rds("saved_maxim_bug.rds") }) test_that("maxim_bug2", { skip_on_cran() run_irace_from_rds("saved_maxim_bug2.rds") }) test_that("non-normal", { skip_on_cran() df <- as.matrix(read.table(text=' 0.007858276 0.007934570 0.007949829 0.009384155 0.009521484 0.009719849 0.011520386 0.012100220 0.012176514 0.214721680 0.219146729 0.219512939 0.005462646 0.005462646 0.005523682 0.005035400 0.005035400 0.005035400 0.011001587 0.011367798 0.011962891 0.153244019 0.155212402 1.157546997 0.008636475 0.008789062 0.008666992 0.008239746 0.008499146 0.008560181 0.010848999 0.011077881 0.011703491 0.191589355 0.191879272 0.191589355 0.011245728 0.011245728 0.011245728 0.006454468 0.006454468 0.006561279 0.010330200 0.010894775 0.011032104 0.160491943 0.164169312 0.167068481', header=FALSE)) ## colnames(df) <- 1:3 ## #df <- log(df) ## library(ggplot2) ## library(tidyr) ## df_long <- as.data.frame(df) ## df_long$run <- 1:16 ## df_long <- df_long %>% gather("config", "y", as.character(1:3)) ## ggplot(df_long, aes(df=y)) + ## geom_histogram(aes(y=..density..), colour="black", fill="white", binwidth=0.01)+ ## geom_density(alpha=.2, fill="#FF6666") + ## facet_grid(config ~ .) ## ggplot(df_long, aes(x=config, y=y)) + geom_boxplot() + geom_jitter() ## aggregate(y ~ config, data = df_long, FUN = function(x) c(mean = mean(x), sd = sd(x), median=median(x), ## shapiro.test = shapiro.test(x)$p.value) ) ## t.test(x[,1], x[,2],paired=TRUE) ## t.test(x[,1], x[,3],paired=TRUE) ## wilcox.test(x[,1], x[,2],paired=TRUE) ## wilcox.test(x[,1], x[,3],paired=TRUE) ## print(all(x[,1] <= x[,2])) ## print(all(x[,1] <= x[,3])) parameters <- readParameters(text='p "" c (1,2,3,4,5)') scenario <- list(instances = 1:16, maxExperiments = 50, testType="t-test") scenario$targetRunner <- function(experiment, scenario, this_df = df) { if (experiment[["id.configuration"]] > 3) cost <- as.numeric(experiment[["id.configuration"]]) else cost <- this_df[as.numeric(experiment[["id.instance"]]), as.numeric(experiment[["id.configuration"]]) ] return(list(cost=cost)) } confs <- irace(scenario = scenario, parameters = parameters) print(confs) expect_equal(confs$.ID., 1L) }) }) # withr::with_output_sink() irace/tests/testthat/configurations.txt0000644000176200001440000000056714157104332020142 0ustar liggesusersparam1 param2 mode real mutation 5 NA "x2" 4.0 "low" 1 NA "x1" 4.0 "low" 1 NA "x2" 4.0 "low" 5 7 "x1" 3.5 "low" 5 6 "x1" 3.5 "low" NA 7 "x3" 3.5 "low" NA 6 "x3" 3.5 "low" 5 NA "x2" 2.0 "low" NA NA "x3" 1.5 "low" NA NA "x3" 4.5 "low" irace/tests/testthat/bug_large_new_instances.Rdata0000644000176200001440000001531214157104332022165 0ustar liggesusers xEߴiLi)PB4GڈR6)ILAUoխzDuG>몫v=Y/73oޤBѾ̼73wy̭Zl\)BM"3΀r$[|ėjhHiR@9Nri;">1G:89I+!;=i^A;'PtP[\7Ĵ[i&]qqTd3d1Ȳ0d-y]f`$)D+_P__yOT?G^qN'ɰv^x pgw33Lɵ+Q,vEbB|`qw8ȄVi&# 7|xuyBy3nfSɚ8['^zUr]nkx݄b]d RAn3)Ɔo}tƺYFR{Hf4z[ob ޙؑqFN+6 ƩsL'mٌF&7 +J1ko:STF{\E)t]FE$*1 w%$[B^9L rl U-gt'y\獣{qz|DJ(}V O!^kMd*$r&澍M퍥;=VE/4ő%){%>juݪZi䵇jh5-Vvꠥ( ]y][U֦}MyVq mr[Ai,y=y5% 8xɅMov]v w4*e'`l]I|Ԣx(hl$08tCg >F=s':K|2ȻTO:?žju-.`,-dr#M}o?7筓M;@k@VvU #ƞ{3FY>S7%iu ޺3+8i-,' /}qEQāu1#&xSZNlCx˜'+)>M7twkuHąwPܫ\IW.9uEC1+*;w,,x] NΑUq|r%HӃQZLŻs->Ȧ][%ΠoQAg۵2nG}7M-"- 0Qw;cl WԏG@|@ (;1*&$ǻ]Cdv+Q;%mB]D? &yvDHk8][]yljT ƎGnDC6>'%`ټ;5yljT6G4 }B c }vJ7r`iDI/DH[I4!G|m\wHutkyljT wJ~LwHutDDq"!7v+;;N:L| Ǝ FF ϕKDq"ao)wHuȾ?T0V;N:d?F`I\wHu~`3wHux#.DC%} H'R;DbT~WS 2 Ic:q')QK:$Fx#ׇ|6t?/vyljTtpǑwHux0;~DþH'RvI'Rz`S"80 ?0 ;N:E6aJ'R&~_ ;N:dA(DC~u7J'R&~?vI'RCq'wJL|Ui~@$~@$~@@D? HD? HD? 臣}͋Z9~@$~@$~':ډ~@$~@$~':ډ~@$~@jCm $~@yD? H'Σ$~@>q-D? HD?h!~@$OG HD? |~HN~}C 00!9ya`KxOD?$'yA? L|00GCrć~{^D?$'yQ{G t\NJ7NF?O~K!>?fH7GV<~8ͣ}@CYW۟z\MZA/2i#;*bP Սה$$QF$$Q*)Mi,4e)MijjjjjjjjjjjQ2QF5ʨF(eTjQ2aaaaaaVaVaVa%AK%KհR +հQ հQ հQ հQ հQ հS ;հS ;հS ;հS ;հS ;(TjSrQN5ʩF9(D#%KYĒf,cI KZYƒvdjL2RVJZ)S+ejL12VʘZS+cjeL12fajfajfajfajfajfejVfejVfejVfejVfejVfcj6fcj6fcj6fcj6fcj6fgjvfgjvfgjvfgjvfgjvVʙZ9S+gjL3rVZbbk%&Zbbk%&Zbbk%&Zbbk%&Zbbk%&ZbbkI\KbG_t]+=`wW6y!6݋OM1 yk$WVY-&+C| mX _Ql!)n& hfy ,X Ц4 |Kb 풘 ùh/&43N!YSeC{bIσ6eFlId|t2@yPsEbɂ lҁ͔WCb)il+\i=4P'|I1 ˕2$ AK8σr-_uv}d9K<+ s> Ʊ4 -r=v@+ dqBy.+g2į9 FĐ Oɯ|{'klr=l, p~A%}&^#6y3dɵ Jr imo v3-qqMzAb!G\k*ɼuK拌h#to O_y% L˓YBzIƘB׹\yA` yVg#Ӛ >+O;f7 Opo1neW|z# l.{owmu|{wKs'W*׶Ω~^ Z^ܻ#>J,G6|S=ذՂړGN>YCsVN?uǰ[mR11tbϊA :~چ[jZ½/gj/Ow⾴ ׭6_ǝ!s"Ŗ w?%N-1B,e/hfkj۳wiɢpd/K}cmG^ms;hӔknVquiw7>;KW[J۟S&O}3?gìfribAnns6^/tZk%۾8uĚm(j^=eMi796kͅ [bK =~A^MW jЗ}š=ߥ5'ϯ#Wuu#2rK's |-#5,߼ᵋ_PmWunޜw׏ l.yg }SOmyo_=mozm? [oYtb}uMk,;y{Օen?㷧Xp=%ƴ=Wh_+0tU7na̮56__[ Ψ;NnȲm4m/rcEVMu;pKUwљMr5VTşU-5{;㊡,fjr~O0jE_Olڃ<׋m7,X 7/{aϭYmڏ5˿>sN)9u7;%{iKf oo8іI.j_|kJ{4W3qEN "cRm gwv;|Eerǽ5:w5ksnny/rsEN,kյ~j"o՟|RNZy[?rox5ϑ2=my-5}5;᧿c}- t{Plu69Ggyjgf6ni[zvcr>V}MsxYW3Ş1KWSěg^<Ğ2Aۥx|mG)ҍCXL(9! 9'5S1N8-girace/tests/testthat/test-bug-13.R0000644000176200001440000000402614157104332016437 0ustar liggesuserscontext("bug") skip_on_cran() # https://github.com/MLopez-Ibanez/irace/issues/13 withr::with_output_sink("test-bug-13.Rout", { parameters.txt <- ' algorithm "--" c (as,mmas,eas,ras,acs) localsearch "--localsearch " c (0, 1, 2, 3) alpha "--alpha " r (0.00, 5.00) beta "--beta " r (0.00, 10.00) rho "--rho " r (0.01, 1.00) ants "--ants " i (5, 100) q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" rasrank "--rasranks " i (1, 100) | algorithm == "ras" elitistants "--elitistants " i (1, 750) | algorithm == "eas" nnls "--nnls " i (5, 50) | localsearch %in% c(1,2,3) dlb "--dlb " c (0, 1) | localsearch %in% c(1,2,3) ' parameters <- irace:::readParameters(text=parameters.txt) lookup <- readRDS("bug-13-lookup.rds") counter <- 0 target.runner <- function(experiment, scenario) { configuration <- experiment$configuration instance <- experiment$instance switches <- experiment$switches debugLevel <- scenario$debugLevel args <- paste(instance, trimws(irace:::buildCommandLine(configuration, switches))) pos <- pmatch(args, lookup[,1]) if(!is.na(pos)) { cost <- as.numeric(lookup[pos, 2]) counter <<- counter + 1 } else { cost <- 10 } if(debugLevel > 0) { cat("# ", args, "\n") cat(cost, "\n") } return(list(cost = cost, call = toString(experiment))) } withr::with_options(list(warning=2), { scenario <- list(instances = scan(what="", quiet=TRUE, text=" 2000-121.tsp 2000-122.tsp 2000-123.tsp 2000-124.tsp 2000-125.tsp 2000-126.tsp 2000-127.tsp 2000-128.tsp 2000-129.tsp 2000-130.tsp"), trainInstancesDir = "", trainInstancesFile = "", maxExperiments = 3000, digits = 3, seed = 2357, deterministic = 1, elitist = 0, targetRunner = target.runner, debugLevel=3) scenario <- checkScenario (scenario) confs <- irace(scenario = scenario, parameters = parameters) expect_equal(counter, 548) }) }) irace/tests/testthat/test-dependencies.R0000644000176200001440000000760214157104332020072 0ustar liggesuserscontext("Test dependencies") withr::with_output_sink("test-dependencies.Rout", { test_that("param depend error checking", { expect_error_readParameters <- function(text, error) expect_error(readParameters(text=text), error) expect_error_readParameters(' p1 "" r (0, 1) p2 "" r (p3, 1) ', "parameter 'p2' is not valid: 'p3' cannot be found") expect_error_readParameters(' p1 "" c (0, 1) p2 "" r (1, p1) ', "parameter 'p2' depends on non-numerical parameters") expect_error_readParameters(' p1 "" r (0, 1) p2 "" r (1, foobar(p1)) ', "parameter 'p2' uses function") expect_error_readParameters(' p1 "" r (0, p3) p2 "" r (1, p1) p3 "" r (p2, 1) ', "cycle detected") expect_error_readParameters(' p1 "" r (0, 1) p2 "" i (0.1, p1) ', "values must be integers") }) checkConditionalAndDependency <- function(configuration, parameters) { namesParameters <- names(parameters$conditions) for (p in namesParameters) { if (!irace:::conditionsSatisfied(parameters, configuration, p)) { expect(is.na(configuration[,p]), paste0("Conditional parameter '", p, "' is not active but it has a value '", configuration[,p], "' assigned.")) } else if (parameters$isDependent[p]) { bounds <- irace:::getDependentBound(parameters, p, configuration) if (anyNA(bounds)) { expect(is.na(configuration[,p]), paste0("Dependent parameter '", p, "' has a value '", configuration[,p], "' but it should be inactive.")) expect_true(anyNA(configuration[parameters$depends[[p]]])) } else { expect (configuration[,p] >= bounds[1], paste0("Parameter '", p, "=", configuration[,p], "' does not comply with dependency: ", parameters$depends[[p]], " and lower bound: ", bounds[1])) expect (configuration[,p] <= bounds[2], paste0("Parameter '", p, " = ", configuration[,p], "' does not comply with dependency: ", parameters$depends[[p]], " and upper bound: ", bounds[2])) } } } } test_that("test inactive dependent", { parameters <- readParameters(text=' p1 "" r (0,1) p2 "" r (0, p1) | p1 < 0.5 p3 "" r (0, p2) ') confs <- irace:::sampleUniform(parameters, 50, digits = 2) for (i in 1:nrow(confs)) { checkConditionalAndDependency(confs[i,], parameters) } }) test_that("checkDependencies", { target.runner <- function(experiment, scenario) { configuration <- experiment$configuration tmax <- as.numeric(configuration[["real"]]) if (configuration[["mode"]] %in% c("x1", "x2")) temp <- as.numeric(configuration[["param1"]]) else temp <- 1 time <- max(1, abs(rnorm(1, mean=(tmax+temp)/10))) return(list(cost = time, time = time, call = toString(experiment))) } test.checkDependencies <- function(parameterFile, ...) { args <- list(...) weights <- rnorm(200, mean = 0.9, sd = 0.02) parameters <- readParameters(parameterFile) scenario <- list(targetRunner = target.runner, instances = weights, seed = 1234567, maxExperiments=200) scenario <- modifyList(scenario, args) scenario <- checkScenario (scenario) nconf <- 100 conf <- irace:::sampleUniform(parameters, nconf, 4) conf <- cbind(seq(1,nrow(conf)), conf) names(conf)[1]<- ".ID." for (i in 1:nconf) checkConditionalAndDependency(conf[i,], parameters) model <- irace:::initialiseModel(parameters, conf, 4) conf2 <- irace:::sampleModel(parameters, conf, model, nconf, 4) for (i in 1:nconf) checkConditionalAndDependency(conf2[i,], parameters) confs <- irace(scenario = scenario, parameters = parameters) for (i in 1:nrow(confs)) { checkConditionalAndDependency(confs[i,], parameters) } } test.checkDependencies(parameterFile="dependencies.txt") test.checkDependencies(parameterFile="dependencies2.txt") }) }) # withr::with_output_sink() irace/tests/testthat/dependencies2.txt0000644000176200001440000000071114157104332017607 0ustar liggesusersparam1 "--param1 " i (1, "real") | mode %in% c("x1", "x2") param2 "--param2 " i ("real", 10) | mode %in% c("x1", "x3") && real > 2.5 && real <= 3.5 mode "--" c ("x1" ,"x2", "x3") real "--paramreal=" r (1.5, 7.5) mutation "--mutation=" o ("none", "very low", "low", "medium", "high", "very high", "all") #unused "-u " c (1, 2, 10, 20) irace/tests/testthat/dummy_wrapper.py0000755000176200001440000000435314157104332017614 0ustar liggesusers#!/usr/bin/env python3 # encoding: utf-8 ''' DummyWrapper -- Dummy wrapper for unit testing This requires the installation of https://github.com/automl/GenericWrapper4AC @author: Manuel López-Ibáñez @copyright: 2018 @license: BSD @contact: manuel.lopez-ibanez@manchester.ac.uk ''' import json import traceback from genericWrapper4AC.generic_wrapper import AbstractWrapper class DummyWrapper(AbstractWrapper): ''' Dummy wrapper for unit testing ''' def __init__(self): AbstractWrapper.__init__(self) self._return_value = None def get_command_line_args(self, runargs, config): ''' ''' runargs.update(config) return "echo '" + json.dumps(runargs) + "'" def process_results(self, filepointer, exit_code): ''' Parse a results file to extract the run's status (SUCCESS/CRASHED/etc) and other optional results. Args: filepointer: a pointer to the file containing the solver execution standard out. exit_code : exit code of target algorithm ''' statuses = ['SUCCESS', 'TIMEOUT', 'CRASHED', 'ABORT'] # If something fails, we a serious problem output = dict(status='ABORT') for line in filepointer: try: argmap = json.loads(str(line.decode('UTF-8')).replace("\n","")) ins = argmap["instance"] seed = argmap["seed"] cost = float(argmap["-cost"]) runtime = float(argmap["-runtime"]) status = statuses[seed % len(statuses)] if ins == "cost": output = dict(status = status, cost = cost) elif ins == "time": cutoff = float(argmap["cutoff"]) output = dict(status=status, runtime=min(cutoff,runtime)) elif ins == "cost+time": cutoff = float(argmap["cutoff"]) output = dict(status=status, cost=cost, runtime=min(cutoff,runtime)) except ValueError: traceback.print_exc() pass return output if __name__ == "__main__": wrapper = DummyWrapper() wrapper.main() irace/tests/testthat/test-targeteval.R0000644000176200001440000000172314157104332017600 0ustar liggesuserscontext("targeteval") withr::with_output_sink("test-targeteval.Rout", { test_that("target.evaluator", { target.runner <- function(experiment, scenario) { return(list(call = toString(experiment))) } target.evaluator <- function(experiment, num.configurations, all.conf.id, scenario, target.runner.call) { result <- list(cost = runif(1), call = toString(experiment)) return(result) } parameters <- readParameters(text = ' algorithm "--" c (as,mmas,eas,ras,acs) ') generate.set.seed() scenario <- checkScenario (list(targetRunner = target.runner, targetEvaluator = target.evaluator, maxExperiments = 200, instances = runif(100))) irace:::checkTargetFiles(scenario = scenario, parameters = parameters) confs <- irace(scenario = scenario, parameters = parameters) expect_gt(nrow(confs), 0L) }) }) # withr::with_output_sink() irace/tests/testthat/bug-13-lookup.rds0000644000176200001440000002172214157104332017362 0ustar liggesusersŝy\UB@@DGD$틢QN!KY0l& A8 #" (* 2 j؄ J'N/˼שk{^>29aiBS6uA΍M~2 ci9-7N1uk7h]yUZCv֩F gWmij-ۏ~i1o<5~q#ϳ_vTg\|Q0!tݺ֤TCI5󮧤y/PB-;,ҧ[T˦l T6i̦Ek1KV$7Y0h)9v.+mWYgO 9M%N]^k6JQb;J(INB!Ҝ7-0C% 0f%AY('YvnZ ]/`R!-^:I|Q(ͱ90fqJjbQB:deT[STFVռ(ƶ)4}unUCNN8eР-B?dRȲ[SiwT3#1ӭPKS}ͣZbO9X;j$#Mzk!A.åg2sX, YNfP*RPoUIK\ܴ_Ό>Εeg][iHj$TC#(GrYG'i $-MpuJrEWrY jI|32밪If֎'6 :(PZdM8y(=T3"eeM2 kdVvIG`45H;\BTAC6QVL*uҴ"56X 3&EoK wu&G#>ܟVn!v2,l]01Xs6ҡ"J^e3+/Iz0t`RՑ8"_0ߠBˊܔşmF.Ԅtؓy|rZͣڼ}=4:% x cЀYjfXoSkj{O)#[Un-&Ųy i|J\N1Y ٣22|Y2wV>%S*irI)H>&\M5bƅ>XWq YU9(pJ!l.rc1sv2ɬ"M5d#HfV~V,NT~[lI91R YnJ>Ef(0:RHH M4Tv,ྯ CVC a6ooUktPkf0^cv@M"s=>#kȻ\lcP2Rgt;MNz#t:D퐝GL۷:bBI[0)2GXU9%3'PL&4aiR .V2-L|hפG̻XSt3c}6a,ZzөCي-4ˋT8TFE&Jc(9˝ @.'6YU;bJTm74q2lyp Oh%sҲ_*jAy-ZAy%g[q6J-(ԴU4[& P^&jAy+cl` [e2@y+)Sqӏܾnj3Gyˢ73@y%F;M(&2Ay+kI,eJP(6(ԁJ6/XkGyE.Wtfu2[%gP^4M(̋eR(,Ay%E;+P^oцJ +jCԉtF?[y73Gyi6WR6:FQ@[3N^ʱ/4JB#3)6EJȴ ;(]j,''TĮ0h:^04Z~2䘕ͥeW=v'P#@7W0@z9}eIS$YSTè"fU@ب;<BK8$,n˷T Ϲz{y?Bq؋J$\'-ϘlF*/s% ZL P&hRQ_7֦bx& R(']9R0(0YPX2MUMv%NU贤-:1JbK2ʫ_*/1zR&: f2b' C%fIeGg)Ӭ8(5zB#$=㸒_ c.PPG =31}-T)kS#Q1Ԉg@*/M;C3 wͧd[ N*VP%#yS v)c7FW8G f@Ƕ/=tMEfx3 `^R$Fm,nnA$s7+%iN:[|qOf7%dGg<9_ id1]9);DvآV|vFgHi-tM>$Y CՎQ2ȚIQ2D-[IˮYBqmK[a&Y*RY-+sT/Q^VyQ$b^rbrC]j d]2^I(SOSgCF4yh(pY)"B$< n9yH5q3ʭ nt,:fh/IJktt#L$Xq^uuǟeQnpb=CX/D\6 ~EHe]7=QĘNْF(18(zZSJgˣ6pTiY3+ShBOCg=^N38Ğ%sgE:2,BN]PK6B` foZ!<2KΖ{!5je}\O~%(2XHÅd5:4)iJ`+GW CykOoɲ@; ɚ:ݚAnkʕ 7; ՘{p֧/@V&t.Zaw|6?t(Nu>' ]#.4 7й wtl|:.3g K·MC]Gm>̀uwÏ&bhC'GCws q皡q;"YK yt\(f3ϸAAϏ^ba 7Aac |?.AB(O8|?.aa`aPgq~3xdhh;CoAO\Bw[/"2ۡ{fz> }~gM2.UxƯ ?'9A>|.0J~t\ 枅N?<7== ^t8tnsq_BEkl9=~<^8y-nǚh?-͸rZ;66t|υ'qgEfW?< XBS'3vZ҅<,u?E^pwAxN[z? ӡ&ܦG'AWC'=~{=Ks6\uy'M; N ÏZk)-:n_{\q۠+Cg 臡n4 g~'hwʆ/ /;Dp*Ok\>pRW t ? IW0wiPxxޜBq@{_MnĦwB}à.kp_ه pҟ.+۰ ácR=q*t<Ӊ'w®' m/ aeC`OîBh5h۶v <^߃g| .9:~6.;z }~ނs[V ϋ󘙘PyL L_Ed칸v[̵˯Y `MCwx4{zq/Ö{#47COOFq|-]n</#@/G`-l0p; <{PX.q t {< zg--8j;gm">HN͇ͫ@y֯ǡ{סe]_cJصw{.7\ JiVm~pP?,"cY?bm(䕭[Zl6m4_q18 5Nirace/tests/testthat/test-capping.R0000644000176200001440000001154714232037511017066 0ustar liggesuserscontext("Capping") withr::with_output_sink("test-capping.Rout", { nsize <- 0.1 ## Functions ########################################################## f_ackley <- function (x,y) { # Transformation of parameter values # from [0,1] to [vmin,vmax] vmin <- -5 vmax <- 5 x <- (x*(vmax-vmin)) + vmin y <- (y*(vmax-vmin)) + vmin a <- -20 * exp (-0.2 * sqrt(0.5 * (x^2 + y^2))) b <- exp(0.5 * (cos(2*pi*x) + cos(2*pi*y))) f <- a - b + exp(1) + 20 # Simulating stochasticity noise <- runif(1, min = (1-nsize), max = (1+nsize)) f <- f*noise # Transform result from [fmin,fmax] # to [0,100] fmin <- 0 fmax <- 15*nsize f <- ((f - fmin) / (fmax-fmin)) * (100-0) + 0 return(f) } f_goldestein_price <- function (x,y) { # Trasfomation of parameter values # from [0,1] to [vmin,vmax] vmin <- -2 vmax <- 2 x <- (x*(vmax-vmin)) + vmin y <- (y*(vmax-vmin)) + vmin a <- 1 + ((x + y + 1)^2) * (19 - 14*x + 3*x^2 - 14*y + 6*x*y + 3*y^2) b <- 30 + ((2*x - 3*y)^2) * (18 - 32*x + 12*x^2 + 48*y - 36*x*y + 27*y^2) f <- a*b # Simulating stochasticity noise <- runif(1, min = (1-nsize), max = (1+nsize)) f <- f*noise # Transform result from [fmin,fmax] # to [0,100] fmin <- 0 fmax <- 1000000*nsize f <- ((f - fmin) / (fmax-fmin)) * (100-0) + 0 return(f) } f_matyas <- function (x,y) { # Trasfomation of parameter values # from [0,1] to [vmin,vmax] vmin <- -10 vmax <- 10 x <- (x*(vmax-vmin)) + vmin y <- (y*(vmax-vmin)) + vmin f <- 0.26 * (x^2 + y^2) - (0.48*x*y) # Simulating stochasticity noise <- runif(1, min = (1-nsize), max = (1+nsize)) f <- f*noise # Transform result from [fmin,fmax] # to [0,100] fmin <- 0 fmax <- 100*nsize f <- ((f - fmin) / (fmax-fmin)) * (100-0) + 0 return(f) } f_himmelblau <- function (x,y) { # Trasfomation of parameter values # from [0,1] to [vmin,vmax] vmin <- -5 vmax <- 5 x <- (x*(vmax-vmin)) + vmin y <- (y*(vmax-vmin)) + vmin f <- (x^2 + y - 11)^2 + (x + y^2 - 7)^2 # Simulating stochasticity noise <- runif(1, min = (1-nsize), max = (1+nsize)) f <- f*noise # Transform result from [fmin,fmax] # to [0,100] fmin <- 0 fmax <- 2000*nsize f <- ((f - fmin) / (fmax-fmin)) * (100-0) + 0 return(f) } ## target runner ########################################################### target.runner <- function(experiment, scenario) { debugLevel <- scenario$debugLevel configuration.id <- experiment$id.configuration instance.id <- experiment$id.instance seed <- experiment$seed configuration <- experiment$configuration instance <- experiment$instance bound <- experiment$bound value <- switch(instance, ackley = f_ackley(as.numeric(configuration[["x"]]), as.numeric(configuration[["y"]])), goldestein = f_goldestein_price(as.numeric(configuration[["x"]]), as.numeric(configuration[["y"]])), matyas = f_matyas(as.numeric(configuration[["x"]]), as.numeric(configuration[["y"]])), himmelblau = f_himmelblau(as.numeric(configuration[["x"]]), as.numeric(configuration[["y"]]))) # Simulate execution bound if (value > bound) value <- bound result <- list(cost = value, time=value, call = toString(experiment)) return(result) } ## target runner ########################################################### target.runner.reject <- function(experiment, scenario) { if (runif(1) <= 0.05) return (list(cost = -Inf, time = 80, call = toString(experiment))) return (target.runner(experiment, scenario)) } cap.irace <- function(...) { args <- list(...) parameters.table <- ' x "" r (0, 1.00) y "" r (0, 1.00) ' parameters <- readParameters(text = parameters.table) scenario <- list(instances = c("ackley", "goldestein", "matyas", "himmelblau"), targetRunner = target.runner, capping = TRUE, boundMax = 80, testType = "t-test", parallel = test_irace_detectCores()) scenario <- modifyList(scenario, args) scenario <- checkScenario (scenario) irace:::checkTargetFiles(scenario = scenario, parameters = parameters) confs <- irace(scenario = scenario, parameters = parameters) best.conf <- getFinalElites(scenario$logFile, n = 1L, drop.metadata = TRUE) expect_identical(removeConfigurationsMetaData(confs[1L, , drop = FALSE]), best.conf) } test_that("cap.irace maxExperiments = 1000", { generate.set.seed() cap.irace(maxExperiments = 1000) }) test_that("cap.irace maxExperiments = 50000", { generate.set.seed() cap.irace(maxTime = 50000) }) test_that("cap.irace targetRunner = target.runner.reject, maxTime = 10000", { skip_on_cran() # This sometimes fails randomly generate.set.seed() cap.irace(targetRunner = target.runner.reject, maxTime = 10000) }) }) # withr::with_output_sink() irace/tests/testthat/parameters.txt0000644000176200001440000000070014157104332017240 0ustar liggesusersparam1 "--param1 " i (1, 10) | mode %in% c("x1", "x2") param2 "--param2 " i (1, 10) | mode %in% c("x1", "x3") && real > 2.5 && real <= 3.5 mode "--" c ("x1" ,"x2", "x3") real "--paramreal=" r (1.5, 4.5) mutation "--mutation=" o ("none", "very low", "low", "medium", "high", "very high", "all") #unused "-u " c (1, 2, 10, 20) irace/tests/testthat/logparameters.txt0000644000176200001440000000071014157104332017743 0ustar liggesusersparam1 "--param1 " i,log (1, 10) | mode %in% c("x1", "x2") param2 "--param2 " i (1, 10) | mode %in% c("x1", "x3") && real > 2.5 && real <= 3.5 mode "--" c ("x1" ,"x2", "x3") real "--paramreal=" r,log (1.5, 4.5) mutation "--mutation=" o ("none", "very low", "low", "medium", "high", "very high", "all") #unused "-u " c (1, 2, 10, 20) irace/tests/testthat/saved_maxim_bug.rds0000644000176200001440000001545014157104332020210 0ustar liggesusers] |OʢȢTmf2i2YeE(Ble\(nCT@®Cq၈ UosLA~}=9uaK?wS.{-u|7[|]SO_oH/ym;c؇8?|p5q]p{mZAFʟLjQ׈}v\uH{kro./}Ec} ^o0FC$.?qc"*w Lhܳ:׏xn["IaWH-bފ~0_uy9oAGC_j )񼺾iQ'28m;̿ Ox^DoW}yNF|>ݿ6*>&[z3g9Gsjs\^b@ mW,g3ߛ0?'^^;ڣ]zBχsx?[.vŷ]38BBX#Dӿt'] C(ݯOg"RBDoqgZv#@~`GbU{yJ_n2 ok86]Qjtf>tz!Y*$}z|mCsP~GqzuoIX؊G ʾ kGP; [Wg_Ք7hHl/K4|v kʣ^KlӃ0ΦA2nMSaNOiZ3n `7):Dli'}IlW?? {yxjΨXWKy(Ô%TO~bx[nr/i}K_K<ǡ}a&";?nadYvݬ}q>L?O6]ZS97j/DYR`(7^k b|qi&!nS??蟟NY'8l?5~6U??8~GS/54O"0?E)j7Lݠ]s~_WoOˡ?̀kj:o51⛹61/"?[F_hCqkvȋ]C,m5s w@?Ú:K)Uz e~I,?~sa[ 4u|7@ _lA.'AA~meڀRϯHa3Z>׻R'z5#OmO|t2?D057_MiD `bĶR[ýr?7z}͚uѦqK66cud"lgT'PV RCЏeHaz>z^N,3O[wczT4ՋJlq:8)XZoq"u7Aƿ^b-{j{,PŧJ~AS4VX bxz8i+u8 yeW&'/AbQWi+EдUE*N8:qiFļd}Т fnhM:_}mƭ~E*0X姄#B: TW|@[m #aX_5`H~o@70ױq*˯xn8 /ilNۀ$~͝q~ϒp7:,í=}kU'ƭoKW |KO&mZ?QzhIuunu2muDqȿsBg_8ܧíLR)*ؙ0R[}9L/I\QqQf?ר1om~ja~H̸eփVj䷇.0~$lk_qv:~q{0}u9j>·DNW+BxOhќ4?|Om=&kuW!?il)%ۮʌ׸qAoI0a=Js:wZoĐ C>ILC~lZy\/locj#z`(zF ur)k `w2|~BPί1?ºi{uCk!p%]04&1S1gsfh/1[Rb,{&?y132T 3錿#k[4ue#ib~qf]ȏ"e8bD#]RAb@D=ym- mJ}%NOl/gߊ1,LR.1Ey߁vS7c ,;=C} _glZiSzz9v.p }1k<|Z?bv8 / #XWiяv`_-p+|Kk<_x|` L}%ݣLD~YϾGQ̳hw&|^O ߅zٕI9p(NrU nGvz$!4rGNzGnzG^zD1$!Q bHCŐ(D1$!Q pP pP pP pP pP 'pR 'pR 'pR 'pR 'pR pQ pQ pQ pQ pQ 7pS 7pS 7pS 7pS 7pS P P P P P /R /R /R /R /R"QD1(F(EbQ"Q`dm6vhg;tC';tC7;C/;dhvfghvfghvfghvfghvfghC$&14I MbhC9999999999yyyyyyyyyyZC+bhE 1"VЊZCc\"1.HK$%qĸDb\"1.HK$%qĸDb\"1.HK$%qĸDb\"1.HK$%%:RcX:x= uvѡ5\.emh5ja˛r{_yo=m^‡dumMMOujVwqn5MF|,ՖЃOKL%'g\eiG[{r='v51}D;z^7(3gN_|I^=(oW-X+RtMZv\FNVkR'n.'W]Z;n:wW[woh@𯓊>s x?}8}-->Sׂ7x~~ۛ7]vNX=kG!-|qd7-mӝKvN.KMfL_דy`Moϼk{93}M|e/uLt9CeGO7Og\7O ;7d7>ؖC̼;Gnzm[>2lywٯkֽautɥ/wK} tǛ7?dmO+xҍ#W/\Yj&޺ON+/]4õ{ŽL68gGY0;YW/|v/v>z>啣.wtx)tEy c}w*_+7nǩMulh0l˭dP(o%k/x&uNcFܲ[6f۴gEvOlμxI=DJtj½6ϿuXq36_ō^?ҦbS/⒭dܟӸo}΂99#_۱(Z'6l|ᕣop-NN5Pr}yjD?˂vfgonWmym:||jˬL*{&sS:ҡΊmk<9 l?oROΓ UE}!3ҤGbҭN80_R٥:&"'eP>jUQEPq$\+  pUU&Iq"v3\Y]ochpr@:Z ʮ.+BC,)n˫#EjuWGƉz߷0$+|TꅉW+ÕU@nsQT d9"V˸Rt9?$K9s/ۯEUeX\n6v7\2jy:ݺ?~. { sk񰝆#UQ' xTIO&ai )Qc}P7 V N L*_jL8R\ D|IxIKm$THvj_$+)~ǭXmlDWӭIWr%;I7%QyD0RI h!L= 1) { : missing value where TRUE/FALSE needed Calls: irace.cmdline -> irace.main -> checkScenario Execution halted #5 --- fixed at revision 1343 Paths in the command-line should be relative to the current working directory. Paths in scenario.txt should be relative to scenario.txt. However, currently the latter are also relative to the working directory of irace. Example: $ ls test test/hook-run test/scenario.txt $ cat test/scenario.txt ######################## hookRun <- "./hook-run" ####################### $ irace -c test/scenario.txt Error in file.check(scenario$hookRun, executable = TRUE, text = "run program hook") : run program hook '/home/manu/./hook-run' does not exist Calls: irace.cmdline -> irace.main -> checkScenario -> file.check #3 -- fixed at revision 1101 $ irace --version (or any other unknown parameter) It should say "unknown parameter" or something like that. #4 -- fixed at revision 827 $ irace --hook-run my-hook-run doesn't work. One needs to use ./my-hook-run #7 -- fixed at revision r882 ------------------------------ When using MPI, if hook-run exits with status 1, the error-handling does not work correctly. It should print the hookRun call, but it doesn't. It prints: Error: The output of `hookRun' is not numeric! The output was: Error : running command '/home/mascia/tuning_paradiseo/experiments/PFSPWT/tuning_2/./hook-run /home/mascia/tuning_paradiseo/experiments/PFSPWT/tuning_2/./Instances//90x20_5 402 --sa-code_definition%ps-initialisation=1 --sa-code_definition%sa-algo_choice@0=7 --sa-code_definition%sa-algo_choice@0%7%sa-perturbation=0 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance=3 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule=0 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule%0%sa-sa_r_initial_temp=1.0607 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule%0%sa-sa_r_factor=0.6901 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule%0%sa-sa_r_max_step=562 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule%0%sa-sa_r_final_temp=825 --sa-code_definition%sa-algo_choice@0%7%sa-time_ratio=7 --sa-code_definition%s Execution halted Warning message: running command 'ls *.26602+1.*.log 2>/dev/null' had status 2 which looks like the master code continued execution even after the slave died and the error output from the master was lost. The truncation is also a problem. In theory, the following program should reproduce the issue, but it doesn't: #!/usr/bin/Rscript # Load the R MPI package if it is not already loaded. if (!is.loaded("mpi_initialize")) { library("Rmpi") } # Spawn as many slaves as possible mpi.spawn.Rslaves(nslaves=5) # In case R exits unexpectedly, have it automatically clean up # resources taken up by Rmpi (slaves, memory, etc...) . Last <- function(){ if (is.loaded("mpi_initialize")){ if (mpi.comm.size(1) > 0){ print("Please use mpi.close.Rslaves() to close slaves.") mpi.close.Rslaves() } print("Please use mpi.quit() to quit R") .Call("mpi_finalize") } } runcommand <- function(command, args) { cat (format(Sys.time(), usetz=TRUE), ":", command, args, "\n") elapsed <- proc.time()["elapsed"] err <- NULL output <- withCallingHandlers( tryCatch(system2(command, args, stdout = TRUE, stderr = TRUE), error=function(e) { err <<- paste(err, conditionMessage(e), sep ="\n") NULL }), warning=function(w) { err <<- paste(err, conditionMessage(w), sep ="\n") invokeRestart("muffleWarning") }) # If e is a warning, the command failed. if (!is.null(err)) { stop (call. = FALSE, format(Sys.time(), usetz=TRUE), ": ERROR (", paste(output, sep="\n"), "):", err,"\n") } cat (format(Sys.time(), usetz=TRUE), ": DONE (", ") Elapsed: ", proc.time()["elapsed"] - elapsed, "\n") return(list(output = output, error = NULL)) } candidates <- list("x", "--xx /home/mascia/tuning_paradiseo/experiments/PFSPWT/tuning_2/./hook-run /home/mascia/tuning_paradiseo/experiments/PFSPWT/tuning_2/./Instances//90x20_5 402 --sa-code_definition%ps-initialisation=1 --sa-code_definition%sa-algo_choice\ @0=7 --sa-code_definition%sa-algo_choice@0%7%sa-perturbation=0 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance=3 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule=0 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cool\ ing_schedule%0%sa-sa_r_initial_temp=1.0607 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule%0%sa-sa_r_factor=0.6901 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule%0%sa-sa_r_max_step=562 --sa-code_definiti\ on%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule%0%sa-sa_r_final_temp=825 --sa-code_definition%sa-algo_choice@0%7%sa-time_ratio=7 --sa-code_definition%something-else and") # Tell all slaves to return a message identifying themselves output <- Rmpi::mpi.applyLB(candidates, runcommand, command = "ls") cat(paste(output, sep="\n")) #mpi.remote.exec(stop("error", mpi.comm.rank(),"of",mpi.comm.size())) # Tell all slaves to close down, and exit the program mpi.close.Rslaves() #mpi.quit() #10 --- fixed at revision 1919 ------------------------------------ Related to #8. With --maxTime, irace assumes that target-runner will report the time, however, this is not possible with --batchmode. If you are confident in modifying the package sources, you need to add a check on scenario$batchmode before checking the scenario$maxTime condition that you quote, you need to handle the case where that function returns no time and you need to handle the case that target-evaluator returns time and how to use that to increment the time budget. However, targetEvaluator is called again for elitist configurations even if no target-runner call is made. This means that we may double count the time? In the meantime, I would suggest that you don't use --maxTime for now. irace/src/0000755000176200001440000000000014325242462012130 5ustar liggesusersirace/src/iracebin/0000755000176200001440000000000014325242462013704 5ustar liggesusersirace/src/iracebin/irace.c0000644000176200001440000000025114157104332015125 0ustar liggesusers#include #include #include #include #include #define R_CODE "library(irace);irace.cmdline()" #include "irace.h" irace/src/iracebin/irace.h0000644000176200001440000000344214157104332015137 0ustar liggesusersvoid *mempcpy(void *dest, const void *src, size_t n) { return (void *) ((char*)memcpy(dest, src, n) + n); } char * concat (const char *str, ...) { va_list ap; size_t allocated = 100; char *result = (char *) malloc (allocated); if (result != NULL) { char *newp; va_start (ap, str); char *wp = result; const char *s; for (s = str; s != NULL; s = va_arg (ap, const char *)) { size_t len = strlen (s); /* Resize the allocated memory if necessary. */ if (wp + len + 1 > result + allocated) { allocated = (allocated + len) * 2; newp = (char *) realloc (result, allocated); if (newp == NULL) { free (result); return NULL; } wp = newp + (wp - result); result = newp; } wp = mempcpy (wp, s, len); } /* Terminate the result string. */ *wp++ = '\0'; /* Resize memory to the optimal size. */ newp = realloc (result, wp - result); if (newp != NULL) result = newp; va_end (ap); } return result; } int main(int argc, char *argv[]) { #ifdef WIN32 # define EXE_EXT ".exe" #else # define EXE_EXT "" #endif const char file[] = "R" EXE_EXT; char * const extra_argv[] = { "--vanilla", "--slave", "-e", R_CODE, "--args"}; int extra_argc = sizeof(extra_argv) / sizeof(char *); char **cmd_args = calloc(extra_argc + argc + 1, sizeof(char *)); cmd_args[0] = argv[0]; for (int i = 0; i < extra_argc; i++) cmd_args[i+1] = extra_argv[i]; for (int i = 1; i < argc; i++) cmd_args[extra_argc + i] = argv[i]; return execvp(file, cmd_args); } irace/src/iracebin/ablation.c0000644000176200001440000000025414233164263015642 0ustar liggesusers#include #include #include #include #include #define R_CODE "library(irace);ablation_cmdline()" #include "irace.h" irace/src/install.libs.R0000644000176200001440000000053114233161272014645 0ustar liggesusersexecs <- file.path("iracebin", c("irace","ablation")) if (WINDOWS) execs <- paste0(execs, ".exe") if (any(file.exists(execs))) { dest <- file.path(R_PACKAGE_DIR, paste0('bin', R_ARCH)) #print(dest) dir.create(dest, recursive = TRUE, showWarnings = FALSE) file.copy(execs, dest, overwrite = TRUE) } else { stop(execs, " not found!") } irace/src/Makevars0000644000176200001440000000007714233171234013624 0ustar liggesusers# -*- mode: makefile -*- EXE= include Makevars-common irace/src/Makevars.win0000644000176200001440000000010214233171222014402 0ustar liggesusers# -*- mode: makefile -*- EXE=.exe include Makevars-common irace/src/Makevars-common0000644000176200001440000000064014233161225015105 0ustar liggesusersiraceEXE= iracebin/irace$(EXE) ablationEXE= iracebin/ablation$(EXE) all: $(iraceEXE) $(ablationEXE) $(iraceEXE): iracebin/irace.o $(CC) $(PKG_CPPFLAGS) $(CFLAGS) iracebin/irace.o -o $@ iracebin/irace.o: iracebin/irace.h $(ablationEXE): iracebin/ablation.o $(CC) $(PKG_CPPFLAGS) $(CFLAGS) iracebin/ablation.o -o $@ iracebin/ablation.o: iracebin/irace.h clean: @$(RM) iracebin/*.o $(iraceEXE) $(ablationEXE) irace/vignettes/0000755000176200001440000000000014325242462013351 5ustar liggesusersirace/vignettes/NEWS.txt0000644000176200001440000007240214321301254014662 0ustar liggesusers# irace 3.5 ## New features and improvements * Handling of dependent parameter domains: These should be specified in the parameter domain definition and, for now, only numerical parameter can define dependent domains. A numerical domain can be dependent on one bound, e.g. `(1, "param1*2")`, where the dependent bound can include basic arithmetic operators. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * The package now provides an `ablation` executable (`ablation.exe` in Windows) that makes easier to perform ablation analysis without having any R knowledge. * The interface to functions `ablation()` and `plotAblation()` has been simplified. The `ablation()` function now allows overriding scenario settings. The `plotAblation()` function will not create the plot if the ablation log does not contain a complete ablation. (Manuel López-Ibáñez) * The argument `n.instances` of `ablation()` has been renamed to `n_instances` and it is now a factor that multiplies `scenario$firstTest`. (Manuel López-Ibáñez) * New command-line option `--quiet` to run without producing any output except errors (also available as a scenario option). (Manuel López-Ibáñez) * New command-line option `--init` to initialize a scenario. (Deyao Chen) * Added support for HTCondor cluster framework to `--batchmode`. (Filippo Bistaffa) * `--check` now also check the contents of `configurationsFile` and runs configurations provided via `initConfigurations`. (Manuel López-Ibáñez, reported by Andreea Avramescu) * New scenario options `targetRunnerLauncher` and `targetRunnerLauncherArgs` to help in cases where the target-runner must be invoked via another software with particular options (such as `python.exe` in Windows). (Manuel López-Ibáñez) * New scenario option `minMeasurableTime`. (Manuel López-Ibáñez) * An error is produced if a variable set in the scenario file is not known to irace. If your scenario file contains R code, then use variable names beginning with a dot `'.'`, which will be ignored by irace. (Manuel López-Ibáñez) * Plotting functions have been moved to the new package [iraceplot](https://auto-optimization.github.io/iraceplot/). In particular, `configurationsBoxplot()` is replaced by `iraceplot::boxplot_training()` and `iraceplot::boxplot_test()`; `parallelCoordinatesPlot()` is replaced by `iraceplot::parallel_cat()` and `iraceplot::parallel_coord()`; and `parameterFrequency()` is replaced by `iraceplot::sampling_frequency()`. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * The user-guide now contains a detailed section on "Hyper-parameter optimization of machine learning methods". (Manuel López-Ibáñez) * When `testType="F-test"` and only two configurations remain, the elimination test now uses the pseudo-median estimated by the Wilcoxon signed-rank test to decide which configuration is the best one instead of comparing the median difference. (Manuel López-Ibáñez) * New functions `testing_fromlog()` and `testing_fromfile()` for independently executing the testing phase. The function `testing.main()` was removed as it is superseded by the new ones. (Manuel López-Ibáñez) * New function `read_logfile()` to easily read the log file produced by irace. (Manuel López-Ibáñez) * New function `printParameters()` that prints a parameters R object as a valid input text. (Manuel López-Ibáñez) * `irace2pyimp` moved to its own R package. (Manuel López-Ibáñez) * Generating the file `irace.Rdata` may be disabled by setting `logFile=""`. (Manuel López-Ibáñez, reported by Johann Dreo) * `path_rel2abs()` and `checkParameters()` are now exported so that other packages may use them. (Manuel López-Ibáñez) * `path_rel2abs()` also searches in system paths. (Manuel López-Ibáñez) * `readConfigurationsFile()` will now detect duplicated configurations and error. (Manuel López-Ibáñez) * The interface to functions `getFinalElites()`, `getConfigurationById()` and `getConfigurationByIteration()` has been simplified. * The package provides a `irace.sindef` file that may be used for building a standalone container of irace using Singularity. See the `README.md` file for instructions. (Contributed by Johann Dreo) * New example `examples/target-runner-python/target-runner-python-win.bat` contributed by Levi Ribeiro. * New helper script in `bin/parallel-irace-slurm` to launch `irace` in [SLURM](https://slurm.schedmd.com/) computer clusters. (Manuel López-Ibáñez) * Rename `scenario.update.paths()` to `scenario_update_paths()`. The old name is deprecated. (Manuel López-Ibáñez) ## Fixes * Correctly handle clear out-performance cases despite strong bi-modality. (Reported by Nguyen Dang, fixed by Manuel López-Ibáñez) * Fix error when recovering from a parallel run on Windows. (Manuel López-Ibáñez, reported by Tarek Gamal) * `testNbElites` now controls how many iteration elites are tested when `testIterationElites=1`. This is the documented behavior in the user guide. (Manuel López-Ibáñez, reported by Marcelo de Souza) * Fixes to the Matlab example. (Manuel López-Ibáñez) * The default of `testType` is now set to `t-test` when capping is enabled. (Manuel López-Ibáñez, reported by Jovana Radjenovic) * Fix various issues in the user guide. (Manuel López-Ibáñez, reported by Jovana Radjenovic) * Remove duplicated elites. (Manuel López-Ibáñez, reported by Federico Naldini) * Fix (#7): warnings with partial matched parameters. (Manuel López-Ibáñez, reported by Marc Becker) * Fix (#10): wrong assert with `elitist=0`. (Manuel López-Ibáñez) * Fix (#12): irace can be run with [FastR](https://www.graalvm.org/22.1/docs/getting-started/#run-r). * Fix (#13): Maximum number configurations immediately rejected reached. (Manuel López-Ibáñez) * Fix: when setting the scenario file in the command-line, `scenarioFile` was not set correctly. The correct scenario was used, however, the debug output and the value stored in the log / recovery file was wrong. (Manuel López-Ibáñez, reported by Richard Schoonhoven) * With `sampleInstances = FALSE`, elitist irace does not change the order of instances already seen. However, if you want to make sure that the order of the instances is enforced, you also need to set `elitistNewInstances=0`. * The function `irace.usage()` was removed. It was not really useful for R users as the same result can be obtained by calling `irace.cmdline("--help")`. (Manuel López-Ibáñez) # irace 3.4.1 (31/03/2020) * `NEWS` converted to markdown. * Fix CRAN error on Solaris. # irace 3.4 (30/03/2020) * `irace2pyimp` function and executable (`irace2pyimp.exe` on Windows) to convert .Rdata files generated by irace to the input files required by the parameter importance analysis tool PyImp (https://github.com/automl/ParameterImportance). (Nguyen Dang, Manuel López-Ibáñez) * Initial configurations may also be provided directly in R using `scenario$initConfigurations` (Manuel López-Ibáñez) * Rdata files are saved in version 2 to keep compatibility with older R versions. (Manuel López-Ibáñez) * Fix invalid assert with ordered parameters: (Leslie Pérez Cáceres) ``` value >= 1L && value <= length(possibleValues) is not TRUE ``` * The `irace` executable (`irace.exe` on Windows) is a compiled binary instead of a script. On Windows, `irace.exe` replaces `irace.bat` (Manuel López-Ibáñez) * `inst/examples/Spear` contains the Spear (SAT solver) configuration scenario. (Manuel López-Ibáñez) * Fixed bug when reporting minimum `maxTime` required. (Reported by Luciana Salete Buriol, fixed by Manuel López-Ibáñez) * Fixed bug detected by assert: ```R all(apply(!is.na(elite.data$experiments), 1, any)) is not TRUE ``` (Reported by Maxim Buzdalov, fixed by Manuel López-Ibáñez) # irace 3.3 (26/04/2019) * Fix buggy test that breaks CRAN. (Manuel López-Ibáñez) * Do not print "23:59:59" when wall-clock time is actually close to zero. (Manuel López-Ibáñez) # irace 3.2 (24/04/2019) * Fix `irace --check --parallel 2` on Windows. (Manuel López-Ibáñez) * Values of real-valued parameter are now printed with sufficient precision to satisfy `digits` (up to `digits=15`). (Manuel López-Ibáñez) * It is possible to specify `boundMax` without capping. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * `irace --check` will exit with code 1 if the check is unsuccessful (Manuel López-Ibáñez) * Print where irace is installed with `--help`. (Manuel López-Ibáñez) * irace will now complain if the output of `target-runner` or `target-evaluator` contains extra lines even if the first line of output is correct. This is to avoid parsing the wrong output. Unfortunately, this may break setups that relied on this behavior. The solution is to only print the output that irace expects. (Manuel López-Ibáñez) * Completely re-implement `log` parameters to fix several bugs. Domains that contain zero or negative values are now rejected. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * New option `aclib=` (`--aclib 1`) enables compatibility with the GenericWrapper4AC (https://github.com/automl/GenericWrapper4AC/) used by AClib (http://aclib.net/). This is EXPERIMENTAL. `--aclib 1` also sets digits to 15 for compatibility with AClib defaults. (Manuel López-Ibáñez) * Fix printing of output when capping is enabled. (Manuel López-Ibáñez) * `checkTargetFiles()` (`--check`) samples an instance unless `sampleInstances` is FALSE. (Manuel López-Ibáñez) * Fix symbol printed in elimination test. (Manuel López-Ibáñez) * Use `dynGet()` to find `targetRunner` and `targetEvaluator`. As a result, we now require R >= 3.2. (Manuel López-Ibáñez) * All tests now use `testthat`. (Manuel López-Ibáñez) * New function `scenario.update.paths()` (Manuel López-Ibáñez) * Fix assert failure that may happen when `elitistNewInstances` is larger than `firstTest`. Reported by Jose Riveaux. (Manuel López-Ibáñez) * Fix bug in `checkTargetFiles()` (`--check`) with capping. (Leslie Pérez Cáceres) * Clarify a few errors/warnings when `maxTime > 0`. (Manuel López-Ibáñez, suggested by Haroldo Gambini Santos) # irace 3.1 (12/07/2018) * Use testthat for unit testing. (Manuel López-Ibáñez) * Allow instances to be a list of arbitrary R objects (`mlr` bugfix). (Manuel López-Ibáñez) # irace 3.0 (05/07/2018) * irace now supports adaptive capping for computation time minimization. The default value of the `testType` option is t-test when adaptive capping is enabled. Please see the user-guide for details. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * The package contains an `ablation()` function implementing the ablation method for parameter importance analysis by Fawcett and Hoos (2016). (Leslie Pérez Cáceres, Manuel López-Ibáñez) * New option `postselection` executes a post-selection race. (Leslie Pérez Cáceres) * At the end of each race, if the race stops before evaluating all instances seen in previous races, then the best overall may be different than the best of the race. We now print the best overall (best-so-far). Elites evaluated on more instances are considered better than those evaluated on fewer. (Manuel López-Ibáñez, Leslie Pérez Cáceres) * Last active parameter values of numerical parameters (`i` and `r`) are carried by the sampling model. When a value must be assigned and the parameter was previously not active, the sampling is performed around the last value. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * R help pages are now generated with Roxygen2. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * The user guide documents `--version`, `--help`, and `--check`. (Manuel López-Ibáñez) * A return value of `Inf` from `targetRunner`/`targetEvaluation` results in the immediate rejection of the configuration without any further evaluation. This is useful for handling unreliable or broken configurations that should not stop irace. (Manuel López-Ibáñez) * Numerical parameters may be sampled on a logarithmic scale using `i,log` or `r,log`. (Alberto Franzin) * New `target-runner.bat` for Windows contributed by André de Souza Andrade. * Fixed all shell scripts calling functions before defining them, which is not portable. (Manuel López-Ibáñez) * Fixed `--parallel` bug in Windows that resulted in `Error in checkForRemoteErrors(val)`. (Manuel López-Ibáñez) * Improve error message when no training instances are given. (Manuel López-Ibáñez) # irace 2.4 (03/08/2017) * The output of irace now specifies in which order, if any, configurations are printed. (Manuel López-Ibáñez, suggested by Markus Wagner) * Several fixes for handling paths in Windows. (Manuel López-Ibáñez) * `readConfigurationsFile()` now has a `text=` argument, which allows reading configurations from a string. (Manuel López-Ibáñez) * User-provided functions (targetRunner, targetEvaluator and repairConfiguration) and user-provided conditions for forbidden configurations are now byte-compiled when read, which should make their evaluation noticeably faster. (Manuel López-Ibáñez) * The argument `'experiment'` passed to the R function `targetRunner` does not contain anymore an element `'extra.params'`. Similarly, the `'scenario'` structure does not contain anymore the elements `'instances.extra.params'` and `'testInstances.extra.params'`. Any instance-specific parameters values now form part of the character string that defines an instance and it is up to the user-defined `targetRunner` to parse them appropriately. These changes make no difference when targetRunner is an external script, or when instances and instance-specific parameter values are read from a file. (Manuel López-Ibáñez) # irace 2.3 * Fix bug that will cause `iraceResults$experimentLog` to count calls to `targetEvaluator` as experiments, even if no call to `targetRunner` was performed. This does not affect the computation of the budget consumed and, thus, it does not affect the termination criteria of irace. The bug triggers an assertion that terminates irace, thus no run that was successful with version 2.2 is affected. (Manuel López-Ibáñez) # irace 2.2 * Command-line parameters are printed to stdout (useful for future replications). (Manuel López-Ibáñez, suggested by Markus Wagner) * Users may provide a function to repair configurations before being evaluated. See the scenario variable repairConfiguration. (Manuel López-Ibáñez) * The option `--sge-cluster` (`sgeCluster`) was removed and replaced by `--batchmode` (`batchmode`). It is now the responsibility of the target-runner to parse the output of the batch job submission command (e.g., `qsub` or `squeue`), and return just the job ID. Values supported are: "sge", "torque", "pbs" and "slurm". (Manuel López-Ibáñez) * The option `--parallel` can now be combined with `--batchmode` to limit the number of jobs submitted by irace at once. This may be useful in batch clusters that have a small queue of jobs. (Manuel López-Ibáñez) * New examples under `inst/examples/batchmode-cluster/`. (Manuel López-Ibáñez) * It is now possible to include scenario definition files from other scenario files by using: ```R eval.parent(source("scenario-common.txt", chdir = TRUE, local = TRUE)) ``` This feature is VERY experimental and the syntax is likely to change in the future. (Manuel López-Ibáñez) * Fix a bug that re-executed elite results under some circumstances. (Leslie Pérez Cáceres) * Restrict the number of maximum configurations per race to 1024. (Leslie Pérez Cáceres) * Do not warn if the last line in the instance file does not terminate with a newline. (Manuel López-Ibáñez) * Fix bug when `deterministic == 1`. (Manuel López-Ibáñez, Leslie Pérez Cáceres) * Update manual and vignette with details about the expected arguments and return value of `targetRunner` and `targetEvaluator`. (Manuel López-Ibáñez) * Many updates to the User Guide vignette. (Manuel López-Ibáñez) * Fix `\dontrun` example in `irace-package.Rd` (Manuel López-Ibáñez) * Fix bug: If testInstances contains duplicates, results of testing are not correctly saved in `iraceResults$testing$experiments` nor reported correctly at the end of a run. Now unique IDs of the form `1t, 2t, ...` are used for each testing instance. These IDs are used for the rownames of `iraceResults$testing$experiments` and the names of the `scenario$testInstances` and `iraceResults$testing$seeds` vectors. (Manuel López-Ibáñez) * Fix bug where irace keeps retrying the `target-runner` call even if it succeeds. (Manuel López-Ibáñez) * New command-line parameter ``` --only-test FILE ``` which just evaluates the configurations given in FILE on the testing instances defined by the scenario. Useful if you decide on the testing instances only after running irace. (Manuel López-Ibáñez) * Bugfix: When using `maxTime != 0`, the number of experiments performed may be miscounted in some cases. (Manuel López-Ibáñez) # irace 2.1 * Fix CRAN errors in tests. (Manuel López-Ibáñez) * Avoid generating too many configurations at once if the initial time estimation is too small. (Manuel López-Ibáñez) # irace 2.0 * Minimum R version is 2.15. * Elitist irace by default, it can be disabled with parameter `--elitist 0`. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * The parameter `--test-type` gains two additional values: (Manuel López-Ibáñez) - `t-test-bonferroni` (t-test with Bonferroni's correction for multiple comparisons), - `t-test-holm` (t-test with Holm's correction for multiple comparisons) * MPI does not create log files with `--debug-level 0`. (Manuel López-Ibáñez) * For simplicity, the `parallel-irace-*` scripts do not use an auxiliary `tune-main` script. For customizing them, make a copy and edit them directly. (Manuel López-Ibáñez) * New parameters: (Manuel López-Ibáñez) ``` --target-runner-retries : Retry target-runner this many times in case of error. ``` * We print diversity measures after evaluating on each instance: (Leslie Pérez Cáceres) - Kendall's W (also known as Kendall's coefficient of concordance) If 1, all candidates have ranked in the same order in all instances. If 0, the ranking of each candidate on each instance is essentially random. W = Friedman / (m * (k-1)) - Spearman's rho: average (Spearman) correlation coefficient computed on the ranks of all pairs of raters. If there are no repeated data values, a perfect Spearman correlation of +1 or -1 occurs when each of the variables is a perfect monotone function of the other. * Many internal and external interfaces have changed. For example, now we consistently use 'scenario' to denote the settings passed to irace and 'configuration' instead of 'candidate' to denote the parameter settings passed to the target algorithm. Other changes are: ```R parameters$boundary -> parameters$domain hookRun -> targetRunner hookEvaluate -> targetEvaluator tune-conf -> scenario.txt instanceDir -> trainInstancesDir instanceFile -> trainInstancesFile testInstanceDir -> testInstancesDir testInstanceFile -> testInstancesFile ``` * Minimal example of configuring a MATLAB program (thanks to Esteban Diaz Leiva) * Paths to files or directories given in the scenario file are relative to the scenario file (except for `--log-file`, which is an output file and it is relative to `--exec-dir`). Paths given in the command-line are relative to the current working directory. Given ```bash $ cat scenario/scenario.txt targetRunner <- "./target-runner" $ irace -s scenario/scenario.txt ``` irace will search for `"./scenario/target-runner"`, but given ```bash $ irace -s scenario/scenario.txt --target-runner ./target-runner ``` irace will search for `"./target-runner"`. (Manuel López-Ibáñez) * New command-line wrapper for Windows installed at `system.file("bin/irace.bat", package="irace")` (thanks to Anthony Antoun) * Budget can be specified as maximum time (`maxTime`, `--max-time`) consumed by the target algorithm. See the documentation for the details about how this is handled. (Leslie Pérez Cáceres, Manuel López-Ibáñez) # irace 1.07 * The best configurations found, either at the end or at each iteration of an irace run, can now be applied to a set of test instances different from the training instances. See options `testInstanceDir`, `testInstanceFile`, `testNbElites`, and `testIterationElites`. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * The R interfaces of `hookRun`, `hookEvaluate` and `hookRunParallel` have changed. See `help(hook.run.default)` and `help(hook.evaluate.default)` for examples of the new interfaces. * Printing of race progress now reports the actual configuration and instance IDs, and numbers are printed in a more human-readable format. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * Reduce memory use for very large values of `maxExperiments`. (Manuel López-Ibáñez, thanks to Federico Caselli for identifying the issue) * New option `--load-balancing` (`loadBalancing`) for disabling load-balancing when executing jobs in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster. (Manuel López-Ibáñez, thanks to Federico Caselli for identifying the issue) * The option `--parallel` in Windows now uses load-balancing by default. (Manuel López-Ibáñez) * The wall-clock time after finishing each task is printed in the output. (Manuel López-Ibáñez, thanks to Federico Caselli for providing an initial patch) # irace 1.06 * Fix bug that could introduce spurious whitespace when printing the final configurations. (Manuel López-Ibáñez) * Fix bug if there are more initial candidates than needed for the first race. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * New configuration options, mainly for R users: - `hookRunParallel`: Optional R function to provide custom parallelization of `hook.run`. - `hookRunData`: Optional data passed to `hookRun`. This is ignored by the default `hookRun` function, but it may be used by custom `hookRun` R functions to pass persistent data around. (Manuel López-Ibáñez) # irace 1.05 * New option `--version`. (Manuel López-Ibáñez) * Terminate early if there is no sufficient budget to run irace with the given settings. (Manuel López-Ibáñez) * The option `--parallel` (without `--mpi`) now works under Windows. (Manuel López-Ibáñez, thanks to Pablo Valledor Pellicer for testing it) * Improved error handling when running under Rmpi. Now irace will terminate as soon as the master node detects at least one failed slave node. This avoids irace reporting two times the same error. Also, irace will print all the unique errors returned by all slaves and not just the first one. (Manuel López-Ibáñez) * Forbidden configurations may be specified in terms of constraints on their values. Forbidden configurations will never be evaluated by irace. See `--forbidden-file` and `inst/templates/forbidden.tmpl`. (Manuel López-Ibáñez) * New option `--recovery-file` (`recoveryFile`) allows resuming a previous irace run. (Leslie Pérez Cáceres) * The confidence level for the elimination test is now configurable with parameter `--confidence`. (Leslie Pérez Cáceres) * Much more robust handling of relative/absolute paths. Improved support for Windows. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * Provide better error messages for incorrect parameter descriptions. (Manuel López-Ibáñez) Examples: ``` x "" i (0, 0) # lower and upper bounds are the same x "" r (1e-4, 5e-4) # given digits=2, ditto x "" i (-1, -2) # lower bound must be smaller than upper bound x "" c ("a", "a") # duplicated values ``` * Print elapsed time for calls to hook-run if `debugLevel >=1`. (Manuel López-Ibáñez) * `examples/hook-run-python/hook-run`: A multi-purpose `hook-run` written in Python. (Franco Mascia) * Parallel mode in an SGE cluster (`--sge-cluster`) is more robust. (Manuel López-Ibáñez) # irace 1.04 * Replace obsolete package multicore by package parallel (requires R >= 2.14.0) * Use load-balancing (`mc.preschedule = FALSE`) in `mclapply`. # irace 1.03 * Use `reg.finalizer` to finish Rmpi properly without clobbering `.Last()`. * Remove uses of deprecated `as.real()`. * Nicer error handling in `readParameters()`. * Add hypervolume (multi-objective) example. * Fix several bugs in the computation of similar candidates. # irace 1.02 * More concise output. * The parameters `expName` and `expDescription` are now useless and they were removed. * Faster computation of similar candidates (Jeremie Dubois-Lacoste and Leslie Pérez Cáceres). * Fix bug when saving instances in `tunerResults$experiments`. * `irace.cmdline ("--help")` does not try to quit R anymore. # irace 1.01 * Fix bug caused by file.exists (and possibly other functions) not handling directory names with a trailing backslash or slash on Windows. * Fix bug using per-instance parameters (Leslie Pérez Cáceres). * Fix bug when reading initial candidates from a file. irace/vignettes/Warning-icon.png0000644000176200001440000012750114157104332016414 0ustar liggesusersPNG  IHDRXPStIME""NbKGDIDATxx4z/Rދ ҋޥ "vD+Ҥ(* (H]AEQHoIH=g7ݙI{}J ٙ7!H,n$ʶ&[J -LW2DH5L4z+rS0?N?6^Ex^DbMAX5BZ^HQ|GQ "yUfRī A,ED jZ-hf*"x~Nϼ1DLa9.':*;7= =DiE|aSc=QέD}, o[BL"bh\@T-2@ZH 3b+Q+ïrq dG> 0 o F46^dgVWҍD-Ƌ,c5 7'@#?Ǖ__%-&L25Dq)DYYH2D!=U*t1uc>߻#>ݓ/̧:ߪWo3gTWRU}@d$  ARBKڰ #j:3Od+[qe1Ѭ0y$o(3K&>ףǰ{"&{dG&Uy$ "^rdي(["A[f.#eXl8g WW;ҤOuLp\ICyVWqXJ#2MՑ+s:`%q%]Kt$Q *#⪂Uqùĕo5*%86>ψ>%,8"cE}*򙮭V3kf%q0DmMK޺U#hʭ_*VWi-A3!*oY@ACDUqMt|y`D , )U,~5*+L#LիM%Z\I/[SxVF\mp _@T$-QPW?$QpjJpW{c[vo%:l=(ڥWvIнvȦ$B$1ψ(rkPB>6U|k6aY:=oΕD'`Yyq-ۈ( SWf輍8ss#-|g,>?.$ #t8r`/U^h9W1w-^wiR+VQaD UվBmm F|4 ,G^h b!"#j[HmanjbO8|]RWq:F1 IT^UP{7<սy-,-yS`-uc9*DV~Ip;jW,:5k29mǸ2|)%qe8h2ofzVɶM̱؟x+xktJJ(%Qc*NxR|\VW?/lo}ٗ0+tHo>ҨzP\.v~O*C8t"NSž-ՍqZl7UZ9)\yVWo 1 2T8 *W|իrw&<ޱ]W֙nmxcP-A3WGT$ R^\UW u+C9$i`Jjm&L8(ڮa|ӓ3j(;~UY`!+/<1v㭙2(DOV a 8 ʭel\\Y\XKY h &~I*Vnq ,*vM ո w+c_W̧$ &z$ ST^1[1{ÇFCW`=`%q%]ItQ *#ДmDDŨ=䲵5D+%5{իFE%q&ιDVaf! QW`{` `+ϱ5Xvlk0~`]ԝgɬ$E%joX%A8J4H/Zڰ6C+WN[6JPXW_BT$hz"o2˃[Kby_~_W{PW 8" Q4Oձ?8cЧ<{^ycp%ѵjDuL"H1CU\?; %y\,>xoc\zyDK|=LOA,eAA|G;quA qc>KxϑMI\ Q7*MxD#Tn Vի[7%[\FX1mWU08B?cQET̙b5nM_3O?q؄*.?,$ C[p~T^Xl1c'JvqqzexwGޘ&uBJVQaVLXUXߋ,L\NW׫$uG,'J-/Uq5$?r\x륄?9c޽|g,VwΩ$t K!8r^*תb}>v+?Q+WvZ}qt)DL"+?\GJTHQڕ=ngMٸ~p6-ӭ%_2F4̷ 3|GTŕdfGf/}:۷q}=ƕBN)}:%qRN &&U0"ڪ + /~X%q\޽w>1 hWaeĕ|4LW‘ ;2H:*Nu>>5{Wg+-V7ɭ$ _'jXepDz5QU\IWs}9Wq]/6xN-yMP-A3&:Id $3[JP*{(m9nϝwqŻw3|o`XFI\"C[pdPFU\m ӝWF=ÎCe\Icn+rC\]^ۙ7e̠$ޣ:=iX%8d@DQaes#׬۸]wO渍+#6yD{ܬb$1;`*Mq~WN+]*WI\MI`Ƒ$!"fMqiT=㘽{/6+Z&W3]Z`[+L\JtQM*#H"DU*v?`giW7;v1X]Z^I\!O[Yp@M?́|[ c#Jز 1{JJLr/&UJ#HDD=զxTUƵŕ9rG[KzE}j ~At Id$rn%*vdHոxߏ+V +޶G]r"ĕD|=-@^}`|6{`8vZo1΁n:ۭ MJI\-w _JT$(f+QPUq<|qt=yիK\RZDK|= ": >׳帺 єJذ 1[rdSWq>M$J  "jZի6iG~oՈWxW\DGz &Kv!u-Aod=-խKqqexQĕ$ dU D0-UWYl1c"7neK]Jz{[^&uBW\7 b;ʸH[[KZ+#4+ްyz?ρշCu)+ֹYʁw`DiDLvpKHr\x%WDฒ:֮+z+mr*83 2SzSU\I]ճ}8)>[Qrb`h۔W$(+N lYN,o wfʡCzY۳"׮ub?-$ M% x^U\X7>`vrPm\ɰ5k8vtOӸXzu&dK.݂I`' Q^e\/Y`{%Wvzg$Wtrĕ%DQZVIa Q9{+eKl_qe1xre8b$quNlTGI\~J*xGH "b겷VҥKYcW6ҥ|!q/8)VUŕhۇ}? /^1s?KrC\IOwjS(+"UDVz\CU\}_ ;_|rJU/Z7?uWzWp6d><ڲ(DojcPkqu}HWHy+V+[q%u̟ϡr)q%gUWq:&iXxuj zE[se*⊿yBǸ2<ڢ`{KD'`Y" oyjW;d=-֭iC\%r\9]|R`]՞w/$_ %H[N2Uq%=Ѻq}$+iGs\Jzs+^&Wae՗uV1aލt* ,Z`{W WIW<r4 =/(+9Dݬb;߳(cjs>ߧ帺KB\%Y\3)_+WrPWÈVaޙ}-[=V`{>sVԔI|wGq%=޲hkŮU1GqUTT&G6r`ݞ5%W7F>g).l{JSWVaFC%"6+[C!WW>wGq؋<^l{˕TW>&xY(@.Uq1 Owne}g! qq%꾸9s8v ۸rֹxu jkn&URw0d3SJlZ̿JqexghӭA#voˇRWfa"*K˗vWaA\x\2ϚWWm(+7^&4x')D\}*G׵0a8滝+_ٳ=mB umS^uX' -AT$ V cUzwys tOJeqN%ZD[Y,! gEj\]u;d<+#W>Wc7/ʁw u}3^*DI\I/5AH'BͼU|gkk΂Oĕ +Uĕi֋=ƕ8? ZANd' !uXU\m s:XlWL!>xTvL̗uWs][YUDL0xITl Z9ݸqI\)N#njXE`i8/V+BT$ B_mUq=kfr`ݚnƕ ++V>WN'OkC~[2O?2*Lw>^Z*'4>>bqa\s̄wwxWiR+;߮G$$b\v*{5x>JӸ2@J^ !ngAF\}NXZ'-Ur\?a2x$/f)ʡ$ !hXe!8P&Uŕ:l Goل={i=eW维͟E pDUL"Q (CQ1WjGl:=s2*ĕ\*.JQW Œ8#P[ZXk#CNz0#IJ;~_c\МK-Ώ3D3f&U0[6uVWJ5dƕVJ*?cb#׮*qw9|pqexJpN7Xpf`Vet-AiR]-Uظ\ak\ɏMṄV4+0yxK\Jad08CH"&+^=חco3;} M;8k\V|EC^hkj?%:_Idm6*vikk0|t=JY)r5K=#Jr:~<}\tOJDVaf)XD@}>-._z2t+s8Qu}DZcnV\[%xnO$J q` J}Uŕl^E-_bW2uENTu kWN~0C\KzlCʛD8cV"B4ܒ>_`oW;!K\Q\9}-U|7{DI\L [DHQW#Z~c~q%J*k q%JE\4՗X\|e#^l+>w۽帺0vȈ-Ǔvq%u{W:=x_%uۆDM0g^=*zuw2JHT(&G<܃^qYmĕVIW[y`b#_;J(>ߥǸݪ1/Ng&#Qq*Jx4DO\JދJ>ƒ)(:s`utcJp* 7XqfT"vjm냃\_)J~;^}/1ulg;4ٲ(\~DM0gHzEU\IYճ}UW2+髯rLux~@WaWNIGT$ )G vKp[|yp/ˁukʻŕ?'q5Q\9}&=.)$ G[pUJzM#q쮝ŕ}C}W" cFzNI\4էzD MaRՓ*bl G,TZeiw5.XU\/3O*؝2UO#f+'8XCND o 7 };[oOXS\ɰrW<]\Q/ȗW.ٕĕ3DOVa<9"ƫX2nu[WDD :UQ5WRWS9D֣8l *.*RU\mϑCX?_\IeB>G~7Fq%WxHާǸ:7 ++D3|0@VUq. Oոv|K~fq%}~8_tk0~`jӄJ6A[KS&U0Z^uv*?.fk=rǕVWR?D~x(c\YYI\O'f+7DU&9Uq)Mп {Jk*?svq#F8ܭǸ2\C^W=oXeqF}k:Eq%=\իrŕQQEC[M %q%|&Ug$MѪվӼ+VWK27k|/FqÇ;Twqu]3JSW#,83D|K+/=Pŕ8tJ|SKX4%ۚr⋚DmL0g(իJJFSҶV./~:! RkWǪ)+7Gp?d9QDWTզŕA֒  ĕV>WNCW>$>%W|= TWsUŕ&[~usǕ +emfq%~z kR`k>/rJ68S!(7q&=bkkܙŕi⊟yyPc\Iϴmߕ-$ G%g D"AA|G;quavJ~)C'dX%S\I [z+̙% v#bXA8s#"j-AUZ|~q%Y>/pVqC8)qexJpo!DL"+/\ì"zDTڒ)_~/,q(&,,+>&'O'KXoppsphVqTVdJzZ^6X"~\HjK,ɗ彮_ŕӧtX ٔĕDM0tĈzGU\Iխno}jJ<>dw3dWqeXzu;/_#_+#W}8+gw q ?ywN3N&jYp]*!)JE\n˙CX_\!pJcXY_ƕa6m̍W:zeXA%qe,2*̄3$"Xq& Ov|r\]{q;١_\?Fl|~WU,,woكϵm1r}a ?$lNI`:nJC#}_\\w~X7%ݷo wxa!W/=v[UcZE%qe8;7XqDd=Q&QTib*l+)?hkg ʰ\r|]/,xo>[5Us&hKFVYaj 5ynիr̺5WFX%e\ɟ+gE*m _S{]}uc-swzW -YDI\#̷ L C("{o 򩮭z~_\ɏ&C^.Xދ!Y\qnۭ+oc\IO>ـȐ>AqMݖI`15PU\IW(mk*/+lٲVFqB|U\9ڕ;X[65+++éD?7GpFH!ꊪژ>_r\|goĕ +Uq%gq%wXiӦ(hNΜgKrڥ _nc\lkNЖ#^#83Wx;o?ֳ0a8vzJz69yqeG9;wY\XGOlLkʼnjDV ֋*ڙ?;s>/)Ngap|FWFXq\qNNomqpWJ|=ΐv: >ۣ帺0vȠ򗸒ae%ǜ@٣}`LkWRGǎ|uq%=d^)c3+QW*Mx#z^U\IT-ok/+7o&8 34cVq"CӮ:dCvU( @E\9J[z &gJ<"T9tpO_\-\/ v>ʐI(.f4+6<ƕyĕ+D|0Θ`}KaexT\,7WC` oGwq%۷mY\+uy^j?7=aXE8kj2+gWs[RB`iƻ)2y۵sֺի]¶*V9 RUq6$h9n>VF\aq%Q?ռys_+nۖ">Ϸll5?_fͬ$tWZ6W SW\X2*̀3)wj[ymS#Cݸ2WJyyy8{, ?j1-OzSMOӤ*fsQDML8aWmT՞ ~q`K0ܦ]l<ƕbi%qe8hUpF&"NuS񅾝~aK\͉5LDnjWN|/X ,ٔĕ@&UF gV@Gһ[kl_L?/""Q+SLZ/E#[,թx_ 5oDsU58R4kJ*=CXNLaq5>#XɓGzwԿhׯQ\9mޜh>KH%q7ţ>tU gX@D\DLvKpu@帺6r0;l/!H(QBյQC!CJg1x`k׮]",Q#mؐZ+ɦ 鼎1 Q{*) -"+Jel^E-qe,oDDX?S⿈<ǎ*Xĕ4qqes {D?7Gphz"z;oe!Cz4帺9uJ?'%Ǐ:~WplkKrڠ_jj!s\ Do{Z:^}*9DZkVWN>}ցuԩ{1Pn|B\Iz VWiDJ1"8b c"n ~?;W}2)c ğs`]IpFq'z',REĕXd> gd Zg{W_~2%ǟ X2z.z\Sq%=q^)a.DL0gfW#TŕTYx~q%֭d u99>noǕVE\Iԯ1 ֨3Tĕ'`Yyqf{\uy+Í3qֻcO?e>x^۸ʘ1cgϺ4+~qurzd=$ _$f0%JzU#p캵ŕs풉~Mʙ3gcB:u8|ʕWҿR؊nJtQ}*& Yw%jbKh~q%=w]vVp c\au_\կk!wLq%qe8y*2]ʁ35AF_jMH{帺WPWLXXUL}qAc|=ƕY2+68z%~ѪJK*6܇,/Op3!00PVZ'jW\ӈn XjVq{W>DL"+?`QaT[nΞ/e}}Ҹĉ+##͓O+d˖M/!x݄ŕZJ:WN6vѼJen 3 իuJh`AXNnlVB 2L|y,W"֬1kgHU:֠6M:![t ѝ:DML8aˈpjbKЈ% lCO/]b_RJZVw|ݻÆiWNkujy\:ٸ=J*qC*C8|%DDXVWkR Xl+t]א[i:}E>p@՝oPc\.ΞUI\'jXep&k&+z5 $+q%}XFz$⊫UZ5\1e!~]gŕtDUM"BUiQw2Pr`ݞ٩帺6r0;6/&S烌1B2e,J?eWDI4 |S4 @zSŖ޲% Ǖ +;q%\QƎe`͝;^lXõ+[{\:٨*VWq%Yʍ3; YAE%Uq6]Z8 {eJL>]R~.}w`iW\vTP1qeJp(ы&UVgx@!"i'Qz/ǬX_\C~3_]t UMWFXx\IVX3-'/4B8SU1[w{ܶç_\}1co!w+Vtz.-ժ&f0 $e QUZIWnmz2t+1;w2{O2,BT4|4x>3QG*% WUŕt_2V/俻~/СCZ1__=U:bq%Gi|ZeW\VU\dDBʛDVIz"A-qWc̙3ZEe_Q6Fq1-tW*2F4HիW)+?=a}}X?]Bwt֭[ ۷o'`,J]۶iW\4;ďg6X',c[VD$J q%$jm^]4_M "EhXOx( T1 T-Sĕ D'`Yp%(eQ^OTՆ,8tpOquk›G1mPE\Hu %K'An-Y,KbK{+yeJ8'f0=ex۰2JWFۇU+ƕ +gm`5pDWIWFX%b\IK j{N[g$8޿nW"oXń*FQ3qH[[Ws؟СVU_F?rsZ(\8ߨZc\9_-YTI\!Zf+' AFDQUqJ|yO'quVĕPsU`W#~4+Cx?ϒ)A[f oXe!BYU|>VUŕy~X#G*3U\F+c)VS;LT$ HPWg!,_\͙#3:0n8k [ WFX)+КULm-^%q5)ΑDlfĕ`RͬyvJ~<5`ƌZ(y?!MJ#~Ω'j]q%=V:J*a?&#51 ℁w͸2,jkk0ŕqBXoVTsϞZŕ?e++ñqTU\b,(3jM|_otEƕVvJv8Nu`XSL?n\ɰJ@\9?\nWd\IOȖEI\ŭbq_gaЩpXYzyWճ}8EŕZݻ q%eǛoWqF--s\Iը +#slnj&UW[DQuWU\mɕC>;ՇaׅOߵ /qI`^z|FqexzeKu≚@^%qe8h̸햠ኀ>ٹ帺6r0;V2ʈ+9/zֹs x%,%Kz5=ƕha4 |#jD$J q%UDUŕtoۣ>P?^e`kX:5 (vU\ުX?q:!Bl8+ñD{ݬb=+ _%*벪Z.-_r\yI=cE;&&Ʃ*U*m?x%9w2K2P+WsfWW^"4^RVյ~ϫr̒/+D߹u`=CֱcǴ+itt4OhWҨ2^2O 5oeEtbQM*+ j"bUնmm O_\͚1G8/p:VbŴ /jXѡMjWתVXҍUW-tUWR8KD0t ƍkW HvU\._c\9cRJm!DM"+0PVuU+#eLϗW7Ǿ_\ɭ={qeX:EVN̙3kXƱxW>d\`iWkUqX" uX1d>WR "櫊+ϱ_/.s'|zw,FEEq̄ ŕn ( G<58XI\M+H-IT$J@ C"jGL]\Is^RJ`KZVI) \a{7^ǸrZo-RPI\<gR{;oe"8`9+4u\X'N"~,a|iWRXGG&UZ+)ckpT9+#)ϑ)*>o~)&"#"8N2W\V*I*D#hYp@s1u-AY2q̙θJiD|>t֭[;F#vbiWjTWeJЖÉ&*L+#Bim`tvqfDxx -[hX={LQѓ'kWR%( g*<-$DI\D L0W!4d%QsqhA[3jW.F\XRPҘ+Èe<^*ĕt,9D֣" "vK87ơCzYﾥ]\9wJ%?qa)?)4 ŕ=ƕʼ+ q f0#LhMJz}ss1˾.b~vKʚ5y ,1>)e!ԪSJpXŏ+;DV ;~WTREmm F@9yyaUB }`Ǩq ~c5* *WNE,YTI\&*V.\Sety+UiR]+t>໛7۷XqT\kǎ)vJ*۵+ó5~5sM? :T!"WWCOԲu+u׮%J`kd5lKn"4紋+idRq!ez*ZO TWqDsU3Rg *'*-A͹sr]\I#w1Xbu/9q2n?0Jq ,yU,xVaf\#@.Uq, Ovie}}``2ʏ*F|lnbmB' x񢶫WvKߑY\IcK]2Ve6WqοېI`l QW˕55gvq%ӉXYFÔ=h֭78' ,e=ƕUywbhݬb=+>l".{;oe:}Z8 =J>o7o.@"+*ֻq% ߵKU,L%hf|= `ՇJzq]s~/,Z&dXH\Ŋֵk+%m?VlٴY&c[һ#GjXQJ:J[<)(PI\I& -LT$ Xo }`+_wѹ?M(?K: (b͓'QuRyqe@>%q%/|h¬{[A"~RWWmW_SUX&͛7u`w0Ka`Ŗ(佱<V>V2J֫2j -:V)ah+QW+7>CJO;/4,6}:{1l!~`]1kY˖W{˔PWoBT$^Auy+õ2{Xc_/Mo7nX,ۄYǎj޼ۃ 2yyG5 *z c\.~(2|htZbJTgۇ_hWq劥JۄW^ҥKX^*ŋ eZU,ŕ`[V|zYM(a$/"aPW 泵5(s0XHSoݻl>1s` 80EoZ gpl͚z ?HG5? WWXq C"갪Z&:;ӫrڪ_xT~WQ˗םYV+mf͚o^J̻?(|/UDBk-syX^rY dHu\GG^&UZ+ɳ5VKSJj@gZ* _-F=eƕCgMX`A z/W=hwʈ+鵫W9[7x#KླྀeX+W`I+Tĕ1D#hYp YB_DTZ53i?/|/ []sU\)|g.vJmBJ*m`Mwk{^`E?3.7quGxwT-KJ#{j>VǭA#ϕ.ɝ3A[LB@Z`|ib?5o\jt|ghwQ "bğ{ʕVRo*V 6>Ǔ^ݞ$|XssU? ?qk0~\IWCB8D MaAT qmBjh/Z!i\ЪG +#+i3.$IM*V6=>ֵ9n] "]sVFXX[d@Ji[Kh%qeF7X9q ӊ:*V }:ͫX^x'*rjg\%v`:޿ 7p{RlJMy;_z`|ᬼ#kn~p&˫W.+9)a3 *# Dd98o{N5W:qZe%7|&_;wr`%6?~X{n*ʕ+էOUL|"O6ko?,ĕX҃ Z%q%Mt2QzW@/TWrdС^~^\X7ZTI$k\Io `_&zw6:䗫W=h5+;ʕK ,V{ +wqB\IgῪWzuRWpUWBgj[=ޡHI\ݳh{c&OL?C/_v^0GVro*ևsᯁu)-ۓj{xH'MJ:?r/WYŕW#*ԩln+M$JPWUwܓdkm\yIZ4'Y\I9¡ &ꫯ6Uի,yi0ꕝ~X`oRWN_!f+(b!Q&VgU4BWVHܕ۸ V58zۉW'k wۄbm޼o^W;Wɵ=hV]Lm|RXNeɔ{ 7 Ը2fj2aekW+ClU*HxG3Q*Vгg#Ke`麊?e\e̘oVs]`P!="vUbĕ[`۲yzuuBJa>Cxs!*%W+OQG㗫{o7Tʫ\q, ψ^g?}'YQv7k׮ϴ+vAE_g} |:5v*r_rELؗ&WNuUSFDfx!2ULh~}+gl/WS]ȕk~Kv3KkLh)M'Xsέ+$X?>Թe-\m(Wv\|!rU-X^Pdh0E1Qԑե7Du+;??(~-l0di_ʗ`%ޝ%oKFlO #ԔY$>MN"WBTwR8pڊ+vQE_g{Ӵ+g6U;5MߊP k?V !p8W+ʱv-j|(2+JQ^IDĄ"(4kΝ,:+t, LG$ʋWKgD݋G|Bd$+pYPi%F V\C/2_rE2˟u }H^}^ܻ J 12QlMOXJbi$Mb%ɕT^V  N"LkE½w-ջrWc|wLYv52BTHvf,}6b4:ңܮ`QN̞iRgBv4Lf% !rfuw8u-Rv7ᗫͲxS.u+IrlMk.E^Pd]HQ!l6~- j^*k ]&6rFaЅree-'S\IFh+p"Wߊ+ʪ!ճuJgwULh)$+%%4rmJxYk' LۻF;S"r(OV32UJ;.:u>G9QrE)؆_& 7 Ct]]u,_8-XbB#LF4h`v|2>ا){Wr1L$W%»zұR"Bd$+ pR4XH?jcrAwX VLk,[//re䲻) ]t1`5ip+Orez%ܮ5t»R#Lٻr+ʖ |Ec镝MUr zK!*;/Rl&B_OA /WYwE^-K\KPm]QM#X:u2U4JxP,ʉ'fv(\Ik%Wfx~H~B=ew_8V^ %W%jH|IezWWjRb }52d}21ZnWJ|w[kFPkE|fؑ%D$~^)V*܁X|P F\ ?M/XݛZc43.W,eIJ)KT+ɚ?ik̘1}UnRjO g5+g؞n(:ߢ1~?6o03Y2՘ wb8V^&J(痫q͂`k,JU,51ewQS,w_ ֲeL#X>AQv+_ăkÇ JH^:\QK oJGd2"R?-i)rEiSDb7j ]02KTxO?mXb-53t;ӣMٻ$XclLr X7ƫS%Vr%1 QaܑX搫yZVO;/W:. ܂勲;oS,Q$~m"XrbjA5K9oMٻzžx+ʙfaB䪚vE`lpgchp((+ᗫ#(k)w{3 Pm])oT+ɢNpp)룏>2\)rzG@\Q*w◫1}F6!XUYsqGL)OHEKAvmޕ+=jA{E|h.+;N+`l\g "WO8692 wh8=Dn#*J6ު%L|h SXKN,ew5(TEFe :uJD+r;s<$X;2gR\Q6D&E^AI4+2҄ȕ}-U('NTk[m6rկuzW]#|` 9UTKzM!X^rg4h镨xPz-uLۻr/;ScAI(Jar<܍0*;3N%WíՔafq%ڣH|qYv7WxbUhhcdYWzEă`}xLٻr+,Ѡ`QjSSx(WvAt0Be2wl89!Tߗ 1Qե%,sa'b Uv{5*KN8ax*++ӥЮ\izWGũ\٧XJx!r%1gs13QrE9۫\iR;R<3kdi 22dя)((ȒQ+WzF^Mke… %*&Yv:#*dM2Űr]p+GF^8}xU*@R'U=%K(g P >_"4NFJAp'cCqQrpr5i0YjeT/w,_ƄFbE> V˖-.WA +]y1lY1xqrfrfB(DQpGc4B\Q._meT.I.{挋`2&˗Q+Z뜚7Q3\QlkzW<( #v(\I :Sٸ?,D("%#X@u48\D$(_7Xb= 6gKiQvc~g!dN prG4h長x"]/Z`ޕ\Ilg8U`Qމ[=a=,#X p+zC\}F8uӫq KӫVi񣲂-bSx׳"Y"EK-]BCCʕ+u+_ȕh+Qv%нzwI(yrctHVތ5$W+:jfR;»/E'&u]Kt]/bfi-5K;J/ٻҺAJm'Ug725ZU>5X+K1˕U}Ɓ

iJiVZ 60KX+ޕhPJjA ^>u ooQdR`Q2,]9BJbBkd\%q)JVEFkSF Ŗ]w^͝nX: KTLȻKk],=D]v܉333*88>[nP+HP\}1R[nJE9t1hR\Q6ƆTJ,q1\rF!B=Ci(Tݙ_F7,LԾ]C\Y~b):&4KT睅"%KIxdB?ΥKqᲲ2"S񸰰Ϟ=^*UZV+bW5cȁxZѠ3_5Ha^I9 "QG/9.#Yy`pDU{¯ZVlL痫cV,ӻ[' `i݉e)Q/$KhʖHԈHr-T3RA+D8S< ֚@|8Y$ԯ'DXx=-'|v"D%Wl6|yd~%~g ژWewXF,ih-[,V\"1ҳ"XC?cޕ\IlMf*-/jG\nV መ^ Jn]/W~]( (|fvJ=Fb%B}劵Ԯg4h镚xӧve]˕r%.@:Ss1feσE.Wdž.["5&4KDT+Y-Qg?JD4Slu7*[SHv9Ta.JT!rU-Xx:BXI` pFB0%W˃񕱃WZ[cTlЗ"6&4KKT(RLo)WF^,{Mۻr+;aoI+ _$V\UkO+ރFO%֒z5i0QjX|f^拤jX y%KmdE|-[,XL WzYW~)˰K[SVɕġr%?7 ? B<75L08^("!_>_:XbOr] b EOލ(YjDUDKϟZʕ޽+3NƃΜظoL12Q •%'BJbB)DQ`pM֋+ʅ=jXw˔)W:Z}1ZIxEKo}S+HP\ {z%>sajvsY"dEy sWsV(i[I,!,嗫8M]}Ϯ]髲X zKhU}LZŊej#WV^:8}(ðDYYV,Ѡ`Q6e +mWbIu>F(%S+*W+BC{4L^l1L\]v7_U.?[$W^]nđw2|ez%1.yzj!#X0 8 R޹-\Mo[wu, W>i ަX" ZB032\Y`(tvOr%q 3K(܀!rE,0 8`1%WSc] /jb .YDKlZxVzʕAQ+#=qrV!1̰Dy䊲*†+L%I "92G]0کo%)^ݫwXcTl|0嫲)?BQOp~ be bzr'9{Wnf{r,\QNgl^2UHÀhp(--䗫i#&іX&(3 8e.2*dy-R+^j~_\+4J`>qo.62QZx^h`ٚY_\ٙ~)V*F>#"uUk$(2< 4i84xW˔)_ϙ(TxZXz }%Y-OH@{{{yJJJxЙwƒ` )oӫ|ތR%Vr%12՘ QwWo+ʩnWA,[Uǎ8Qv7r]; J*\c޶m~'p޽qII 4AAA~M6xxѢE1Py+rG4镞OɕľѠ`sldžk0YloAgQr~=~ӫi)7G].Fb)YY%A+V~Z2C&MycQbe ^],ĩݻ(.呫 ӫ3E$dt$Ɓqԡ4BDi[I,%? |7/`hy5nFֲѦXzE,-M駟ɚxȑ#^JXIDF^B(kR\IE2)MHCCU b ̣DSEeR~2(LrA8>}FewXQ?$'2T-^٢3<ㅋ;䇟Cڿ"JRSju4(zzjSxsB.+9^AIٙnGVz:N^sMe$>G݈TEk$(**_2?Zdev'h)»RTOSv؁ u+wbbbo(U"VzɕU]^I[hr)6:SO1Ѫa CI{8R٣ jto|#fg>0i%"*Qz,!hz!DrL =V"ʗѠW,{}@S؝5tyq[z%2bC_E՗ j|E$X>ygEֲS,EZ$KihѯMn*V ;MĊ%4\,ky栿 Ǜw$WdlfXU/IU$(V(D,xv"Dm66DȕQwh)P; "YjE~ #WʌrG4hā?ce,̉M \rEBU뗻!#X% 0Mf+er5y(QnR;}sQڽŻ_S,#Jhђd~ 'W鸢BXDJޕk2imaȕġ9+Ξ^Xre~* 0kӉ+RŽѠ\QNe'l6!rEy UFNL~u)WTG-L>ʱ+>EQ)BJ7ђdk٦+ +O]HUq^Uϔ+w, y O \Y"D$ ^ab0"PF\- W -ӻ|3/fbMX#C5k.l#?M#zhU*W< EFV^I|e9{W!!6)t0?CK V K$+2J*+{v5Bj劲44d+;yYxUr D!*+#j ~7^U]bΦ+S,B,d,-!h+, ~ѠWket H+ʱ»Qxk6Be$+Ƿ`G"Qj%deՌ1zzW"']rCOS,Gf,ڹs%+==]XYEZ]SkLUjOˉ0Ven^JJb>8"LD(ZbGL^lrB= "XzJOdFf9jٲ(rW4Slk+~;C/e+/b"R?"DFV ! Hhp(lӌz5y(^f\gg,)/B4KN:wl95jp\ ֕U O aޕ'_OV< V%]i»3 HV0 }w^5 2}+Uq1ƴQӫNyX&*qYB»K; IȐGx K/UL|!WzDui-7YWFLٻrO‚i"N,+*WDk+CzA!*ozR\Qz5eRZ*NYb(5*%Y"#CEٱckBʗrw օ LMٻr+q\rEGm6!ruV#Bؐ""A/ }4,26|f:q!4rTh$R+ZeeedM4y4\A]5 LG(aUNN"Wڨ0J#x"FkQr4$_7?D%2\yXBHH$[ekرJMߊG*C4W^62Q9$X5Qa\rE9Mx>,[S;Be f$n-ӢrVӫ.5IU%bU%Yk ZjX +=zW0̦N &{+'{i= @\~12H\bEYoL?jee5s~:y!T[xU]d-ٲcsk>W^\+ӫ xETJJY "2r% .F\I Gu0 I{}(\p\ :+ZloSg (YZD~,999 &Vf+(u`Sʎ M1 r*X'ǂljf53ws(k[0p@U N7L>z:}!4bT(ZX#CѢh"UII rѠ'o+Mٻr+'# {]SȕDv)L)8ZQr,,8a(fX&*u#_Q ,4EeW^ eU li{Wr%yx(>*WՂEy9!wn?_olR];t !ˌ,-%B(soذkΜ9LR%'VF+}Þ~c +%k4,W 4P@C"Be$+)jLZJ/%3OfXb$W N B_IPOڶm 1\i5X+w B4;,DYWYOdex%8>V\ID}0̉+^OE_҇_vLYν\E޵H^&[o& ?#UQQ{FDNDʕ/A,WNc8>6)tDʞR%VrUA`GcED{GFQ~+ ~Y>z.~ }-Y"YZDa%+;;*Vf+ W9M>w%B< Pz*r:T/YʝRbA)U<(ZN?zW.-j,>%Khy{pմiS|I!bW$\A]G;fޕ';DA$.D$"B= LJ^ :SqWG~_ϷYw"X|.> )YzlQ~mnݻk*Ube4hЇvDʝّbzg!di l9KCB'Z[oe})VF+}ñ5 g5+w|+n,,QBJbB 0BDŽʞ]Wzk vmT\y\M.YYZEKlQ<Νucbb*==s=_ħN$TJRX\Y=[Ԯ$X ٚ<8Q?D~!W)52WHQg"V-W3k͓-LTI4 a_XZ%Kddhi-g۷/Y3gδ3~xhV)\P0JQri _=]Zw&WosٕQ|!Y"Ci7-s!V+v+vNI[4,Xw"j!BTYFΜ{merUCa>KpѬhKx?6oH(rѕ+Fٖ\&#W⢙W5(D9 ,B`5"ԅb{Or5?[E}4^`8ȕ`IBCDPdFz%_ZLϋW W0Rɽ{eFk-I,W4#t6)L--W  #(Z Ӳd䪆,po{|qաpQթ+5l k2x |?/+bBmj7auze'L` ³!jw!4YFJ Vȏj@| DUɕDo׷ChrFUޘJ N I>4YʇpڍSk-fO#2Usע93TrRRS!)frp㳦pNGOŇd8xL$3j;mG0nbdvv0m0_/f!CLG$v%%0˕nZx"WS޿Ed5(G\!G VлR,IdZ \9/9g2\9+TPN}]p48)ÉTW8.Qϕc Hq = IUbmɰW"= 얈we;݉sC히uM |6KDfS-QlP`;P`;ʰFᮬRs0WVʰB"6eXEZC¢@|\)VE5GG=107h+ ֗Z#Ag. >e F\"Wʣ`NNYdrIxjj*F\m\\) \Y!μ+wX4pلe2BiP*T`r8_֍ejAruA\U!WU"AN ^q?!Uvg&@+5ӫ*KN ~""DyJPZhBJb BX,!W"MD(Z8? J,+?9?iUv:g: @i r&(W_ +F ~+/+v镝4;/;Xy+{z@a!V^oLyWck2QJU*߽]%]MK4D"䪮>s./IJM) ?&"ZFM-WK'A꿢j]j2\I3qUmR_zW"JлޕRzWKP]( m&oѠ$W cUEr iQa) #@jHPba꬟KPj7m6]E]ADYY(k"AgNǂdmP_*"1'J({ZI"M[jޕvij o=ҿv]w.WvȽ{JhНI X-i*#YiE%y5,p|uh[ ,һBϥv]i/oRiKлLG((W^S%WeУ 03M%WSwu^`(,,eL\j$W [bI& jVLDF}n\S6D~ Jr|ۯJeл,5E}KлeZKjQ:V"='BJbwxCrlDOl6݈~rua(,eL2QX&R'X%WSABj_"4XF 6#G3E嫖MNGCwD-Rjrq*W{VXN#T&#Y醔־Ċ(|mHb{MѠL4ſv]A BT媢"…ȕĽ=F{X\Qz/W#{:л8CwD!л[ʝ&WNu85?(D(kBd+P^P"Sj%6e7/O5Ov w%Bw+.ozWuw.Wȵ j)V!](%WˣOzeL2QX& Da*ɕ`QV2-:ImHXkGFV !)B+v58q0VD!g(CwD}ܻ+] ^AINU=)Aq!Wv"t.f2`C"Uڷ<>_6z!Aqeuqe~-sF|dNZ=fdjW #9Z#J(/W8C CwD!]ɱ>*k4,W@!r5QxgBd_ Dz%6ךC zWL`}~.eLڍػ:}z&bgE\HPmQb%A(Z7zգ,eLzWL2Q,e+BBAg:IG"z@h`}YQrE9ҡ \5 ,`(<zWY& 2O$vUEGR,B-e$S@?"Ċ|0qETe2QX& Da(,eL`DyYH)reP-Vr%17h_L6EӫA!g]2Qx3weR"ýFrE9^/ ?d "W|P;EG%SGk1Pj +]AʯD[j`ǁf:lgM|( CZ0Ū`*WU$, 8q4xWSX& Da(a(,e_&`hY(υh+IVcBrI\Qwm?{/lR;`_K-C zW~]&R;Ա56yz%Dȕ-Qbe!Xh%֑/\UL2QX& Da(,@f2\!Q"AOE5W8ar0"S!W _l zWL +x3<zW]ɱ:<]9(OI³m6r%qB3(jz_\Q4-^MGA ^R;`h].CDհ/1Y$6 +P*%M`o%42_ ol :s^~1,oq1]"X&b>rEхz57geл,5d}Kлef.+9:yzEr"% O&W Ew"D_4ղH|}hL2QX& Da(,`]@ɉ^YNCXX8:%*W}-G}C zWLzWлe:лʰP+XrE+G;˴ĩ58y(Zs8CL4ſv]2QX& q6\I!*W:A$#`5Q"E\Q.5rH0|}L& ve'9@c\a-ƸhFS^όOpwz;P??3<vxbݮJ?g頯d*O/.zVsKtv-ɕe[::ˍ;;x\C+]%nMv.|Y-mtr-:p{KWKsޙ$N+m9Ɲ2izV4s -ӼT;g$yA;EzAw 4vPNiڔ8,LQ<..39vJ3Y24LNf-NIdK4p%K"6 %]8Df}Wx"Avݩ_O4WR%w )IvXAgjq?/bTUE䨧VZHqP.ahWIreSGf;#Lvg-&9V0`ㇸpUb`yre;4ڝN#`-~tg!->v~pgDoW;˕A=\=(пGwUWrr镜\ݫBLjɕ镌\)NJ+9^W䪕 0!W\+r镜\+镜\^JNLO+߻`Rf܋9+AM+]M$NbhA!hA!+V%VrE3-XO#1B"WODT hA!hA)Ok+JGsVwn"W ײ D B4 D B4Ѡh9.V\~IA`-F-'7;!hA!hAͦo(.rT6ZJbqX>7D B4 D B4Ѡh@R<~i5XѰS(GHdo$(G0 (D B4 D B4ѠhpkB,~z8r%v"JC::I.׈A4 D B4 D }+xR@POqUc$\P3"V+g> _=ſ\jh@_3)[CSC!6nnr1B2Eyx3+6Ր렀rc5FDG-rYWCkYȖhP5rdIdPՄU.;dX)~5PA},H4~G걐?s&H,HJSJbI!dzs8$b刖'ƕjegD4 vHDEF?tDX3?P1*FZZCF>`} .bV-d F yYȋ x #yx232WyG #sd <,dFg>Ff0 Lcd S$&22A#V1+ N*&[ [(WqFzՋ:.WOk'\=rI$W-Wtz 4@ZFVw4[u\^$Wρ\i'\(WqQqQ W*+I`I ruE\}rIިrFz4ռ:.Ws5գu\fi\M(W\q.f+Jq>B"OMԷ2\oeF2RʌreHHHX,Pf2;١ev(Cf[I@|"Bm4V:y05"Sev(CPf2;(WPW rEDrltu١ev(CPf2Nru5U#!7P:ɂ2;١ev(CPf7z2KQ(1HC$7D~2;١ev(CPf7b2 ) WWHVw"So}+o}+[A VF[s/BoxΠܳ#/("ZI,or}+[A Vзgj8B V_S3J/ Wзo}+[Aבޅ+ <{Ԟ* зo}+[A V:/:ŊRDAF:EI.S t-?PX Cay(,,塂"Pr"X7!M"t}?XIn+Mx 'qP  + A /32_yA 3{cY<#VFNR%QBG?6B_BI-Շo&z0dC77QJHB&?Ӭ d}cz57_Ґ,|; S? @/hM)L%vSjZwY:}M8SslhB!Z%xxC]zUp8p8peIENDB`irace/vignettes/section/0000755000176200001440000000000014320017006015002 5ustar liggesusersirace/vignettes/section/irace-options.tex0000644000176200001440000004603714321271071020317 0ustar liggesusers \subsection[General options]{General options} \begin{description} \defparameter[h]{-{}-help}{help}{}% Show the list of command-line options of \irace. \defparameter[v]{-{}-version}{version}{}% Show the version of \irace. \defparameter[c]{-{}-check}{check}{}% Check that the scenario and parameter definitions are correct and test the execution of the target algorithm. See \autoref{sec:execution}. \defparameter[i]{-{}-init}{init}{}% Initialize the working directory with the template config files. This copies the files in \code{\$IRACE_HOME/templates} to the working directory without overwriting the files with the same names as those of the template files. \defparameter[s]{scenarioFile}{scenario}{./scenario.txt}% File that contains the scenario setup and other irace options. All options listed in this section can be included in this file. See \IRACEHOME{/templates/} for an example. Relative file-system paths specified in the scenario file are relative to the scenario file itself. \defparameter{execDir}{exec-dir}{./}% Directory where the target algorithm executions will be performed. The default execution directory is the current directory. \begin{xwarningbox} The execution directory must exist before executing \irace, it will not be created automatically. \end{xwarningbox} \defparameter[l]{logFile}{log-file}{./irace.Rdata}% File to save tuning results as an \aR dataset. The provided path must be either an absolute path or relative to \parameter{execDir}. See \autoref{sec:output r} for details on the format of the \aR dataset. \defparameter{debugLevel}{debug-level}{0}% Level of information to display in the text output of \irace. A value of 0 silences all debug messages. Higher values provide more verbose debug messages. Details about the text output of \irace are given in \autoref{sec:output text}. \defparameter{seed}{seed}{}% Seed to initiallize the random number generator. The seed must be a positive integer. If the seed is \code{""} or \code{NULL}, a random seed will be generated. \defparameter{repairConfiguration}{}{}% User-defined \aR function that takes a configuration generated by \irace and repairs it. See \autoref{sec:repairconf} for details. \defparameter{postselection}{postselection}{0}% Percentage of the configuration budget used to perform a postselection race of the best configurations of each iteration after the execution of \irace. See \autoref{sec:postselection}. \defparameter{aclib}{aclib}{0}% Enable/disable AClib mode. This option enables compatibility with \texttt{GenericWrapper4AC} (\url{https://github.com/automl/GenericWrapper4AC/}) as \parameter{targetRunner} script. \end{description} \subsection[Elitist irace]{Elitist \irace} \begin{description} \defparameter[e]{elitist}{elitist}{1}% Enable/disable elitist \irace. In the \textbf{elitist} version of \code{irace}~\citep{LopDubPerStuBir2016irace}, elite configurations are not discarded from the race until non-elite configurations have been executed on the same instances as the elite configurations. Each race begins by evaluating all configurations on a number of new instances. This number is defined by the option \parameter{elitistNewInstances}. After the new instances have been evaluated, configurations are evaluated on instances seen in the previous race. Elite configurations already have results for most of these previous instances and, therefore, do not need to be re-evaluated. Finally, after configurations have been evaluated on all these instances, the race continues by evaluating additional new instances. The statistical tests can be performed at any moment during the race according to the setting of the options \parameter{firstTest} and \parameter{eachTest}. The elitist rule forbids discarding elite configurations, even if the show poor performance, until the last of the previous instances is seen in the race. The \textbf{non-elitist} version of \irace can discard elite configurations at any point of the race, instances are not re-used from one race to the next, and new instances are sampled for each race. \defparameter{elitistNewInstances}{elitist-new-instances}{1}% Number of new instances added to each race before evaluating instances from previous races (only for elitist \irace). \begin{xwarningbox} If \parameter{deterministic} is \code{TRUE} then the number of \parameter{elitistNewInstances} will be reduced or set to \code{0} once all instances have been evaluated. \end{xwarningbox} \defparameter{elitistLimit}{elitist-limit}{2}% Maximum number of statistical tests performed without successful elimination after all instances from the previous race have been evaluated. If the limit is reached, the current race is stopped. Only valid for elitist \irace. Use \code{0} to disable the limit. \end{description} \subsection[Internal irace options]{Internal \irace options} \begin{description} \defparameter{sampleInstances}{sample-instances}{1}% Enable/disable the sampling of the training instances. If the option \parameter{sampleInstances} is disabled, the instances are used in the order provided in the \parameter{trainInstancesFile} or in the order they are read from the \parameter{trainInstancesDir} when\parameter{trainInstancesFile} is not provided. For more information about training instances see \autoref{sec:training}. \defparameter{softRestart}{soft-restart}{1}% Enable/disable the soft-restart strategy that avoids premature convergence of the probabilistic model. When a sampled configuration is \emph{similar} to its parent configuration, the probabilistic model of these configurations is soft restarted. The soft-restart mechanism is explained in the \irace paper~\citep{LopDubPerStuBir2016irace}. The similarity of categorical and ordinal parameters is given by the hamming distance, and the option \parameter{softRestartThreshold} defines the similarity of numerical parameters. \defparameter{softRestartThreshold}{soft-restart-threshold}{}% Soft restart threshold value for numerical parameters. By default, it is computed as $10^{-digits}$, where \parameter{digits} corresponds to the \irace option explained in this section. \defparameter{nbIterations}{iterations}{0}% Maximum number of iterations to be executed. Each iteration involves the generation of new configurations and the use of racing to select the best configurations. By default (with 0), \irace calculates a \emph{minimum} number of iterations as $\Niter = \lfloor 2 + \log_{2}\Nparam \rfloor$, where $\Nparam$ is the number of non-fixed parameters to be tuned. Setting this parameter may make \irace stop sooner than it should without using all the available budget. We recommend to use the default value. \defparameter{nbExperimentsPerIteration}{experiments-per-iteration}{0}% Number of runs of the target algorithm per iteration. By default (when equal to 0), this value changes for each iteration and depends on the iteration index and the remaining budget. Further details are provided in the \irace paper~\citep{LopDubPerStuBir2016irace}. We recommend to use the default value. \defparameter{minNbSurvival}{min-survival}{0}% Minimum number of configurations needed to continue the execution of each race (iteration). If the number of configurations alive in the race is not larger than this value, the current iteration will stop and a new iteration will start, even if there is budget left to continue the current race. By default (when equal to 0), the value is calculated automatically as $\lfloor 2 + \log_{2}\Nparam \rfloor$, where $\Nparam$ is the number of non-fixed parameters to be tuned. \defparameter{nbConfigurations}{num-configurations}{0}% The number of configurations that will be raced at each iteration. By default (when equal to 0), this value changes for each iteration and depends on \parameter{nbExperimentsPerIteration}, the iteration index and \parameter{mu}. The precise details are given in the \irace paper~\citep{LopDubPerStuBir2016irace}. We recommend to use the default value. \defparameter{mu}{mu}{5}% Parameter used to define the number of configurations to be sampled and evaluated at each iteration. The number of configurations will be calculated such that there is enough budget in each race to evaluate all configurations on at least $\mu + \min(5,j)$ training instances, where $j$ is the index of the current iteration. The value of $\mu$ will be adjusted to never be lower than the value of \parameter{firstTest}. We recommend to use the default value and, if needed, adjust \parameter{firstTest}and \parameter{eachTest}, instead. \end{description} \subsection[Target algorithm parameters]{Target algorithm parameters} \begin{description} \defparameter[p]{parameterFile}{parameter-file}{./parameters.txt}% File that contains the description of the parameters of the target algorithm. See \autoref{sec:target parameters}. \defparameter{forbiddenFile}{forbidden-file}{}% File containing a list of logical expressions that cannot be true for any evaluated configuration. If empty or \code{NULL}, no forbidden configurations are considered. See \autoref{sec:forbidden} for more information. \defparameter{digits}{digits}{4}% Maximum number of decimal places that are significant for numerical (real) parameters. \end{description} \subsection[Target algorithm execution]{Target algorithm execution} \begin{description} \defparameter{targetRunner}{target-runner}{./target-runner}% Executable or \aR function that evaluates a configuration of the target algorithm on a particular instance. See \autoref{sec:runner} for details. \defparameter{targetRunnerLauncher}{target-runner-launcher}{}% Executable that will be used to launch the target runner, when \parameter{targetRunner} cannot be executed directly (.e.g, a Python script in Windows). \defparameter{targetRunnerLauncherArgs}{target-runner-args}{{targetRunner} {targetRunnerArgs}}% Command-line arguments provided to \parameter{targetRunnerLauncher}. The substrings \code{\{targetRunner\}} and \code{\{targetRunnerArgs\}} will be replaced by the value of the option \parameter{targetRunner} and by the arguments usually passed when calling \parameter{targetRunner}, respectively. Example: \code{"-m \{targetRunner\} --args \{targetRunnerArgs\}"}. \defparameter{targetRunnerRetries}{target-runner-retries}{0}% Number of times to retry a call to \parameter{targetRunner} if the call failed. \defparameter{targetRunnerData}{}{}% Optional data passed to \parameter{targetRunner}. This is ignored by the default \parameter{targetRunner} function, but it may be used by custom \parameter{targetRunner} functions to pass persistent data around. \defparameter{targetRunnerParallel}{}{}% Optional \aR function to provide custom parallelization of \parameter{targetRunner}. See \autoref{sec:parallel} for more information. \defparameter{targetEvaluator}{target-evaluator}{}% Optional script or \aR function that returns a numerical value for an experiment after all configurations have been executed on a given instance using \parameter{targetRunner}. See \autoref{sec:evaluator} for details. \defparameter{deterministic}{deterministic}{0}% Enable/disable deterministic target algorithm mode. If the target algorithm is deterministic, configurations will be evaluated only once per instance. See \autoref{sec:training} for more information. \begin{xwarningbox} If the number of instances provided is less than the value specified for the option \parameter{firstTest}, no statistical test will be performed. \end{xwarningbox} \defparameter{parallel}{parallel}{0}% Number of calls of the \parameter{targetRunner} to execute in parallel. Values 0 or 1 mean no parallelization. For more information on parallelization, see \autoref{sec:parallel}. \defparameter{loadBalancing}{load-balancing}{1}% Enable/disable load-balancing when executing experiments in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster. See \autoref{sec:parallel}. \defparameter{mpi}{mpi}{0}% Enable/disable use of \pkg{Rmpi} to execute the \parameter{targetRunner} in parallel using MPI protocol. When \parameter{mpi} is enabled, the option \parameter{parallel} is the number of slave nodes. See \autoref{sec:parallel}. \defparameter{batchmode}{batchmode}{0}% Specify how irace waits for jobs to finish when \parameter{targetRunner} submits jobs to a batch cluster: \code{sge}, \code{pbs}, \code{torque}, \code{slurm} or \code{htcondor} (\parameter{targetRunner} must submit jobs to the cluster using. for example, \code{qsub}). See \autoref{sec:parallel}. \end{description} \subsection[Initial configurations]{Initial configurations} \begin{description} \defparameter{configurationsFile}{configurations-file}{}% File containing a table of initial configurations. If empty or \code{NULL}, \irace will not use initial configurations. See \autoref{sec:initial}. \begin{xwarningbox} The provided configurations must not violate the constraints described in \parameter{parameterFile} and \parameter{forbiddenFile}. \end{xwarningbox} \end{description} \subsection[Training instances]{Training instances} \begin{description} \defparameter{trainInstancesDir}{train-instances-dir}{./Instances}% Directory where training instances are located; either absolute path or relative to current directory. See \autoref{sec:training}. \defparameter{trainInstancesFile}{train-instances-file}{}% File that contains a list of instances and optionally additional parameters for them. See \autoref{sec:training}. \begin{xwarningbox} The list of instances in \parameter{trainInstancesFile} is interpreted as file-system paths relative to \parameter{trainInstancesDir}. When using an absolute path or instances that are not files, set \code{trainInstancesDir=""}. \end{xwarningbox} \end{description} \subsection[Tuning budget]{Tuning budget} \begin{description} \defparameter{maxExperiments}{max-experiments}{0}% The maximum number of runs (invocations of \parameter{targetRunner}) that will be performed. It determines the maximum budget of experiments for the tuning. See \autoref{sec:budget}. \defparameter{maxTime}{max-time}{0}% The maximum total time in seconds for the runs of \parameter{targetRunner} that will be performed. The mean execution time of each run is estimated in order to calculate the maximum number of experiments (see option \parameter{budgetEstimation}). When \parameter{maxTime} is positive, then \parameter{targetRunner} \textbf{must} return the execution time as its second output. See \autoref{sec:budget}. \defparameter{budgetEstimation}{budget-estimation}{0.02}% Fraction (smaller than 1) of the budget used to estimate the mean execution time of a configuration. Only used when \parameter{maxTime} $> 0$. See \autoref{sec:budget}. \defparameter{minMeasurableTime}{min-measurable-time}{0.01}% Minimum time unit that is still (significantly) measureable. \end{description} \subsection[Statistical test]{Statistical test} \begin{description} \defparameter{testType}{test-type}{}% Specifies the statistical test used for elimination: \begin{itemize} \item[] \code{F-test} (Friedman test) \item[] \code{t-test} (pairwise t-tests with no correction) \item[] \code{t-test-bonferroni} (t-test with Bonferroni's correction for multiple comparisons) \item[] \code{t-test-holm} (t-test with Holm's correction for multiple comparisons). \end{itemize} We recommend to not use corrections for multiple comparisons because the test typically becomes too strict and the search stagnates. See \autoref{sec:stat test} for details about choosing the statistical test most appropriate for your scenario. \begin{xwarningbox} The default setting of \parameter{testType} is \code{F-test} unless the \parameter{capping} option is enabled in which case, the default setting is \code{t-test}. \end{xwarningbox} \defparameter{firstTest}{first-test}{5}% Specifies how many instances are evaluated before the first elimination test. \begin{xwarningbox} The value of \parameter{firstTest} must be a multiple of \parameter{eachTest}. \end{xwarningbox} \defparameter{eachTest}{each-test}{1}% Specifies how many instances are evaluated between elimination tests. \defparameter{confidence}{confidence}{0.95}% Confidence level for the elimination test. \end{description} \subsection[Adaptive capping]{Adaptive capping} \begin{description} \defparameter{capping}{capping}{0}% Enable the use of adaptive capping. This option is only available when \parameter{elitist} is active. When using this option, \irace provides an execution bound to each target algorithm execution (See \autoref{sec:runner}). For more details about this option See \autoref{sec:capping}. \defparameter{cappingType}{capping-type}{median}% Specifies the measure used to define the execution bound: \begin{itemize} \item[] \code{median} (the median of the performance of the elite configurations) \item[] \code{mean} (the mean of the performance of the elite configurations) \item[] \code{best} (the best performance of the elite configurations) \item[] \code{worst} (the worst performance of the elite configurations). \end{itemize} \defparameter{boundType}{bound-type}{candidate}% Specifies how to calculate the performance of elite configurations for the execution bound: \begin{itemize} \item[] \code{candidate} (performance of candidates is aggregated across the instances already executed) \item[] \code{instance} (performance of candidates on each instance). \end{itemize} \defparameter{boundMax}{bound-max}{0}% Maximum execution bound for \code{targetRunner}. It must be specified when capping is enabled. \defparameter{boundDigits}{bound-digits}{0}% Precision used for calculating the execution time. It must be specified when capping is enabled. \defparameter{boundPar}{bound-par}{1}% Penalty used for PARX. This value is used to penalize timed out executions, see \autoref{sec:capping}. \defparameter{boundAsTimeout}{bound-as-timeout}{1}% Replace the configuration cost of bounded executions with \parameter{boundMax}. See \autoref{sec:capping}. \end{description} \subsection[Recovery]{Recovery} \begin{description} \defparameter{recoveryFile}{recovery-file}{}% Previously saved \irace log file that should be used to recover the execution of \irace; either absolute path or relative to the current directory. If empty or \code{NULL}, recovery is not performed. For more details about recovery, see \autoref{sec:recovery}. \end{description} \subsection[Testing]{Testing} \begin{description} \defparameter{-{}-only-test}{only-test}{}% Run the configurations contained in the file provided as argument on the test instances. See \autoref{sec:testing}. \defparameter{testInstancesDir}{test-instances-dir}{}% Directory where testing instances are located, either absolute or relative to the current directory. \defparameter{testInstancesFile}{test-instances-file}{}% File containing a list of test instances and, optionally, additional parameters for them. \defparameter{testNbElites}{test-num-elites}{1}% Number of elite configurations returned by irace that will be tested if test instances are provided. For more information about the testing, see \autoref{sec:testing}. \defparameter{testIterationElites}{test-iteration-elites}{0}% Enable/disable testing the elite configurations found at each iteration. \end{description} irace/vignettes/irace-scheme.pdf0000644000176200001440000005432714157104332016400 0ustar liggesusers%PDF-1.4 % 4 0 obj << /Length 39 /Filter /FlateDecode >> stream x+2T0BC]C#S]SC\.}\C|@.l( endstream endobj 3 0 obj << /Type /Page /Contents 4 0 R /Resources 2 0 R /MediaBox [0 0 380 239] /Parent 6 0 R /Group 5 0 R >> endobj 1 0 obj << /Type /XObject /Subtype /Form /FormType 1 /PTEX.FileName (./irace-scheme-simple.pdf) /PTEX.PageNumber 1 /PTEX.InfoDict 7 0 R /BBox [0 0 595 842] /Group 5 0 R /Resources << /Font << /F1 8 0 R/F2 9 0 R>> /ProcSet [ /PDF /Text ] >> /Length 4301 /Filter /FlateDecode >> stream x[ˎ% W:t  ĉ;'dd5y!o!aս31>XEQɛÿt$C:C8~q!{Hgr|C|:}qFu ͝Թ0vF/K˅;υsw΅*(^qD}O}.5ޥ;bzn1ljsG2o٬ͫ3Gݴ |? m_Um3K%di>.ӛ쎚ӛn> +tkK{ޔvr_-fn1 +odYcoMtZ((7KRo :P^\U30"^)66q6~^'N2v2˅{L_׌1߈afTΨDz8>\׌R4/#PQ! F-V'jwB%Top dy7Ob;Ur7 XX4b ZilmVnA#}$[aaS$` LK%G&%L)|ӎm9+d:XB!M5RF)(ӘH*ׄȭ`7 =o&fST,|K11P:1 6iރشa0jսlTePB*MFqүbr4||xW]'#X2؅%#v#Dyuf'D;QB}@'u%8S9l`o!;r $ewG1-e,H4)a=eƦ^ F7PX;Ɯ!'8~>֝+vs= e=MjГs(0_ NQs-V PxE*=|e!'-櫖M`4`FOFCJXN &w\p7pw>_5nlpg[ V<5\^3r `wʣyBr N7 ݱ{L}]F P*n\qmn`&@0M =Rᣖ92(ȓcȈh]KS8%#P:@p:32ox^2QԣCAdo24q'8K653`ub酉g{Κn߿Bt$t^4d+i qhVW桛1(G,.RӠ3cV܅ƗB;wwqL1{NyS;C_pRi8ə OQg G$.t_*&׽}zy3>)X9^M}bGKVDJ( N }nPwca߿~~w7A¤0 ؝`'O$c+B]㑌Se"08J둎x#QG)'ɳc:S<[1쎠7;ocNh #!iR n?hDs×ry̶/ 6=jA\oϗ$ X;OwI{iyq=_25nKSNIɽ*t7Hjf9Ӝc&𬖫%Y`']^2b씆C*yP|l-Kr Bpc#ǖrz ))t P߹R(`ڕ⹿RH9V'b༿z'+!^ t0hjPgӢ A=.K6\`쨊uq3.+㞊 D#i+gM6E7f"4iu.vx]!XEf'5̮I8͠Шe7LMF3RdB{Ǻ Qsnj8꧔Wa`*#4vzZӸ48S&1b[ F T@AǺȭEV+' F#q&>҈3:>#^zj!i<ϋX0=xDltF_Ŋ!((r<" +((R^n=hFy46`HCX ciup݁:}X7k#/H2/e_5(~&+W1 Ò ̪00',̀yA(;М0l),1H͆ 0*t+wL8&V'ذwϠ&Q&Ԝ8*SrρA YFN&d`uӝ6hG}AAS8qݴd f,>nH?136΄ACYZ:șj"F"*,*jܘ8{z3 L ;rTT{G{diFG< xE>6n!7({;R;8$j2Nް`{ FxmX2Jh0 &*0w[h "iㆥ> $#Vcn&l0Qatm083dX׊ESޑ2n2d̢M t( } >8-ae0%t\.IiSK!ARqf\6ZѩcVSǚ-"]?%P#TQQv8s 5tnQ./JK-eSMQG;Gɜinjٟ9.=r_Fl4(!&Q\(ͅDIc]Po|Z)K)q(tTP8=n0Jjt;id\Дb;2O^&YL[+.dVlʈoA)%Ju#07q (QȠp/P (A&ٕ.K+F;< V:Y댫mxGPi.F+rj;,Qdy`Dệ>/r=Gj{o&܈&D?Z0-Z7ODSr\p*IɎ?)Tsl)F` endstream endobj 7 0 obj << /Creator /Producer /CreationDate (D:20160720164806+01'00') >> endobj 8 0 obj << /Type /Font /Subtype /TrueType /BaseFont /BAAAAA+LiberationSans /FirstChar 0 /LastChar 18 /Widths [ 365 610 333 556 222 556 556 500 277 500 556 666 833 556 722 556 277 556 722] /FontDescriptor 10 0 R /ToUnicode 11 0 R >> endobj 9 0 obj << /Type /Font /Subtype /TrueType /BaseFont /CAAAAA+LiberationSans-Italic /FirstChar 0 /LastChar 17 /Widths [ 365 500 556 222 500 277 722 222 277 556 542 277 333 556 556 556 333 333] /FontDescriptor 12 0 R /ToUnicode 13 0 R >> endobj 10 0 obj << /Type /FontDescriptor /FontName /BAAAAA+LiberationSans /Flags 4 /FontBBox [ -203 -303 1049 910] /ItalicAngle 0 /Ascent 905 /Descent -211 /CapHeight 910 /StemV 80 /FontFile2 14 0 R >> endobj 11 0 obj << /Length 305 /Filter /FlateDecode >> stream x]n0~ CMBH$f8tQiCj8޾OJ=Y,١96քՏ{ciyWc̹6*ܭWCX<ۏUŲb > endobj 13 0 obj << /Length 302 /Filter /FlateDecode >> stream x]Mk0 >v; !Ц-8!~mC#}%eusl ٫U jx 2smTG鯆α,zy 04˒eo6?^x`ًGƸ9*}: %ײѱl¼?)5j\w \1_MdQ*TKqN\$&q./ )FQy:$>PԿ&}|$Dgʟ"KA^#. { ƻqxi=cntJ7nL endstream endobj 14 0 obj << /Length 16 0 R /Filter /FlateDecode /Length1 11508 >> stream xzkxS׵#%ɖ:?u,ǀe6l [,$$D^GـcAqG ߍN-h+P#VC;Ip } 9h/c_Eo( L$or[<3Qǧ;Z7nhq7_FWvjQ#VWUVZ|Yͺ ?/לmJ3uڄ^Vq,C0*7;9[f.R@B 9onEL]RmwHJIiQjTD7 uf2ㅶLc"=}r^PMi%7#w;:?Cp:74I-[e@Ü*3[߱ [lwOP XK{POY$,2馜Nr-pwa66l^c||+@< ҩ0?gfBC> shukd\ Um-7l`/ ԙӒys_D]w¸ܴq*7G"QF4ULelvzaÊV$ѵ<@oܼZ rW"jMk@2[3eZ$|`cgTZLjtL{B%J(B5mDg^SBP)4 $B-MRhYJC10AcGelk0Q1f9ּU)xiɞq]t$n}\TjdV}Vjf˦T/2@4P2Gg*곘l}^>suP@V^K.<~+\q#y<Γ?Yg;o~z="ZD HdgrNt q^JKԩP;l-mevr+)g1^l_ $K"064f &aMejjUMfKQͼqzxVV祲QHFRǻb Xx[e9%+Xo*fU혡!zېĕh01)%'jq-eYfWvKaJE gػ f0AYn`Z0a}bm1alXf3te̖ٖz, x5ātzClȜdC ߝlyZRjykM1 _Y 9Fk2߽˼i[?gvke֮\[X&dlYJx.` L-GBR<.cKYFR'XVG e~'yxbR5Sq}NYYSMF!ph^ܑBV}lYiR=_uNx/g?ZG0yh3?#c]yNɍq43t\gR*s:x]|iLxNrtIxeD-el}t֒aY@b*R9#Ldltv>Z@^m\3sjK\IEXr}7lin3smBP=HrY6V?&#$YD\ԃʘbm|6TS%\I c*'G98&Ch 5 rҀpsk8@ rH\]NN** xf%PJꉦ `Y)ZVpa+] rvteФ`J;ޠJZ!Pa&%OsCU[-=ai+%+{WLRO݊ esz3ZrjQS~:7fe![ |hmw)}7w3oܬDoImaVHޗ4fĊ ( hi{<ȃ oH [f&+S|u O7Ⱥ㩓c^gu.~w_pKڬ{–u;V[:ۚVec{Z,FɿִEWtؔ/߶ȱ+ڻJ* tmM{ķk+W\ x?b1+@J>Ǿ̲Ź)RHJ\| s #N ıjn, kU8G In&k4`]vljݖi3Σ\o.'ۓѴLV.'?;25_=?>tmC"]Kh2ã72XpL ZSN23L>aB"4A͘fF!Nӭ@Pk4;!T=(B-0oN2t^"Y2uq=45*,Nʔkx nD!aĿ_$9kv޷b߻S6K?StM.,]vͤ3k01>2y>A2$wXZJVfT[+ 84KQ} [>Zv[;HJ"F\̍#]W/n{vǵk ~|liok%WDVҥ[m8 g-ΊlD&ȵ~bpT ܔ\~KOͿiN [0m;~>ry߾i߾z}󁾊㎿c.W˶8Z#RըpP %/!SM!Ѹ6GbLra5DפksJsqr؅J.No0ieEK#eX:^,em|{WO8k[M;JkI's;)uN ==ߣq` {KLLR=6 dPp.]"86#'Ǎj%U 1FiS(cȜQɈ$0NWi1N3aeF@N3VÅJ #'c-+Ws QʑJJ.V@ p0z[)IqHp툒Wjj$c֮Xr8hz ØW~ya!YWy( L9 6d`.#F*A>tP'D s ~ 3c c;y3f}cP71>YĬ;ccҗ%}~RBK¤pB`ҙN2jVGl]t6=uq V㉜y$V[wgMO6^В d⠅sO5^"O{5n)iw\~ýHsG>cV_\=|?}ԥ";x7:۬2s uG]5sQ͟y6| +y `Ch31:O ob R%Ly&*"0c4@eo#\%d0wos˹*Mc\1UHBΨ 1PMv/ڍ!0Aj @0u49ȓǣ WApyQn±`]Xq( ǣ~> '16MGa(LPS, Ea-aGa/Fa5?Ga *dGa-agp,Z~( >Pm 8: EBYiJaOpy%BPUXc_7g6߰y ^!@Շ@6_ zHv-?ojk 9%h ?G(TzQ)K7`A8u{z $Oۅae?YȌ[;!F=4;Ю[ߊ2򌬘3K۔~"^ =~)޶*~e~C?@Gg?JӚ[n˾H0_xm4+㘕71ױp.4Yhb5uuU|3WO?2NX3郙+3Wgiƾ9L3J?L̴7iaQnһi5]mO|.4_cZϟO,s95 w3'gY<7F㵫1cLdyZ,333++d˯ˎx/STԥSp(tuMHiߑ#G''ؓ8816A&' |p0stCLMCrSNkKL .\f2u3,eL[Z--Dj)_Z HnL.3N|yIƜ8,Mm2m #l2i.>-ڴڀvB{E;U;vU@89|ONnXΫ74j6һ)ȨsKOt$ 3N5Px^8 QS) bp.8zo8 Y"A( @>Ci[!7Bǟ endstream endobj 15 0 obj << /Length 17 0 R /Filter /FlateDecode /Length1 11236 >> stream xyxSו=~Zz- ۲-Y-Ho [&e,ABh&BB ]%M@"C8@'o4v2[N~|{}5Igfgog9s9sKL 2EX" E&B!97!vcaJ&v|#!\!J~辡5FOeF+O KOQ*)f%̇#8ƓhQ%؊أ$!c}$Jb(yy?ODʭ'}GFbh;!ɴGAv"N#3@^N> 5}-HuF%S/=,T3EKV,a|F3~vTX%"cOo|īT#cU=6߷?.[{yEҺ-K{{e]{斮Zrl|Wспe ׹RD:10U8*XfZ2=}k=_tqM _@ <(-kM&;hjlX6˄ x]r ҚmUV5fk|";w.&у}Yu}?u\=SUˋd8XlBʎU6-e|ԱI@amXqO>_)y宾&|Vў jp~6uVڲKJ6?JP7њ]UK3u䖙nMK66X7;0  /Ybjq6Cφ0'&^a3W0+2]mk"րm,_'dE\Q)C|) 2pBnaW$SfvpCUNO >. AR\QON辭cu [4f5lƭݏֻpKHyAE,(0ukl5n73^4*I%+5djk9`O9,MNtzNkcͩ͋F \e&n10H.A\jтZ!x7^9^/ފr!^C|J¥pywD?Pށgq*wy^iĖ\W?7VeJl2Bl7;yIP3j#ՠ){3C!ބFQü:h՝]q:]a.)# s(bϷ,h)'zuy6#} cI#5;L^kU饭o_?߽ z Bo-K폭\m psHqJfҢd'Uh@C2\ 5͎ c<2w.~W9(6ɧzzBhJY#|yϾ~7I*~$OZ_36~" +M}#;kX,/;b0([ĆQQfxtrotL\hji(ň F"$Fbbgd<ޒ 4IT]L1|1uSt2NIeeoP_P 9OD'82.vumDt=!C[F[;G-Hv#ۀr#L"W/%»I@]//(_1c)KIȲS#n蔹_ yq Fld}wpȲ-)1Ӟ߅R`Phf_0O4F'1JcxjWsLO1Gۆel]Kʐnie24&d?F9߉/e|O{h\^C{1/ǀNѻЙowQxZ!y&CGeoԝ#8#ݩiڽ,r=/ƚd7x;"i%n^y8kp&\ H\O.'7؍7ݸ|' ץm'O]WֿE*_ֿ^vktͷ"p-cokv| }a | Տk~Vdk⭩XivbGclfe@xuf2Ͽj΂ 5Vi._qU''3$;30/|9]xl,=;gZϜ>s }*VM<pY%5nÛ@.}ov!*:<~?»>ar 85~}$`-?"a@-FKt4c#c )=Wx5N2 z&UxKE%q|!6wtr>["3c0P?vߎwA *h ~ @DP]XgA%U}[*֭-ΪqzJ;Ƃֳ̉Hv\AgBGtvFjZ ]m n,T nՆ?r;J9 /gBjkK@јx 3sevnCVP/l xV!& i7+)30 tuz> endobj 2 0 obj << /XObject << /Im1 1 0 R >> /ProcSet [ /PDF ] >> endobj 6 0 obj << /Type /Pages /Count 1 /Kids [3 0 R] >> endobj 18 0 obj << /Type /Catalog /Pages 6 0 R >> endobj 19 0 obj << /Producer (pdfTeX-1.40.16) /Creator (TeX) /CreationDate (D:20160720164946+01'00') /ModDate (D:20160720164946+01'00') /Trapped /False /PTEX.Fullbanner (This is pdfTeX, Version 3.14159265-2.6-1.40.16 (TeX Live 2015/Debian) kpathsea version 6.2.1) >> endobj xref 0 20 0000000000 65535 f 0000000250 00000 n 0000021747 00000 n 0000000133 00000 n 0000000015 00000 n 0000021686 00000 n 0000021812 00000 n 0000004853 00000 n 0000005023 00000 n 0000005262 00000 n 0000005504 00000 n 0000005706 00000 n 0000006084 00000 n 0000006298 00000 n 0000006673 00000 n 0000014287 00000 n 0000021644 00000 n 0000021665 00000 n 0000021869 00000 n 0000021919 00000 n trailer << /Size 20 /Root 18 0 R /Info 19 0 R /ID [ ] >> startxref 22186 %%EOF irace/vignettes/irace-package.Rnw0000644000176200001440000050055014325242447016525 0ustar liggesusers% !Rnw weave = knitr %%% DO NOT EDIT the .tex file directly since it is generated from the .Rnw %%% sources. %\VignetteEngine{knitr::knitr} %\VignetteIndexEntry{irace package: User Guide} %\VignetteDepends{knitr} %\VignetteCompiler{knitr} \synctex=1 \RequirePackage{xparse} \RequirePackage[dvipsnames]{xcolor} \documentclass[a4paper,english]{article} \usepackage[T1]{fontenc} \usepackage[utf8]{inputenc} \usepackage[a4paper]{geometry} % It saves some pages \usepackage[english]{babel} \usepackage{ifthen} \newboolean{Release} \setboolean{Release}{true} \usepackage{calc} \usepackage{afterpage} \usepackage{algorithm,algorithmic} \usepackage{booktabs} \usepackage{tabularx} \usepackage{xspace} \usepackage{amsmath,amssymb} \usepackage{relsize} \usepackage{fancyvrb} \usepackage{microtype} % \texttt{test -- test} keeps the "--" as "--" (and does not convert it to an en dash) \DisableLigatures{encoding = T1, family = tt* } \usepackage[hyphens]{url} \usepackage{hyperref} \usepackage[numbers]{natbib} \usepackage[nottoc]{tocbibind} %% For autoref \hypersetup{ colorlinks, linkcolor={red!50!black}, citecolor={blue!50!black}, urlcolor={blue!70!black} } \addto\extrasenglish{% \def\sectionautorefname{Section} \let\subsectionautorefname\sectionautorefname \let\subsubsectionautorefname\sectionautorefname } \usepackage[titletoc, title]{appendix} % Fix use with \autoref \newcommand*{\Appendixautorefname}{Appendix} \usepackage{tocloft} \setlength{\cftsubsecnumwidth}{3em}% Set length of number width in ToC for \subsection \usepackage[inline]{enumitem} \setlist[enumerate]{leftmargin=*,widest=00} \setlist[itemize]{leftmargin=1.5em} %% FIXME: listing is very limited, we should use 'minted' \usepackage{listings} \lstdefinestyle{BashInputStyle}{ language=bash,% basicstyle=\ttfamily,% numbers=none,% frame=tb,% rulecolor=\color{lightgray}, % framesep=1ex, framexleftmargin=1ex, columns=fullflexible,% backgroundcolor=\color{yellow!05},% linewidth=\linewidth,% % xleftmargin=1\linewidth,% identifierstyle=\color{darkgray},% keywordstyle=\color{darkgray},% keywordstyle={[2]\color{Cyan}},% keywordstyle={[3]\color{olive}},% stringstyle=\color{MidnightBlue},% commentstyle=\color{RedOrange},% morestring=[b]',% showstringspaces=false } \DefineVerbatimEnvironment{Code}{Verbatim}{} \DefineVerbatimEnvironment{CodeInput}{Verbatim}{fontshape=rm} \DefineVerbatimEnvironment{CodeOutput}{Verbatim}{} \newenvironment{CodeChunk}{}{} \newcommand{\IRACEHOME}[1]{\hyperlink{irace_home}{\path{$IRACE_HOME}}\path{#1}} \providecommand{\keywords}[1]{\textbf{\textit{Index terms---}} #1} % Simple font selection is not good enough. For example, |\texttt{--}| % gives `\texttt{--}', i.e., an endash in typewriter font. Hence, we % need to turn off ligatures, which currently only happens for commands % |\code| and |\samp| and the ones derived from them. Hyphenation is % another issue; it should really be turned off inside |\samp|. And % most importantly, \LaTeX{} special characters are a nightmare. E.g., % one needs |\~{}| to produce a tilde in a file name marked by |\file|. % Perhaps a few years ago, most users would have agreed that this may be % unfortunate but should not be changed to ensure consistency. But with % the advent of the WWW and the need for getting `|~|' and `|#|' into % URLs, commands which only treat the escape and grouping characters % specially have gained acceptance \makeatletter \DeclareRobustCommand\code{\bgroup\@makeother\_\@makeother\~\@makeother\$\@noligs\@codex} \def\@codex#1{\texorpdfstring% {{\text{\normalfont\ttfamily\hyphenchar\font=-1 #1}}}% {#1}\egroup} \makeatother \let\proglang=\textsf \newcommand{\pkg}[1]{{\fontseries{b}\selectfont #1}} \newcommand{\aR}{\proglang{R}\xspace} \newcommand{\MATLAB}{\proglang{MATLAB}\xspace} \newcommand{\eg}{e.g.,\xspace} \newcommand{\SoftwarePackage}{\pkg} \newcommand{\ACOTSP}{\SoftwarePackage{ACOTSP}\xspace} %% How to use this command: % Parameter with one short switch: \defparameter[short]{paramName}{long}{default} % Parameter without short switch: \defparameter{paramName}{long}{default} % Parameter without switch: \defparameter{paramName}{}{default} \newcommand{\defparameter}[4][]{% \item[\code{#2}]\hypertarget{opt:#2}{} ~~ % \ifthenelse{\equal{#3}{}}{}{% \emph{flag:} % \ifthenelse{\equal{#1}{}}{}{% \code{-#1}~~~\emph{or}~~~}% \code{--#3} ~~ }% \emph{default:}~\texttt{#4} \\ } \newcommand{\parameter}[1]{\hyperlink{opt:#1}{\code{#1}}} %\usepackage{showlabels} %\showlabels{hypertarget} \newcommand{\irace}{\pkg{irace}\xspace} \newcommand{\Irace}{\pkg{Irace}\xspace} \newcommand{\race}{\pkg{race}\xspace} \newcommand{\FRACE}{\text{F-Race}\xspace} \newcommand{\IFRACE}{\text{I/F-Race}\xspace} \newcommand{\PyImp}{\pkg{PyImp}\xspace} \newcommand{\iraceversion}{\Sexpr{packageVersion("irace")}} \newcommand{\Niter}{\ensuremath{N^\text{iter}}\xspace} \newcommand{\Nparam}{\ensuremath{{N^\text{param}}}\xspace} \newcommand{\iter}{\ensuremath{j}\xspace} \newcommand{\Budget}{\ensuremath{B}\xspace} \newcommand{\Budgetj}{\ensuremath{\Budget_{\iter}}\xspace} \newcommand{\Bused}{\ensuremath{\Budget_\text{used}}\xspace} \newcommand{\Ncand}[1][]{\ensuremath{N_{#1}}\xspace} \newcommand{\Mui}{\ensuremath{\mu_{\iter}}\xspace} \newcommand{\Nmin}{\ensuremath{N^\text{min}}\xspace} \newcommand{\Nsurv}{\ensuremath{N^\text{surv}}\xspace} \newcommand{\Nelite}{\ensuremath{N^\text{elite}}\xspace} \newcommand{\Nnew}{\ensuremath{N^\text{new}}\xspace} \newcommand{\bmax}{\ensuremath{b^\text{max}}\xspace} \newcommand{\bmin}{\ensuremath{b^\text{min}}\xspace} \newcommand{\Celite}{\ensuremath{\Theta^\text{elite}}\xspace} \ifthenelse {\boolean{Release}}{% \newcommand{\MANUEL}[1]{} \newcommand{\LESLIE}[1]{} \newcommand{\THOMAS}[1]{} }{% \newcommand{\MANUEL}[1]{{\footnotesize\noindent\textbf{\color{red}[~MANUEL: #1~]}}} \newcommand{\LESLIE}[1]{\footnote{\noindent\textbf{[ LESLIE: #1 ]}}} \newcommand{\THOMAS}[1]{\footnote{\noindent\textbf{[ THOMAS: #1 ]}}} } \newcommand{\hide}[1]{} \usepackage{tcolorbox} \newcommand{\infoicon}{% \parbox[c]{0.75cm}{\includegraphics[keepaspectratio=true,width=0.75cm]{light-bulb-icon}}% \hspace{1em}} \newcommand{\warningicon}{% \parbox[c]{0.75cm}{\includegraphics[keepaspectratio=true,width=0.75cm]{Warning-icon}}% \hspace{1em}} \definecolor{LightGray}{RGB}{193,193,193} \definecolor{LightYellow}{RGB}{253,247,172} \newlength\macroiconwidth \newenvironment{xwarningbox}{% \setlength{\fboxrule}{3.0\fboxrule}% \setlength{\fboxsep}{0\fboxsep}% \begin{tcolorbox}[colback=LightYellow,colframe=LightGray,boxrule=\fboxrule,boxsep=\fboxsep]% \infoicon% \settowidth{\macroiconwidth}{\infoicon}% \begin{minipage}[c]{\columnwidth - \macroiconwidth - 2.0\fboxrule - 2.0\fboxsep} \raggedright\footnotesize % }{% \end{minipage} \end{tcolorbox} % } \begin{document} <>= library(knitr) @ \author{Manuel L\'opez-Ib\'a\~nez, Leslie P\'erez C\'aceres, J\'er\'emie Dubois-Lacoste,\\ Thomas St\"utzle and Mauro Birattari \\IRIDIA, CoDE, Universit\'e Libre de Bruxelles, Brussels, Belgium} \title{The \irace Package: User Guide} \date{Version \iraceversion, \today} %\keywords{automatic % algorithm configuration, racing, parameter tuning, \aR} \maketitle \tableofcontents %Load files needed for examples <>= library("irace") load("examples.Rdata") iraceResults <- irace::read_logfile("irace-acotsp.Rdata") log_ablation_file <- file.path(system.file(package="irace"), "exdata", "log-ablation.Rdata") load(log_ablation_file) options(width = 70) @ \newpage %% %% %% %% General info %% %% %% \section{General information} \MANUEL{Some things could be taken from the intro of the irace paper and reformulated.} \MANUEL{It would be good to mention that not only opt algorithms can be configured with irace, we say this in the paper.} \subsection{Background} \MANUEL{I would add a paragraph defining what is irace (a bit longer than the abstract above) and references to the literature so people can find more info. The first reference should be the irace TR.} \LESLIE{Here i guess we should say why tune an algorithm is a good idea, and why using irace is a better one.} The \irace package implements an \emph{iterated racing} procedure, which is an extension of Iterated F-race (\IFRACE)~\cite{BirYuaBal2010:emaoa}. The main use of \irace is the automatic configuration of optimization and decision algorithms, that is, finding the most appropriate settings of an algorithm given a set of instances of a problem. However, it may also be useful for configuring other types of algorithms when performance depends on the used parameter settings. It builds upon the \pkg{race} package by Birattari and it is implemented in \aR. The \irace package is available from CRAN: % \begin{center} \url{https://cran.r-project.org/package=irace} \end{center} % More information about \irace is available at \url{https://mlopez-ibanez.github.io/irace}. \subsection{Version} The current version of the \irace package is \iraceversion. Previous versions of the package can also be found in the \href{https://cran.r-project.org/package=irace}{CRAN website}. The algorithm underlying the current version of \irace and its motivation are described by \citet{LopDubPerStuBir2016irace}. The \textbf{adaptive capping mechanism} available from version $3.0$ is described by \citet{PerLopHooStu2017:lion}. Details of the implementation before version 2.0 can be found in a previous technical report~\cite{LopDubStu2011irace}. % \begin{xwarningbox} Versions of \irace before 2.0 are not compatible with the file formats detailed in this document. \end{xwarningbox} \subsection{License} The \irace package is Copyright \copyright{} \the\year\ and distributed under the GNU General Public License version 3.0 (\url{http://www.gnu.org/licenses/gpl-3.0.en.html}). The \irace package is free software (software libre): You can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. The \irace package is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Please be aware that the fact that this program is released as Free Software does not excuse you from scientific propriety, which obligates you to give appropriate credit! If you write a scientific paper describing research that made substantive use of this program, it is your obligation as a scientist to (a) mention the fashion in which this software was used in the Methods section; (b) mention the algorithm in the References section. The appropriate citation is: \begin{itemize}[leftmargin=3em] \item[] Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, Thomas Stützle, and Mauro Birattari. The \irace package: Iterated Racing for Automatic Algorithm Configuration. \emph{Operations Research Perspectives}, 3:43--58, 2016. doi:~\href{http://dx.doi.org/10.1016/j.orp.2016.09.002}{10.1016/j.orp.2016.09.002} \end{itemize} \section{Before starting} \MANUEL{I think this could be a bit more detailed by defining what is a parameter, a configuration, an instance, etc. but ok for now.} The \irace package provides an automatic configuration tool for tuning optimization algorithms, that is, automatically finding good configurations for the parameters values of a (target) algorithm saving the effort that normally requires manual tuning. \begin{figure}[t] \centering \includegraphics[width=0.6\textwidth]{irace-scheme} \caption{Scheme of \irace flow of information.} \label{fig:irace-scheme} \end{figure} Figure~\ref{fig:irace-scheme} gives a general scheme of how \irace works. \Irace receives as input a \emph{parameter space definition} corresponding to the parameters of the target algorithm that will be tuned, a set of \emph{instances} for which the parameters must be tuned for and a set of options for \irace that define the \emph{configuration scenario}. Then, \irace searches in the parameter search space for good performing algorithm configurations by executing the target algorithm on different instances and with different parameter configurations. A \parameter{targetRunner} must be provided to execute the target algorithm with a specific parameter configuration ($\theta$) and instance ($i$). The \parameter{targetRunner} function (or program) acts as an interface between the execution of the target algorithm and \irace: It receives the instance and configuration as arguments and must return the evaluation of the execution of the target algorithm. The following user guide contains guidelines for installing \irace, defining configuration scenarios, and using \irace to automatically configure your algorithms. %% %% %% %% Installation %% %% %% \section{Installation} \subsection{System requirements} \begin{itemize} \item \aR ($\text{version} \geq 3.2.0$) is required for running irace, but you don't need to know the \aR language to use it. \aR is freely available and you can download it from the \aR project website (\url{https://www.r-project.org}). See \autoref{sec:installation} for a quick installation guide of \aR. \item For GNU/Linux and OS X, the command-line executable \code{parallel-irace} requires GNU Bash. Individual examples may require additional software. \end{itemize} \subsection{\irace installation} \label{sec:irace install} The \irace package can be installed automatically within \aR or by manual download and installation. We advise to use the automatic installation unless particular circumstances do not allow it. The instructions to install \irace with the two mentioned methods are the following: \subsubsection[Install automatically within R]{Install automatically within \aR{}} Execute the following line in the \aR console to install the package: <>= install.packages("irace") @ Select a mirror close to your location, and test the installation in the \aR console with: <>= library("irace") q() # To exit R @ Alternatively, within the \aR graphical interface, you may use the \code{Packages and data->Package installer} menu on OS X or the \code{Packages} menu on Windows. \subsubsection{Manual download and installation} From the \irace package CRAN website (\url{https://cran.r-project.org/package=irace}), download one of the three versions available depending on your operating system: \begin{itemize} \item \code{irace_\iraceversion.tar.gz} (Unix/BSD/GNU/Linux) \item \code{irace_\iraceversion.tgz} (OS X) \item \code{irace_\iraceversion.zip} (Windows) \end{itemize} To install the package on GNU/Linux and OS X, you must execute the following command at the shell (replace \code{} with the path to the downloaded file, either \code{irace_\iraceversion.tar.gz} or \code{irace_\iraceversion.zip}): % \begin{lstlisting}[style=BashInputStyle] R CMD INSTALL \end{lstlisting} To install the package on Windows, open \aR and execute the following line on the \aR console (replace \code{} with the path to the downloaded file \code{irace_\iraceversion.zip}): %\LESLIE{Check that this actually works on internet says that this: \code{Rscript -e "install.packages('foo.zip', repos = NULL)"} also works} % <>= install.packages("", repos = NULL) @ If the previous installation instructions fail because of insufficient permissions and you do not have sufficient admin rights to install \irace system-wide, then you need to force a local installation. \subsubsection{Local installation} Let's assume you wish to install \irace on a path denoted by \code{}, which is a filesystem path for which you have sufficient rights. This directory \textbf{must} exist before attempting the installation. Moreover, you must provide to \aR the path to this library when loading the package. However, the latter can be avoided by adding the path to the system variable \code{R_LIBS} or to the \aR internal variable \code{.libPaths}, as we will see below.\footnote{% On Windows, see also \url{https://cran.r-project.org/bin/windows/base/rw-FAQ.html\#I-don_0027t-have-permission-to-write-to-the-R_002d3_002e3_002e1_005clibrary-directory}.} On GNU/Linux or OS X, execute the following commands to install the package on a local directory: \begin{lstlisting}[style=BashInputStyle] export R_LIBS_USER="" # Create R_LIBS_USER if it doesn't exist mkdir $R_LIBS_USER # Replace with the path to the downloaded file. R CMD INSTALL --library=$R_LIBS_USER # Tell R where to find R_LIBS_USER export R_LIBS=${R_LIBS_USER}:${R_LIBS} \end{lstlisting} On Windows, you can install the package on a local directory by executing the following lines in the \aR console: <>= # Replace with the path to the downloaded file. # Replace with the path used for installation. install.packages("", repos = NULL, lib = "") # Tell R where to find R_LIBS_USER. # This must be executed for every new session. .libPaths(c("", .libPaths())) @ \subsubsection{Testing the installation and invoking irace} Once \irace has been installed, load the package and test that the installation was successful by opening an \aR console and executing: <>= # Load the package library("irace") # Obtain the installation path system.file(package = "irace") @ The last command must print out the filesystem path where \irace is installed. In the remainder of this guide, the variable \code{\$IRACE_HOME} is used to denote this path. When executing any provided command that includes the \code{\$IRACE_HOME} variable do not forget to replace this variable with the installation path of \irace. On GNU/Linux or OS X, you can let the operating system know where to find \irace by defining the \code{\$IRACE_HOME} variable and adding it to the system \code{PATH}. Append the following commands to \path{~/.bash_profile}, \path{~/.bashrc} or \path{~/.profile}: % %<>= \begin{lstlisting}[style=BashInputStyle] # Replace with the irace installation path export IRACE_HOME= export PATH=${IRACE_HOME}/bin/:$PATH # Tell R where to find R_LIBS_USER # Use the following line only if local installation was forced export R_LIBS=${R_LIBS_USER}:${R_LIBS} \end{lstlisting} %@ Then, open a new terminal and launch \irace as follows: %<>= \begin{lstlisting}[style=BashInputStyle] irace --help \end{lstlisting} %@ On Windows, you need to add both \aR and the installation path of \irace to the environment variable \code{PATH}. To edit the \code{PATH}, search for ``Environment variables'' in the control panel, edit \code{PATH} and add a string similar to \path{C:\R_PATH\bin;C:\IRACE_HOME\bin\x64\} where \code{R_PATH} is the installation path of \aR and \code{IRACE_HOME} is the installation path of \irace. If \irace was installed locally, you also need to edit the environment variable \code{R_LIBS} to add \code{R_LIBS_USER}. Then, open a new terminal (run program \code{cmd.exe}) and launch \irace as: % %<>= \begin{lstlisting}[style=BashInputStyle] irace.exe --help \end{lstlisting} %@ Alternatively, you may directly invoke \irace from within the \aR console by executing: <>= library("irace") irace.cmdline("--help") @ \section{Running irace}\label{sec:execution} Before performing the tuning of your algorithm, it is necessary to define a tuning scenario that will give \irace all the necessary information to optimize the parameters of the algorithm. The tuning scenario is composed of the following elements: \begin{enumerate} \item Target algorithm parameter description (see \autoref{sec:target parameters}). \item Target algorithm runner (see \autoref{sec:runner}). \item Training instances list (see \autoref{sec:training}) \item \irace options (see \autoref{sec:irace options}). \item \textit{Optional:} Initial configurations (see \autoref{sec:initial}). \item \textit{Optional:} Forbidden configurations (see \autoref{sec:forbidden}). \item \textit{Optional:} Target algorithm evaluator (see \autoref{sec:evaluator}). \end{enumerate} These scenario elements can be provided as plain text files or as \aR objects. This user guide provides examples of both types, but we advise the use of plain text files, which we consider the simpler option. For a step-by-step guide to create the scenario elements for your target algorithm continue to \autoref{sec:step}. For an example execution of \irace using the \ACOTSP scenario go to \autoref{sec:example}. \subsection{Step-by-step setup guide}\label{sec:step} This section provides a guide to setup a basic execution of \irace. The template files provided in the package (\IRACEHOME{/templates}) will be used as basis for creating your new scenario. Please follow carefully the indications provided in each step and in the template files used; if you have doubts check the the sections that describe each option in detail. \begin{enumerate}[leftmargin=*] \item Create a directory (\eg~\path{~/tuning/}) for the scenario setup. This directory will contain all the files that describe the scenario. On GNU/Linux or OS X, you can do this as follows: %<>= \begin{lstlisting}[style=BashInputStyle] mkdir ~/tuning cd ~/tuning \end{lstlisting} %@ \item Initialize the tuning directory with template config files. On GNU/Linux or OS X, you can do this as follows: %<>= \begin{lstlisting}[style=BashInputStyle] irace --init \end{lstlisting} %@ \item Define the target algorithm parameters to be tuned by following the instructions in \code{parameters.txt}. Available parameter types and other guidelines can be found in \autoref{sec:target parameters}. \item \textit{Optional}: Define the initial parameter configuration(s) of your algorithm, which allows you to provide good starting configurations (if you know some) for the tuning. Follow the instructions in \code{configurations.txt} and set \parameter{configurationsFile}\code{="configurations.txt"} in \code{scenario.txt}. More information in \autoref{sec:initial}. If you do not need to define initial configurations remove this file from the directory. \item \textit{Optional}: Define forbidden parameter value combinations, that is, configurations that \irace must not consider in the tuning. Follow the instructions in \code{forbidden.txt} and update \code{scenario.txt} with \parameter{forbiddenFile} \code{=} \code{"forbidden.txt"}. More information about forbidden configurations in \autoref{sec:forbidden}. If you do not need to define forbidden configurations remove this file from the directory. \item Place the instances you would like to use for the tuning of your algorithm in the folder \path{~/tuning/Instances/}. In addition, you can create a file (\eg~\code{instances-list.txt}) that specifies which instances from that directory should be run and which instance-specific parameters to use. To use such an instance file, set the appropriate option in \code{scenario.txt}, e.g., \parameter{trainInstancesFile} \code{= "instances-list.txt"}. See \autoref{sec:training} for guidelines. \item Uncomment and assign in \code{scenario.txt} only the options for which you need a value different from the default. %% MANUEL: I'm not sure what this means. % The names of the template files match the default names of the scenario % options. Some common parameters that you might want to adjust are: \begin{description} \item[\parameter{execDir}] (\code{--exec-dir}): the directory in which \irace will execute the target algorithm; the default value is the current directory. %\item[\parameter{logFile}] (\code{--log-file}): a file the name of the results \aR data file that produces \irace. \item[\parameter{maxExperiments}] (\code{--max-experiments}): the maximum number of executions of the target algorithm that \irace will perform. \item[\parameter{maxTime}] (\code{--max-time}): maximum total execution time in seconds for the executions of \code{targetRunner}. In this case, \code{targetRunner} must return two values: cost and time. Note that you must provide either \parameter{maxTime} or \parameter{maxExperiments}. \end{description} For setting the tuning budget see \autoref{sec:budget}. For more information on \irace options and their default values, see \autoref{sec:irace options}. \item Modify the \code{target-runner} script to run your algorithm. This script must execute your algorithm with the parameters and instance specified by \irace and return the evaluation of the execution and \textit{optionally} the execution time (\code{cost [time]}). When the \parameter{maxTime} option is used, returning \code{time} is mandatory. The \code{target-runner} template is written in \proglang{GNU Bash} scripting language, which can be executed easily in GNU/Linux and OS X systems. However, you may use any other programming language. We provide examples written in \proglang{Python}, \proglang{MATLAB} and other languages in \IRACEHOME{/examples/}. % Follow these instructions to adjust the given \code{target-runner} template to your algorithm: \begin{enumerate} \item Set the \code{EXE} variable with the path to the executable of the target algorithm. \item Set the \code{FIXED_PARAMS} if you need extra arguments in the execution line of your algorithm. An example could be the time that your algorithm is required to run (\code{FIXED_PARAMS}\hspace{0pt}\code{=}\hspace{0pt}\code{"--time 60"}) or the number of evaluations required (\code{FIXED_PARAMS}\hspace{0pt}\code{=}\hspace{0pt}\code{"--evaluations 10000"}). \item The line provided in the template executes the executable described in the \code{EXE} variable. % \begin{center} \code{\$EXE \$\{FIXED_PARAMS\} -i \$\{INSTANCE\} --seed \$\{SEED\} \$\{CONFIG_PARAMS\}} \end{center} % You must change this line according to the way your algorithm is executed. In this example, the algorithm receives the instance to solve with the flag \code{-i} and the seed of the random number generator with the flag \code{--seed}. The variable \code{CONFIG_PARAMS} adds to the command line the parameters that \irace has given for the execution. You must set the command line execution as needed. For example, the instance might not need a flag and might need to be the first argument: \begin{center} \code{\$EXE \$\{INSTANCE\} \$\{FIXED_PARAMS\} --seed \$\{SEED\} \$\{CONFIG_PARAMS\}} \end{center} The output of your algorithm is saved to the file defined in the \code{\$STDOUT} variable, and error output is saved in the file given by \code{\$STDERR}. The line: \begin{center} \code{if [ -s "\${STDOUT}" ]; then} \end{center} checks if the file containing the output of your algorithm is not empty. The example provided in the template assumes that your algorithm prints in the last output line the best result found (only a number). The line: \begin{center} \code{COST=\$(cat \$\{STDOUT\} | grep -e '\^{}[[:space:]]*[+-]\textbackslash{}?[0-9]' | cut -f1)} \end{center} parses the output of your algorithm to obtain the result from the last line. The \code{target-runner} script must print \textbf{only} one number. In the template example, the result is printed with \code{echo "\$COST"} (assuming \parameter{maxExperiments} is used) and the generated files are deleted (you may remove that line if you wish to keep them). \begin{xwarningbox} The \code{target-runner} script must be an executable file, unless you specify \parameter{targetRunnerLauncher} and \parameter{targetRunnerLauncherArgs}. \end{xwarningbox} You can test the target runner from the \aR console by checking the scenario as explained earlier in \autoref{sec:execution}. If you have problems related to the \code{target-runner} script when executing \irace, see \autoref{sec:check list} for a check list to help diagnose common problems. For more information about the \parameter{targetRunner}, please see \autoref{sec:runner}, \end{enumerate} \item \textit{Optional}: Modify the \code{target-evaluator} file. This is rarely needed and the \code{target-runner} template does not use it. \autoref{sec:evaluator} explains when a \parameter{targetEvaluator} is needed and how to define it. \item The \irace executable provides an option (\parameter{--check}) to check that the scenario is correctly defined. We recommend to perform a check every time you create a new scenario. When performing the check, \irace will verify that the scenario and parameter definitions are correct and will test the execution of the target algorithm. To check your scenario execute the following commands: \begin{itemize} \item From the command-line (on Windows, execute \code{irace.bat}): %<>= \begin{lstlisting}[style=BashInputStyle] # $IRACE_HOME is the installation directory of irace. $IRACE_HOME/bin/irace --scenario scenario.txt --check \end{lstlisting} %@ \item Or from the \aR console: <>= library("irace") scenario <- readScenario(filename = "scenario.txt", scenario = defaultScenario()) checkIraceScenario(scenario = scenario) @ \end{itemize} \item Once all the scenario elements are prepared you can execute \irace, either using the command-line wrappers provided by the package or directly from the \aR console: \begin{itemize} \item \textbf{From the command-line console}, call the command (on Windows, you should execute \code{irace.exe}): \begin{lstlisting}[style=BashInputStyle] cd ~/tuning/ # $IRACE_HOME is the installation directory of irace # By default, irace reads scenario.txt, you can specify a different file # with --scenario. $IRACE_HOME/bin/irace \end{lstlisting} For this example we assume that the needed scenario files have been set properly in the \code{scenario.txt} file using the options described in \autoref{sec:irace options}. Most \irace options can be specified in the command line or directly in the \code{scenario.txt} file. \item \textbf{From the \aR console}, evaluate: <>= library("irace") # Go to the directory containing the scenario files setwd("~/tuning") scenario <- readScenario(filename = "scenario.txt", scenario = defaultScenario()) irace.main(scenario = scenario) @ \end{itemize} This will perform one run of \irace. See the output of \code{irace --help} in the command-line or \code{irace.usage()} in \aR for quick information on additional \irace parameters. For more information about \irace options, see \autoref{sec:irace options}. \end{enumerate} \begin{xwarningbox} Command-line options override the same options specified in the \code{scenario.txt} file. \end{xwarningbox} \subsection{Setup example for ACOTSP}\label{sec:example} The \ACOTSP tuning example can be found in the package installation in the folder \IRACEHOME{/examples/acotsp}. % Other example scenarios can be found in the same folder. More examples of tuning scenarios can be found in the Algorithm Configuration Library (AClib, \url{http://www.aclib.net/}). In this section, we describe how to execute the \ACOTSP scenario. If you wish to start setting up your own scenario, continue to the next section. For this example, we assume a GNU/Linux system such as Ubuntu with a working \proglang{C} compiler such as \code{gcc}. To execute this scenario follow these steps: \begin{enumerate} \item Create a directory for the tuning (\eg~\path{~/tuning/}) and copy the example scenario files located in the \code{examples} folder to the created directory: %<>= \begin{lstlisting}[style=BashInputStyle] mkdir ~/tuning cd ~/tuning # $IRACE_HOME is the installation directory of irace. cp $IRACE_HOME/examples/acotsp/* ~/tuning/ \end{lstlisting} %@ \item Download the training instances from \url{https://iridia.ulb.ac.be/supp/IridiaSupp2016-003/index.html} to the \path{~/tuning/} directory. \item Create the instance directory (\eg~\path{~/tuning/Instances}) and decompress the instance files on it. %<>= \begin{lstlisting}[style=BashInputStyle] mkdir ~/tuning/Instances/ cd ~/tuning/ tar -xvf tsp-instances-training.tar.bz2 Instances/ \end{lstlisting} %@ \item Download the \ACOTSP software from \url{http://www.aco-metaheuristic.org/aco-code/} to the \path{~/tuning/} directory and compile it. %<>= \begin{lstlisting}[style=BashInputStyle] cd ~/tuning/ tar -xvf ACOTSP-1.03.tgz cd ~/tuning/ACOTSP-1.03 make \end{lstlisting} %@ \item Create a directory for executing the experiments and execute \irace: %<>= \begin{lstlisting}[style=BashInputStyle] mkdir ~/tuning/acotsp-arena/ cd ~/tuning/ # $IRACE_HOME is the installation directory of irace. $IRACE_HOME/bin/irace \end{lstlisting} %@ \item Or you can also execute \irace from the \aR console using: <>= library("irace") setwd("~/tuning/") irace.cmdline() @ \end{enumerate} %% %% %% %% Scenario settings %% %% %% \section{Defining a configuration scenario}\label{sec:scenario} \subsection{Target algorithm parameters} \label{sec:target parameters} The parameters of the target algorithm are defined by a parameter file as described in \autoref{sec:parameters file}. Optionally, when executing \irace from the \aR console, the parameters can be specified directly as an \aR object (see \autoref{sec:parameters object}). For defining your parameters follow the guidelines provided in the following sections. \subsubsection{Parameter types} Each target parameter has an associated type that defines its domain and the way \irace handles them internally. Understanding the nature of the domains of the target parameters is important to select appropriate types. The four basic types supported by \irace are the following: \begin{itemize} \item \textit{Real} parameters are numerical parameters that can take floating-point values within a given range. The range is specified as an interval `\code{(,)}'. This interval is closed, that is, the parameter value may eventually be one of the bounds. The possible values are rounded to a number of \emph{decimal places} specified by option \parameter{digits}. For example, given the default number of digits of $4$, the values $0.12345$ and $0.12341$ are both rounded to $0.1234$. Selected real-valued parameters can be optionally sampled on a logarithmic scale (base $e$). \item \textit{Integer} parameters are numerical parameters that can take only integer values within the given range. Their range is specified as the range of real parameters and they can also be optionally sampled on a logarithmic scale (base $e$). \item \textit{Categorical} parameters are defined by a set of possible values specified as `\code{(,} \code{...,} \code{)}'. The values are quoted or unquoted character strings. Empty strings and strings containing commas or spaces must be quoted. \item \emph{Ordinal} parameters are defined by an \emph{ordered} set of possible values in the same format as for categorical parameters. They are handled internally as integer parameters, where the integers correspond to the indexes of the values. \end{itemize} \subsubsection{Parameter domains} For each target parameter, an interval or a set of values must be defined according to its type, as described above. There is no limit for the size of the set or the length of the interval, but keep in mind that larger domains could increase the difficulty of the tuning task. Choose always values that you consider relevant for the tuning. In case of doubt, we recommend to choose larger intervals, as occasionally best parameter settings may be not intuitive a priori. All intervals are considered as closed intervals. It is possible to define parameters that will have always the same value. Such ``\emph{fixed}'' parameters will not be tuned but their values are used when executing the target algorithm and they are affected by constraints defined on them. All fixed parameters must be defined as categorical parameters and have a domain of one element. \subsubsection{Parameter dependent domains}\label{sec:paramdependant} Domains that are dependent on the values of other parameters can be specified only for numerical parameters (both integer and real). To do so, the dependent domain must be expressed in function of another parameter, which must be a numerical parameter. The expression that defines a dependency must be written between quotes: \code{(value, "expression")} or \code{("expression", value)} or \code{("expression", "expression")}. The expressions can only use the following operators and \aR functions: \code{+}, \code{-}, \code{*}, \code{/}, \code{\%\%}, \code{min}, \code{max}, \code{round}, \code{floor}, \code{ceiling}, \code{trunc}. If you need to use an operator or function not listed here, please contact us. \begin{xwarningbox} The user must ensure that the defined domain is valid at all times since \irace currently is not able to detect possible invalid domains based on the expressions provided. \end{xwarningbox} If you have a parameter \code{p2} that is just a transformation of another \code{p1}, then instead of using a dependent domain (left-hand side of the following example), it will be better to create a dummy parameter that controls the transformation (right-hand side) and do the transformation within \code{target-runner}. For example: % \begin{center} \begin{minipage}{0.4\linewidth} \small\centering% \begin{CodeInput}[frame=single] # With dependent domains p1 "" r (0, 100) p2 "" r ("p1", "p1 + 10") \end{CodeInput} \end{minipage} \hspace{\stretch{1}} should be \hspace{\stretch{1}} \begin{minipage}{0.4\linewidth} \small\centering% \begin{CodeInput}[frame=single] # With a dummy parameter p1 "" r (0, 100) p2dum "" r (0, 10) \end{CodeInput} \end{minipage} \end{center} % and \code{target-runner} will compute $\code{p2} = \code{p2dum} \cdot \code{p1}$. \subsubsection{Conditional parameters}\label{sec:conditional} Conditional parameters are active only when others have certain values. These dependencies define a hierarchical relation between parameters. For example, the target algorithm may have a parameter \code{localsearch} that takes values \code{(sa, ts)} and another parameter \code{ts-length} that only needs to be set if the first parameter takes precisely the value \code{ts}. Thus, parameter \code{ts-length} is conditional on \code{localsearch == "ts"}. \subsubsection{Parameter file format}\label{sec:parameters file} For simplicity, the description of the parameters space is given as a table. Each line of the table defines a configurable parameter % \begin{center} \code{\

C׿\q?^_kN|s.c;%Se1*xK<}{ Yθم{<]|^UzvMu[wC?Wβ;oKo6|WSFe{/kCzg=r_ѫ9~{7z=c؝;?z%9.p@_~_~ϟ1O[q<{`s\F}w|sܩ/Uuxj֟j]cΟ#ޖv%vɻo_vqŨî:7oy:_V k2/仮N}mE?<6s&ݱwC=g[_,og>sK.˩~OsGM}f^3=gE[>v={;ݸ+^w;cO>(n~o*T7?~|+Z\X|~0ͩ?C;wq3ovms(7mn:|Czg ;/{캯% 6^MGy矽9wmL]e>5m/z~÷|g/zȆݳܕE#szTuG~ӄI׾znjזϫmQ/*⒉KK{w9げ׭x[߽j-YkpҾxoɻxsGɋ.]~ W7 z |jiwfI|}}^q):Uw<}߆uns/Wtٰ~UG\p;N_̿O=.LǸA9gv Zu9sM {ۊ¯Rv=}Yww>u3 -|B r ~3CNcE`{mNxH)Dw! 3j9zYڎҮ |q+hB"ǷXK" n*ව&h`\ &G_q ( L蠛Lo.@~6),k ;Nʌa2p%Q@PtbE)+N.|G{|iI?}z)%7iy.O/XZ Z{rԓ-ڴg}|蜘{4wl>t%VShƌv[;`7u { =pwՃ+7Ξx9pgWޣ˺N毖ZΟ>^!y_u{n;#!3gerV>w;.ew,Gm#>4z}[+~k:yRGKzϾo>BeO=Wח=eAvm7߽9cѫdS'ϙ&aQy_>0}Bޖ'ߛ/F.]eq￢ac~;b׆dwƔfޱd@/ʯ?c`N+y+GmxQ_ϙ?ME|zhk/s˓]W{YلӇrޫa?Ov =kk/ {QM]9;ξcewƐ.>]g}>{i'MF._WRtۯzmyu3n˖ٽ/uV>.K3i7xgLOq̜d{'(7M~jK/ yCj>y-,{y^9ᗇOn\m+7.\w'qQ^yh)kwz?;E5Uko:7qOU}e*N:5˧ޜ1W,m`z /?lּu%˯}=I?yʸ|]LfIӦ|cy߾n*zUGμh=N'A.dhz$&(eTrIb`R?gC]2ىI)@o^fa $TҶY{ЪhtUa!t;3O i g̉;;޳IDvN@7 솿=@+ n2OF= a d0ly;pCDg(? \˱SGr7ny#EJ^ _FC ߠ Jɷ+F9y@o2,é,wo`<i%@c07\Bo0ʍN k>kI.ƮmevvQ:g5𺃮'0}A L' C (iv@bBp`lMRkfUL{p"QClX%ذJXL 0&`$lH$o1khi9q?* tw诿UmǗ*%%m&Iu NR%"J5sΦf2mhg6d|},X<pVrk#9gi[^߹cnt ڒMfmU,`lߢ*4,I4V W*js"mHaד!f rUQWVۘч2S4%e5vy>:mmj(aݬYɲ#k\9$~(|}4XM)h/ԥ9~h;EO mmup}]mMKN3qLoVt Ż _?]PM7 5Xd m=Jjά@> z$hY]2b́ZOd)Joa$|or}ˎ1kcu{#H9V3<l,n6z4/o u ?(~#1U~pSlty` uEʊ B!` KYo9~fnP7s"f.d6psiN p.y:ӥLm?z!Ctzx\pc9ЖB9[ ޿ʦa:k>DWhPMKA|ăkA=C џ.mUAȖ0iX"6:uԂvQ;Xং 59n05o^=xmLp3M 0-i1ϚVH5O7S.$h Kx3ap),HO+e_>'9*R߬AOCG2/5?#5n,npۃY5fwC9Ò<| s6cs #MDNyp$WIBt5WA0´~λfDA ӸTm+9Gq9V1'{e?aIhw0}/'!}4&0#VͥXtid߰`r'4+x\m!VC<_Ȓ |k~i[dT6LHvfC[Ap0fM1-dׂE(;v6H^IL[yҲn>0^-r"ө-[ ;B-tX_T-` Jƍ pJ3:V bJڀô Z.ӫ֗K$(UzIpͫ*kM42+3zK*+Hvo'yrԞ ǪG0ڿ¸i# HzȲiAxq}R~~VwHE3w%K l6g LC6"!yD, ~9eZO37C i:'o)0}8yo.k;Q;;X-"xb~ĝDp'{LStpg;YӀcz0w||e.K-숸+=pWXq׀pW;|k1D-p7Y6PYG<ȥ"Y*BP. $Jjpko >iL/ S2ɶCZ<7 0\J!a)¬.OI޶,ûDlӠO(n,ugwluڔpX"=tCbAXNbAX)`=(so*{loV.Xr=na'mvy/rxFn1Jafm nXn#T3\[;]4}'3Y~4Aw!Gt%_I?Np8K^ga\M|@0.R{$ X~V'ljoU<`% ,NpߜB' ~xo5`8,rm,[lȋDg>>~rP3PxO>5>~thFɳvr[<\wrTS,Jz¸Gu:ዌFOG(w[Ka_NaY7([K&K qXKc+XNl^hZ A~߉{o;@GPIG]#iM q2( |gЍ@Gx;Ա7@? ЗK "x T :@>P@R(zx $ &L!0-JK/gG`b ,u8]jX0<@*6@*TQ mf#wrY:T*ԗbSZeµpe,9ߜHng ؋XGPc7:å @خ1 ɵ+d?VFg2hd3X h)LF%VB:^ ҷH x\lxgn L?)ld^L&Mߎ$qi.w| 㶆^@Gb;˅.Lo&O*z >%+Tۃ]Cr"W8wAXoLTWgv.?i=y:D:-[؁9}k\A!MS?5vP:S;-AbGy^WiBWBW"2:L+K\7#6h 7I5UL ?tql6.^k7/;7:[gG#D5ƏF.eO*^<5Uʪu,}{Ӷ(* kM/$P2,~4YiN+vD;‰G:,8,afF4MDV4J!h0'Zx!^kyk!La>#c6"rJ#Êqx$ / ֦ A,01@rb2-A󺛱xo'̧7詐@\=>ZF#mLCWh[)5# a\R*eULF e8lw'߾*.vEh݌&rfYEZ^ ( ja0&]Y3_/dz?UaCюm|Fw e@w3t6]ЯRmVС NW |ꈭgnywi oO /uPصiʶH]=q?8之 g #s-Gy!8_!+e{ пH Ž+2a73pܫ :q-zH("0>,-n 6],\MzaBn>p_{?p/ /両5B0O S3!THp~ ih[(u Dc3cbw  =xO& 0{{>|<|~ ybǓDx;\anZEx + u9˳_Qm&$kCd4aҫ%u5vm˟I71-TǯgQ6K6H.]+{&HZ+u̯lB1qܔYxY28]Vo#(Z6diЕN]H[Bafixms&;xIHz^`+ %`MWxگ֦y-~m3gkm2yLϜ;Iý7i]8?IKc6Y'K3 D|V"trVj:7rj+8~?F_˃|jz6v6K163m/𢡄T$JZ?ᾤW[MxՓg#(K"P*"d%`<7Y熮WАaӥΎ89swS%h<嗦[k%2~9>b<\@mmN;5G|#3Ͽ1QuCȿ4an8oO0/"D3~"d_f]}S(45Fc6i&`݁e&MQc8ntv7+MGouʂiGD͝`j{DW{3Kطv-f5j[ PqVւ ^3ϣ^+BAcBn#SFx tKʧH!Z#Vx/!hח9ȽܛQ qZw+!N4uzvƲJMJtV e{z%`ZRE⦸=^i"զ]xٔISeek7e7kd͖w@5Fq_8αPCe~/&2sINEYeS|mW/RB fBM,ќQ28KOSkw V\_t~4t䭫s%Nl [lD:CWaGAGƟ)U,,5["vU'BYgcRs.SMsݎ[\*l**sC En禚) ?4MMx1jēy3\LGG|)Opi"(͒p}7D$7Ta'maJJ_ÅoZ[Q'{!ڱNgքiBH ~dx&Qoԗ%Gpӏ#t| b5-oILJ>CF>d؂aX7:$@r?FwKDvOL{K7=*ݾr-|S$>daL.8s3E>L7ѡ|Bǡ:G8 u-ȋ<<|=Cl)`_,)"?U |O^2Mį\LFGD })LT.fr1 :2q;)1|'ljoU<"];c?NVŃ7܅zSi{ "=a)'Jzҿح0+vma9>mlbIb\(lhuYQg>5>=j;yRǒtWJ<%_Io5kk+u?ų8K:=]q@ߍb:׻w7ŏ=aP"eDyK$Ri\wl0}x:%=)_@M˷ѻɷA>on|F=|:5 /wm`o -:ۑ_;vSѮƒW nUPulPpZzo#^j"V@5kD3|%u:pkwvm$=ł=/qQmD=m3ʸxSpʒ:U=iQ-PcVKn5lN6rމrBnpO/e NZQ0=Gn* o☶wwƃtLpW~:TA,Ʃr[9yb*rޢ. "ܸϖܥM &BExh'kEARՠ qjA'lmdK"+E--wm56re6ЁAnz8@o}`?C>`!0p<4@ !-X&`Z)fpfBo`>򤶘QTcI"@[,#P`% [`#<  P`oA P PClw;U{~'85p&<p.<?O \$p\p<nXյ;ܥx  L!G@{?Fqx6pQ w4% JM?*=["} PO]'B_8KmKx02'Gu c#;S 2w3UBN[>$I2^g8rv%Gr]@ΎAx:QAx&x|>>W]т =\4 ۈQY]o42;SDaljke{|7s|RcS- nؠH. ~.ux9taowENhGYa#f*>j/YݍsD~`Ur2&hW $=<| ?RH1ž>Y_/}!0y d'>FIw>-&{*,qφ@Q^,w_4/,*OkX4VU6U5 [3w_ <?Tl(V5Q4N~1B<.*`|^Oxo/1GMr|AaԚuJoβGl!7H[ ;cTv Te!1VW[,u^omkmBmeˑ`a$}_m2o_鶵_Z|+c6*aV@piګ,Oí6c6'Ѷkl6@:k(0\WeօcR61y! -/_F[‰vmoWbh;W-S+;31V;dI;;-u? _TXdc~ZtX?VhWYƩrIZ㕯 Զt ?KS<_xTh .+L!-?:p1r;Djt&l%{P?vm#բe|ֱ&ᶨ亴wEs5<㱳veӱAYY(Q2H]umS,ZQ]O&ԕP7j=ⳅL{5&Eإ9wnz$p+tz L 0e%l3+=3 '89m'ܷ*h[72 G!"*??ΩdƏxtkوzT&' uڸ`r׍uٓ, 6N/0 bRi߳3ˍ5>?J}ʙ-RURX.=[3`,t0{QU,o`/9,Ӷsc(傊B^䎡(/EchheheR]`,䎃]Ztݖ7;E[55r;_􎵼CewG{8QO.k,`͂6|v|=_xԨj#T\f;'D mk('uQo*مX~s53Gj,[.h lsWP/l}^>T'&e-_8{?J7y(yQj,jQ+3^ƏS+xU :*X䪢$2AXr\]Rt]!zHtTRvRBaߐ"sO6S dWM1rf|z֜i*,Z!HJ52M؛_\t e|0BݜKWvg{?۶yj?L<5v~fe;{X=_x o85e4Ba]}}XAWU W$-v_"_,Ry5Ӹɥ:,8QRYrytEs`|hl0?- Kh51vBuo[I@u0rB UHi,YS=-7jB2rKqb6ޱ{%^Avq ݠcM*`?M̱ZP25"ύK[+v%{Sd.ocMl"Y܎$Ut\L'B\{؁[.Eu3 yٓR*aNew4U\Õcɬy Κr6rwhK-[2`}@*?|#^#|m|mxh/^{co~`_o|'5>]\L 7igV2m}h7d;fneɖoZ63u_uk栃J_o1f*閩}ͭV#n4|`,|76]KNՌr3eSؠ628K6QLkkrnԚ;̣iW'˯p9ttd6ۅ ?1;zSz O@Hb7MjzŻZEf;}U.w:s*;pfZvo w9ww0+?BhT.ng(+|Gu9z8'I?>L.z&p (Y۷wi@ttY@HyrH`ʻ}e{(oM!71%/yr_xͅĚLSv4<.'m#X5K-Ot饾ZN|H"mex6't4~ ('H~$bFGvS~DD\R.nT{DDewX+n8n>nnT w ;O^axl}`/E9G.Sd(t57S/r\Y1Wn͍GP "_0 }2}/ p{At3UމAq$HTa~SݚA` 53 XE v:tP~\$x}8?ON *e%ô7L Gc;)cc 8>A~ ?z i>hq/.?=vYӸ|r(86!Q?-/0>/ҧZ_Yp&R9#qL;pG nAvy;pڨo\Cp|68{qFƝqb,| b,1ṉFp±#c n4v6N׺_'a0v]aDd-Oq냲?/15t2:H1]"4d бݠuJ?jLy=tF> bK|p0[Z8]#IZlpBNI,#yzkHxʉd~ۓ)ƙ4v9X :/=::3@nE7j;7"" ܶѤ7˥Fಇ?Wy夳0f^w7vqN nOpWߎR%{ sCs[v`t>x݉ p`pl@gp =Fw1' cv%[OH\Q˸dR=o*G*JWa<Z I82.Ct5dzG-,fPO{UlsLItBo!^1}Ąj5܆ۃ$a6pCg٨"\/8ܠ#2 dT^!f9+V6IQp1T905NRǙ>#/?Ƽ> )MAR3h߅641O'N|m}ҾGӟH&s |EzŊm!87~Ÿ"[ ٪-l|q $[];6ʷ$>W #N}ԇ nq/MAWaoW4Jf:qyFɈgmIMl䶼Do;.oHY"6A9~{nG؃‘u+S#􈃝~oH8<aGvDdf~3GڃǼ];Gt1OؼR!ޔ^/vw{َ_TO[<ryzGghM;kh|kEüp' ݷI }GoГQ/H%̸ج!,?^B9a=p m#<҅)~OyXz޽LPӱvtpυ}vό}p7Ճ]dFg%uapymc].J:t߸ގ؎W;ҸoUfܝǍ)AƖJ\4}7vpyzz>ҋނޜ }#}]ȇn vs9:oҫzK"rmKR4\ v_ȴ,t${B Ei./@} p0%b(%XH}4NO $ ^(c`XM-`"c `N&`e)V||0dtQ0\ QpW s;K[6 -nO_{a>G->I>t_5M[Y\G0E) C:Rg.r <77]cӾ~tľ;ɫu^}nzb\u=/*(wOOMꟂe}CsK%2?Ek'Vhmy zuvhlQI+ΥyWz'qS;ֆp.e7=D5+/@* o?!Ӎڷ ho= FZfObu_xo@Ĵn1q_1N_N_?dp-!ep b4b;`yp9Y(>a1IDgQoMw6sZؗyH|+.v{ CQkGcyw_Bp烻 ܥ.2utSLD(q)\Xg^:v$x\gFEAS@GY8źы[^cujH{GoywZ_>'S <$L'r~#I77 ҠY^^B% )n$TAAnu0r`BSh+Q.7 2x1(p Hi0&-9o&Bp쓀+\'>Kzr,ߛLJ>|i2w¼ֈS&&6 d_.uf!eY s/p<R/91VҝdeeŀVkׇF A{,V)td ae"rRh7xV*pꅹLՁEDPE6j'}LD Nz']@&kˏSA2&rabmtnѨw8>[~'Dy9kkl;b헭E|s֖#b6S[-ְ󑭩=ֺxVTJ+;9Oʁk6vz˝fNmEі{jyqy3}y'&ey|aE9]2Kg%6(C 䚺@A`Kފ?Uȟ}ğOJzSHt'FzOo}2ٲ~z2Y>OG O3ؖv\1t{݅|{z zqKe|&Uz*{1ǟğ}B×>J'jGU<~|^H9ا:߁¾A¾j=lD"#?Ԍg !<0܌%~P (=7Bط_hGG9(᩾{@(nh?şihXXxpƙaS?og6>^0>R"Lj8䓛~&|%|a|8Eqm%LxQ1x̷1qox`x-Nnİ3.~Հ-ՙ~A$aBOH:r5@6]rB4r("W2>i6Cà/\5d)3mHs|Jc85tNilߑUTO|E|;3\(1_pUꝸf,\|`зJSLC9;oC0¸,2u[q>23V+K~l0pyo5zrC0Ot?k7ôX-7`Eb͇$aD=H;yvS9;rNoȣ-3<;[Ì_қ.oֽ1Bl?Diws ;8d{^g4H`p0 ` .K 0 &Nh),`Y0h~<%0XXDXC`gXO% l('PJUPC(.#av;.8@q9*'ɧƎ'hM{RbV ]{6|CJ:ҿu[-ź*b:271 AodX0UaՂc)5t #}䮢DwD?FmQaD HQkqvsR3S=Y'@~>e{(\'nMG*ŏk}>d/!"nTfުy']B zG6Nw(T ݏ[P+bW\3|Cw#:pz\A,ѓ7. v׺cC|.+'ICùK(M͡KXZ{/Te5~-1GՂw.k:*o28IĶڥX*oyN"٩HbtGFGIpj.jE=$jY{x>]-f|U+:8Ǜ@\gp#[?tn| ܦpE!||%eE FIte_̰[k 13Պb!dH+_Hw*)TN'+V^0]ɤ+H\ڸB,kYLtt9 BەRJWGqŴʈ~J V7i7(#ZEV+ Tt(^-\JppeWWeI4 `F?p%=tg2<=][?!p6  ;. p! -#OKK.'p` O\C5p->HuBxz n[`u;m3>p/ <3-6[3@=Eig\%< +O1xb  2u_BK)21?g0ǐJH\oJ]EYeIJ ͌/?*u%gzr(gofta+] f:,p<ߗ,Zs:3 9 ru`9K] =!lZfp%E}A~~9xZr y@c):[5p0fR-({a-{7C$T3!OB>,% YBך(4[-0P#߲sg{zLW., nؑϖ]w4{2MM$&eb{'<>ϟgw!ӱaSV(~33唪eS%z,wͅ^mSdSqvy]Mm@v]CBt9C?+"w* B]ZuY`[o,X^~$ӱ+#lmU&d?4W]xtN`aovMmub]a_`h Z,,;3fzPSqRU]Y, :AɅ2j\#0ՕuU,q;uڛ3yJZWTYZl 2 QsNpVR*ti8kUt,wVX맃 .̟E!g7{w9%}v@R!7Y@_Z]QY1tC]EreEvme.Jla9ZimI LA9hD}8C5%K\ ^B{mc珴|eU"VrUaϿᦪQzÄ}km`Aч x>$"gZd`&sKp[ \&)͂jkn:43}l%UY,(Eo9FĸX̲H@&#lbH)( Kbc*]n7pÂ) I1lJBg/K#\4_Qc×+A!?(~ltZK;ZZHE$?_} ȧp;B81_ǃ7"1HXgp Ruͨ%d?!z hnY3 NrcGGHp̓Ϥ,嵐X$Z5~n.Hk$ވxa,t|vlRjGG)@'TE,,kI>0>L!pim A9$$ᲅiq t 2 }t!"/d? '~Gs_* YXr 0g#lQ1ƊO&8@!l L֟AYiv0"0_/eZ."?+BG% z VY~P啰m `Z#[w)~o!nx0 ,N֬-NqK|(F{v50/rjJl '\QdI zv͊_}I H#ՖW [oɓ[3kEs/3rdnA0PjGڇd2=l_UMuɆ`Y_jO3H^ T6?& M0OC #{9dryX\T0eph|uI n7rCT :xʱY˶Me%f5Adw hm//2l RIRõZWӅ`42k42|W/)/11`)r8K+: a|ЉGTFCShtOҭ`Iv! _h /9!25DF5 -l`P<6s[.dJlkr* }sx5'oNʰf\R%'LinmL^Vou" or~iQRt|O!C~>mv|$~[rI,a8( 6I!]AH܊^{s:!cvee%u jrq6A't1.ѫS[d-|Z/CqBI6kaA w'r>YE79{CD^ڋE?t2wC9нub'UZCqE p@.&[+ANeOĕ21 o7&}j]>gW"?aQF;vxŶ\%~l{u^s=uĭ=GgS6X-ߓ V(Jc`Bo6 KTVln [zMٷ%o8c~tXjB06Yl*]A+Vjo.g=o]<"@m C+CaxpF~ה}H$H ̜RY^(- QY֗IHIsn|cYnyoy%<$:D< 'I+aiXyh=tߒG[fp>+ہ_HOrt9f9l3 a2mw =n_pqM|c ,gfA7xJ9@7#y2^yk0#"*V8FC"zViI](W<y<^hMH7Cn: 2e:rY䟧H7/nd@[doԓM0ؗ)bY aE,^vIe?d_ wȶւgqh~=%8/ӗ1\d& gt[&dCzIw08U8TV)6m+P jBwZS6WGl8;v ] g'Ȼ=?)0CaSv+"vƐtr hܳQ9Y#v\^w}0 BW2m/0A{9We(og^ W s:$uZ : ܟ{ýQD֟aB[Y[,x/m> Y¹›,:s':#˅+-3A{} A{oh| <<}Jd iBDanKhlf~,V }}Y@IC~7fRbgς=}=OxQ?W;䵖zx1^/ :Hx],_xMV߀xR?/[ ۛB{owmdzxl}ǧh%O!a#B!(Daw~ LxP3) Kȑ?~vw? 6HJ@~ ? F'.áM 6:Yo@?!g)qq(` ͱ|rLa r|vTght3NwGL2+**J\k+B*C$+6F//ǁ!;tI{qk姐1+2J 7fIlEivM]UZ,XYIӬֿ, /b7kɘiBYîH7 YCviQ[tzhVBr+[&ajϯ0)N23|\; {; rx.Jjg9(NzRu.Ha{a_,ܺ:-uh7sNfM΂ GJIR|NF.$N帛Pr\!g^?یW\_~j$.ʩ?ܣVs|n䵒{Bn?1;e'PVۡ&Z;qXsǏ3ǯgrl7:Lys<0NL4'IxVov*f0`[9e.uo]R]biiGD`j{{wX}/˨q=G#mJ0lDrJWAx*_5ϣ^+BAcBncSj tNfSڋE yE،ě|vV'hח9Ƚܛv|'hW8ȽқvMljfmk;Xs[:=u=_Sa'Z5UtKSETl?f|Wӝ&KؽLz._*bM}rmɮoxQM'bɌ?jWPuo J-[=+zfottQe zT7tUOJopUUtI vrW+~Scƿá)!Xיſڪכ)9Ȼ6a_w{J_~h9^y&cYY4"mmMu偊ښ!jk7l8d돓X2If?Y07p***N`LcT)^x&$|6˩nP0FtQg+8* g@FJ_Å0p:vl%jGߚ0@)@{a~e0iDVy* Ғ4 DĹ_ ;\My:zDDBXL2G٥ɏoŪDLJ>z[2l0\ 'e$ݯٓ9 H]m$3S2-tX_/"0P/Q酗R`~T1O~A ڀ_@MȓҞ2 Lfaxрx~!W#]e]ӄ.e Kץ6".)dxFĮLfǪ*, ѓ SV09wzӉ.̷\*RX 7Ń/g0?5Nx#|>y xۅa:bW+ӭGtvD3MDnZI.ƴ.<\In0x.%7Q}pUO$Ϥ})N&66Au>L7ѡ|Bǡ:G8 u-ȋKZ X'mbKD da4dOTL~(Ri"~b0:"Z.fLa*5?r1CYs$H88N|\?qc̷*|%y.ӟ|OS #O8Uғn̈́_nۗAn3fK OR"pEafTG'blஈ>+9 OW>'V#ɳ8|}=ʝWR,Jz]Np}]sl_Io;-IX_F?ɻNў0(F[2l嚆I6W07[ȯܝ|)hWcWҫ*v:v(_Io^V/ﵩXMߋ)Tx)G\07$v_:pkwvmo$=ł6B `;z7Q/_Hq=^ ,X%?f<{ Z BFeHk . Qk>U߅_*إfwҊ>to_t>B.54c>HD=3= \mSK:7KoY1{@{8'8'rKM &BExih'kEARՠpjA'lmdK"+E--[׭ @km,"`ǡ=@m.p8~h| }z +|'c` 6hq&xV3+i{Y\ЛC`<-xa3`!X`)E,XIÖ1X b~3 6;@9@ATs]Na`՞DN 4j 8O.$S9.s+ \:ps3[qLWvǃ/GzL<@U27T{^ r.,˜< d|*V u"WJpeYmY gB:\IDvS?/C'#y`"L/>E]N eGY6#v)>j{YbsDu~(sZvŵIЮ4Izx~N"]O}-[xxR m0eJV`\GZV"r}4a^[Lmൽ;_>7=am ˰dkarw.Ʋn,!dXЃ0Y{60>Yj퀎 z>fy'Gz8#x+]la~-]wm @z)S)KaARJ[amll.]i.7". WW8 d ˃0'*ʼ!ׯVf1+J/p0jmh]tꁎ-$asҎW$irTҧ#{eS^Dd¥y s0חi`!K L5yF(8SD.l/2lN[;7Dg)hWE |Tض6H:lҖ;).F N a+!l0'ۺuĦk KB2^d,K~7٤Hd>!\Bt.'E* Yd* \Ӕq-d?V+@.$g`zv6[ov O 6M2l'"-L;q_v R]S=$ ˙`,O mfGcy#9!c ie1iTm⤝. њtQNJX.v@ֽK .'q])u_VWMBPe>7vK{.=׀ͯ]qu`#Kh0ʒ׃eFpl7dپ4n_Ӏspn||FsŽiz[{p>LaW1}¨SCN;Sr ? 녾08쏃h\< N'ݤ|OE6:n@ِ3h>wu\:ߋ KE%/Y^?¡lm; {[E/f\hXHvOdJ|[x']qJ4sC)WOsrE{No3і+i 6u?Ш㯾xɏݖ~}˲o/!qS(qN 3~˙E\F3!x9yՏjl s|m)|v ֖fQ[\ ?nvc7.6* WZ{YCW}Sa'Sz'\ rqsOX%9W-ǻyZ'V+Y߹ϲ 6on,hWi9W#Q?Œ4_;%Wm; ^;.>ᆯg~.zp?hT{db}O0e_@QNze8.WExq'(tksղetdWRBa"sO6s^dW}L1+σKf7,Xha!R//O.%BY^3<~P ޱFPIst=MK%f:yj>n*{>v{Vuv9lm UEU7◈Ɨ+LAe"x1;#\uإ:,6M f|9󿭚D;7c1r.G.^F͡XPa3<΅>;Fm׾{8VkɡCˣPn/8T(۠oVU(5uUj[ 'Dl!_TGgw|Uq#|~\ ]ȇn+z§]K;~hN}+._?XRE*_9rj,߿mMA>_m{3;~ښv#c<~[GP1"͝:\OlO|0[ܚe S&[G>=;=y~l% ]p^Ns~4wY8޼]=k ap.wSLA 9{'~DmUu[* lMݜ_iߣj$}/W eo+iXOѾvj=@NӑX}5zUqr} |_g3t\`t^D[o;ČjJR3b'`ߨK o_[ɴA%2~93&s^&stFU :ȽG36tVr}+__WI|wnlP%aj)l|[e.SԱZaN6Ɯ.1ECy<麙[Q~u¾ clI^AGoz](?zCx;1^o!ToԷa=nGXU3+ʎ,2雮v)SفǛDr߿}x rzx /JVU?aW^v|д~M5]zMK9Aǻ\<"v!ŏe Data['GџA?>JwE.+? |AKGjBRV .C:H_n*ÃZ9(n0$)6*̏s[3HLAE&+ߎ@N6Ё@ʏ :LKYXw0D1AD +Ouy;Dqhq1]84X[#Ʃy<w!E,ƸzXb#>Ƕ6xM8_Iu=S1fNuǷd\Z,8>q[Pw|ă/ҙBr#t !6pBvv~ԥK+nH 1jV9)w2Kz2O--ClitT'iiF 8%4Z[!">0mg:`.HSqQ_hxM)wmucApC0p{udNLۚ+Qv¼VYsUL/û np8`nv'}oG^}9)1=vGhw^ N%owvKq= iY`?A-MrU}Kd?-۞LHVK{&e<0֌?ٟzo>Q&B0A!>z3z 4Zh }L-dYwˉM0oYC<ͧI8gǘg4Y8 @8HW t& І&PډTOڗHۛ6{7&Hi {TQO7Ӊ'OFl}t5SGTHmb# ~[qyE) u;xԟ[9wdž􈃝~H8|c.t\K;t۹7Uh%}9Pҁ6ץAKp ) ^oC d@ :=`zw!d"a|_o?AC`m1@ hM$pyk''F/g10,&X0@0O'+>P>f }Jc:( M`.ُcic_mN%qEǖc7'0#@Ϥd& ޭ,Sx#S"!)kc }ig"3!,߲t!vh>3VOp9~x1yib?:bߝ?qW}ȫ>BF=j_գtKA/>u,\I\DR\&[ʗYy ;ꧧ`OADz>ѡm~"h{r+k6ӼMG:;irAljvΤzR+= a㸩 km8gu̗|pG[ vrx_̟FF~7A#-_'/7S bF 8pbp'axp'cD&s2Eo`TjBÐ2n ӄy_j]̋ `¼f Jݜ ,a |$"dz;ܹ]_-< ]$e>\ؕy|_;N|__I=!ޡ5#Dxӱݎ/w!]Rp:}iJ_"v}ӸRkE3/;}<3ݢ|y,_bE-[{YݱB}:e~$=z#Y<;A\i-t)dav& 9{@\?a|ޑ^niP,H/ vZ K! t7 trow H7:Tr9@s0)m(恛Gors@{$4Zpcha|yc7G^!ELI@WF%=9M^BWL ~4l;a^kD)hy`Pk/Y:XD,K9{]M)œG] Ndвb]5gvCqu@ =@n:o2Y^dRV) <ˀVpWWm[ 8\@"c|dk j&b"u'Zd= ܓ.{Jw ǩ ku0 9{h;mrl-ָ#~w/[抭-Gl7. [a;#[SS>ZoUVX~R\{\431}^OW{˛!; x)6'tmY:Ӝ/&PA߬:X (Pk N.U+Kx+FS&jU{>}rt%=)QTK:#G=ا7>nlY\ F=n,dG#rlK;.xBY~>=}= d|82W~*{ e=JOw>K]ZWO *?Z>tS/S@a_e a_e`d6| j3iLG \vnW?O(ב F!m/\#xT= X mfPc?ƌϟ 4NlsPsZOxUO3BA6xuOUM]}qƾe'w|7j{aMS5/1 ͨ4CU}' P矌/1/==B/N6c{VLS2M(ˋe?]!ePYS~؝T\^fWݲW/+/+}0%`ExpMg?YX~yƧ W'Cޡ>."_K' V[\]䍧P/I8lԊ٨%`ΨDF+ẏV}}-7$?ȗj_<_̵1{6:Al2IlrmZ]F,I0 K|,84S:XnQޣp[+Ӆ/?@?^ 3x_Ï<%xj%n¸$ă6TNf sSv4ԁK}!bZ>D݅KoݿtN?2C}(8#/qχL ӡ3M+vtOc񐩈R x">1P1`e[劷v/GqtSXsÄ~s}}:GziaaI/W)ycDM?K^0>8^L<ۨoc<ۘ8Ko1 B@'\o4h**'e"}./*lN\3.H>n0[]%E{ ZZߜ_J!M`a\ :D8V`+KSf SON=qPE5MEaZvA1uopv?ٝ<)~ ۜS t 7і\[-aF /M7^t|ہ]I!EDCV"4a {ǻi ڹzO = @/|ˁ޳w$08 0p%FcS\O'a4i0,fo| ]K ,V,ep!3'PP@%*!P `; `BTSTc4\&=])K1+]W =kp>t!S%_ :bg}1ܛ_ 72,Yߪ0jr챔 >rWzqT6騰I(5֎8C;fZǹw)), zr?2?C[oƓQXk7#MyRnq'Gm-1#n;9w-X p+Es Ypgpd֞Da#-1ce+nr:prbv: C]ĵ>tsX7sd3DWowj=ңf^lv';k}f҅Z].!nnps p_IUXSk]1!>QD_$vܥM &%d=XtזޘVjAvC5exQ$bcZ{Yx<'Thh:#q#ˣ$8 ZMDkf]"[pu t= uoMU} . j7HDFPnP q"HWf>" $/fح5ׄљHjE`2ĕC/$`;jW*]KedPK_.i m\!EZ Lg,&UNJ`\!]MJ)]UbZBeD?]%@Mڍ P" j*:A.]fx2$pNq`ƞ{h3`E8\Ho'ץU jx'!Vي `~:\m n<[lV Ϻ6 m p  Q-yݞ"43.ixTh^xSJܺG1*FO sb9s791YWAx0v ܮg ~:CJC8x^;`69j7b?wԈM~ς. v[+̻5`+ 0M0N?59B@^ xHБ2ev=qJxg@tx!ex>@Wg0>O3sN+YZ\iOO0n?i|hf'W +w s Вs\dCO?ԁ0 \^F2(a] ex荇|$;/̷| S K y˜7J沶Hc!0n asגPτ74 ` ~ U\zg ӱKSV(~33䔪eS%z՞*wͅ^mSdSqvy]Mm@v]CBt9C?+"w* B]ZuY`f,X^Y$ӱ#nlmJ4X\/gV+6:0LYv6 qs.YҩXN-6%ػ+I:V1$$$ %B 5H!Z{ogrw޽ޛ7}쀵}:7[ DcxS٩5 05;_0a'k%pg9rGś{M*1qB~.ۦ=,J/nbP78t\b+};ԯ&zo,qq~}~ok9Ko[@ u4o?tv_u~M7ŗŗ)~Υ zp]ehzX):BP]ZMW0D(?|Q9r|bNwAr=C>IXT)|[P,RzCYH}Z#xǮgsp9"ъ~t/p&. h;oAg>(&[#< /QcxH .CG|hـ @;RZH'L8GPӅ_复,>H,Rz#&ĸ G;((U D=T*,)RF|d?R/,,`=<BL ƎuW.\ XXh 'Ky-4HECCUEF',~һHWLQODXxѡ|jàゅ)MGМ\goaq9q';T<H"ۊ` 2m Cl:i8gy{>{W< 0o- @7?5!O|/ Wx_B7Q X2rxVp4*пx4h/>> KM]!+! `!n ě7HgE+VU,$Rh6}`Zt#kw N ih|ύ'q>;aaM-~MgAK`Ffc &%:8֓T~Wpo0V%x?G? .:ňun&eˤwImPEPPrPt2H;9nW(iK$=H:hFp}1}H_nr d8(4nO(0'ѱ# xVA jihOߧ92aCتZ0$J#Oa!X swM`Jļ=yC`-Miom46wn`h2O 8'tq"hu 9$ejmbAfï3uK^N %OLIlQJhUo*fCFauG1p vW3AsB-9ce,G[a͝_[۳ޯ:Iu); CIPY#z3K֓8aQ>&aEjZ:T3 \u؍gyD?w2\$ќ PI[=`zO^ʇ =vxB!De2ĥӈ~.GmqoɄ# HGeG=d/<'zY/'39 ;̟yR)HyߕJ6L(]xdŅ,-xTJ֑)\>(yPVA9r`]`7'E@XqM%'屜yxzļ*c8 "y6m&0g!@ Yg3*!R+ S-J ,jP'PUz:S&@=J`ؽATJ O@b/<+g!Unׇ7tcr[G4K)?1K.c^Fs;s[H*f'c%nάg.8",t't;9 v;s"tJ7DHu$P:薏 н(&;cB􃤰;c!c_8׷uZci_ox/ϊ6060Mb j]>Z7?L-]SW Ġ.c!#h5uY0/Z3 ]B2ID:`Wa^bq߂uj!8±X"vm.mM{aT29/܄gR-J8*-UଘdJaO[`%M-x^K?do52}ӆy2w R><1A}ŖA>2k^}@@g9sȉ@6Xq}[IY bPk],|T ZџBc1!nKXcWmYk5üe$:,': b`OH'b*jv%sX_#ՃGqrd~`93yҜup*Y* AIDEV&VJ!+P?^SLxH9~.@DsHǭCĖ~EB`ŞsjThfF|zhif!pYOTLbOEWR9~*3N֤OŶNŎ//È/(p| +pE#Vt%-My1t&U}Wm?Skm|nHU8E3*@b/EWR9~i`L;*IU_TlT4FX]0t}\Փh`C3`z zr(\8ECQ5j7[nvb{i[|f=HDVK8)ADo{C]>^-#h~bѽ+,3D(+> k[.bBQ}EP8k@\ x -~ RL"QGB@#3Aq௏ 8  ޕl N-@L>>%D{Sf4B` `| __ȁ]`)7[>as9X Pk"h}VJϑ|Q΢FG!F"B4=„@L#$̈=IX#<+FZPW=~>> uŶ{cس~mK^fblc<#YtO{E:JQʩLބ.鬯!.ҙ^ў @ ~/Agv "^!F{:#s ơ::c3g&}C@x/ Nz $&qstˈk}KJ> sY6RqmJwdShlɄwPl%J1O'Տt ]ԗT9wÎ[MHgL^m5O?Ht?z(ȥ)/&w K0ںΉuQ< B3s3,DxVA z_'s (7RD  ,BJ.ԎmB=.j$UIQ.0($\azQ(ҧkEzpӥpӥcxS(CJ;Hթ 1*yeV(rqÍt]#w,_x0clg,@N8_ՙwGh?jLb#OM:E*DnAqrB(NRԸ1q Zp_!odZT9 "E:g 9n Bg';!CgdOǺseFw ?SßDsI@:9!OPb@Ñ>hCJ:K1/ :Da]s.2X;;ʹP9 7)}қb ZW$6@v 0M//t~] bR Dhl-vÄ._&0]Y76vZ<,6_ $GLMbI,-Ӽ]=qsp%DtK䐴Լd%7HS{;Fz4H=, U%l a\ TKf=nN7 >^ sÂ?RwG頶76Wv]s磻wX@s.7)1qrcVs4/tuT[,(M /WNُ¬hDcHk,׀a=MfĚQbN?VC*=yu55Uc66QJ1^婩uUqiWK(a}" Rd 1vi-JA=۝x]~p-RLtjP~ĭJ>%~#Oqp.F!BL?qYg gyIN*#p˅nQu F6t?MԽvAdo!V`I2|ݴBB8$$W4Km-VEA's;}-Ӧ7duUuD%F(YnV1焮Dں\xq;! ]"7(.pvyi8=]NѦn4r95A.p+ x:A$@Y ZOÎy9r)k@ BcPǷur4<3ђ5IJO}zql7 c{d:gbM= 9]L|x&(͝>k IU꠿K)տ:q3 jb00oOrp"C'A)I+ D=;AX5'!CZAWB?.qe.hk&+p G{ ^\?HsM0Ryѷ;^B|8K^0?T,x ѯqz o<]xz~8YǺ1|!aul\ x Y Vg/man [ ^c#x1@+7YI JM "P}b1)f!7D_)3ATgU{JDHJP`(R}+h`4n Bۀb80(3!baz8=J.U1՛#p9i@_8fu{|]F1pyɭksGyo̓BhI.uwlRDfu0i4Jat in)r r>l7o R%r _+tL" {4wv%jZ¬0O[–vUHvrNe.4 _uNX=(&yyLĽ{Dmk*Ş)2v<>,B| .&o.Ѳg#}S:M6 {A~噴I 1ђKEuOH (\21Bm}s3ng ?GldknAc#n>uQq{S bKSӟ85OI<iz5o:/05%oovtv=}oϹ@oG+Tphsh 7\ qns(# 4;pBT8%VڨVAAJu1J '7 B™9dwf_{/pIrS!hэ1ZtWDdx{S baՎ/xZ! cdޛ-Ǿ0>YT=T#kEJuP bPءvXΥhYG4JVD$l9GXXj#5{ X(EKĠ'*셌m=O_s3D JHme+t7],z^\M+g6#8,4K 4]}gZ3pt rZ:;W9HFht:X҉Ë+zZqFxU-p!j/pèf-;$\%. :¸wI(ǵ1ѝ*pJxX]1c#pꁚv'i,{$ۺA-@;|]j.EМj^#r6'TlU ,]#AXdmClƖ4գ [Bv7 EuXF1SW}X)V,|b bnei_99Du[֡D21j<.+ iy[Ƭ.ar&t5_=x8@Wp?J13M4NIt29,r8@<s@8~p{9A<9( J9(s Rf:j_/90ᤙ239t.~=Hsx>̋`[&36$`o3y@ 0 0G5DJaB0% 4€ˬ0A|!h'kU_Q}2gr |a=*Czc㪳&N|Sl>~+8A|U/iݾ/VqKn~Otm1o[tL0&p_ eV1V8ܙÞäK3{ˬ<קouM0/ˬ-?`olGנw̲"yͼ[gS+>2ÔS3K+|z#YYcCg:Ǯ z^1]Vܺ_-33(f@*tsL/sܕF7[3=DN:{[{z[Z=ޞN/?]m^ EJ7'u͗vgQ|Ϙ 8TLo9{s}͊7XU6n/6-~6gw0WzsoW?3E͒6_d>9T,\ŝ3|(rFר`)*Z۵m'@%>&ev;|$<4ͦ@n׃yOoOCr:ZZmDϔΎZݷgZÓi H0,sbՕ'. g.\7`6Zi1x x"IqDF ) n\OtpO pLp,pw={{s=s=1{{!ʀQA"_   |1/pm—@_K |)/—A2_ ,? ς |9/W@ _+ |%WA*_ |5>j_ k | 5TgC~67Bx#7B8n^oZ 1V}}}} }oM F }3o͐7!}3Kw:+V\xbsX|ZwpZsYšXP܂J2rYǵν;< )Cxl#w2yu8<)-k[PRrv .s:T:pϵh#Dte /ױӲãqXs*SNJSP)ٜρ9>9!Cc֪Y?pwA:9ˡ<8h,O(t:FI>pC=e)'uMS Y|%فkPvL>)K/;rjқSaoK2G<4vYr՟.9'>Zq?0_NQ WJGՃDO 'FNOqC^O;ziY,rv~Kiw)!{.W(Z-kɿ9ߜC9 5X F>-9{Qiދ嬞Wnt ]aީ/~:~ki~{lg$mr{釜9_MOz(muCQmA5rbҟOLq{UKGՈ%3⥃7]U}Kpxʭ9G/RNQ͑>@:QeK+|CXEIۤ }}R:WIA9Iv͔WS'n>e'rڨҊ&,ԌIǰugnkߗw' ݬX 5crR쌗NQR__tKoPp5(g~0Z-v)}҅$%ug;_Y/{۞^h_xtOvÊr"ՓI/~+/<+S?H;qB.v)z{K>W#E{.rtٿ@>j)r;m*''4 i|- {tϾϧi)ltڋ֧JW*-% i^'|"|[vӳ]+WNzM'}=֏nSWO># 4 _~^N飨r;3T~-kftշz{Nyt!w8@v}ez󛲔t!h;uZ5C 2C?5wJ/Se痓oV7v2eOj ]0G6?%g;'Oir&߁X[o3%u%߹(Y@;'gZՆ {Ng_7WNV偗%Kέe q9{Eͧ=tl)H=rߧ<|h/|Ju: LzUn:u#}66Q)jk߆ycQ7-EmHҋj6k]j\"mql*g/HW&}Ul]欔v(= 05Wʣ0 ?d=M ^?=Kz%Ko?56nHOIBu|i[ix:HE0 W]f)rr[[evĬQ,i+kw~ߎ9Bz^ie"z͗!>kV0.W%mcH۔Nl쏤]TBz;KTw^)ΦҾ ʸW󺽗.r}yxu z3hUxnx>~֫/zI ZVv.%ȻvH>nkᏱ8逺5:(^,,[~5x_\|+]M]^һ`}8cv>Xu^/ưV~ OVy_3/^ͼ#mң5?s߅}'Wڤ۷(҃TAtVǡ|(T #Oo5I捯Kl{q/u=[sj_w8ci2P~, :bbO,)}Z,Mb%lP" (p ,P|B^ ^U%\XU8tRyGG7,QRc_8 V 5̩nC$n 卹>r [Uv=tuD:<ֆNڝ1tO|Y6p}mT?F_lWL8 ߱A`n'X|]9[?vK鈧nӋؤ"|9_nG/1/z1E>"]LDz0[%W`1_b|]҉tx|1dkµ%1xwuE*|%]5./Ӟʵ R{ҞV{Ӟ4@ GƣBQxThWטBFwn%WL LCFF ~q!ԡe!dѯezb=wE,~WIkV/yxF?n #?-ѝ{똜n$&Q_0:2~-XjƯZ&gLO>njoIމ>/Zҭazv'W!{5KW\2NVq/eF{U&O70zGľN/qS'Ve=+a2~Kb~&YY͌;ZF-?km`baz^L?`~/ftWS3\}Kw[LIVi{|2=~V|Mu32y,^=L/332o/dYSd<&ke?[̬ge>VMޔ1zÿ—^2@;[ g dr!.ncrlg)WMwFA/sL#~|YI0/敁qi{?`y,7&O쪚{+uSX7ߛ"Ooi¦)+V7qW" ^p"]t[G5uNzOIO7vN׽vjf)T,/`M\Wva3FV$屧-J2}G(irace/vignettes/examples.Rdata0000644000176200001440000000070514324745245016154 0ustar liggesusersT=O05NIց!Eb$fXM)+@ &F/ {*'%]ޝqO.{=B%AlQY#-҅ -BYmA$Y(? 2c9K|DHP0^EM-7lr ?ڭ>,>puaT`}*ҦSgy3#NRL,厴.E40W V*CLobKۚ5.)2iej]F~+G%7uUo-& [ԞnG{R&8fȚ4;Dsh1Ϝv+AfȣУ6=ZtIXuЏpi'\Dr%ȫV !'!o4tj s+m oՋs;çWOtvx?+ZJ|j5^\,~[f irace/vignettes/irace-acotsp-stdout.txt0000644000176200001440000006652714324745245020032 0ustar liggesusers#------------------------------------------------------------------------------ # irace: An implementation in R of (Elitist) Iterated Racing # Version: 3.5.6863679 # Copyright (C) 2010-2020 # Manuel Lopez-Ibanez # Jeremie Dubois-Lacoste # Leslie Perez Caceres # # This is free software, and you are welcome to redistribute it under certain # conditions. See the GNU General Public License for details. There is NO # WARRANTY; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # # irace builds upon previous code from the race package: # race: Racing methods for the selection of the best # Copyright (C) 2003 Mauro Birattari #------------------------------------------------------------------------------ # installed at: /home/manu/R/x86_64-pc-linux-gnu-library/3.6/irace # called with: --parallel 2 == irace == WARNING: A default scenario file '/home/manu/work/irace/git/examples/vignette-example/scenario.txt' has been found and will be read. # Read 1 configuration(s) from file '/home/manu/work/irace/git/examples/vignette-example/default.txt' # 2022-10-22 10:04:33 BST: Initialization # Elitist race # Elitist new instances: 1 # Elitist limit: 2 # nbIterations: 5 # minNbSurvival: 5 # nbParameters: 11 # seed: 1406598565 # confidence level: 0.95 # budget: 1000 # mu: 5 # deterministic: FALSE # 2022-10-22 10:04:33 BST: Iteration 1 of 5 # experimentsUsedSoFar: 0 # remainingBudget: 1000 # currentBudget: 200 # nbConfigurations: 33 # Markers: x No test is performed. c Configurations are discarded only due to capping. - The test is performed and some configurations are discarded. = The test is performed but no configuration is discarded. ! The test is performed and configurations could be discarded but elite configurations are preserved. . All alive configurations are elite and nothing is discarded +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ |x| 1| 33| 14| 23436382.00| 33|00:02:51| NA| NA| NA| |x| 2| 33| 11| 23376711.50| 66|00:02:52|+0.94|0.97|0.0066| |x| 3| 33| 14| 23336841.67| 99|00:02:51|+0.91|0.94|0.0146| |x| 4| 33| 14| 23278031.50| 132|00:02:51|+0.93|0.94|0.0127| |-| 5| 4| 14| 23207001.40| 165|00:02:51|-0.08|0.14|0.8158| +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ Best-so-far configuration: 14 mean value: 23207001.40 Description of the best-so-far configuration: .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. 14 14 ras 3 3.7374 4.7749 0.1279 5 17 NA 0 37 NA NA # 2022-10-22 10:18:52 BST: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants 14 ras 3 3.7374 4.7749 0.1279 5 17 NA 0 37 NA 11 acs 3 2.6159 1.0514 0.5986 12 21 0.2211 0 NA NA 31 acs 1 2.3896 7.8790 0.3748 34 36 0.9029 1 NA NA 6 acs 3 2.2422 7.8826 0.1091 14 40 0.0174 0 NA NA # 2022-10-22 10:18:52 BST: Iteration 2 of 5 # experimentsUsedSoFar: 165 # remainingBudget: 835 # currentBudget: 208 # nbConfigurations: 32 # Markers: x No test is performed. c Configurations are discarded only due to capping. - The test is performed and some configurations are discarded. = The test is performed but no configuration is discarded. ! The test is performed and configurations could be discarded but elite configurations are preserved. . All alive configurations are elite and nothing is discarded +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ |x| 6| 32| 6| 23326585.00| 32|00:02:41| NA| NA| NA| |x| 2| 32| 11| 23327784.00| 60|00:02:21|+0.94|0.97|0.0053| |x| 5| 32| 55| 23179654.00| 88|00:02:21|+0.93|0.96|0.0065| |x| 3| 32| 55| 23157937.50| 116|00:02:21|+0.92|0.94|0.0068| |-| 1| 10| 55| 23231073.00| 144|00:02:21|-0.10|0.12|1.0265| |=| 4| 10| 55| 23195411.50| 150|00:00:30|+0.02|0.18|0.9224| |=| 7| 10| 57| 23198073.14| 160|00:00:50|+0.04|0.18|0.9016| +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ Best-so-far configuration: 57 mean value: 23198073.14 Description of the best-so-far configuration: .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. 57 57 acs 3 1.0519 6.5794 0.6649 11 33 0.2283 1 NA NA 11 # 2022-10-22 10:32:21 BST: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants 57 acs 3 1.0519 6.5794 0.6649 11 33 0.2283 1 NA NA 55 mmas 3 1.0832 5.5967 0.9317 18 46 NA 0 NA NA 6 acs 3 2.2422 7.8826 0.1091 14 40 0.0174 0 NA NA 39 acs 2 2.5486 3.7878 0.0262 13 12 0.7621 0 NA NA 11 acs 3 2.6159 1.0514 0.5986 12 21 0.2211 0 NA NA # 2022-10-22 10:32:21 BST: Iteration 3 of 5 # experimentsUsedSoFar: 325 # remainingBudget: 675 # currentBudget: 225 # nbConfigurations: 32 # Markers: x No test is performed. c Configurations are discarded only due to capping. - The test is performed and some configurations are discarded. = The test is performed but no configuration is discarded. ! The test is performed and configurations could be discarded but elite configurations are preserved. . All alive configurations are elite and nothing is discarded +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ |x| 8| 32| 62| 23338867.00| 32|00:02:41| NA| NA| NA| |x| 1| 32| 62| 23375111.50| 59|00:02:21|+0.94|0.97|0.0075| |x| 5| 32| 62| 23196573.67| 86|00:02:21|+0.95|0.97|0.0068| |x| 6| 32| 78| 23240373.50| 113|00:02:21|+0.82|0.87|0.0165| |-| 4| 10| 78| 23206857.00| 140|00:02:21|+0.23|0.38|0.7064| |!| 3| 10| 62| 23192587.50| 145|00:00:30|+0.20|0.33|0.7410| |!| 2| 10| 78| 23195256.86| 150|00:00:30|+0.24|0.35|0.6748| |-| 7| 5| 78| 23193481.75| 155|00:00:30|-0.08|0.06|0.8453| +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ Best-so-far configuration: 78 mean value: 23193481.75 Description of the best-so-far configuration: .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. 78 78 acs 3 1.5444 6.6646 0.8947 20 18 0.2581 1 NA NA 57 # 2022-10-22 10:45:59 BST: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants 78 acs 3 1.5444 6.6646 0.8947 20 18 0.2581 1 NA NA 62 mmas 3 1.2260 2.0146 0.7659 8 16 NA 0 NA NA 85 acs 3 3.0740 4.9115 0.0734 18 47 0.4206 1 NA NA 71 acs 3 1.7574 6.9647 0.6223 13 19 0.4806 0 NA NA 72 acs 3 2.0489 7.1428 0.5326 16 6 0.2179 0 NA NA # 2022-10-22 10:45:59 BST: Iteration 4 of 5 # experimentsUsedSoFar: 480 # remainingBudget: 520 # currentBudget: 260 # nbConfigurations: 33 # Markers: x No test is performed. c Configurations are discarded only due to capping. - The test is performed and some configurations are discarded. = The test is performed but no configuration is discarded. ! The test is performed and configurations could be discarded but elite configurations are preserved. . All alive configurations are elite and nothing is discarded +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ |x| 9| 33| 71| 23073306.00| 33|00:02:51| NA| NA| NA| |x| 7| 33| 78| 23137315.50| 61|00:02:21|+0.74|0.87|0.0148| |x| 6| 33| 78| 23196511.00| 89|00:02:21|+0.75|0.83|0.0236| |x| 1| 33| 78| 23254301.25| 117|00:02:22|+0.76|0.82|0.0233| |-| 3| 14| 113| 23225975.00| 145|00:02:21|-0.02|0.18|0.9419| |=| 8| 14| 113| 23242633.33| 154|00:00:50|+0.01|0.18|0.9115| |=| 4| 14| 113| 23210932.71| 163|00:00:50|-0.03|0.12|0.9145| |=| 2| 14| 113| 23212340.75| 172|00:00:50|+0.03|0.15|0.8593| |=| 5| 14| 113| 23174747.00| 181|00:00:50|+0.04|0.15|0.8558| |=| 10| 14| 78| 23180941.50| 195|00:01:10|+0.04|0.13|0.8711| +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ Best-so-far configuration: 78 mean value: 23180941.50 Description of the best-so-far configuration: .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. 78 78 acs 3 1.5444 6.6646 0.8947 20 18 0.2581 1 NA NA 57 # 2022-10-22 11:02:50 BST: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants 78 acs 3 1.5444 6.6646 0.8947 20 18 0.2581 1 NA NA 113 acs 3 1.9861 6.6955 0.7803 26 15 0.0512 1 NA NA 110 mmas 3 1.1100 2.0382 0.6833 15 11 NA 0 NA NA 85 acs 3 3.0740 4.9115 0.0734 18 47 0.4206 1 NA NA 91 acs 3 1.5186 5.7143 0.5523 14 26 0.8361 1 NA NA # 2022-10-22 11:02:50 BST: Iteration 5 of 5 # experimentsUsedSoFar: 675 # remainingBudget: 325 # currentBudget: 325 # nbConfigurations: 34 # Markers: x No test is performed. c Configurations are discarded only due to capping. - The test is performed and some configurations are discarded. = The test is performed but no configuration is discarded. ! The test is performed and configurations could be discarded but elite configurations are preserved. . All alive configurations are elite and nothing is discarded +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ |x| 11| 34| 138| 23258118.00| 34|00:02:51| NA| NA| NA| |x| 4| 34| 138| 23112149.00| 63|00:02:31|+0.42|0.71|0.5375| |x| 7| 34| 127| 23142940.33| 92|00:02:31|+0.55|0.70|0.3724| |x| 6| 34| 127| 23173748.75| 121|00:02:31|+0.51|0.63|0.3385| |-| 8| 11| 127| 23202123.40| 150|00:02:31|+0.17|0.34|0.7210| |=| 3| 11| 127| 23177660.67| 156|00:00:30|+0.12|0.26|0.7576| |=| 1| 11| 127| 23216506.14| 162|00:00:30|+0.02|0.16|0.8691| |=| 10| 11| 127| 23212485.75| 168|00:00:30|-0.01|0.11|0.8993| |=| 5| 11| 127| 23171591.78| 174|00:00:30|-0.02|0.10|0.9045| |=| 9| 11| 127| 23162579.40| 180|00:00:30|+0.02|0.11|0.8709| |=| 2| 11| 127| 23171928.36| 186|00:00:30|-0.00|0.09|0.8872| |=| 12| 11| 138| 23170655.33| 197|00:01:00|+0.02|0.10|0.8676| +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ Best-so-far configuration: 138 mean value: 23170655.33 Description of the best-so-far configuration: .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. 138 138 acs 3 1.4294 6.0012 0.9344 16 21 0.2217 1 NA NA 78 # 2022-10-22 11:19:51 BST: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants 138 acs 3 1.4294 6.0012 0.9344 16 21 0.2217 1 NA NA 127 acs 3 1.6457 5.9689 0.7436 13 17 0.0080 1 NA NA 120 acs 3 1.3726 6.8320 0.8051 24 16 0.2054 1 NA NA 124 acs 3 1.6169 7.9049 0.7166 20 8 0.2282 1 NA NA 113 acs 3 1.9861 6.6955 0.7803 26 15 0.0512 1 NA NA # 2022-10-22 11:19:51 BST: Iteration 6 of 6 # experimentsUsedSoFar: 872 # remainingBudget: 128 # currentBudget: 128 # nbConfigurations: 14 # Markers: x No test is performed. c Configurations are discarded only due to capping. - The test is performed and some configurations are discarded. = The test is performed but no configuration is discarded. ! The test is performed and configurations could be discarded but elite configurations are preserved. . All alive configurations are elite and nothing is discarded +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ |x| 13| 14| 149| 23168071.00| 14|00:01:10| NA| NA| NA| |x| 9| 14| 138| 23147346.50| 23|00:00:50|+0.28|0.64|0.2294| |x| 2| 14| 138| 23168709.00| 32|00:00:50|+0.33|0.56|0.1781| |x| 1| 14| 138| 23222390.50| 41|00:00:50|+0.35|0.51|0.1570| |-| 11| 7| 138| 23229536.00| 50|00:00:50|+0.20|0.36|0.8842| |=| 5| 7| 138| 23175555.00| 52|00:00:10|+0.08|0.23|0.8989| |=| 8| 7| 138| 23200975.71| 54|00:00:10|+0.02|0.16|0.8807| |=| 4| 7| 138| 23171626.25| 56|00:00:10|+0.10|0.21|0.8201| |=| 3| 7| 120| 23156247.00| 58|00:00:10|+0.08|0.18|0.8132| |=| 10| 7| 138| 23163756.60| 60|00:00:10|+0.04|0.14|0.8714| |=| 7| 7| 138| 23169372.45| 62|00:00:10|+0.03|0.11|0.8612| |=| 12| 7| 138| 23163537.83| 64|00:00:10|+0.04|0.12|0.8393| |=| 6| 7| 138| 23174511.15| 66|00:00:10|-0.01|0.07|0.8624| |=| 14| 7| 138| 23178603.86| 73|00:00:40|-0.02|0.05|0.8731| +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ Best-so-far configuration: 138 mean value: 23178603.86 Description of the best-so-far configuration: .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. 138 138 acs 3 1.4294 6.0012 0.9344 16 21 0.2217 1 NA NA 78 # 2022-10-22 11:26:25 BST: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants 138 acs 3 1.4294 6.0012 0.9344 16 21 0.2217 1 NA NA 149 acs 3 1.8645 8.8636 0.8623 31 18 0.0324 1 NA NA 127 acs 3 1.6457 5.9689 0.7436 13 17 0.0080 1 NA NA 120 acs 3 1.3726 6.8320 0.8051 24 16 0.2054 1 NA NA 147 acs 3 1.3012 6.1776 0.7881 22 10 0.1276 1 NA NA # 2022-10-22 11:26:25 BST: Iteration 7 of 7 # experimentsUsedSoFar: 945 # remainingBudget: 55 # currentBudget: 55 # nbConfigurations: 8 # Markers: x No test is performed. c Configurations are discarded only due to capping. - The test is performed and some configurations are discarded. = The test is performed but no configuration is discarded. ! The test is performed and configurations could be discarded but elite configurations are preserved. . All alive configurations are elite and nothing is discarded +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ |x| 15| 8| 120| 23314657.00| 8|00:00:40| NA| NA| NA| |x| 12| 8| 127| 23220366.50| 11|00:00:20|+0.24|0.62|0.3659| |x| 5| 8| 127| 23095057.67| 14|00:00:20|+0.40|0.60|0.2741| |x| 2| 8| 155| 23128548.75| 17|00:00:20|+0.40|0.55|0.2772| |-| 1| 6| 155| 23176037.40| 20|00:00:20|+0.15|0.32|0.7974| |=| 9| 6| 155| 23163208.83| 21|00:00:10|+0.02|0.18|0.8719| |=| 13| 6| 155| 23167865.00| 22|00:00:10|+0.00|0.14|0.8751| |=| 4| 6| 155| 23146280.75| 23|00:00:10|+0.02|0.15|0.8457| |=| 3| 6| 155| 23137118.56| 24|00:00:10|+0.02|0.13|0.8488| |=| 14| 6| 155| 23145861.20| 25|00:00:10|-0.01|0.09|0.8678| |=| 7| 6| 155| 23150676.36| 26|00:00:10|-0.01|0.08|0.8466| |=| 10| 6| 155| 23149829.50| 27|00:00:10|+0.00|0.09|0.8466| |=| 11| 6| 155| 23158077.23| 28|00:00:10|+0.04|0.11|0.8411| |=| 6| 6| 155| 23166610.93| 29|00:00:10|+0.04|0.11|0.8286| |=| 8| 6| 155| 23180093.40| 30|00:00:10|+0.03|0.09|0.8325| |=| 16| 6| 155| 23192615.19| 36|00:00:30|+0.05|0.11|0.8122| +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ Best-so-far configuration: 155 mean value: 23192615.19 Description of the best-so-far configuration: .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. 155 155 acs 3 1.7343 8.2833 0.7587 31 19 0.0859 1 NA NA 120 # 2022-10-22 11:30:38 BST: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants 155 acs 3 1.7343 8.2833 0.7587 31 19 0.0859 1 NA NA 127 acs 3 1.6457 5.9689 0.7436 13 17 0.0080 1 NA NA 138 acs 3 1.4294 6.0012 0.9344 16 21 0.2217 1 NA NA 120 acs 3 1.3726 6.8320 0.8051 24 16 0.2054 1 NA NA 149 acs 3 1.8645 8.8636 0.8623 31 18 0.0324 1 NA NA # 2022-10-22 11:30:38 BST: Stopped because there is not enough budget left to race more than the minimum (5) # You may either increase the budget or set 'minNbSurvival' to a lower value # Iteration: 8 # nbIterations: 8 # experimentsUsedSoFar: 981 # timeUsed: 0 # remainingBudget: 19 # currentBudget: 19 # number of elites: 5 # nbConfigurations: 5 # Total CPU user time: 9871.301, CPU sys time: 46.151, Wall-clock time: 5165.205 # Best configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants 155 acs 3 1.7343 8.2833 0.7587 31 19 0.0859 1 NA NA 127 acs 3 1.6457 5.9689 0.7436 13 17 0.0080 1 NA NA 138 acs 3 1.4294 6.0012 0.9344 16 21 0.2217 1 NA NA 120 acs 3 1.3726 6.8320 0.8051 24 16 0.2054 1 NA NA 149 acs 3 1.8645 8.8636 0.8623 31 18 0.0324 1 NA NA # Best configurations as commandlines (first number is the configuration ID; same order as above): 155 --acs --localsearch 3 --alpha 1.7343 --beta 8.2833 --rho 0.7587 --ants 31 --nnls 19 --q0 0.0859 --dlb 1 127 --acs --localsearch 3 --alpha 1.6457 --beta 5.9689 --rho 0.7436 --ants 13 --nnls 17 --q0 0.008 --dlb 1 138 --acs --localsearch 3 --alpha 1.4294 --beta 6.0012 --rho 0.9344 --ants 16 --nnls 21 --q0 0.2217 --dlb 1 120 --acs --localsearch 3 --alpha 1.3726 --beta 6.832 --rho 0.8051 --ants 24 --nnls 16 --q0 0.2054 --dlb 1 149 --acs --localsearch 3 --alpha 1.8645 --beta 8.8636 --rho 0.8623 --ants 31 --nnls 18 --q0 0.0324 --dlb 1 # Testing of elite configurations: 5 # Testing iteration configurations: TRUE # 2022-10-22 11:30:38 BST: Testing configurations (in no particular order): 14 11 31 6 57 55 39 78 62 85 71 72 113 110 91 138 127 120 124 149 147 155 algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants 14 ras 3 3.7374 4.7749 0.1279 5 17 NA 0 37 NA 11 acs 3 2.6159 1.0514 0.5986 12 21 0.2211 0 NA NA 31 acs 1 2.3896 7.8790 0.3748 34 36 0.9029 1 NA NA 6 acs 3 2.2422 7.8826 0.1091 14 40 0.0174 0 NA NA 57 acs 3 1.0519 6.5794 0.6649 11 33 0.2283 1 NA NA 55 mmas 3 1.0832 5.5967 0.9317 18 46 NA 0 NA NA 39 acs 2 2.5486 3.7878 0.0262 13 12 0.7621 0 NA NA 78 acs 3 1.5444 6.6646 0.8947 20 18 0.2581 1 NA NA 62 mmas 3 1.2260 2.0146 0.7659 8 16 NA 0 NA NA 85 acs 3 3.0740 4.9115 0.0734 18 47 0.4206 1 NA NA 71 acs 3 1.7574 6.9647 0.6223 13 19 0.4806 0 NA NA 72 acs 3 2.0489 7.1428 0.5326 16 6 0.2179 0 NA NA 113 acs 3 1.9861 6.6955 0.7803 26 15 0.0512 1 NA NA 110 mmas 3 1.1100 2.0382 0.6833 15 11 NA 0 NA NA 91 acs 3 1.5186 5.7143 0.5523 14 26 0.8361 1 NA NA 138 acs 3 1.4294 6.0012 0.9344 16 21 0.2217 1 NA NA 127 acs 3 1.6457 5.9689 0.7436 13 17 0.0080 1 NA NA 120 acs 3 1.3726 6.8320 0.8051 24 16 0.2054 1 NA NA 124 acs 3 1.6169 7.9049 0.7166 20 8 0.2282 1 NA NA 149 acs 3 1.8645 8.8636 0.8623 31 18 0.0324 1 NA NA 147 acs 3 1.3012 6.1776 0.7881 22 10 0.1276 1 NA NA 155 acs 3 1.7343 8.2833 0.7587 31 19 0.0859 1 NA NA # 2022-10-22 11:49:09 BST: Testing results (column number is configuration ID in no particular order): 14 11 31 6 57 55 39 78 62 85 71 72 113 110 91 138 127 120 124 149 147 155 1t 23391860 23332695 23537760 23414477 23364455 23485970 23376575 23368630 23370336 23416359 23412859 23402853 23413652 23358323 23443305 23390347 23324399 23359717 23360665 23356925 23301025 23356774 2t 23229539 23167464 23337793 23272324 23147927 23195166 23258960 23114234 23263064 23232588 23183375 23155695 23140836 23206760 23136299 23126209 23117499 23142083 23141203 23118103 23131024 23175108 3t 23184984 23094701 23252927 23171748 22982248 23026437 23038220 23083312 23053955 23082028 23023432 23067583 23011134 23000956 23053830 23047801 22994436 23001269 23028544 22977398 23032405 23032211 4t 23116701 23126420 23232767 23157322 23044199 23063903 23136997 23053899 23115735 23102111 23108032 23010549 23051643 23022518 23074969 23031784 23023703 23031107 23042387 23081150 23015695 23027772 5t 23237519 23373072 23380589 23320207 23217884 23233250 23316907 23211425 23173925 23170016 23276775 23209411 23189520 23219876 23261978 23196505 23197529 23174024 23222521 23185052 23184322 23163568 6t 23481721 23475588 23508760 23479494 23467320 23495025 23549932 23409175 23483799 23423813 23479579 23446369 23410700 23415018 23491880 23444264 23436888 23416332 23419003 23463089 23427505 23394824 7t 23369278 23676371 23494491 23392883 23401441 23380877 23319057 23343664 23389619 23383997 23405404 23350024 23312217 23354092 23396796 23343027 23343136 23331633 23354417 23361221 23324993 23375510 8t 23311915 23288024 23387646 23337527 23249894 23333692 23377228 23215822 23324171 23316955 23267699 23285411 23242464 23268552 23284711 23208403 23233397 23247441 23247049 23209032 23217163 23267663 9t 23358261 23341818 23449508 23430565 23374655 23293488 23476762 23328162 23428572 23284190 23280730 23371465 23318878 23303311 23287065 23288033 23334631 23296373 23256521 23308818 23264005 23299111 10t 23095703 23296750 23286985 23147297 23078823 23051369 23124520 23112076 23066743 23077135 23189020 23065758 23087598 23053962 23028421 23066774 23087940 23041424 23132030 23062589 23014124 23038023 # 2022-10-22 11:49:09 BST: Finished testing irace/R/0000755000176200001440000000000014325242462011542 5ustar liggesusersirace/R/parameterAnalysis.R0000644000176200001440000001023514244705607015357 0ustar liggesusers#' Return the elite configurations of the final iteration. #' #' @template arg_iraceresults #' @param n Number of elite configurations to return, if \code{n} is larger than the #' number of configurations, then only the existing ones are returned. The default (\code{n=0}) returns all of them. #' @param drop.metadata Remove metadata, such the configuration ID and #' the ID of the parent, from the returned configurations. See #' \code{\link{removeConfigurationsMetaData}}. #' #' @return A data frame containing the elite configurations required. #' #' @examples #' log_file <- system.file("exdata/irace-acotsp.Rdata", package="irace", mustWork=TRUE) #' print(removeConfigurationsMetaData(getFinalElites(log_file, n=1))) #' #' @author Manuel López-Ibáñez and Leslie Pérez Cáceres #' @concept analysis #' @export getFinalElites <- function(iraceResults, n = 0L, drop.metadata = FALSE) { if (missing(iraceResults)) stop("argument 'iraceResults' is missing") iraceResults <- read_logfile(iraceResults) last.elites <- iraceResults$allElites[[length(iraceResults$allElites)]] if (n == 0) n <- length(last.elites) if (length(last.elites) < n) { cat("Only", length(last.elites), "configurations available, reducing n,") n <- length(last.elites) } last.elites <- last.elites[1:n] configurations <- subset(iraceResults$allConfigurations, get(".ID.") %in% as.character(last.elites), drop = FALSE) if (drop.metadata) configurations <- removeConfigurationsMetaData(configurations) return(configurations) } #' Returns the configurations selected by ID. #' #' @template arg_iraceresults #' @param ids The id or a vector of ids of the candidates configurations to obtain. #' @param drop.metadata Remove metadata, such the configuration ID and #' the ID of the parent, from the returned configurations. See #' [removeConfigurationsMetaData()]. #' #' @return A data frame containing the elite configurations required. #' #' @author Manuel López-Ibáñez and Leslie Pérez Cáceres #' @concept analysis #' @export getConfigurationById <- function(iraceResults, ids, drop.metadata = FALSE) { if (missing(iraceResults)) stop("argument 'iraceResults' is missing") iraceResults <- read_logfile(iraceResults) if (length(ids) < 1) stop("You must provide at least one configuration id.") selection <- iraceResults$allConfigurations[,".ID."] %in% ids if (length(selection) < 1) stop("No configuration found with id", ids,".") configurations <-iraceResults$allConfigurations[selection, , drop = FALSE] if (drop.metadata) configurations <- removeConfigurationsMetaData(configurations) return(configurations) } #' Returns the configurations by the iteration in which they were executed. #' #' @template arg_iraceresults #' @param iterations The iteration number or a vector of iteration numbers from where #' the configurations should be obtained. #' @param drop.metadata (\code{FALSE}) Remove metadata, such the configuration ID and #' the ID of the parent, from the returned configurations. See #' \code{\link{removeConfigurationsMetaData}}. #' #' @return A data frame containing the elite configurations required. #' #' @author Manuel López-Ibáñez and Leslie Pérez Cáceres #' @concept analysis #' @export getConfigurationByIteration <- function(iraceResults, iterations, drop.metadata = FALSE) { if (missing(iraceResults)) stop("argument 'iraceResults' is missing") iraceResults <- read_logfile(iraceResults) if (length(iterations) < 1) stop("You must provide at least one configuration id.") # To silence warning. iteration <- NULL ids <- unique(subset(as.data.frame(iraceResults$experimentLog), iteration %in% iterations, select=c("configuration"), drop=TRUE)) selection <- iraceResults$allConfigurations[,".ID."] %in% ids if (length(selection) < 1) stop("No configuration found with id", ids,".") configurations <- iraceResults$allConfigurations[selection, , drop=FALSE] if (drop.metadata) configurations <- removeConfigurationsMetaData(configurations) return(configurations) } irace/R/path_rel2abs.R0000644000176200001440000001024414321266642014236 0ustar liggesusers#' Converts a relative path to an absolute path. It tries really hard to create #' canonical paths. #' #' @param path (`character(1)`) Character string representing a relative path. #' @param cwd (`character(1)`) Current working directory. #' #' @return (`character(1)`) Character string representing the absolute path #' #' @examples #' path_rel2abs("..") #' @export path_rel2abs <- function (path, cwd = getwd()) { # Keep doing gsub as long as x keeps changing. gsub.all <- function(pattern, repl, x, ...) { repeat { newx <- gsub(pattern, repl, x, ...) if (newx == x) return(newx) x <- newx } } # FIXME: Why NA and not FALSE? irace_normalize_path <- function(path) suppressWarnings(normalizePath(path, winslash = "/", mustWork = NA)) if (is.null.or.na(path)) { return (NULL) } else if (path == "") { return ("") } # Using .Platform$file.sep is too fragile. Better just use "/" everywhere. s <- "/" # Possibly expand ~/path to /home/user/path. path <- path.expand(path) # Remove winslashes if given. path <- gsub("\\", s, path, fixed = TRUE) # Detect a Windows drive windrive.regex <- "^[A-Za-z]:" windrive <- "" if (grepl(paste0(windrive.regex, "($|", s, ")"), path)) { m <- regexpr(windrive.regex, path) windrive <- regmatches(path, m) path <- sub(windrive.regex, "", path) } # Change "/./" to "/" to get a canonical form path <- gsub.all(paste0(s, ".", s), s, path, fixed = TRUE) # Change "//" to "/" to get a canonical form path <- gsub(paste0(s, s, "+"), s, path) # Change "/.$" to "/" to get a canonical form path <- sub(paste0(s, "\\.$"), s, path) # Drop final "/" path <- sub(paste0(s, "$"), "", path) if (path == "") path <- s # Prefix the current cwd to the path if it doesn't start with # / \\ or whatever separator. if (path == "." || !startsWith(path, s)) { # There is no need to normalize cwd if it was returned by getwd() if (!missing(cwd)) { # Recurse to get absolute cwd cwd <- path_rel2abs(cwd) } # Speed-up the most common cases. # If it is just "." if (path == ".") return (irace_normalize_path(cwd)) # If it does not contain separators at all and does not start with ".." if (!startsWith(path, "..") && !grepl(s, path)) { # it may be a command in the path. sys_path <- suppressWarnings(Sys.which(path)) if (nchar(sys_path) > 0 && file.access(sys_path, mode = 1) == 0 && !file.info(sys_path)$isdir) { return(irace_normalize_path(as.vector(sys_path))) } } # Remove "./" from the start of path. path <- sub(paste0("^\\.", s), "", path) # Make it absolute but avoid doubling s if (substring(cwd, nchar(cwd)) == s) path <- paste0(cwd, path) else path <- paste0(cwd, s, path) # If it is just a path without ".." inside if (!grepl(paste0(s,"\\.\\."), path)) { return (irace_normalize_path(path)) } # Detect a Windows drive if (grepl(paste0(windrive.regex, "($|", s, ")"), path)) { m <- regexpr(windrive.regex, path) windrive <- regmatches(path, m) path <- sub(windrive.regex, "", path) } } # else # Change "/x/.." to "/" to get a canonical form prevdir.regex <- paste0(s, "[^", s,"]+", s, "\\.\\.") repeat { # We need to do it one by one so "a/b/c/../../../" is not converted to "a/b/../" tmp <- sub(paste0(prevdir.regex, s), s, path) if (tmp == path) break path <- tmp } # Handle "/something/..$" to "/" that is, when ".." is the last thing in the path. path <- sub(paste0(prevdir.regex, "$"), s, path) # Handle "^/../../.." to "/" that is, going up at the root just returns the root. repeat { # We need to do it one by one so "a/b/c/../../../" is not converted to "a/b/../" tmp <- sub(paste0("^", s, "\\.\\.", s), s, path) if (tmp == path) break path <- tmp } # Handle "^/..$" to "/" that is, when ".." is the last thing in the path. path <- sub(paste0("^", s, "\\.\\.$"), s, path) # Add back Windows drive, if any. path <- paste0(windrive, path) # We use normalizePath, which will further simplify the path if # the path exists. irace_normalize_path(path) } irace/R/readConfiguration.R0000644000176200001440000013705214321271071015332 0ustar liggesusers#' Read parameter configurations from a file #' #' Reads a set of target-algorithm configurations from a file and puts them in #' \pkg{irace} format. The configurations are checked to match the parameters #' description provided. #' #' @param filename (`character(1)`) \cr Filename from which the configurations should be read. The contents should be readable by `read.table( , header=TRUE)`. #' @template arg_parameters #' @template arg_debuglevel #' @template arg_text #' #' @return A data frame containing the obtained configurations. #' Each row of the data frame is a candidate configuration, #' the columns correspond to the parameter names in `parameters`. #' #' @details #' Example of an input file: #' ``` #' # This is a comment line #' ... #' 0.5 "value_1" ... #' 1.0 "value_2" ... #' 1.2 "value_3" ... #' ... ... #' ``` #' The order of the columns does not necessarily have to be the same #' as in the file containing the definition of the parameters. #' #' @seealso #' [readParameters()] to obtain a valid parameter structure from a parameters file. #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export ## FIXME: What about digits? readConfigurationsFile <- function(filename, parameters, debugLevel = 0, text) { if (missing(filename) && !missing(text)) { filename <- strcat("text=", deparse(substitute(text))) configurationTable <- read.table(text = text, header = TRUE, colClasses = "character", stringsAsFactors = FALSE) } else { # Read the file. configurationTable <- read.table(filename, header = TRUE, colClasses = "character", stringsAsFactors = FALSE) } irace.assert(is.data.frame(configurationTable)) nbConfigurations <- nrow(configurationTable) # Print the table that has been read. cat("# Read ", nbConfigurations, " configuration(s) from file '", filename, "'\n", sep="") if (debugLevel >= 2) { print(as.data.frame(configurationTable, stringsAsFactor = FALSE), digits=15) } namesParameters <- names(parameters$conditions) # This ignores fixed parameters unless they are given with a different value. if (ncol(configurationTable) != length(namesParameters) || !setequal (colnames(configurationTable), namesParameters)) { # Column names must match a parameter, including fixed ones. missing <- setdiff (colnames(configurationTable), namesParameters) if (length(missing) > 0) { irace.error("The parameter names (", strlimit(paste(missing, collapse=", ")), ") given in the first row of file ", filename, " do not match the parameter names: ", paste(namesParameters, collapse=", ")) return(NULL) } # All non-fixed parameters must appear in column names. varParameters <- parameters$names[!parameters$isFixed] missing <- setdiff (varParameters, colnames(configurationTable)) if (length(missing) > 0) { irace.error("The parameter names (", strlimit(paste(missing, collapse=", ")), ") are missing from the first row of file ", filename) return(NULL) } # Add any missing fixed parameters. missing <- setdiff (namesParameters, colnames(configurationTable)) if (length(missing) > 0) { irace.assert (all(parameters$isFixed[missing])) tmp <- lapply(missing, function(x) get.fixed.value(x, parameters)) names(tmp) <- missing configurationTable <- cbind.data.frame(configurationTable, tmp, stringsAsFactors = FALSE) } } # Reorder columns. configurationTable <- configurationTable[, namesParameters, drop = FALSE] # Fix up numeric columns. for (currentParameter in namesParameters) { type <- parameters$types[[currentParameter]] if (type == "i" || type == "r") { configurationTable[, currentParameter] <- suppressWarnings(as.numeric(configurationTable[, currentParameter])) } } # Loop over all configurations in configurationTable for (k in seq_len(nbConfigurations)) { # Loop over all parameters, in hierarchical order. for (currentParameter in namesParameters) { currentValue <- configurationTable[k, currentParameter] type <- parameters$types[[currentParameter]] # Check the status of the conditions for this parameter to know # whether it must be enabled. if (conditionsSatisfied(parameters, configurationTable[k, ], currentParameter)) { # Check that the value is among the valid ones. if (type == "i" || type == "r") { currentValue <- as.numeric(currentValue) lower <- paramLowerBound(currentParameter, parameters) upper <- paramUpperBound(currentParameter, parameters) if (is.na(currentValue) || currentValue < lower || currentValue > upper) { irace.error ("Configuration number ", k, " from file ", filename, " is invalid because the value \"", configurationTable[k, currentParameter], "\" for the parameter ", currentParameter, " is not within the valid range [", lower,", ", upper,"]") return(NULL) } # For integers, only accept an integer. if (type == "i" && as.integer(currentValue) != currentValue) { irace.error ("Configuration number ", k, " from file ", filename, " is invalid because parameter ", currentParameter, " is of type integer but its value ", currentValue, " is not an integer") return(NULL) } # type == "o" or "c" } else if (!(currentValue %in% paramDomain(currentParameter, parameters))) { irace.error ("Configuration number ", k, " from file ", filename, " is invalid because the value \"", currentValue, "\" for the parameter \"", currentParameter, "\" is not among the valid values: (\"", paste(paramDomain(currentParameter, parameters), collapse="\", \""), "\")") return(NULL) } } else if (!is.na(currentValue)) { irace.error ("Configuration number ", k, " from file ", filename, " is invalid because parameter \"", currentParameter, "\" is not enabled because of condition \"", parameters$conditions[[currentParameter]], "\" but its value is \"", currentValue, "\" instead of NA") return(NULL) } } } if (anyDuplicated(configurationTable)) { irace.error("Duplicated configurations in file ", filename, " :\n", paste0(capture.output( configurationTable[duplicated(configurationTable), , drop=FALSE]), "\n")) } configurationTable } # FIXME: It may be faster to create a single expression that concatenates all # the elements of forbidden using '|' checkForbidden <- function(configurations, forbidden) { # We have to use a variable name that will never appear in # configurations, so .FORBIDDEN . for (.FORBIDDEN in forbidden) { #print(.FORBIDDEN) configurations <- subset(configurations, eval(.FORBIDDEN)) #print(configurations) #print(str(configurations)) ## FIXME: This is normally called with a single configuration. Thus, it ## would be faster to break as soon as nrow(configurations) < 1 } #print(nrow(configurations)) configurations } compile.forbidden <- function(x) { if (is.bytecode(x)) return(x) # If we are given an expression, it must be a single one. irace.assert(is.language(x) && (!is.expression(x) || length(x) == 1)) if (is.expression(x)) x <- x[[1]] # When a is NA and we check a == 5, we would get NA, which is # always FALSE, when we actually want to be TRUE, so we test # is.na() first below. # We expect that there will be undefined variables, since the expressions # will be evaluated within a data.frame later. expr <- compiler::compile(substitute(is.na(x) | !(x), list(x = x)), options = list(suppressUndefined=TRUE)) attr(expr, "source") <- as.character(as.expression(x)) expr } readForbiddenFile <- function(filename) { forbiddenExps <- parse(file = filename) # FIXME: Using && or || instead of & and | will not work. Detect # this and give an error to the user. # FIXME: Check that the parameter names that appear in forbidden # all appear in parameters$names to catch typos. # FIXME: Instead of a list, we should generate a single expression that is # the logical-OR of all elements of the list. # First we would need to handle the "is.na(x) | !(x)" case here. # Maybe: sapply(forbiddenExps, function(x) substitute(is.na(x) | !(x), list(x=x))) # x <- parse(text=paste0("(", paste0(forbiddenExps,collapse=")||("), ")")) # Byte-compile them. sapply(forbiddenExps, compile.forbidden) } buildForbiddenExp <- function(configurations, parameters) { if (nrow(configurations) < 1) return(NULL) pnames <- parameters$names lines <- c() # We cannot use apply() because it converts numeric to character. for (k in 1:nrow(configurations)) { values <- as.list(configurations[k, pnames]) has.value <- !is.na(values) values <- lapply(values[has.value], function(x) deparse(substitute(x, list(x=x)))) lines <- c(lines, paste0("(", pnames[has.value]," == ", values, ")", collapse = "&")) } exps <- parse(text = lines) # print(exps) sapply(exps, compile.forbidden) } #' Reads from a file the scenario settings to be used by \pkg{irace}. #' #' @param filename (`character(1)`) \cr Filename from which the scenario will #' be read. If empty, the default `scenarioFile` is used. An example #' scenario file is provided in `system.file(``package="irace",` #' `"templates/scenario.txt.tmpl")`. #' @templateVar arg_appendix This is an initial scenario that is overwritten #' for every setting specified in the file to be read. #' @template arg_scenario #' @template arg_params_def #' #' @return The scenario list read from the file. The scenario settings not #' present in the file are not present in the list, i.e., they are `NULL`. #' #' @seealso #' \describe{ #' \item{[printScenario()]}{prints the given scenario.} #' \item{[defaultScenario()]}{returns the default scenario settings of \pkg{irace}.} #' \item{[checkScenario()]}{to check that the scenario is valid.} #' } #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export readScenario <- function(filename = "", scenario = list(), params_def = .irace.params.def) { # This function allows recursively including scenario files. scenario_env <- new.env() include.scenario <- function(rfilename, topfile = filename, envir. = scenario_env) { if (!file.exists (rfilename)) { irace.error ("The scenario file ", shQuote(rfilename), " included from ", shQuote(topfile), " does not exist.") } handle.source.error <- function(e) { irace.error("Reading scenario file ", shQuote(rfilename), " included from ", shQuote(topfile), " produced the following errors or warnings:\n", paste0(conditionMessage(e), collapse="\n")) return(NULL) } withCallingHandlers( tryCatch(source(rfilename, local = envir., chdir = TRUE), error = handle.source.error, warning = handle.source.error)) } # First find out which file... if (filename == "") { filename <- path_rel2abs(params_def["scenarioFile","default"]) if (file.exists(filename)) { irace.warning("A default scenario file ", shQuote(filename), " has been found and will be read.") } else { irace.error ("Not scenario file given (use ", params_def["scenarioFile", "short"], " or ", params_def["scenarioFile", "long"], ") and no default scenario file ", shQuote(filename), " has been found.") } } else { filename <- path_rel2abs(filename) } if (file.exists (filename)) { debug.level <- getOption(".irace.debug.level", default = 0) if (debug.level >= 1) cat("# Reading scenario file", shQuote(filename), ".......") # chdir = TRUE to allow recursive sourcing. handle.source.error <- function(e) { irace.error("Reading scenario file ", shQuote(filename), " produced the following errors or warnings:\n", paste0(conditionMessage(e), collapse="\n")) return(NULL) } withCallingHandlers( tryCatch(source(filename, local = scenario_env, chdir = TRUE), error = handle.source.error, warning = handle.source.error)) if (debug.level >= 1) cat (" done!\n") } else { irace.error ("The scenario file ", shQuote(filename), " does not exist.") } ## Read scenario file variables. scenario[["scenarioFile"]] <- filename # If these are given and relative, they should be relative to the # scenario file (except logFile, which is relative to execDir). pathParams <- setdiff(params_def[params_def[, "type"] == "p", "name"], "logFile") params_names <- params_def[!startsWith(params_def[,"name"], "."), "name"] for (param in params_names) { if (exists (param, envir = scenario_env, inherits = FALSE)) { value <- get(param, envir = scenario_env, inherits = FALSE) if (!is.null.or.empty(value) && is.character(value) && (param %in% pathParams)) { value <- path_rel2abs(value, cwd = dirname(filename)) } scenario[[param]] <- value } } unknown_scenario_vars <- setdiff(ls(scenario_env), params_names) if (length(unknown_scenario_vars) > 0) { # We only accept variables that match irace.params.names and if the user # wants to define their own, they should use names starting with ".", which # are ignored by ls() irace.error("Scenario file ", shQuote(filename), " contains unknown variables: ", paste0(unknown_scenario_vars, collapse=", "), "\nMAKE SURE NO VARIABLE NAME IS MISSPELL (for example, 'parameterFile' is correct, while 'parametersFile' is not)", "\nIf you wish to use your own variables in the scenario file, use names beginning with a dot `.'") } scenario } setup_test_instances <- function(scenario) { if (is.null.or.empty(scenario[["testInstances"]])) { if (!is.null.or.empty(scenario$testInstancesDir) || !is.null.or.empty(scenario$testInstancesFile)) { scenario$testInstancesDir <- path_rel2abs(scenario$testInstancesDir) if (!is.null.or.empty(scenario$testInstancesFile)) { scenario$testInstancesFile <- path_rel2abs(scenario$testInstancesFile) } scenario[["testInstances"]] <- readInstances(instancesDir = scenario$testInstancesDir, instancesFile = scenario$testInstancesFile) } else { scenario[["testInstances"]] <- NULL } } if (!is.null(scenario[["testInstances"]]) && is.null(names(scenario[["testInstances"]]))) { # Create unique IDs for testInstances names(scenario[["testInstances"]]) <- paste0(1:length(scenario[["testInstances"]]), "t") } scenario } #' Check and correct the given scenario #' #' Checks for errors a (possibly incomplete) scenario setup of #' \pkg{irace} and transforms it into a valid scenario. #' #' @template arg_scenario #' #' @return The scenario received as a parameter, possibly corrected. Unset #' scenario settings are set to their default values. #' #' @details This function checks that the directories and the file names #' provided and required by the \pkg{irace} exist. It also checks that the #' settings are of the proper type, e.g. that settings expected to be integers #' are really integers. Finally, it also checks that there is no inconsistency #' between settings. If an error is found that prevents \pkg{irace} from #' running properly, it will stop with an error. #' #' @seealso #' \describe{ #' \item{[readScenario()]}{for reading a configuration scenario from a file.} #' \item{[printScenario()]}{prints the given scenario.} #' \item{[defaultScenario()]}{returns the default scenario settings of \pkg{irace}.} #' \item{[checkScenario()]}{to check that the scenario is valid.} #' } #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export ## FIXME: This function should only do checks and return TRUE/FALSE. There ## should be other function that does the various transformations. checkScenario <- function(scenario = defaultScenario()) { quote.param <- function(name) { if (.irace.params.def[name, "long"] != "") { return(paste0("'", name, "' (", .irace.params.def[name, "long"], ")")) } paste0("'", name, "'") } as.boolean.param <- function(x, name) { x <- as.integer(x) if (is.na (x) || (x != 0 && x != 1)) { irace.error (quote.param(name), " must be either 0 or 1.") } as.logical(x) } check.valid.param <- function(x) { cat(x, "\n") cat(.irace.params.def[x, "domain"], "\n") valid <- trimws(strsplit(.irace.params.def[x, "domain"],",",fixed=TRUE)[[1]]) if (scenario[[x]] %!in% valid) { irace.error ("Invalid value '", scenario[[x]], "' of ", quote.param(x), ", valid values are: ", paste0(valid, collapse = ", ")) } } # Fill possible unset (NULL) with default settings. scenario <- defaultScenario(scenario) # Duplicated entries will cause confusion. dups <- anyDuplicated(names(scenario)) if (dups > 0) irace.error("scenario contains duplicated entries: ", names(scenario)[dups]) # Boolean control parameters. boolParams <- .irace.params.def[.irace.params.def[, "type"] == "b", "name"] for (p in boolParams) { scenario[[p]] <- as.boolean.param (scenario[[p]], p) } options(.irace.quiet = scenario$quiet) ## Check that everything is fine with external parameters # Check that the files exist and are readable. scenario$parameterFile <- path_rel2abs(scenario$parameterFile) # We don't read parameterFile here because the user may give the parameters # explicitly. And it is validated in readParameters anyway. scenario$execDir <- path_rel2abs(scenario$execDir) file.check (scenario$execDir, isdir = TRUE, text = paste0("execution directory ", quote.param("execDir"))) options(.irace.execdir = scenario$execDir) if (!is.null.or.empty(scenario$logFile)) { scenario$logFile <- path_rel2abs(scenario$logFile, cwd = scenario$execDir) file.check(scenario$logFile, writeable = TRUE, text = quote.param('logFile')) } else { # We cannot use NULL because defaultScenario() would override it. scenario$logFile <- "" } if (!is.null.or.empty(scenario$recoveryFile)) { scenario$recoveryFile <- path_rel2abs(scenario$recoveryFile) file.check(scenario$recoveryFile, readable = TRUE, text = paste0("recovery file ", quote.param("recoveryFile"))) if (!is.null.or.empty(scenario$logFile) # Must have been set "" above. && scenario$recoveryFile == scenario$logFile) { irace.error("log file and recovery file should be different '", scenario$logFile, "'") } } else { # We cannot use NULL because defaultScenario() would override it. scenario$recoveryFile <- "" } if (is.null.or.empty(scenario$targetRunnerParallel)) { scenario$targetRunnerParallel <- NULL } else if (is.function.name(scenario$targetRunnerParallel)) { scenario$targetRunnerParallel <- get.function(scenario$targetRunnerParallel) } else { irace.error("'targetRunnerParallel' must be a function") } if (is.null.or.empty(scenario$repairConfiguration)) { scenario$repairConfiguration <- NULL } else if (!is.function.name(scenario$repairConfiguration)) { irace.error("'repairConfiguration' must be a function") } else { # Byte-compile it. scenario$repairConfiguration <- bytecompile(get.function(scenario$repairConfiguration)) } if (is.function.name(scenario$targetRunner)) { scenario$targetRunner <- get.function(scenario$targetRunner) .irace$target.runner <- bytecompile(scenario$targetRunner) } else if (is.null(scenario$targetRunnerParallel)) { if (is.character(scenario$targetRunner)) { scenario$targetRunner <- path_rel2abs(scenario$targetRunner) .irace$target.runner <- if (scenario$aclib) target.runner.aclib else target.runner.default if (is.null.or.empty(scenario$targetRunnerLauncher)) { file.check (scenario$targetRunner, executable = TRUE, text = paste0("target runner ", quote.param("targetRunner"))) } else { scenario$targetRunnerLauncher <- path_rel2abs(scenario$targetRunnerLauncher) file.check (scenario$targetRunnerLauncher, executable = TRUE, text = paste0("target runner launcher ", quote.param("targetRunnerLauncher"))) check_launcher_args(scenario$targetRunnerLauncherArgs) } } else { irace.error(quote.param ('targetRunner'), " must be a function or an executable program") } } if (is.null.or.empty(scenario$targetEvaluator)) { scenario$targetEvaluator <- NULL .irace$target.evaluator <- NULL } else if (is.function.name(scenario$targetEvaluator)) { scenario$targetEvaluator <- get.function(scenario$targetEvaluator) .irace$target.evaluator <- bytecompile(scenario$targetEvaluator) } else if (is.character(scenario$targetEvaluator)) { scenario$targetEvaluator <- path_rel2abs(scenario$targetEvaluator) file.check (scenario$targetEvaluator, executable = TRUE, text = "target evaluator") .irace$target.evaluator <- target.evaluator.default } else { irace.error(quote.param('targetEvaluator'), " must be a function or an executable program") } irace.assert(is.null(scenario$targetEvaluator) == is.null(.irace$target.evaluator)) # Training instances if (is.null.or.empty(scenario$instances)) { scenario$trainInstancesDir <- path_rel2abs(scenario$trainInstancesDir) if (!is.null.or.empty(scenario$trainInstancesFile)) { scenario$trainInstancesFile <- path_rel2abs(scenario$trainInstancesFile) } if (is.null.or.empty(scenario$trainInstancesDir) && is.null.or.empty(scenario$trainInstancesFile)) irace.error("Both ", quote.param ("trainInstancesDir"), " and ", quote.param ("trainInstancesFile"), " are empty: No instances provided") scenario$instances <- readInstances(instancesDir = scenario$trainInstancesDir, instancesFile = scenario$trainInstancesFile) } # Testing instances scenario <- setup_test_instances(scenario) # Configurations file if (!is.null.or.empty(scenario$configurationsFile)) { scenario$configurationsFile <- path_rel2abs(scenario$configurationsFile) file.check (scenario$configurationsFile, readable = TRUE, text = "configurations file") # We cannot read the configurations here because we need the parameters. # FIXME: We should have the parameters inside scenario. } if (is.null.or.empty(scenario$initConfigurations)) { scenario$initConfigurations <- NULL } else if (!is.data.frame(scenario$initConfigurations) && !is.matrix(scenario$initConfigurations)) { irace.error("if given, initConfigurations must be a matrix or data.frame") } # This prevents loading the file two times and overriding forbiddenExps if # the user specified them explicitly. if (is.null.or.empty(scenario$forbiddenExps) && !is.null.or.empty(scenario$forbiddenFile)) { scenario$forbiddenFile <- path_rel2abs(scenario$forbiddenFile) file.check (scenario$forbiddenFile, readable = TRUE, text = "forbidden configurations file") scenario$forbiddenExps <- readForbiddenFile(scenario$forbiddenFile) irace.note(length(scenario$forbiddenExps), " expression(s) specifying forbidden configurations read from '", scenario$forbiddenFile, "'\n") } # Make it NULL if it is "" or NA # FIXME: If it is a non-empty vector of strings, parse them as above. if (is_null_or_empty_or_na(scenario$forbiddenExps)) scenario$forbiddenExps <- NULL # We have characters everywhere, set to the right types to avoid # problems later. # Integer control parameters intParams <- .irace.params.def[.irace.params.def[, "type"] == "i", "name"] for (param in intParams) { if (is.null.or.empty(scenario[[param]])) scenario[[param]] <- NA if (is.na(scenario[[param]])) next # Allow NA default values scenario[[param]] <- suppressWarnings(as.numeric(scenario[[param]])) if (is.null(scenario[[param]]) || is.na (scenario[[param]]) || !is.wholenumber(scenario[[param]])) irace.error (quote.param (param), " must be an integer.") } if (scenario$firstTest <= 1) { irace.error(quote.param ("firstTest"), " must be larger than 1.") } if (scenario$firstTest %% scenario$eachTest != 0) { irace.error(quote.param("firstTest"), " must be a multiple of ", quote.param("eachTest"), ".") } if (scenario$mu < scenario$firstTest) { if (scenario$debugLevel >= 1) { irace.warning("Assuming 'mu = firstTest' because 'mu' cannot be lower than 'firstTest'\n") } scenario$mu <- scenario$firstTest } # AClib benchmarks use 15 digits if (scenario$aclib) scenario$digits <- 15 if (scenario$digits > 15 || scenario$digits <= 0) irace.error (quote.param ("digits"), " must be within [1,15].") # Real [0, 1] control parameters realParams <- .irace.params.def[.irace.params.def[, "type"] == "r", "name"] for (param in realParams) { if (is.null.or.empty(scenario[[param]])) scenario[[param]] <- NA if (is.na(scenario[[param]])) next # Allow NA default values scenario[[param]] <- suppressWarnings(as.numeric(scenario[[param]])) if (is.null(scenario[[param]]) || is.na (scenario[[param]]) || scenario[[param]] < 0.0 || scenario[[param]] > 1.0) irace.error (quote.param(param), " must be a real value within [0, 1].") } ## Only maxExperiments or maxTime should be set. Negative values are not ## allowed. if (scenario$maxExperiments == 0 && scenario$maxTime == 0) { irace.error("Tuning budget was not provided. Set ", quote.param("maxExperiments"), "or ", quote.param("maxTime"), ".") } else if (scenario$maxExperiments > 0 && scenario$maxTime > 0) { irace.error("Two different tuning budgets provided, please set only ", quote.param("maxExperiments"), " or only ", quote.param ("maxTime"), ".") } else if (scenario$maxExperiments < 0 ) { irace.error("Negative budget provided, ", quote.param("maxExperiments"), "must be >= 0." ) } else if (scenario$maxTime < 0) { irace.error("Negative budget provided, ", quote.param("maxTime"), " must be >= 0.") } if (scenario$maxTime > 0 && (scenario$budgetEstimation <= 0 || scenario$budgetEstimation >= 1)) irace.error(quote.param("budgetEstimation"), " must be within (0,1).") if (is.na (scenario$softRestartThreshold)) { scenario$softRestartThreshold <- 10^(- scenario$digits) } if (scenario$deterministic && scenario$firstTest > length(scenario$instances)) { irace.error("When deterministic == TRUE, the number of instances (", length(scenario$instances), ") cannot be smaller than firstTest (", scenario$firstTest, ")") } if (scenario$mpi && scenario$parallel < 2) { irace.error (quote.param("parallel"), " must be larger than 1 when mpi is enabled.") } if (is.null.or.empty(scenario$batchmode)) scenario$batchmode <- 0 if (scenario$batchmode != 0) { scenario$batchmode <- tolower(scenario$batchmode) check.valid.param("batchmode") } # Currently batchmode requires a targetEvaluator if (scenario$batchmode != 0 && is.null(scenario$targetEvaluator)) { irace.error(quote.param("batchmode"), " requires using ", quote.param("targetEvaluator"), ".") } if (scenario$batchmode != 0 && scenario$mpi) { irace.error(quote.param("mpi"), " and ", quote.param("batchmode"), " cannot be enabled at the same time.") } if (scenario$capping) { if (!scenario$elitist) irace.error("When capping == TRUE, elitist must be enabled.") if (scenario$boundMax <= 0) irace.error("When capping == TRUE, boundMax (", scenario$boundMax, ") must be > 0") check.valid.param("cappingType") check.valid.param("boundType") if (scenario$boundPar < 1) irace.error("Invalid value (", scenario$boundPar, ") ", quote.param("boundPar"), " must be >= 1") } else if (scenario$boundMax <= 0 || is.na(scenario$boundMax)) { # no capping scenario$boundMax <- NULL } if (is_null_or_empty_or_na(scenario$testType)) { if (scenario$capping) scenario$testType <- "t-test" else scenario$testType <- "f-test" } scenario$testType <- switch(tolower(scenario$testType), "f-test" =, # Fall-through "friedman" = "friedman", "t-test" =, # Fall-through "t.none" = "t.none", "t-test-holm" =, # Fall-through, "t.holm" = "t.holm", "t-test-bonferroni" =, # Fall-through, "t.bonferroni" = "t.bonferroni", check.valid.param("testType")) scenario } #' Prints the given scenario #' #' @template arg_scenario #' #' @seealso #' \describe{ #' \item{[readScenario()]}{for reading a configuration scenario from a file.} #' \item{[printScenario()]}{prints the given scenario.} #' \item{[defaultScenario()]}{returns the default scenario settings of \pkg{irace}.} #' \item{[checkScenario()]}{to check that the scenario is valid.} #' } #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export printScenario <- function(scenario) { params_names <- .irace.params.names cat("## irace scenario:\n") for (param in params_names) { if (param == "forbiddenExps") extra <- paste0(" = expression(", paste0(collapse=", ", sapply(scenario[[param]], attr, "source")), ")") else extra <- "" cat(param, " = ", deparse(scenario[[param]]), extra, "\n", sep = "") } cat("## end of irace scenario\n") } #' Default scenario settings #' #' Return scenario object with default values. #' #' @template arg_scenario #' #' @template arg_params_def #' #' @return A list indexed by the \pkg{irace} parameter names, #' containing the default values for each parameter, except for those #' already present in the scenario passed as argument. #' The scenario list contains the following elements: # __IRACE_OPTIONS__BEGIN__ #' \itemize{ #' \item General options: #' \describe{ #' \item{`scenarioFile`}{Path of the file that describes the configuration scenario setup and other irace settings. (Default: `"./scenario.txt"`)} #' \item{`execDir`}{Directory where the programs will be run. (Default: `"./"`)} #' \item{`logFile`}{File to save tuning results as an R dataset, either absolute path or relative to execDir. (Default: `"./irace.Rdata"`)} #' \item{`quiet`}{Reduce the output generated by irace to a minimum. (Default: `0`)} #' \item{`debugLevel`}{Debug level of the output of \code{irace}. Set this to 0 to silence all debug messages. Higher values provide more verbose debug messages. (Default: `0`)} #' \item{`seed`}{Seed of the random number generator (by default, generate a random seed). (Default: `NA`)} #' \item{`repairConfiguration`}{User-defined R function that takes a configuration generated by irace and repairs it. (Default: `""`)} #' \item{`postselection`}{Percentage of the configuration budget used to perform a postselection race of the best configurations of each iteration after the execution of irace. (Default: `0`)} #' \item{`aclib`}{Enable/disable AClib mode. This option enables compatibility with GenericWrapper4AC as targetRunner script. (Default: `0`)} #' } #' \item Elitist `irace`: #' \describe{ #' \item{`elitist`}{Enable/disable elitist irace. (Default: `1`)} #' \item{`elitistNewInstances`}{Number of instances added to the execution list before previous instances in elitist irace. (Default: `1`)} #' \item{`elitistLimit`}{In elitist irace, maximum number per race of elimination tests that do not eliminate a configuration. Use 0 for no limit. (Default: `2`)} #' } #' \item Internal `irace` options: #' \describe{ #' \item{`sampleInstances`}{Randomly sample the training instances or use them in the order given. (Default: `1`)} #' \item{`softRestart`}{Enable/disable the soft restart strategy that avoids premature convergence of the probabilistic model. (Default: `1`)} #' \item{`softRestartThreshold`}{Soft restart threshold value for numerical parameters. If \code{NA}, \code{NULL} or \code{""}, it is computed as \code{10^-digits}. (Default: `""`)} #' \item{`nbIterations`}{Maximum number of iterations. (Default: `0`)} #' \item{`nbExperimentsPerIteration`}{Number of runs of the target algorithm per iteration. (Default: `0`)} #' \item{`minNbSurvival`}{Minimum number of configurations needed to continue the execution of each race (iteration). (Default: `0`)} #' \item{`nbConfigurations`}{Number of configurations to be sampled and evaluated at each iteration. (Default: `0`)} #' \item{`mu`}{Parameter used to define the number of configurations sampled and evaluated at each iteration. (Default: `5`)} #' } #' \item Target algorithm parameters: #' \describe{ #' \item{`parameterFile`}{File that contains the description of the parameters of the target algorithm. (Default: `"./parameters.txt"`)} #' \item{`forbiddenExps`}{Vector of R logical expressions that cannot evaluate to \code{TRUE} for any evaluated configuration. (Default: `""`)} #' \item{`forbiddenFile`}{File that contains a list of logical expressions that cannot be \code{TRUE} for any evaluated configuration. If empty or \code{NULL}, do not use forbidden expressions. (Default: `""`)} #' \item{`digits`}{Maximum number of decimal places that are significant for numerical (real) parameters. (Default: `4`)} #' } #' \item Target algorithm execution: #' \describe{ #' \item{`targetRunner`}{Executable called for each configuration that executes the target algorithm to be tuned. See the templates and examples provided. (Default: `"./target-runner"`)} #' \item{`targetRunnerLauncher`}{Executable that will be used to launch the target runner, when \code{targetRunner} cannot be executed directly (.e.g, a Python script in Windows). (Default: `""`)} #' \item{`targetRunnerLauncherArgs`}{Command-line arguments provided to \code{targetRunnerLauncher}. The substrings \code{\{targetRunner\}} and \code{\{targetRunnerArgs\}} will be replaced by the value of the option \code{targetRunner} and by the arguments usually passed when calling \code{targetRunner}, respectively. Example: \code{"-m {targetRunner} --args {targetRunnerArgs}"}. (Default: `"{targetRunner} {targetRunnerArgs}"`)} #' \item{`targetRunnerRetries`}{Number of times to retry a call to \code{targetRunner} if the call failed. (Default: `0`)} #' \item{`targetRunnerData`}{Optional data passed to \code{targetRunner}. This is ignored by the default \code{targetRunner} function, but it may be used by custom \code{targetRunner} functions to pass persistent data around. (Default: `""`)} #' \item{`targetRunnerParallel`}{Optional R function to provide custom parallelization of \code{targetRunner}. (Default: `""`)} #' \item{`targetEvaluator`}{Optional script or R function that provides a numeric value for each configuration. See templates/target-evaluator.tmpl (Default: `""`)} #' \item{`deterministic`}{If the target algorithm is deterministic, configurations will be evaluated only once per instance. (Default: `0`)} #' \item{`parallel`}{Number of calls to \code{targetRunner} to execute in parallel. Values \code{0} or \code{1} mean no parallelization. (Default: `0`)} #' \item{`loadBalancing`}{Enable/disable load-balancing when executing experiments in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster. (Default: `1`)} #' \item{`mpi`}{Enable/disable MPI. Use \code{Rmpi} to execute \code{targetRunner} in parallel (parameter \code{parallel} is the number of slaves). (Default: `0`)} #' \item{`batchmode`}{Specify how irace waits for jobs to finish when \code{targetRunner} submits jobs to a batch cluster: sge, pbs, torque, slurm or htcondor. \code{targetRunner} must submit jobs to the cluster using, for example, \code{qsub}. (Default: `0`)} #' } #' \item Initial configurations: #' \describe{ #' \item{`initConfigurations`}{Data frame describing initial configurations (usually read from a file using \code{readConfigurations}). (Default: `""`)} #' \item{`configurationsFile`}{File that contains a table of initial configurations. If empty or \code{NULL}, all initial configurations are randomly generated. (Default: `""`)} #' } #' \item Training instances: #' \describe{ #' \item{`instances`}{Character vector of the instances to be used in the \code{targetRunner}. (Default: `""`)} #' \item{`trainInstancesDir`}{Directory where training instances are located; either absolute path or relative to current directory. If no \code{trainInstancesFiles} is provided, all the files in \code{trainInstancesDir} will be listed as instances. (Default: `"./Instances"`)} #' \item{`trainInstancesFile`}{File that contains a list of training instances and optionally additional parameters for them. If \code{trainInstancesDir} is provided, \code{irace} will search for the files in this folder. (Default: `""`)} #' } #' \item Tuning budget: #' \describe{ #' \item{`maxExperiments`}{Maximum number of runs (invocations of \code{targetRunner}) that will be performed. It determines the maximum budget of experiments for the tuning. (Default: `0`)} #' \item{`maxTime`}{Maximum total execution time in seconds for the executions of \code{targetRunner}. \code{targetRunner} must return two values: cost and time. (Default: `0`)} #' \item{`budgetEstimation`}{Fraction (smaller than 1) of the budget used to estimate the mean computation time of a configuration. Only used when \code{maxTime} > 0 (Default: `0.02`)} #' \item{`minMeasurableTime`}{Minimum time unit that is still (significantly) measureable. (Default: `0.01`)} #' } #' \item Statistical test: #' \describe{ #' \item{`testType`}{Statistical test used for elimination. The default value selects \code{t-test} if \code{capping} is enabled or \code{F-test}, otherwise. Valid values are: F-test (Friedman test), t-test (pairwise t-tests with no correction), t-test-bonferroni (t-test with Bonferroni's correction for multiple comparisons), t-test-holm (t-test with Holm's correction for multiple comparisons). (Default: `""`)} #' \item{`firstTest`}{Number of instances evaluated before the first elimination test. It must be a multiple of \code{eachTest}. (Default: `5`)} #' \item{`eachTest`}{Number of instances evaluated between elimination tests. (Default: `1`)} #' \item{`confidence`}{Confidence level for the elimination test. (Default: `0.95`)} #' } #' \item Adaptive capping: #' \describe{ #' \item{`capping`}{Enable the use of adaptive capping, a technique designed for minimizing the computation time of configurations. This is only available when \code{elitist} is active. (Default: `0`)} #' \item{`cappingType`}{Measure used to obtain the execution bound from the performance of the elite configurations.\itemize{\item median: Median performance of the elite configurations.\item mean: Mean performance of the elite configurations.\item best: Best performance of the elite configurations.\item worst: Worst performance of the elite configurations.} (Default: `"median"`)} #' \item{`boundType`}{Method to calculate the mean performance of elite configurations.\itemize{\item candidate: Mean execution times across the executed instances and the current one.\item instance: Execution time of the current instance.} (Default: `"candidate"`)} #' \item{`boundMax`}{Maximum execution bound for \code{targetRunner}. It must be specified when capping is enabled. (Default: `0`)} #' \item{`boundDigits`}{Precision used for calculating the execution time. It must be specified when capping is enabled. (Default: `0`)} #' \item{`boundPar`}{Penalization constant for timed out executions (executions that reach \code{boundMax} execution time). (Default: `1`)} #' \item{`boundAsTimeout`}{Replace the configuration cost of bounded executions with \code{boundMax}. (Default: `1`)} #' } #' \item Recovery: #' \describe{ #' \item{`recoveryFile`}{Previously saved log file to recover the execution of \code{irace}, either absolute path or relative to the current directory. If empty or \code{NULL}, recovery is not performed. (Default: `""`)} #' } #' \item Testing: #' \describe{ #' \item{`testInstancesDir`}{Directory where testing instances are located, either absolute or relative to current directory. (Default: `""`)} #' \item{`testInstancesFile`}{File containing a list of test instances and optionally additional parameters for them. (Default: `""`)} #' \item{`testInstances`}{Character vector of the instances to be used in the \code{targetRunner} when executing the testing. (Default: `""`)} #' \item{`testNbElites`}{Number of elite configurations returned by irace that will be tested if test instances are provided. (Default: `1`)} #' \item{`testIterationElites`}{Enable/disable testing the elite configurations found at each iteration. (Default: `0`)} #' } #' } # __IRACE_OPTIONS__END__ #' #' @seealso #' \describe{ #' \item{[readScenario()]}{for reading a configuration scenario from a file.} #' \item{[printScenario()]}{prints the given scenario.} #' \item{[defaultScenario()]}{returns the default scenario settings of \pkg{irace}.} #' \item{[checkScenario()]}{to check that the scenario is valid.} #' } #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export defaultScenario <- function(scenario = list(), params_def = .irace.params.def) { params_names <- params_def[!startsWith(params_def[,"name"], "."), "name"] if (is.null(names(scenario))) { scenario <- setNames(as.list(params_def[params_names,"default"]), params_names) } else if (!all(names(scenario) %in% params_names)) { irace.error("Unknown scenario parameters: ", paste(names(scenario)[!(names(scenario) %in% params_names)], collapse = ", ")) } else { for (k in params_names) { if (is.null.or.na(scenario[[k]])) { scenario[[k]] <- params_def[k, "default"] } } } scenario } readInstances <- function(instancesDir = NULL, instancesFile = NULL) { if (is.null.or.empty(instancesDir) && is.null.or.empty(instancesFile)) irace.error("Both instancesDir and instancesFile are empty: No instances provided") instances <- NULL if (!is.null.or.empty(instancesFile)) { file.check (instancesFile, readable = TRUE, text = "instance file") # We do not warn if the last line does not finish with a newline. instances <- readLines (instancesFile, warn = FALSE) instances <- sub("#.*$", "", instances) # Remove comments instances <- sub("^[[:space:]]+", "", instances) # Remove leading whitespace instances <- instances[instances != ""] # Delete empty lines if (is.null.or.empty(instances)) irace.error("No instances found in '", instancesFile, "' (whitespace and comments starting with '#' are ignored)") if (!is.null.or.empty(instancesDir)) instances <- paste0 (instancesDir, "/", instances) } else { file.check (instancesDir, isdir = TRUE, notempty = TRUE, text = "instances directory") # The files are sorted in alphabetical order, on the full path if # 'full.names = TRUE'. instances <- list.files (path = instancesDir, full.names = TRUE, recursive = TRUE) if (length (instances) == 0) irace.error("No instances found in `", instancesDir, "'") } instances } ## Check targetRunner execution checkTargetFiles <- function(scenario, parameters) { ## Create two random configurations conf.id <- c("testConfig1", "testConfig2") configurations <- sampleUniform(parameters, length(conf.id), digits = scenario$digits, forbidden = scenario$forbiddenExps, repair = scenario$repairConfiguration) configurations <- cbind(.ID. = conf.id, configurations, stringsAsFactors=FALSE) # Read initial configurations provided by the user. initConfigurations <- allConfigurationsInit(scenario, parameters) if (nrow(initConfigurations) > 0L) { irace.assert(all(colnames(configurations) == colnames(initConfigurations))) configurations <- rbind(configurations, initConfigurations) } bounds <- rep(scenario$boundMax, nrow(configurations)) instances.ID <- if (scenario$sampleInstances) sample.int(length(scenario$instances), 1L) else 1L experiments <- createExperimentList( configurations, parameters, instances = scenario$instances, instances.ID = instances.ID, seeds = 1234567, scenario, bounds = bounds) startParallel(scenario) on.exit(stopParallel(), add = TRUE) # FIXME: Create a function try.call(err.msg,warn.msg, fun, ...) # Executing targetRunner cat("# Executing targetRunner (", nrow(configurations), "times)...\n") result <- TRUE output <- withCallingHandlers( tryCatch(execute.experiments(experiments, scenario), error = function(e) { cat(sep = "\n", "\n# Error occurred while executing targetRunner:", paste0(conditionMessage(e), collapse="\n")) result <<- FALSE NULL }), warning = function(w) { cat(sep = "\n", "\n# Warning occurred while executing targetRunner:", paste0(conditionMessage(w), collapse="\n")) invokeRestart("muffleWarning")}) if (scenario$debugLevel >= 1) { cat ("# targetRunner returned:\n") print(output, digits = 15) } irace.assert(is.null(scenario$targetEvaluator) == is.null(.irace$target.evaluator)) if (!result) return(FALSE) if (!is.null(scenario$targetEvaluator)) { cat("# Executing targetEvaluator...\n") output <- withCallingHandlers( tryCatch(execute.evaluator(experiments, scenario, output, configurations[, ".ID."]), error = function(e) { cat(sep = "\n", "\n# Error ocurred while executing targetEvaluator:", paste0(conditionMessage(e), collapse="\n")) result <<- FALSE NULL }), warning = function(w) { cat(sep = "\n", "\n# Warning ocurred while executing targetEvaluator:", paste0(conditionMessage(w), collapse="\n")) invokeRestart("muffleWarning")}) if (scenario$debugLevel >= 1) { cat ("# targetEvaluator returned:\n") print(output, digits = 15) } } result } irace/R/main.R0000644000176200001440000003730414312612254012614 0ustar liggesusers# ========================================================================= # irace: An implementation in R of Iterated Race. # ------------------------------------------------------------------------- # # Copyright (C) 2010-2020 # Manuel López-Ibáñez # Jérémie Dubois-Lacoste # Leslie Perez Caceres # # ------------------------------------------------------------------------- # This program is free software (software libre); you can redistribute # it and/or modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 2 of the # License, or (at your option) any later version. # # This program is distributed in the hope that it will be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, you can obtain a copy of the GNU # General Public License at: # http://www.gnu.org/copyleft/gpl.html # or by writing to the Free Software Foundation, Inc., 59 Temple Place, # Suite 330, Boston, MA 02111-1307 USA # ------------------------------------------------------------------------- # $Revision$ # ========================================================================= #' irace.license #' #' A character string containing the license information of \pkg{irace}. #' #' @export ## __VERSION__ below will be replaced by the version defined in R/version.R ## This avoids constant conflicts within this file. irace.license <- '#------------------------------------------------------------------------------ # irace: An implementation in R of (Elitist) Iterated Racing # Version: __VERSION__ # Copyright (C) 2010-2020 # Manuel Lopez-Ibanez # Jeremie Dubois-Lacoste # Leslie Perez Caceres # # This is free software, and you are welcome to redistribute it under certain # conditions. See the GNU General Public License for details. There is NO # WARRANTY; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # # irace builds upon previous code from the race package: # race: Racing methods for the selection of the best # Copyright (C) 2003 Mauro Birattari #------------------------------------------------------------------------------ ' cat_irace_license <- function() { cat(sub("__VERSION__", irace.version, irace.license, fixed=TRUE)) } #' Higher-level interface to launch irace. #' #' @template arg_scenario #' #' @param output.width (\code{integer(1)}) The width used for the screen #' output. #' #' @details This function checks the correctness of the scenario, reads the #' parameter space from \code{scenario$parameterFile}, invokes [irace()], #' prints its results in various formatted ways, (optionally) calls #' [psRace()] and, finally, evaluates the best configurations on the test #' instances (if provided). If you want a lower-level interface that just #' runs irace, please see function [irace()]. #' #' @templateVar return_invisible TRUE #' @template return_irace #' @seealso #' [irace.cmdline()] a higher-level command-line interface to #' [irace()] #' [readScenario()] to read the scenario setup from a file. #' [defaultScenario()] to provide a default scenario for \pkg{irace}. #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @concept running #' @export irace.main <- function(scenario, output.width = 9999L) irace_common(scenario = scenario, simple=FALSE, output.width = output.width) #' Test configurations given in `.Rdata` file #' #' `testing_fromlog` executes the testing of the target algorithm configurations #' found by an \pkg{irace} execution. #' #' @param logFile Path to the `.Rdata` file produced by \pkg{irace}. #' #' @param testNbElites Number of (final) elite configurations to test. Overrides #' the value found in `logFile`. #' #' @param testIterationElites (`logical(1)`) If `FALSE`, only the final #' `testNbElites` configurations are tested; otherwise, also test the best #' configurations of each iteration. Overrides the value found in `logFile`. #' #' @param testInstancesDir Directory where testing instances are located, either absolute or relative to current directory. #' #' @param testInstancesFile File containing a list of test instances and optionally additional parameters for them. #' #' @param testInstances Character vector of the instances to be used in the `targetRunner` when executing the testing. #' #' @return Boolean. `TRUE` if the testing ended successfully otherwise, `FALSE`. #' #' @details The function `testing_fromlog` loads the `logFile` and obtains the #' testing setup and configurations to be tested. Within the `logFile`, the #' variable `scenario$testNbElites` specifies how many final elite #' configurations to test and `scenario$testIterationElites` indicates #' whether test the best configuration of each iteration. The values may be #' overridden by setting the corresponding arguments in this function. The #' set of testing instances must appear in `scenario[["testInstances"]]`. #' #' @seealso [defaultScenario()] to provide a default scenario for \pkg{irace}. #' [testing_fromfile()] provides a different interface for testing. #' #' @author Manuel López-Ibáñez and Leslie Pérez Cáceres #' @concept running #' @export testing_fromlog <- function(logFile, testNbElites, testIterationElites, testInstancesDir, testInstancesFile, testInstances) { if (is.null.or.empty(logFile)) { irace.note("No logFile provided to perform the testing of configurations. Skipping testing.\n") return(FALSE) } iraceResults <- read_logfile(logFile) scenario <- iraceResults[["scenario"]] parameters <- iraceResults[["parameters"]] instances_changed <- FALSE if (!missing(testNbElites)) scenario[["testNbElites"]] <- testNbElites if (!missing(testIterationElites)) scenario$testIterationElites <- testIterationElites if (!missing(testInstances)) scenario[["testInstances"]] <- testInstances if (!missing(testInstancesDir)) { scenario$testInstancesDir <- testInstancesDir instances_changed <- TRUE } if (!missing(testInstancesFile)) { scenario$testInstancesFile <- testInstancesFile instances_changed <- TRUE } cat("\n\n# Testing of elite configurations:", scenario$testNbElites, "\n# Testing iteration configurations:", scenario$testIterationElites,"\n") if (scenario$testNbElites <= 0) return (FALSE) # If they are already setup, don't change them. if (instances_changed || is.null.or.empty(scenario[["testInstances"]])) { scenario <- setup_test_instances(scenario) if (is.null.or.empty(scenario[["testInstances"]])) { irace.note("No test instances, skip testing\n") return(FALSE) } } # Get configurations that will be tested if (scenario$testIterationElites) testing_id <- sapply(iraceResults$allElites, function(x) x[1:min(length(x), scenario$testNbElites)]) else { tmp <- iraceResults$allElites[[length(iraceResults$allElites)]] testing_id <- tmp[1:min(length(tmp), scenario$testNbElites)] } testing_id <- unique.default(unlist(testing_id)) configurations <- iraceResults$allConfigurations[testing_id, , drop=FALSE] irace.note ("Testing configurations (in no particular order): ", paste(testing_id, collapse=" "), "\n") testing_common(configurations, scenario, parameters, iraceResults) return(TRUE) } #' Test configurations given an explicit table of configurations and a scenario file #' #' Executes the testing of an explicit list of configurations given in #' `filename` (same format as in [readConfigurationsFile()]). A `logFile` is #' created unless disabled in `scenario`. This may overwrite an existing one! #' #' @param filename Path to a file containing configurations: one configuration #' per line, one parameter per column, parameter names in header. #' #' @template arg_scenario #' #' @return iraceResults #' #' @seealso [testing_fromlog()] provides a different interface for testing. #' #' @author Manuel López-Ibáñez #' @concept running #' @export testing_fromfile <- function(filename, scenario) { irace.note ("Checking scenario\n") scenario <- checkScenario(scenario) if (!scenario$quiet) printScenario(scenario) irace.note("Reading parameter file '", scenario$parameterFile, "'.\n") parameters <- readParameters (file = scenario$parameterFile, digits = scenario$digits) configurations <- readConfigurationsFile (filename, parameters) configurations <- cbind(.ID. = 1:nrow(configurations), configurations, .PARENT. = NA) rownames(configurations) <- configurations$.ID. num <- nrow(configurations) configurations <- checkForbidden(configurations, scenario$forbiddenExps) if (nrow(configurations) < num) { irace.warning("Some of the configurations in the configurations file were forbidden", "and, thus, discarded") } # To save the logs iraceResults <- list(scenario = scenario, irace.version = irace.version, parameters = parameters, allConfigurations = configurations) irace.note ("Testing configurations (in the order given as input): \n") iraceResults <- testing_common(configurations, scenario, parameters, iraceResults) return(iraceResults) } testing_common <- function(configurations, scenario, parameters, iraceResults) { verbose <- !scenario$quiet if (verbose) configurations.print(configurations) iraceResults$testing <- testConfigurations(configurations, scenario, parameters) irace_save_logfile (iraceResults, scenario) # FIXME : We should print the seeds also. As an additional column? irace.note ("Testing results (column number is configuration ID in no particular order):\n") if (verbose) print(iraceResults$testing$experiments) irace.note ("Finished testing\n") return(iraceResults) } #' Test that the given irace scenario can be run. #' #' Test that the given irace scenario can be run by checking the scenario #' settings provided and trying to run the target-algorithm. #' #' @template arg_scenario #' @template arg_parameters #' #' @return returns \code{TRUE} if successful and gives an error and returns #' \code{FALSE} otherwise. #' #' @details If the `parameters` argument is missing, then the parameters #' will be read from the file `parameterFile` given by `scenario`. If #' `parameters` is provided, then `parameterFile` will not be read. This function will #' try to execute the target-algorithm. #' #' @seealso #' \describe{ #' \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} #' \item{\code{\link{printScenario}}}{prints the given scenario.} #' \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} #' \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} #' } #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export checkIraceScenario <- function(scenario, parameters) { irace.note ("Checking scenario\n") scenario$debugLevel <- 2 scenario <- checkScenario(scenario) if (!scenario$quiet) printScenario(scenario) if (missing(parameters)) { irace.note("Reading parameter file '", scenario$parameterFile, "'.\n") parameters <- readParameters (file = scenario$parameterFile, digits = scenario$digits, debugLevel = 2) } else if (!is.null.or.empty(scenario$parameterFile)) { if (!scenario$quiet) cat("# checkIraceScenario(): 'parameters' provided by user. ", "Parameter file '", scenario$parameterFile, "' will be ignored\n", sep = "") } checkParameters(parameters) irace.note("Checking target runner.\n") if (checkTargetFiles(scenario = scenario, parameters = parameters)) { irace.note("Check successful.\n") return(TRUE) } else { irace.error("Check unsuccessful.\n") return(FALSE) } } init <- function() { irace.note("Initializing working directory...\n") libPath <- system.file(package = "irace") tmplFiles <- list.files(file.path(libPath, "templates")) for (file in tmplFiles) { if (grepl(".tmpl", file) && (file != "target-evaluator.tmpl")) { newFile <- gsub(".tmpl", "", file) if ((file == "target-runner.tmpl") && .Platform$OS.type == 'windows') { file.copy(file.path(libPath, "templates", "windows", "target-runner.bat"), file.path(getwd(), "target-runner.bat"), overwrite = FALSE) } else { file.copy(file.path(libPath, "templates", file), file.path(getwd(), newFile), overwrite = FALSE) } } } } #' Launch `irace` with command-line options. #' #' Calls [irace.main()] using command-line options, maybe parsed from the #' command line used to invoke R. #' #' @param argv (\code{character()}) \cr The arguments #' provided on the R command line as a character vector, e.g., #' \code{c("--scenario", "scenario.txt", "-p", "parameters.txt")}. #' Using the default value (not providing the parameter) is the #' easiest way to call \code{irace.cmdline}. #' #' @details The function reads the parameters given on the command line #' used to invoke R, finds the name of the scenario file, #' initializes the scenario from the file (with the function #' \code{\link{readScenario}}) and possibly from parameters passed in #' the command line. It finally starts \pkg{irace} by calling #' \code{\link{irace.main}}. #' #' List of command-line options: #' ```{r echo=FALSE,comment=NA} #' cmdline_usage(.irace.params.def) #' ``` #' #' @templateVar return_invisible TRUE #' @template return_irace #' #' @seealso #' [irace.main()] to start \pkg{irace} with a given scenario. #' @examples #' irace.cmdline("--version") #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @concept running #' @export irace.cmdline <- function(argv = commandArgs(trailingOnly = TRUE)) { parser <- CommandArgsParser$new(argv = argv, argsdef = .irace.params.def) quiet <- !is.null(parser$readArg (short = "-q", long = "--quiet")) if (quiet) { op <- options(.irace.quiet = TRUE) on.exit(options(op)) } else { cat_irace_license() cat("# installed at: ", system.file(package="irace"), "\n", "# called with: ", paste(argv, collapse = " "), "\n", sep = "") } if (!is.null(parser$readArg (short = "-h", long = "--help"))) { parser$cmdline_usage() return(invisible(NULL)) } if (!is.null(parser$readArg(short = "-v", long = "--version"))) { print(citation(package="irace")) return(invisible(NULL)) } if (!is.null(parser$readArg(short = "-i", long = "--init"))) { init() return(invisible(NULL)) } # Read the scenario file and the command line scenarioFile <- parser$readCmdLineParameter ("scenarioFile", default = "") scenario <- readScenario(scenarioFile) for (param in .irace.params.names) { scenario[[param]] <- parser$readCmdLineParameter(paramName = param, default = scenario[[param]]) } if (quiet) scenario$quiet <- TRUE # Check scenario if (!is.null(parser$readArg (short = "-c", long = "--check"))) { checkIraceScenario(scenario) return(invisible(NULL)) } # Only do testing testFile <- parser$readArg (long = "--only-test") if (!is.null(testFile)) { return(invisible(testing_fromfile(testFile, scenario))) } if (length(parser$argv) > 0) { irace.error ("Unknown command-line options: ", paste(parser$argv, collapse = " ")) } irace_common(scenario = scenario, simple=FALSE) } irace/R/parameterExploration.R0000644000176200001440000001407214252663377016111 0ustar liggesusers#' psRace #' #' \code{psRace} performs a postselection race a set of configurations. #' #' @param iraceLogFile NULL Log file created by \pkg{irace}, this file must contain the #' \code{iraceResults} object. #' @param iraceResults NULL Object created by \pkg{irace} and saved in \code{scenario$logFile}. #' @param conf.ids NULL IDs of the configurations in iraceResults$allConfigurations to be used for ablation. #' If NULL, the elites argument will be used. #' @param postselection NULL Percentage of the maxExperiments provided in the scenario to be used in the race. #' @param max.experiments NULL Number of experiments available for the race. If NULL budget for the race is set #' by the parameter scenario$postselection, which defines the percentage of the total budget of \pkg{irace} #' (iraceResults$scenario$maxExperiments or iraceResults$scenario$maxTime/iraceResults$state$timeEstimate) to use #' for the postselection. #' @param elites FALSE Flag for selecting configurations. If FALSE, the best configurations of each #' iteration are used for the race. If TRUE, the elite configurtions of each iteration are used for the race. #' @param seed 1234567 Numerical value to use as seed for the random number generation. #' #' @return If iraceLogFile is NULL, it returns a list with the following elements: #' \describe{ #' \item{configurations}{Configurations used in the race.} #' \item{instances}{A matrix with the instances used in the experiments. First column has the #' instances ids from iraceResults$scenario$instances, second column the seed assigned to the instance.} #' \item{maxExperiments}{Maximum number of experiments set for the race.} #' \item{experiments}{A matrix with the results of the experiments (columns are configurations, rows are instances).} #' \item{elites}{Best configurations found in the experiments.} #' } #' If \code{iraceLogFile} is provided this list object will be saved in \code{iraceResults$psrace.log}. #' #' @examples #' \dontrun{ #' # Execute the postselection automatically after irace #' scenario <- readScenario(filename="scenario.txt") #' parameters <- readParameters("parameters.txt") #' # Use 10% of the total budget #' scenario$postselection <- 0.1 #' irace(scenario=scenario, parameters=parameters) #' # Execute the postselection after the execution of \pkg{irace}. #' psRace(iraceLogFile="irace.Rdata", max.experiments=120) #' } #' #' @author Leslie Pérez Cáceres #' @export psRace <- function(iraceLogFile=NULL, iraceResults=NULL, conf.ids=NULL, postselection=NULL, max.experiments=NULL, elites=FALSE, seed=1234567) { # Input check if (is.null(iraceLogFile) && is.null(iraceResults)) irace.error("You must provide a Rdata file or an iraceResults object.") irace.note ("Starting post-selection:\n# Seed:", seed, "\n") if (!is.null(iraceLogFile)) cat("# Log file:",iraceLogFile,"\n") # Load the data of the log file if (!is.null(iraceLogFile)) iraceResults <- read_logfile(iraceLogFile) parameters <- iraceResults$parameters scenario <- iraceResults$scenario # Get selected configurations if (!is.null(conf.ids)) { if (!all(conf.ids %in% iraceResults$allConfigurations$.ID.)) irace.error("Configuration ids provided", conf.ids,"cannot be found in the configurations.") configurations <- iraceResults$allConfigurations[iraceResults$allConfigurations$.ID.%in% conf.ids,,drop=FALSE] } else { which.elites <- if (elites) unlist(iraceResults$allElites) else iraceResults$iterationElites which.elites <- unique(which.elites) configurations <- iraceResults$allConfigurations[which.elites, ] } if (nrow(configurations) <= 1) irace.error ("The number configurations should be > 1.") # LESLIE: Should we use testing instances? # Generate new instances instances <- generateInstances(scenario, 1000) .irace$instancesList <- instances .irace$next.instance <- 1 # MANUEL: Does this really work? It looks very strange. if (!is.null(scenario$boundMax)) scenario$instances <- paste(scenario$instances, scenario$boundMax, sep=" ") scenario$elitist <- scenario$capping <- FALSE if (is.null(postselection)) postselection <- scenario$postselection # Calculate available budget # FIXME: add numerical checks if (is.null(max.experiments)) { budget <- if (scenario$maxExperiments > 0) scenario$maxExperiments else (scenario$maxTime/iraceResults$state$timeEstimate) max.experiments <- ceiling(postselection * budget) } cat("# configurations:", nrow(configurations), "\n") cat("# postselection %", postselection, "\n") cat("# scenario experiments:",scenario$maxExperiments,"\n" ) cat("# available experiments:",max.experiments,"\n" ) cat("# minSurvival: 1\n") if (!is.null(seed)) set.seed(seed) # Should we fix the paramenters for the race? race.output <- race(maxExp = max.experiments, minSurvival = 1, elite.data = NULL, configurations = configurations, parameters = parameters, scenario = scenario, elitistNewInstances = 0) experiments <- race.output$experiments elite.configurations <- extractElites(scenario, parameters, race.output$configurations, min(race.output$nbAlive, 1)) irace.note("Elite configurations (first number is the configuration ID;", " listed from best to worst according to the ", test.type.order.str(scenario$testType), "):\n") configurations.print(elite.configurations, metadata = scenario$debugLevel >= 1) psrace.log <- list(configurations = configurations, instances = instances[1:nrow(experiments),], maxExperiments = max.experiments, experiments = experiments, elites = elite.configurations$.ID.) iraceResults$psrace.log <- psrace.log if (!is.null(iraceLogFile)) irace_save_logfile(iraceResults, scenario) return(psrace.log) } irace/R/utils.R0000644000176200001440000006155414321266642013042 0ustar liggesusers# An internal function to reload irace and set options for debugging # errors. It may also be used to reload other packages. # FIXME: Reload dynamic libraries? See ?dyn.load irace.reload.debug <- function(package = "irace") { pkg <- paste0("package:", package) try(detach(pkg, character.only = TRUE, unload = TRUE)) library(package, character.only = TRUE) options(error = if (interactive()) utils::recover else quote(utils::dump.frames("iracedump", TRUE))) } irace.print.memUsed <- function(objects) { object.size.kb <- function (name, envir) { object.size(get(name, envir = envir)) / 1024 } envir <- parent.frame() if (missing(objects)) { objects <- ls(envir = envir, all.names = TRUE) } x <- sapply(objects, object.size.kb, envir = envir) y <- sapply(ls(envir = .irace, all.names = TRUE), object.size.kb, envir = .irace) names(y) <- paste0(".irace$", names(y)) x <- c(x, y) # Do not print anything that is smaller than 32 Kb x <- x[x > 32] cat(sep="", sprintf("%30s : %17.1f Kb\n", names(x), x)) cat(sep="", sprintf("%30s : %17.1f Mb\n", "Total", sum(x) / 1024)) # This does garbage collection and also prints memory used by R. cat(sep="", sprintf("%30s : %17.1f Mb\n", "gc", sum(gc()[,2]))) } # Print a user-level warning message, when the calling context # cannot help the user to understand why the program failed. irace.warning <- function(...) { if (getOption(".irace.quiet", default=FALSE)) return() cat(sep="", .msg.prefix, "WARNING: ", ..., "\n") } # Print a user-level fatal error message, when the calling context # cannot help the user to understand why the program failed. irace.error <- function(...) { # The default is only 1000, which is too small. 8170 is the maximum # value allowed up to R 3.0.2 op <- options(warning.length = 8170) on.exit(options(op)) stop (.msg.prefix, ..., call. = FALSE) } ## utils::dump.frames is broken and cannot be used with bquote, so we need a wrapper. When irace crashes, it generates a file "iracedump.rda". To debug the crash use: ## R> load("iracedump.rda") ## R> debugger(iracedump) ## ## See help(dump.frames) for more details. irace.dump.frames <- function() { execDir <- getOption(".irace.execdir") if (!is.null(execDir)) cwd <- setwd(execDir) ## Only a very recent R version allows saving GlovalEnv: ## https://stat.ethz.ch/pipermail/r-devel/2016-November/073378.html # utils::dump.frames(dumpto = "iracedump", to.file = TRUE, include.GlobalEnv = TRUE) ## For now, we use the following work-around: ## http://stackoverflow.com/questions/40421552/r-how-make-dump-frames-include-all-variables-for-later-post-mortem-debugging utils::dump.frames(dumpto = "iracedump") save.image(file = "iracedump.rda") if (!is.null(execDir)) setwd(cwd) # We need this to signal an error in R CMD check. if (!interactive()) q("no", status = 1, runLast = FALSE) } # Print an internal fatal error message that signals a bug in irace. irace.internal.error <- function(...) { .irace.bug.report <- paste0("An unexpected condition occurred. ", "Please report this bug to the authors of the irace package ") op <- options(warning.length = 8170, error = if (interactive()) utils::recover else irace.dump.frames) on.exit(options(op)) # 6 to not show anything below irace.assert() bt <- capture.output(traceback(6)) warnings() stop (.msg.prefix, paste0(..., collapse = "\n"), paste0(bt, collapse= "\n"), "\n", .msg.prefix, "\n", .irace.bug.report, call. = FALSE) invisible() } irace.assert <- function(exp, eval.after = NULL) { # FIXME: It would be great if we could save into a file the state of # the function that called this one. if (exp) return(invisible()) mc <- match.call()[[2]] msg <- paste0(deparse(mc), " is not TRUE\n") if (!is.null(eval.after)) { msg.after <- eval.parent(capture.output(eval.after)) msg <- paste0(msg, "\n", paste0(msg.after, collapse="\n")) } irace.internal.error(msg) invisible() } irace.note <- function(...) { # FIXME: If this was a function within an irace object, we could replace it # when using quiet. if (getOption(".irace.quiet", default=FALSE)) return() cat ("# ", format(Sys.time(), usetz=TRUE), ": ", paste0(..., collapse = ""), sep = "") } file.check <- function (file, executable = FALSE, readable = executable, writeable = FALSE, isdir = FALSE, notempty = FALSE, text = NULL) { EXEC <- 1 # See documentation of the function file.access() WRITE <- 2 READ <- 4 if (!is.character(file) || is.null.or.empty(file)) { irace.error (text, " ", shQuote(file), " is not a vaild filename") } file <- path_rel2abs(file) ## The above should remove the trailing separator if present for windows OS ## compatibility, except when we have just C:/, where the trailing separator ## must remain. if (!file.exists(file)) { if (writeable) { if (tryCatch({ suppressWarnings(file.create(file) && file.remove(file)) }, error=function(e) FALSE)) return(TRUE) irace.error("cannot create ", text, " ", shQuote(file)) return (FALSE) } irace.error (text, " '", file, "' does not exist") return(FALSE) } if (writeable && (file.access(file, mode = WRITE) != 0)) { irace.error(text, " '", file, "' cannot be written into") return(FALSE) } if (readable && (file.access(file, mode = READ) != 0)) { irace.error(text, " '", file, "' is not readable") return (FALSE) } if (executable && file.access(file, mode = EXEC) != 0) { irace.error(text, " '", file, "' is not executable") return (FALSE) } if (isdir) { if (!file.info(file)$isdir) { irace.error(text, " '", file, "' is not a directory") return (FALSE) } if (notempty && length(list.files (file, recursive=TRUE)) == 0) { irace.error(text, " '", file, "' does not contain any file") return (FALSE) } } else if (file.info(file)$isdir) { irace.error(text, " '", file, "' is a directory, not a file") return (FALSE) } return (TRUE) } # Returns the smallest multiple of d that is higher than or equal to x. round.to.next.multiple <- function(x, d) return(x + d - 1 - (x - 1) %% d) # This returns FALSE for Inf/-Inf/NA is.wholenumber <- function(x, tol = .Machine$double.eps^0.5) { is.finite(x) & (abs(x - round(x)) < tol) } is.na.nowarn <- function(x) { length(x) == 1 && suppressWarnings(is.na(x)) } is.na.or.empty <- function(x) { (length(x) == 0) || is.na.nowarn(x) } is.null.or.na <- function(x) { is.null(x) || is.na.nowarn(x) } is.null.or.empty <- function(x) { (length(x) == 0) || (length(x) == 1 && !suppressWarnings(is.na(x)) && is.character(x) && x == "") } is_null_or_empty_or_na <- function(x) { (length(x) == 0) || is.na.nowarn(x) || (length(x) == 1 && !suppressWarnings(is.na(x)) && is.character(x) && x == "") } is.function.name <- function(FUN) { # FIXME: Is there a simpler way to do this check? is.function(FUN) || (!is.null(FUN) && !is.na(FUN) && as.character(FUN) != "" && !is.null(get.function(FUN))) } get.function <- function(FUN) { if (is.function(FUN)) return(FUN) FUN <- dynGet(as.character(FUN), ifnotfound = NULL, inherits = TRUE) if (is.function(FUN)) return(FUN) return (NULL) } is.bytecode <- function(x) typeof(x) == "bytecode" bytecompile <- function(x) { if (is.bytecode(x)) return(x) return(compiler::cmpfun(x)) } # FIXME: Use stringr function and replace this function str_sub <- function(x, start=0, stop=nchar(x)) { negs <- start < 0 if (any(negs)) start[negs] <- nchar(x[negs]) + 1 - start[negs] negs <- stop < 0 if (any(negs)) stop[negs] <- nchar(x[negs]) + 1 - stop[negs] return(substr(x, start, stop)) } strcat <- function(...) { do.call(paste0, args = list(..., collapse = NULL)) } #' Update filesystem paths of a scenario consistently. #' #' This function should be used to change the filesystem paths stored in a #' scenario object. Useful when moving a scenario from one computer to another. #' #' @template arg_scenario #' @param from character string containing a regular expression (or character #' string for `fixed = TRUE`) to be matched. #' @param to the replacement string.character string. For `fixed = FALSE` #' this can include backreferences `"\1"` to `"\9"` to #' parenthesized subexpressions of `from`. #' @param fixed logical. If `TRUE`, `from` is a string to be matched #' as is. #' @return The updated scenario #' @examples #' \dontrun{ #' scenario <- readScenario(filename = "scenario.txt") #' scenario <- scenario_update_paths(scenario, from = "/home/manuel/", to = "/home/leslie") #' } #' @seealso [base::grep()] #' @export scenario_update_paths <- function(scenario, from, to, fixed = TRUE) { pathParams <- .irace.params.def[.irace.params.def[, "type"] == "p", "name"] # Only consider the ones that actually appear in scenario. pathParams <- intersect(pathParams, names(scenario)) scenario[pathParams] <- lapply(scenario[pathParams], sub, pattern = from, replacement = to, fixed = fixed) scenario } #' @rdname scenario_update_paths #' @export scenario.update.paths <- function(scenario, from, to, fixed = TRUE) { .Deprecated("scenario_update_paths") scenario_update_paths(scenario=scenario, from=from, to=to, fixed=fixed) } # This function is used to trim potentially large strings for printing, since # the maximum error/warning length is 8170 characters (R 3.0.2) strlimit <- function(str, limit = 5000) { if (nchar(str) > limit) return(paste0(substr(str, 1, limit - 3), "...")) return(str) } test.type.order.str <- function(test.type) { switch(test.type, friedman = "sum of ranks", t.none =, # Fall-throught t.holm =, # Fall-throught t.bonferroni = "mean value", irace.internal.error ("test.type.order.str() Invalid value '", test.type, "' of test.type")) } trim.leading <- function(str) sub('^[[:space:]]+', '', str) ## white space, POSIX-style trim.trailing <- function(str) sub('[[:space:]]+$', '', str) ## white space, POSIX-style # remove leading and trailing white space characters trim <- function(str) trim.trailing(trim.leading(str)) isFixed <- function (paramName, parameters) as.logical(parameters$isFixed[paramName]) paramDomain <- function (paramName, parameters) parameters$domain[[paramName]] paramLowerBound <- function (paramName, parameters) as.numeric(parameters$domain[[paramName]][1]) paramUpperBound <- function (paramName, parameters) as.numeric(parameters$domain[[paramName]][2]) inNumericDomain <- function(value, domain) (value >= domain[1] && value <= domain[2]) nbParam <- function (parameters) length(parameters$names) ## This function takes two matrices x and y and merges them such that the ## resulting matrix z has: # rownames(z) <- setunion(rownames(x), rownames(y)) and # rownames(z) <- setunion(rownames(x), rownames(y)) and # z[rownames(x), colnames(x)] <- x and z[rownames(y), colnames(y)] <- y, and # z[i, j] <- NA for all i,j not in x nor y. merge.matrix <- function(x, y) { new.cols <- setdiff(colnames(y), colnames(x)) new.rows <- setdiff(rownames(y), rownames(x)) if (is.null(rownames(x)) || is.null(colnames(x))) return(y) if (is.null(rownames(y)) || is.null(colnames(y))) return(x) # Add columns x <- cbind(x, matrix(NA, ncol = length(new.cols), nrow = nrow(x), dimnames = list(rownames(x), new.cols))) # Add rows x <- rbind(x, matrix(NA, ncol = ncol(x), nrow = length(new.rows), dimnames = list(new.rows, colnames(x)))) # Update x[rownames(y), colnames(y)] <- y # There must be a non-NA entry for each instance. irace.assert(all(apply(!is.na(x), 1, any))) return(x) } ## extractElites # Input: the configurations with the .RANK. field filled. # the number of elites wished # Output: nbElites elites, sorted by ranks, with the weights assigned. extractElites <- function(scenario, parameters, configurations, nbElites) { # Keep only alive configurations. ## FIXME: Shouldn't this be done by the caller? configurations <- configurations[configurations$.ALIVE., , drop = FALSE] if (nbElites < 1) { irace.internal.error("nbElites is lower or equal to zero.") } # Remove duplicated. Duplicated configurations may be generated, however, it # is too slow to check at generation time. Nevertheless, we can check now # since we typically have very few elites. ## FIXME: Use a variant of similarConfigurations. configurations <- configurations[order(configurations$.ID.), , drop = FALSE] before <- nrow(configurations) configurations <- configurations[!duplicated(removeConfigurationsMetaData(configurations)), , drop = FALSE] after <- nrow(configurations) if (after < before && scenario$debugLevel >= 1) { irace.note("Dropped ", before - after, " duplicated elites\n") } nbElites <- min(after, nbElites) # Sort by rank. elites <- configurations[order(configurations$.RANK.), , drop = FALSE] elites <- elites[1:nbElites, , drop = FALSE] elites[, ".WEIGHT."] <- ((nbElites - (1:nbElites) + 1) / (nbElites * (nbElites + 1) / 2)) elites } #' removeConfigurationsMetaData #' #' Remove the columns with "metadata" of a matrix containing some #' configuration configurations. These "metadata" are used internaly #' by \pkg{irace}. This function can be used e.g. before printing #' the configurations, to output only the values for the parameters #' of the configuration without data possibly useless to the user. #' #' @template arg_configurations #' #' @return The same matrix without the "metadata". #' #' @seealso #' [configurations.print.command()] to print the configurations as command lines. #' [configurations.print()] to print the configurations as a data frame. #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export ## Keep only parameters values removeConfigurationsMetaData <- function(configurations) { # Meta-data colnames begin with "." configurations[, grep("^\\.", colnames(configurations), invert = TRUE), drop = FALSE] } #' Print configurations as a data frame #' #' @template arg_configurations #' @param metadata A Boolean specifying whether to print the metadata or #' not. The metadata are data for the configurations (additionally to the #' value of each parameter) used by \pkg{irace}. #' #' @return None. #' #' @seealso #' [configurations.print.command()] to print the configurations as command-line strings. #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export configurations.print <- function(configurations, metadata = FALSE) { rownames(configurations) <- configurations$.ID. if (!metadata) { configurations <- removeConfigurationsMetaData(configurations) } print(as.data.frame(configurations, stringsAsFactors = FALSE), digits = 15) } #' Print configurations as command-line strings. #' #' Prints configurations after converting them into a representation for the #' command-line. #' #' @template arg_configurations #' @template arg_parameters #' #' @return None. #' #' @seealso #' [configurations.print()] to print the configurations as a data frame. #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export configurations.print.command <- function(configurations, parameters) { if (nrow(configurations) <= 0) return(invisible()) ids <- as.numeric(configurations$.ID.) configurations <- removeConfigurationsMetaData(configurations) # Re-sort the columns configurations <- configurations[, parameters$names, drop = FALSE] # A better way to do this? We cannot use apply() because that coerces # to a character matrix thus messing up numerical values. len <- nchar(max(ids)) for (i in seq_len (nrow(configurations))) { cat(sprintf("%-*d %s\n", len, ids[i], buildCommandLine(configurations[i, , drop=FALSE], parameters$switches))) } } # FIXME: This may not work when working interactively. For example, # one cannot change the number of slaves. A more robust function # would try to close any open slaves, and then re-spawn a different # number. ## # FIXME2: Slaves will load the irace namespace independently of how the master # loaded it. This can be seen by doing in the slaves: # ## print(loadedNamespaces()) ## try(print(as.list(get(".__NAMESPACE__.", envir = asNamespace("irace", base.OK = FALSE), ## inherits = FALSE))$path)) ## try(print(path.package("irace"))) # # That is, neither R_LIBS, .libPaths or whether library was called with lib.loc # will affect the slaves. It also happens before we can set those variables on # the slaves. Thus, one may end up running a different version of irace on the # slaves than on the master. I wasted more than 12 hours trying to find a # work-around but nothing seems to work. mpiInit <- function(nslaves, debugLevel = 0) { # Load the Rmpi package if it is not already loaded. if (! ("Rmpi" %in% loadedNamespaces())) { if (! suppressPackageStartupMessages (requireNamespace("Rmpi", quietly = TRUE))) irace.error("The 'Rmpi' package is required for using MPI") # When R exits, finalize MPI. reg.finalizer(environment(Rmpi::mpi.exit), function(e) { # Rmpi already prints a message, so we don't need this. # cat("# Finalize MPI...\n") if (Rmpi::mpi.comm.size(1) > 0) # FIXME: dellog == TRUE tries to delete log files, but it does # not take into account that we may have changed directory and # it does not fail gracefully but produces an annoying: # Warning message: running command 'ls *.30577+1.*.log 2>/dev/null' had status 2 Rmpi::mpi.close.Rslaves(dellog = FALSE) # We would like to use .Call("mpi_finalize", PACKAGE = "Rmpi"), which is # what mpi.finalize does, minus the annoying message: "Exiting Rmpi. Rmpi # cannot be used unless relaunching R", which we do not care about # because this finalizer should only be called when exiting R. capture.output(Rmpi::mpi.finalize(), file = if (.Platform$OS.type == 'windows') 'NUL' else '/dev/null') }, onexit = TRUE) # Create slaves # needlog: a logical. If TRUE, R BATCH outputs will be saved in log # files. If FALSE, the outputs will send to /dev/null. # quiet: a logical. If TRUE, do not print anything unless an error occurs. # If FALSE, prints to stdio how many slaves are successfully # spawned and where they are running. Rmpi::mpi.spawn.Rslaves(nslaves = nslaves, quiet = (debugLevel < 2), needlog = (debugLevel > 0)) } } ## FIXME: Move this to the manual page. # Computes: # * Kendall's W (also known as Kendall's coefficient of concordance) # If 1, all configurations have ranked in the same order in all instances. # If 0, the ranking of each configuration on each instance is essentially random. # W = Friedman / (m * (k-1)) # # * Spearman's rho: average (Spearman) correlation coefficient computed on the # ranks of all pairs of raters. If there are no repeated data values, a # perfect Spearman correlation of +1 or −1 occurs when each of the variables # is a perfect monotone function of the other. # data: matrix with the data, instances in rows (judges), configurations in # columns. concordance <- function(data) { irace.assert (is.matrix(data) && is.numeric(data)) n <- nrow(data) #judges k <- ncol(data) #objects if (n <= 1 || k <= 1) return(list(kendall.w = NA, spearman.rho = NA)) # Get rankings by rows (per instance) r <- t(apply(data, 1L, rank)) R <- colSums(r) TIES <- tapply(r, row(r), table) # If everything is tied, then W=1, perfect homogeneity. if (all(unlist(TIES) == ncol(data))) { W <- 1 } else { # FIXME: This formula seems slightly different from the one in # friedman.test. Why? T <- sum(unlist(lapply(TIES, function (u) {u^3 - u}))) W <- ((12 * sum((R - n * (k + 1) / 2)^2)) / ((n^2 * (k^3 - k)) - (n * T))) } # Spearman's rho rho <- (n * W - 1) / (n - 1) ## Same as in friedman test #STATISTIC <- n * (k - 1) * W #PARAMETER <- k - 1 #pvalue <- pchisq(PARAMETER, df = PARAMETER, lower.tail = FALSE) return(list(kendall.w = W, spearman.rho = rho)) } ## FIXME: Move this to the manual page. ## FIXME: Reference! Explain a bit what is computed! # Calculates Performance similarity of instances # data: matrix with the data, instances in rows (judges), configurations # in columns. # Returns: variance value [0,1], where 0 is a homogeneous set of instances and # 1 is a heterogeneous set. dataVariance <- function(data) { irace.assert (is.matrix(data) && is.numeric(data)) # LESLIE: should we rank data?? # MANUEL: Why? if (nrow(data) <= 1 || ncol(data) <= 1) return(NA) # Normalize #datamin <- apply(data,1,min,na.rm=TRUE) #datamax <- apply(data,1,max,na.rm=TRUE) #normdata <- (data - datamin) / (datamax-datamin) #standardize meandata <- rowMeans(data) stddata <- apply(data, 1L, sd) # If stddata == 0, then data is constant and it doesn't matter as long as it # is non-zero. stddata[stddata == 0] <- 1 zscoredata <- (data - meandata) / stddata # We could log-tranform if needed # Variance of configurations qvar <- mean(apply(zscoredata, 2L, var)) return(qvar) } runcommand <- function(command, args, id, debugLevel) { if (debugLevel >= 2L) { irace.note (command, " ", args, "\n") elapsed <- proc.time()["elapsed"] } err <- NULL output <- withCallingHandlers( tryCatch(system2(command, args, stdout = TRUE, stderr = TRUE), error = function(e) { err <<- c(err, paste(conditionMessage(e), collapse="\n")) NULL }), warning = function(w) { err <<- c(err, paste(conditionMessage(w), collapse="\n")) invokeRestart("muffleWarning") }) if (is.null(output)) output <- "" # If the command could not be run an R error is generated. If ‘command’ # runs but gives a non-zero exit status this will be reported with a # warning and in the attribute ‘"status"’ of the result: an attribute # ‘"errmsg"’ may also be available. if (!is.null(err)) { err <- paste(err, collapse = "\n") if (!is.null(attr(output, "errmsg"))) err <- paste(sep = "\n", err, attr(output, "errmsg")) if (debugLevel >= 2L) irace.note ("ERROR (", id, "): ", err, "\n") return(list(output = output, error = err)) } if (debugLevel >= 2L) { irace.note ("DONE (", id, ") Elapsed wall-clock seconds: ", formatC(proc.time()["elapsed"] - elapsed, format = "f", digits = 2), "\n") } # TODO: Return elapsed time so that we can report at the end the total # elapsed time taken by irace vs the time taken by the target-runner. list(output = output, error = NULL) } # Safe sampling of vector: resample <- function(x, ...) x[sample.int(length(x), ...)] # Rounds up the number x to the specified number of decimal places 'digits'. ceiling.digits <- function(x, digits) { multiple <- 10^-digits div <- x / multiple int_div <- trunc(div) return (int_div * multiple + ceiling(div - int_div) * multiple) } # ceil.decimal <- function(x, d) { # # get the significant digits in the integer part. # ssd <- x * 10^(d) # # get the non significant digits # nsd <- ssd - floor(ssd) # ssd <- trunc(ssd) # sel <- nsd > 0 | ssd==0 # ssd[sel] <- ssd[sel] + 1 # x2 <- ssd/10^(d) # return(x2) # } is.file.extension <- function(filename, ext) substring(filename, nchar(filename) + 1 - nchar(ext)) == ext # Same as !(x %in% table) "%!in%" <- function(x, table) match(x, table, nomatch = 0L) == 0L irace_save_logfile <- function(iraceResults, scenario) { if (is.null.or.empty(scenario$logFile)) return(invisible()) cwd <- setwd(scenario$execDir) # FIXME: Use saveRDS # FIXME: Bump to version=3 when we bump the minimum R version to >=3.6 save(iraceResults, file = scenario$logFile, version = 2) setwd(cwd) } valid_iracelog <- function(x) { is.list(x) && ("scenario" %in% names(x)) } #' Read the log file produced by irace (`irace.Rdata`). #' #' @param filename Filename that contains the log file saved by irace. Example: `irace.Rdata`. #' #' @param name Optional argument that allows overriding the default name of the object in the file. #' #' @return (`list()`) #' @concept analysis #' @export read_logfile <- function(filename, name = "iraceResults") { # If filename is already the iraceResults object, just return it. if (valid_iracelog(filename)) return(filename) if (file.access(filename, mode=4) != 0) stop("read_logfile: Cannot read file '", filename, "'") load(filename) iraceResults <- get0(name, inherits=FALSE) if (!valid_iracelog(iraceResults)) stop("The file '", filename, "' does not contain the '", name, "' object.") iraceResults } do_nothing <- function(...) invisible() irace/R/zzz.R0000644000176200001440000000066114255304511012521 0ustar liggesusers# Uses 10 decimal places at most so that there is space for '-', '.', and # 'e-NN' within the 16 spaces. .irace.format.perf <- "%#16.10g" .onLoad <- function(libname, pkgname) { # FIXME: We would like to use %#16.10g but this causes problems with # https://github.com/oracle/fastr/issues/191 R_engine <- R.version$engine if (!is.null(R_engine) && R_engine == "FastR") .irace.format.perf <<- "%16.10g" invisible() } irace/R/argparser.R0000644000176200001440000000661214312613520013651 0ustar liggesusers#' R6 Class for parsing command-line arguments #' #' @export CommandArgsParser <- R6::R6Class("CommandArgsParser", cloneable = FALSE, list( argv = NULL, argsdef = NULL, initialize = function(argv, argsdef) { # Handle the case where we are given a single character string like a # command-line. if (!missing(argv) && length(argv) == 1) { argv <- strsplit(trim(argv), " +")[[1]] } self$argv <- argv required_colnames <- c("name", "short", "long", "type", "default") if (any(required_colnames %!in% colnames(argsdef))) { stop("argsdef must contain the column names: ", paste0(required_colnames, collapse=", ")) } self$argsdef <- argsdef rownames(self$argsdef) <- argsdef$name self }, readCmdLineParameter = function (paramName, default = NULL) { value <- self$readArg(short = self$argsdef[paramName, "short"], long = self$argsdef[paramName,"long"]) if (is.null(value)) { value <- if (is.null(default)) self$argsdef[paramName, "default"] else default } else if (is.na(value) && self$argsdef[paramName,"type"] != 'x') { stop("option '", self$argsdef[paramName,"long"],"' requires an argument", call. = FALSE) } return(value) }, # Function to read command-line arguments. ## FIXME: This function always consumes two arguments. This is problematic ## for flags that have no arguments, like --check. readArg = function(short = "", long = "") { argv <- self$argv pos <- c() if (length(short) > 0) { # FIXME: use match() pos <- grep(paste0("^", short, "$"), argv) if (length(pos) == 0) { # FIXME: use pmatch() pos <- grep(paste0("^", short, "="), argv) } } if (length(long) > 0 && length(pos) == 0) { pos <- grep(paste0("^", long, "$"), argv) if (length(pos) == 0) { pos <- grep(paste0("^", long, "="), argv) } } if (length(pos) == 0) { return(NULL) } else if(length(pos) > 0) { # Allow repeated parameters pos <- max(pos) } value <- unlist(strsplit(argv[pos], '=', fixed = TRUE))[2] if (is.null (value) || is.na(value)) { value <- argv[pos + 1] self$argv <- argv[-(pos + 1)] } self$argv <- self$argv[-pos] return (value) }, readAll = function() { params <- list() for (param in self$argsdef$name[self$argsdef$type != 'x']) { value <- self$readCmdLineParameter(paramName = param) if (is.na(value) || (length(value) > 0 && value == "")) value <- NULL params[[param]] <- value } params }, cmdline_usage = function(){ irace::cmdline_usage(self$argsdef) }) ) #' `cmdline_usage()` prints the output of `--help` #' #' @param cmdline_args Definition of the command-line arguments. #' #' @rdname CommandArgsParser #' @export cmdline_usage <- function(cmdline_args) { for (i in seq_len(nrow(cmdline_args))) { short <- cmdline_args[i,"short"] long <- cmdline_args[i,"long"] desc <- cmdline_args[i,"description"] if (desc == "" || (short == "" && long == "")) next if (short != "") short <- paste0(short,",") default <- cmdline_args[i,"default"] if (!is_null_or_empty_or_na(default)) { desc <- paste0(desc, " Default: ", default, ".") } cat(sep = "\n", strwrap(desc, width = 80, initial = sprintf("%3s%-20s ", short, long), exdent = 25)) } } irace/R/irace-options.R0000644000176200001440000003250614321271071014441 0ustar liggesusers## This file was generated by scripts/generate-options.R # Non-variable options (such as --help and --version) have names starting with "." # Variables that do not have a command-line option have description == "" # Types are b(oolean), i(nteger), s(tring), r(eal), p(ath), x (R object or no value) # FIXME: Add special type for R functions. # FIXME: For i and r add their domain. .irace.params.def <- structure(list(name = c(".help", ".version", ".check", ".init", ".onlytest", "scenarioFile", "execDir", "parameterFile", "forbiddenExps", "forbiddenFile", "initConfigurations", "configurationsFile", "logFile", "recoveryFile", "instances", "trainInstancesDir", "trainInstancesFile", "sampleInstances", "testInstancesDir", "testInstancesFile", "testInstances", "testNbElites", "testIterationElites", "testType", "firstTest", "eachTest", "targetRunner", "targetRunnerLauncher", "targetRunnerLauncherArgs", "targetRunnerRetries", "targetRunnerData", "targetRunnerParallel", "targetEvaluator", "deterministic", "maxExperiments", "maxTime", "budgetEstimation", "minMeasurableTime", "parallel", "loadBalancing", "mpi", "batchmode", "digits", "quiet", "debugLevel", "seed", "softRestart", "softRestartThreshold", "elitist", "elitistNewInstances", "elitistLimit", "repairConfiguration", "capping", "cappingType", "boundType", "boundMax", "boundDigits", "boundPar", "boundAsTimeout", "postselection", "aclib", "nbIterations", "nbExperimentsPerIteration", "minNbSurvival", "nbConfigurations", "mu", "confidence"), type = c("x", "x", "x", "x", "p", "p", "p", "p", "x", "p", "x", "p", "p", "p", "s", "p", "p", "b", "p", "p", "x", "i", "b", "s", "i", "i", "p", "p", "s", "i", "x", "x", "p", "b", "i", "i", "r", "r", "i", "b", "b", "s", "i", "b", "i", "i", "b", "r", "b", "i", "i", "x", "b", "s", "s", "i", "i", "i", "b", "r", "b", "i", "i", "i", "i", "i", "r"), short = c("-h", "-v", "-c", "-i", "", "-s", "", "-p", "", "", "", "", "-l", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "-q", "", "", "", "", "-e", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", ""), long = c("--help", "--version", "--check", "--init", "--only-test", "--scenario", "--exec-dir", "--parameter-file", "", "--forbidden-file", "", "--configurations-file", "--log-file", "--recovery-file", "", "--train-instances-dir", "--train-instances-file", "--sample-instances", "--test-instances-dir", "--test-instances-file", "", "--test-num-elites", "--test-iteration-elites", "--test-type", "--first-test", "--each-test", "--target-runner", "--target-runner-launcher", "--target-runner-args", "--target-runner-retries", "", "", "--target-evaluator", "--deterministic", "--max-experiments", "--max-time", "--budget-estimation", "--min-measurable-time", "--parallel", "--load-balancing", "--mpi", "--batchmode", "--digits", "--quiet", "--debug-level", "--seed", "--soft-restart", "--soft-restart-threshold", "--elitist", "--elitist-new-instances", "--elitist-limit", "", "--capping", "--capping-type", "--bound-type", "--bound-max", "--bound-digits", "--bound-par", "--bound-as-timeout", "--postselection", "--aclib", "--iterations", "--experiments-per-iteration", "--min-survival", "--num-configurations", "--mu", "--confidence"), default = c(NA, NA, NA, "", "", "./scenario.txt", "./", "./parameters.txt", "", "", "", "", "./irace.Rdata", "", "", "./Instances", "", "1", "", "", "", "1", "0", "", "5", "1", "./target-runner", "", "{targetRunner} {targetRunnerArgs}", "0", "", "", "", "0", "0", "0", "0.02", "0.01", "0", "1", "0", "0", "4", "0", "0", NA, "1", "", "1", "1", "2", "", "0", "median", "candidate", "0", "0", "1", "1", "0", "0", "0", "0", "0", "0", "5", "0.95"), domain = c(NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, "F-test,t-test,t-test-holm,t-test-bonferroni", NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, "sge,pbs,torque,slurm,htcondor", NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, "median,mean,worst,best", "instance,candidate", NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA), description = c("Show this help.", "Show irace package version.", "Check scenario.", "Initialize the working directory with template config files.", "Only test the configurations given in the file passed as argument.", "File that describes the configuration scenario setup and other irace settings.", "Directory where the programs will be run.", "File that contains the description of the parameters of the target algorithm.", "", "File that contains a list of logical expressions that cannot be TRUE for any evaluated configuration. If empty or NULL, do not use forbidden expressions.", "", "File that contains a table of initial configurations. If empty or NULL, all initial configurations are randomly generated.", "File to save tuning results as an R dataset, either absolute path or relative to execDir.", "Previously saved log file to recover the execution of irace, either absolute path or relative to the current directory. If empty or NULL, recovery is not performed.", "", "Directory where training instances are located; either absolute path or relative to current directory. If no trainInstancesFiles is provided, all the files in trainInstancesDir will be listed as instances.", "File that contains a list of training instances and optionally additional parameters for them. If trainInstancesDir is provided, irace will search for the files in this folder.", "Randomly sample the training instances or use them in the order given.", "Directory where testing instances are located, either absolute or relative to current directory.", "File containing a list of test instances and optionally additional parameters for them.", "", "Number of elite configurations returned by irace that will be tested if test instances are provided.", "Enable/disable testing the elite configurations found at each iteration.", "Statistical test used for elimination. The default value selects t-test if capping is enabled or F-test, otherwise. Valid values are: F-test (Friedman test), t-test (pairwise t-tests with no correction), t-test-bonferroni (t-test with Bonferroni's correction for multiple comparisons), t-test-holm (t-test with Holm's correction for multiple comparisons).", "Number of instances evaluated before the first elimination test. It must be a multiple of eachTest.", "Number of instances evaluated between elimination tests.", "Executable called for each configuration that executes the target algorithm to be tuned. See the templates and examples provided.", "Executable that will be used to launch the target runner, when targetRunner cannot be executed directly (.e.g, a Python script in Windows).", "Command-line arguments provided to targetRunnerLauncher. The substrings \\{targetRunner\\} and \\{targetRunnerArgs\\} will be replaced by the value of the option targetRunner and by the arguments usually passed when calling targetRunner, respectively. Example: \"-m {targetRunner --args {targetRunnerArgs}\"}.", "Number of times to retry a call to targetRunner if the call failed.", "Optional data passed to targetRunner. This is ignored by the default targetRunner function, but it may be used by custom targetRunner functions to pass persistent data around.", "Optional R function to provide custom parallelization of targetRunner.", "Optional script or R function that provides a numeric value for each configuration. See templates/target-evaluator.tmpl", "If the target algorithm is deterministic, configurations will be evaluated only once per instance.", "Maximum number of runs (invocations of targetRunner) that will be performed. It determines the maximum budget of experiments for the tuning.", "Maximum total execution time in seconds for the executions of targetRunner. targetRunner must return two values: cost and time.", "Fraction (smaller than 1) of the budget used to estimate the mean computation time of a configuration. Only used when maxTime > 0", "Minimum time unit that is still (significantly) measureable.", "Number of calls to targetRunner to execute in parallel. Values 0 or 1 mean no parallelization.", "Enable/disable load-balancing when executing experiments in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster.", "Enable/disable MPI. Use Rmpi to execute targetRunner in parallel (parameter parallel is the number of slaves).", "Specify how irace waits for jobs to finish when targetRunner submits jobs to a batch cluster: sge, pbs, torque, slurm or htcondor. targetRunner must submit jobs to the cluster using, for example, qsub.", "Maximum number of decimal places that are significant for numerical (real) parameters.", "Reduce the output generated by irace to a minimum.", "Debug level of the output of irace. Set this to 0 to silence all debug messages. Higher values provide more verbose debug messages.", "Seed of the random number generator (by default, generate a random seed).", "Enable/disable the soft restart strategy that avoids premature convergence of the probabilistic model.", "Soft restart threshold value for numerical parameters. If NA, NULL or \"\", it is computed as 10^-digits.", "Enable/disable elitist irace.", "Number of instances added to the execution list before previous instances in elitist irace.", "In elitist irace, maximum number per race of elimination tests that do not eliminate a configuration. Use 0 for no limit.", "User-defined R function that takes a configuration generated by irace and repairs it.", "Enable the use of adaptive capping, a technique designed for minimizing the computation time of configurations. This is only available when elitist is active.", "Measure used to obtain the execution bound from the performance of the elite configurations: median, mean, worst, best.", "Method to calculate the mean performance of elite configurations: candidate or instance.", "Maximum execution bound for targetRunner. It must be specified when capping is enabled.", "Precision used for calculating the execution time. It must be specified when capping is enabled.", "Penalization constant for timed out executions (executions that reach boundMax execution time).", "Replace the configuration cost of bounded executions with boundMax.", "Percentage of the configuration budget used to perform a postselection race of the best configurations of each iteration after the execution of irace.", "Enable/disable AClib mode. This option enables compatibility with GenericWrapper4AC as targetRunner script.", "Maximum number of iterations.", "Number of runs of the target algorithm per iteration.", "Minimum number of configurations needed to continue the execution of each race (iteration).", "Number of configurations to be sampled and evaluated at each iteration.", "Parameter used to define the number of configurations sampled and evaluated at each iteration.", "Confidence level for the elimination test.")), row.names = c(".help", ".version", ".check", ".init", ".onlytest", "scenarioFile", "execDir", "parameterFile", "forbiddenExps", "forbiddenFile", "initConfigurations", "configurationsFile", "logFile", "recoveryFile", "instances", "trainInstancesDir", "trainInstancesFile", "sampleInstances", "testInstancesDir", "testInstancesFile", "testInstances", "testNbElites", "testIterationElites", "testType", "firstTest", "eachTest", "targetRunner", "targetRunnerLauncher", "targetRunnerLauncherArgs", "targetRunnerRetries", "targetRunnerData", "targetRunnerParallel", "targetEvaluator", "deterministic", "maxExperiments", "maxTime", "budgetEstimation", "minMeasurableTime", "parallel", "loadBalancing", "mpi", "batchmode", "digits", "quiet", "debugLevel", "seed", "softRestart", "softRestartThreshold", "elitist", "elitistNewInstances", "elitistLimit", "repairConfiguration", "capping", "cappingType", "boundType", "boundMax", "boundDigits", "boundPar", "boundAsTimeout", "postselection", "aclib", "nbIterations", "nbExperimentsPerIteration", "minNbSurvival", "nbConfigurations", "mu", "confidence"), class = "data.frame") .irace.params.names <- c("scenarioFile", "execDir", "parameterFile", "forbiddenExps", "forbiddenFile", "initConfigurations", "configurationsFile", "logFile", "recoveryFile", "instances", "trainInstancesDir", "trainInstancesFile", "sampleInstances", "testInstancesDir", "testInstancesFile", "testInstances", "testNbElites", "testIterationElites", "testType", "firstTest", "eachTest", "targetRunner", "targetRunnerLauncher", "targetRunnerLauncherArgs", "targetRunnerRetries", "targetRunnerData", "targetRunnerParallel", "targetEvaluator", "deterministic", "maxExperiments", "maxTime", "budgetEstimation", "minMeasurableTime", "parallel", "loadBalancing", "mpi", "batchmode", "digits", "quiet", "debugLevel", "seed", "softRestart", "softRestartThreshold", "elitist", "elitistNewInstances", "elitistLimit", "repairConfiguration", "capping", "cappingType", "boundType", "boundMax", "boundDigits", "boundPar", "boundAsTimeout", "postselection", "aclib", "nbIterations", "nbExperimentsPerIteration", "minNbSurvival", "nbConfigurations", "mu", "confidence") ## FIXME: If these values are special perhaps they should be saved in $state ? .irace.params.recover <- c("instances", "seed", "testInstances", # We need this because this data may mutate "targetRunnerData", "elitist", "deterministic") irace/R/version.R0000644000176200001440000000016714324731005013351 0ustar liggesusers#' irace.version #' #' A character string containing the version of `irace`. #' @export irace.version <- '3.5.6863679' irace/R/irace-package.R0000644000176200001440000001502714252636656014361 0ustar liggesusers#' The irace package: \packageTitle{irace} #' #' \packageDescription{irace} #' #' @name irace-package #' @docType package #' @import stats utils compiler #' @importFrom R6 R6Class #' @importFrom grDevices dev.new dev.off pdf #' @importFrom graphics abline axis barplot boxplot lines matplot mtext par plot points strwidth text bxp grid #' #' #' @details License: GPL (>= 2) #' #' @author Maintainers: Manuel López-Ibáñez and Leslie Pérez Cáceres #' \email{irace-package@googlegroups.com} #' #' @keywords package optimize tuning automatic configuration #' #' @references #' Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, #' Thomas Stützle, and Mauro Birattari. The irace package: Iterated #' Racing for Automatic Algorithm Configuration. \emph{Operations Research #' Perspectives}, 2016. \doi{10.1016/j.orp.2016.09.002} #' #' Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Thomas Stützle, and Mauro #' Birattari. \emph{The irace package, Iterated Race for Automatic #' Algorithm Configuration}. Technical Report TR/IRIDIA/2011-004, IRIDIA, #' Université Libre de Bruxelles, Belgium, 2011. #' #' Manuel López-Ibáñez and Thomas Stützle. The Automatic Design of #' Multi-Objective Ant Colony Optimization Algorithms. \emph{IEEE Transactions #' on Evolutionary Computation}, 2012. #' #' #' @examples #' ####################################################################### #' # This example illustrates how to tune the parameters of the simulated #' # annealing algorithm (SANN) provided by the optim() function in the #' # R base package. The goal in this example is to optimize instances of #' # the following family: #' # f(x) = lambda * f_rastrigin(x) + (1 - lambda) * f_rosenbrock(x) #' # where lambda follows a normal distribution whose mean is 0.9 and #' # standard deviation is 0.02. f_rastrigin and f_rosenbrock are the #' # well-known Rastrigin and Rosenbrock benchmark functions (taken from #' # the cmaes package). In this scenario, different instances are given #' # by different values of lambda. #' ####################################################################### #' ## First we provide an implementation of the functions to be optimized: #' f_rosenbrock <- function (x) { #' d <- length(x) #' z <- x + 1 #' hz <- z[1:(d - 1)] #' tz <- z[2:d] #' s <- sum(100 * (hz^2 - tz)^2 + (hz - 1)^2) #' return(s) #' } #' f_rastrigin <- function (x) { #' sum(x * x - 10 * cos(2 * pi * x) + 10) #' } #' #' ## We generate 20 instances (in this case, weights): #' weights <- rnorm(20, mean = 0.9, sd = 0.02) #' #' ## On this set of instances, we are interested in optimizing two #' ## parameters of the SANN algorithm: tmax and temp. We setup the #' ## parameter space as follows: #' parameters_table <- ' #' tmax "" i,log (1, 5000) #' temp "" r (0, 100) #' ' #' #' ## We use the irace function readParameters to read this table: #' parameters <- readParameters(text = parameters_table) #' #' ## Next, we define the function that will evaluate each candidate #' ## configuration on a single instance. For simplicity, we restrict to #' ## three-dimensional functions and we set the maximum number of #' ## iterations of SANN to 1000. #' target_runner <- function(experiment, scenario) #' { #' instance <- experiment$instance #' configuration <- experiment$configuration #' #' D <- 3 #' par <- runif(D, min=-1, max=1) #' fn <- function(x) { #' weight <- instance #' return(weight * f_rastrigin(x) + (1 - weight) * f_rosenbrock(x)) #' } #' res <- stats::optim(par,fn, method="SANN", #' control=list(maxit=1000 #' , tmax = as.numeric(configuration[["tmax"]]) #' , temp = as.numeric(configuration[["temp"]]) #' )) #' ## New output interface in irace 2.0. This list may also contain: #' ## - 'time' if irace is called with 'maxTime' #' ## - 'error' is a string used to report an error #' ## - 'outputRaw' is a string used to report the raw output of calls to #' ## an external program or function. #' ## - 'call' is a string used to report how target_runner called the #' ## external program or function. #' return(list(cost = res$value)) #' } #' #' ## We define a configuration scenario by setting targetRunner to the #' ## function define above, instances to the first 10 random weights, and #' ## a maximum budget of 'maxExperiments' calls to targetRunner. #' scenario <- list(targetRunner = target_runner, #' instances = weights[1:10], #' maxExperiments = 500, #' # Do not create a logFile #' logFile = "") #' #' ## We check that the scenario is valid. This will also try to execute #' ## target_runner. #' checkIraceScenario(scenario, parameters = parameters) #' #' \donttest{ #' ## We are now ready to launch irace. We do it by means of the irace #' ## function. The function will print information about its #' ## progress. This may require a few minutes, so it is not run by default. #' tuned_confs <- irace(scenario = scenario, parameters = parameters) #' #' ## We can print the best configurations found by irace as follows: #' configurations.print(tuned_confs) #' #' ## We can evaluate the quality of the best configuration found by #' ## irace versus the default configuration of the SANN algorithm on #' ## the other 10 instances previously generated. #' ## To do so, first we apply the default configuration of the SANN #' ## algorithm to these instances: #' test <- function(configuration) #' { #' res <- lapply(weights[11:20], #' function(x) target_runner( #' experiment = list(instance = x, #' configuration = configuration), #' scenario = scenario)) #' return (sapply(res, getElement, name = "cost")) #' } #' default <- test(data.frame(tmax=10, temp=10)) #' ## We extract and apply the winning configuration found by irace #' ## to these instances: #' tuned <- test(removeConfigurationsMetaData(tuned_confs[1,])) #' #' ## Finally, we can compare using a boxplot the quality obtained with the #' ## default parametrization of SANN and the quality obtained with the #' ## best configuration found by irace. #' boxplot(list(default = default, tuned = tuned)) #' } #' #' @seealso #' \code{\link{irace.main}} to start \pkg{irace} with a given scenario. #' NULL # Prefix for printing messages to the user. .msg.prefix <- "== irace == " irace/R/irace.R0000644000176200001440000015274314321267614012766 0ustar liggesusers# Sets irace variables from a recovery file. It is executed in the # parent environment which must be irace(). # # FIXME: Restoring occurs after reading the command-line/scenario file. At # least for the irace command-line parameters (scenario), it should occur # before. We would need to: # # 1) Read recovery file settings from command-line/scenario file # # 2) if set, then recover irace scenario # 3) then read other settings from command-line/scenario file being # careful to not override with defaults whatever the recovery has set. # # 4) checkSchenario() # # A work-around is to modify the recovery file (you can load it in R, # modify scenario then save it again). recoverFromFile <- function(filename) { # substitute() is needed to evaluate filename here. eval.parent(substitute({ # This restores iraceResults, thus it doesn't need restoring. load (filename) # .Random.seed and .irace are special for (name in setdiff(names(iraceResults$state), c(".Random.seed", ".irace"))) assign(name, iraceResults$state[[name]]) # FIXME: Check that irace.version matches and warn if not. assign(".Random.seed", iraceResults$state$.Random.seed, .GlobalEnv) for (name in ls(iraceResults$state$.irace)) assign(name, get(name, envir = iraceResults$state$.irace), envir = .irace) # These variables are not state, but they are used directly by irace. for (name in c("parameters", "allConfigurations")) assign(name, iraceResults[[name]]) # Restore part of scenario but not all. for (name in .irace.params.recover) scenario[[name]] <- iraceResults$scenario[[name]] options(.race.debug.level = scenario$debugLevel) options(.irace.debug.level = scenario$debugLevel) })) } ## ## Numerical configurations similarity function ## # FIXME: This function is too slow and it shows up in profiles. numeric.configurations.equal <- function(x, configurations, parameters, threshold, param.names) { d <- rep(0.0, nrow(configurations)) isSimilar.mat <- matrix(TRUE, nrow = nrow(configurations), ncol = length(param.names)) selected <- 1:nrow(configurations) for (i in seq_along(param.names)) { param <- param.names[i] x.domain <- getDependentBound(parameters, param, x) x.range <- diff(x.domain) X <- x[[param]] # FIXME: Since at the end we select a subset of configurations, we could use selected here. y <- configurations[, param] ## FIXME: This can probably done much faster by doing a matrix operation that updates ## isSimilar.mat[, i] in one step instead of the for-loop. ## We would need to handle the NAs first. for (j in seq_len(nrow(isSimilar.mat))) { # Configurations loop Y <- y[selected[j]] if (is.na (X) && is.na(Y)) { # Both NA, just ignore this param next } else if (xor(is.na (X), is.na(Y))) { # Distance is 1.0, so not equal isSimilar.mat[j,i] <- FALSE } else { # FIXME: Why is this updating d[j]? It seems that if the difference is # large for one configuration, then it will be assumed to be large for # the rest. if (parameters$isDependent[param]) { # Compare depedent domains by normalising their values to their own ranges first # and calculating the difference. (When possible) y.domain <- getDependentBound(parameters, param, configurations[selected[j],]) y.range <- diff(x.domain) dx <- ifelse (x.range == 0, 0, (as.numeric(X) - x.domain[1]) / x.range) dy <- ifelse (y.range == 0, 0, (as.numeric(Y) - y.domain[1]) / y.range) d[j] <- max(d[j], abs(dx - dy)) } else { # FIXME: We should calculate (X - x.domain[1]) / x.range once for all configurations # and all parameters, then calculate the differences using vectorization. d[j] <- max(d[j], abs((as.numeric(X) - as.numeric(Y)) / x.range)) } if (d[j] > threshold) isSimilar.mat[j,i] <- FALSE } } index <- which(apply(isSimilar.mat,1,all)) isSimilar.mat <- isSimilar.mat[index, , drop=FALSE] d <- d[index] selected <- selected[index] if (nrow(isSimilar.mat) == 0) break } if (length(selected) != 0) return(c(x[[".ID."]], configurations[selected,".ID."])) return(NULL) } ## ## Identify which configurations are similar. ## # FIXME: It would be nice to print the minimum similarity found to the user. similarConfigurations <- function(configurations, parameters, threshold) { debug.level <- getOption(".irace.debug.level", 0) if (debug.level >= 1) irace.note ("Computing similarity of configurations .") # Create vectors of categorical and numerical p <- parameters$types %in% c("c","o") vecCat <- parameters$names[p & !parameters$isFixed] vecNum <- parameters$names[!p & !parameters$isFixed] irace.assert(all(parameters$types[vecCat] %in% c("c","o"))) irace.assert(all(!(parameters$types[vecNum] %in% c("c","o")))) irace.assert(length(intersect(vecCat, vecNum)) == 0) nbCater <- length(vecCat) nbNumer <- length(vecNum) ### Categorical/Ordinal filtering #### if (nbCater > 0) { ## Build a vector with the categorical appended together in a string strings <- do.call(paste, c(configurations[, vecCat, drop=FALSE], sep = " ; ")) if (nbNumer != 0) configurations <- configurations[, c(".ID.", vecNum)] ord.strings <- order(strings) configurations <- configurations[ord.strings, ] strings <- strings[ord.strings] ## keep similar (index i == true means is the same as i + 1) similarIdx <- strings[-length(strings)] == strings[-1] ## Now let's get just a FALSE if we remove it, TRUE otherwise: keepIdx <- c(similarIdx[1], (similarIdx[-1] | similarIdx[-length(similarIdx)]), similarIdx[length(similarIdx)]) ## filtering them out: configurations <- configurations [keepIdx, , drop=FALSE] ## filtering their strings out (to use them to define blocks): strings <- strings[keepIdx] ## if everything is already filtered out, return if (nrow(configurations) == 0) { if (debug.level >= 1) cat(" DONE\n") return(NULL) } } ### Numerical parameters within blocks of the same string ### if (nbNumer > 0) { similar <- c() if (nbCater > 0) { ## In this case the object "string" is available to define blocks ## Loop over blocks: beginBlock <- 1 while (beginBlock < nrow(configurations)) { ## The current block is made of all configurations that have the same ## categorical string as the one of configuration[beginBlock, ] blockIds <- which(strings == strings[beginBlock]) endBlock <- blockIds[length(blockIds)] irace.assert (endBlock > beginBlock) ## Loop inside blocks: for (i in seq(beginBlock, endBlock - 1)) { ## Compare configuration i with all the ones that are after in the block similar <- c(similar, numeric.configurations.equal(configurations[i, ], configurations[(i+1):endBlock,], parameters, threshold = threshold, param.names = vecNum)) if (debug.level >= 1) cat(".") } beginBlock <- endBlock + 1 # Next block starts after the end of the current one } } else { ## No categorical, so no blocks, just do the basic check without blocks for (i in seq_len(nrow(configurations) - 1)) { similar <- c(similar, numeric.configurations.equal(configurations[i, ], configurations[(i+1):nrow(configurations),], parameters, threshold = threshold, param.names = vecNum)) if (debug.level >= 1) cat(".") } } # FIXME: We have to use unique because we return the same configuration # more than once in different calls to numeric.configurations.equal. # Currently, we compare each configuration k=1...n with every configuration # k+1...n. Instead, we should compare k=1...n with ((k+1...n) notin # similar). It may happen that A ~ B and A ~ C and B /= C, but this is OK # because we still return A, B and C. It may also happen that A ~ B, B ~ C # and A /= C, but this is also OK because we will compare A with B,C then B # with C. similar <- unique(similar) configurations <- configurations[configurations[, ".ID."] %in% similar, ] } if (debug.level >= 1) cat(" DONE\n") if (nrow(configurations) == 0) { return (NULL) } else { return(configurations[,".ID."]) } } ## Number of iterations. computeNbIterations <- function(nbParameters) (2 + log2(nbParameters)) ## Computational budget at each iteration. computeComputationalBudget <- function(remainingBudget, indexIteration, nbIterations) { return (floor (remainingBudget / (nbIterations - indexIteration + 1))) } ## The number of configurations computeNbConfigurations <- function(currentBudget, indexIteration, firstTest, eachTest, nElites = 0, nOldInstances = 0, newInstances = 0, maxConfigurations = 1024) { # FIXME: This is slightly incorrect, because we may have elites that have not # been executed on all nOldInstances. Thus, we need to pass explicitly the # budget that we save (that is, number of entries that are not NA). savedBudget <- nElites * nOldInstances n <- max (firstTest + eachTest * min(5, indexIteration), round.to.next.multiple(nOldInstances + newInstances, eachTest)) return (min (floor ((currentBudget + savedBudget) / n), maxConfigurations) ) } ## Termination of a race at each iteration. The race will stop if the ## number of surviving configurations is equal or less than this number. computeTerminationOfRace <- function(nbParameters) (2 + log2(nbParameters)) ## Compute the minimum budget required, and exit early in case the ## budget given by the user is insufficient. computeMinimumBudget <- function(scenario, minSurvival, nbIterations, boundEstimate) { eachTest <- scenario$eachTest Tnew <- scenario$elitistNewInstances mu <- max(scenario$mu, scenario$firstTest) # This is computed from the default formulas as follows: # B_1 = B / I # B_2 = B - (B/I) / (I - 1) = B / I # B_3 = B - 2(B/I) / (I - 2) = B / I # thus # B_i = B / I # and # C_i = B_i / T_i = B / (I * T_i). # # We want to enforce that C_i >= min_surv + 1, thus # B / (I * T_i) >= min_surv + 1 (1) # becomes # B >= (min_surv + 1) * I * T_i # # This is an over-estimation, since actually B_1 = floor(B/I) and if # floor(B/I) < B/I, then B_i < B/I, and we could still satisfy Eq. (1) # with a smaller budget. However, the exact formula requires computing B_i # taking into account the floor() function, which is not obvious. minimumBudget <- (minSurvival + 1) * nbIterations # We need to compute T_i: if (scenario$elitist) { # T_i = max(mu + Teach * min (5, i), # ceiling((T_{i-1} + Tnew) / Teach) * Teach) # T_1 = mu + Teach # T_2 ~ mu + Teach + max (Teach, Tnew) # T_3 ~ max(mu + 3 * Teach, # mu + Teach + max(Teach, Tnew) + T_new) # = mu + Teach + max(Teach + max(Teach, Tnew), 2 * Tnew) # if Teach > Tnew then 2*Teach > 2*Tnew then max = 2*Teach # if Teach < Tnew then Teach + Tnew < 2*Tnew then max = 2*Tnew # hence: T_3 = mu + Teach + 2 * max(Teach, Tnew) # T_4 = max(mu + 4 * Teach, # ceiling((mu + Teach + 2 * max(Teach, Tnew)) + Tnew) / Teach) * Teach) # ~ mu + Teach + max(2 * Teach + max(Teach, Tnew), 3 * Tnew) # = mu + Teach + 3 * max(Teach, Tnew) # T_i = mu + Teach + (i - 1) * max(Teach, Tnew) # T_6 = max (mu + 5*Teach, # mu + Teach + 5 * max(Teach, Tnew) + Tnew) # = mu + Teach + Tnew + 5 * max (Teach, Tnew) # T_i = mu + Teach + max(I-5, 0) * Tnew + 5 * max (Teach, Tnew) if (nbIterations > 5) { minimumBudget <- minimumBudget * (mu + eachTest + (nbIterations - 5) * Tnew + 5 * max(eachTest, Tnew)) } else { minimumBudget <- minimumBudget * (mu + eachTest + (nbIterations - 1) * max(eachTest, Tnew)) } } else { # T_i = mu + T_each * min (5, i) # and the most strict value is for i >= 5, thus # B >= (min_surv + 1) * I * (mu + 5 * T_each) minimumBudget <- minimumBudget * (mu + 5 * eachTest) } return(minimumBudget) } checkMinimumBudget <- function(scenario, remainingBudget, minSurvival, nbIterations, boundEstimate, timeUsed) { minimumBudget <- computeMinimumBudget (scenario, minSurvival, nbIterations, boundEstimate) if (remainingBudget < minimumBudget) { if (scenario$maxTime == 0) { irace.error("Insufficient budget: ", "With the current settings, irace will require a value of ", "'maxExperiments' of at least '", minimumBudget, "'.") } else if (nbIterations == 1) { irace.error("Insufficient budget: ", "With the current settings and estimated time per run (", boundEstimate, ") irace will require a value of 'maxTime' of at least '", (minimumBudget * boundEstimate) + timeUsed, "'.") } return(FALSE) } return(TRUE) } startParallel <- function(scenario) { cwd <- setwd (scenario$execDir) on.exit(setwd(cwd), add = TRUE) parallel <- scenario$parallel if (parallel > 1) { if (scenario$mpi) { mpiInit(parallel, scenario$debugLevel) } else { requireNamespace("parallel", quietly = TRUE) if (.Platform$OS.type == 'windows' && is.null(.irace$cluster)) { # FIXME: makeCluster does not print the output generated by the workers # on Windows. We need to use the future package for that: # https://stackoverflow.com/questions/56501937/how-to-print-from-clusterapply .irace$cluster <- parallel::makeCluster(parallel) if (scenario$debugLevel >= 1) irace.note("makeCluster initialized for ", parallel, " jobs.") # In Windows, this needs to be exported, or we get: ## Error in checkForRemoteErrors(val) : ## 2 nodes produced errors; first error: could not find function "target.runner" parallel::clusterExport(.irace$cluster, ls(environment(startParallel)), envir=environment(startParallel)) parallel::clusterExport(.irace$cluster, list("target.runner"), envir=.irace) # In addition, we export the global environment because the user may # have defined stuff there. There must be a better way to do this, but # I cannot figure it out. R sucks sometimes. parallel::clusterExport(.irace$cluster, ls(envir=.GlobalEnv)) } } } } stopParallel <- function() { if (!is.null(.irace$cluster)) { try(parallel::stopCluster(.irace$cluster), silent=TRUE) .irace$cluster <- NULL } } irace.init <- function(scenario) { # We need to do this here to use/recover .Random.seed later. if (is.na(scenario$seed)) { # FIXME: We should store this seed in state not in scenario. We should not modify scenario. scenario$seed <- trunc(runif(1, 1, .Machine$integer.max)) } set.seed(scenario$seed) if (scenario$debugLevel > 2) { irace.note("RNGkind: ", paste0(RNGkind(), collapse = " "), "\n") irace.note(".Random.seed: ", paste0(.Random.seed, collapse = ", "), "\n") } scenario } ## Generate instances + seed. generateInstances <- function(scenario, remainingBudget) { instances <- scenario$instances ntimes <- if (scenario$deterministic) 1 else # "Upper bound"" of instances needed # FIXME: We could bound it even further if maxExperiments >> nInstances ceiling (remainingBudget / length(instances)) # Get instances order if (scenario$sampleInstances) { # Sample instances index in groups (ntimes) sindex <- as.vector(sapply(rep(length(instances), ntimes), sample.int, replace = FALSE)) } else { sindex <- rep(1L:length(instances), ntimes) } # Sample seeds. # 2147483647 is the maximum value for a 32-bit signed integer. # We use replace = TRUE, because replace = FALSE allocates memory for each possible number. data.frame(instance = sindex, seed = sample.int(2147483647L, size = length(sindex), replace = TRUE), stringsAsFactors=FALSE) } addInstances <- function(scenario, instancesList, n.instances) { # Generate instance + seed list if (is.null.or.empty(instancesList)) instancesList <- generateInstances(scenario, n.instances) # If deterministic, we have already added all instances. else if (!scenario$deterministic) instancesList <- rbind(instancesList, generateInstances(scenario, n.instances)) # FIXME: Something is adding rownames. Clear them to avoid future problems. rownames(instancesList) <- NULL return(instancesList) } ## Estimate the mean execution time do.experiments <- function(configurations, ninstances, scenario, parameters) { output <- lapply(1:ninstances, race.wrapper, configurations = configurations, bounds = rep(scenario$boundMax, nrow(configurations)), which.alive = 1:nrow(configurations), which.exe = 1:nrow(configurations), parameters = parameters, scenario = scenario) Results <- matrix(nrow = ninstances, ncol = nrow(configurations), dimnames = list(1:ninstances, as.character(configurations[, ".ID."]))) experimentLog <- matrix(nrow = 0, ncol = 4, dimnames = list(NULL, c("instance", "configuration", "time", "bound"))) # Extract results for (j in seq_len(ninstances)) { vcost <- unlist(lapply(output[[j]], "[[", "cost")) if (scenario$capping) vcost <- applyPAR(vcost, boundMax = scenario$boundMax, boundPar = scenario$boundPar) Results[j, ] <- vcost vtimes <- unlist(lapply(output[[j]], "[[", "time")) irace.assert(!any(is.null(vtimes))) experimentLog <- rbind(experimentLog, cbind(j, configurations$.ID., vtimes, if(!is.null(scenario$boundMax)) scenario$boundMax else NA)) } rejectedIDs <- configurations[apply(is.infinite(Results), 2, any), ".ID."] return (list(experiments = Results, experimentLog = experimentLog, rejectedIDs = rejectedIDs)) } ## Gets the elite configurations time matrix from the experiment log generateTimeMatrix <- function(elites, experimentLog) { is.elite <- experimentLog[,"configuration"] %in% elites$.ID. # Remove everything that we don't need. experimentLog <- experimentLog[is.elite, c("configuration", "instance", "time", "bound"), drop = FALSE] experimentLog[, "time"] <- pmin(experimentLog[,"time"], experimentLog[, "bound"]) # FIXME: It would be better to use spread() from tidyr resultsTime <- reshape(as.data.frame(experimentLog), direction = "wide", idvar = "instance", timevar = "configuration", drop = "bound") rownames(resultsTime) <- resultsTime$instance resultsTime <- resultsTime[order(resultsTime$instance), , drop = FALSE] colnames(resultsTime) <- substring(colnames(resultsTime), nchar("time.") + 1) resultsTime <- as.matrix(resultsTime[, as.character(elites$.ID.), drop = FALSE]) return(resultsTime) } ## Initialize allConfigurations with any initial configurations provided. allConfigurationsInit <- function(scenario, parameters) { initConfigurations <- scenario$initConfigurations confs_from_file <- NULL if (!is.null.or.empty(scenario$configurationsFile)) { confs_from_file <- readConfigurationsFile(scenario$configurationsFile, parameters, scenario$debugLevel) } if (!is.null.or.empty(initConfigurations)) { if (!identical(initConfigurations, confs_from_file)) irace.warning("'initConfigurations' provided in 'scenario',", " thus ignoring configurations from file '", scenario$configurationsFile, "'.") cat("# Adding", nrow(initConfigurations), "initial configuration(s)\n") if (scenario$debugLevel >= 2) print(as.data.frame(scenario$initConfigurations, stringsAsFactors = FALSE), digits=15) } else { initConfigurations <- confs_from_file } if (!is.null.or.empty(initConfigurations)) { allConfigurations <- initConfigurations allConfigurations <- cbind(.ID. = 1:nrow(allConfigurations), allConfigurations, .PARENT. = NA) rownames(allConfigurations) <- allConfigurations$.ID. num <- nrow(allConfigurations) allConfigurations <- checkForbidden(allConfigurations, scenario$forbiddenExps) if (nrow(allConfigurations) < num) { irace.warning(num - nrow(allConfigurations), " of the ", num, " initial configurations were forbidden", " and, thus, discarded") } } else { configurations.colnames <- c(".ID.", names(parameters$conditions), ".PARENT.") allConfigurations <- as.data.frame(matrix(ncol = length(configurations.colnames), nrow = 0, dimnames = list(NULL, configurations.colnames)), stringsAsFactors=FALSE) } allConfigurations } #' irace #' #' `irace` implements iterated Race. It receives some parameters to be tuned #' and returns the best configurations found, namely, the elite configurations #' obtained from the last iterations (and sorted by rank). #' #' @template arg_scenario #' @template arg_parameters #' #' @details The function `irace` executes the tuning procedure using #' the information provided in `scenario` and `parameters`. Initially it checks #' the correctness of `scenario` and recovers a previous execution if #' `scenario$recoveryFile` is set. A R data file log of the execution is created #' in `scenario$logFile`. #' #' @template return_irace #' @examples #' \dontrun{ #' parameters <- readParameters("parameters.txt") #' scenario <- readScenario(filename = "scenario.txt") #' irace(scenario = scenario, parameters = parameters) #' } #' #' @seealso #' \describe{ #' \item{[irace.main()]}{a higher-level interface to `irace`.} #' \item{[irace.cmdline()]}{a command-line interface to `irace`.} #' \item{[readScenario()]}{for reading a configuration scenario from a file.} #' \item{[readParameters()]}{read the target algorithm parameters from a file.} #' \item{[defaultScenario()]}{returns the default scenario settings of \pkg{irace}.} #' \item{[checkScenario()]}{to check that the scenario is valid.} #' } #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @concept running #' @export irace <- function(scenario, parameters) irace_common(scenario, parameters, simple = TRUE) irace_common <- function(scenario, parameters, simple, output.width = 9999L) { if (!simple) { op <- options(width = output.width) # Do not wrap the output. on.exit(options(op), add = TRUE) } scenario <- checkScenario(scenario) debugLevel <- scenario$debugLevel if (debugLevel >= 1) { op.debug <- options(warning.length = 8170, error = if (interactive()) utils::recover else irace.dump.frames) on.exit(options(op.debug), add = TRUE) printScenario (scenario) } if (missing(parameters)) { # Read parameters definition parameters <- readParameters (file = scenario$parameterFile, digits = scenario$digits, debugLevel = debugLevel) } else { parameters <- checkParameters(parameters) } eliteConfigurations <- irace_run(scenario = scenario, parameters = parameters) if (simple) return(eliteConfigurations) if (!scenario$quiet) { cat("# Best configurations (first number is the configuration ID;", " listed from best to worst according to the ", test.type.order.str(scenario$testType), "):\n", sep = "") configurations.print(eliteConfigurations) cat("# Best configurations as commandlines (first number is the configuration ID; same order as above):\n") configurations.print.command (eliteConfigurations, parameters) } if (scenario$postselection > 0) psRace(iraceLogFile=scenario$logFile, postselection=scenario$postselection, elites=TRUE) testing_fromlog(logFile = scenario$logFile) invisible(eliteConfigurations) } irace_run <- function(scenario, parameters) { quiet <- scenario$quiet catInfo <- if (quiet) do_nothing else function(..., verbose = TRUE) { irace.note (..., "\n") if (verbose) { cat ("# Iteration: ", indexIteration, "\n", "# nbIterations: ", nbIterations, "\n", "# experimentsUsedSoFar: ", experimentsUsedSoFar, "\n", "# timeUsed: ", timeUsed, "\n", "# remainingBudget: ", remainingBudget, "\n", "# currentBudget: ", currentBudget, "\n", "# number of elites: ", nrow(eliteConfigurations), "\n", "# nbConfigurations: ", nbConfigurations, "\n", sep = "") } } irace_finish <- function(iraceResults, scenario, reason) { elapsed <- timer$elapsed() if (!quiet) cat("# Total CPU user time: ", elapsed["user"], ", CPU sys time: ", elapsed["system"], ", Wall-clock time: ", elapsed["wallclock"], "\n", sep="") iraceResults$state$elapsed = elapsed iraceResults$state$completed = reason irace_save_logfile(iraceResults, scenario) iraceResults$state$eliteConfigurations } timer <- Timer$new() debugLevel <- scenario$debugLevel # Recover state from file? if (!is.null.or.empty(scenario$recoveryFile)) { irace.note ("Resuming from file: '", scenario$recoveryFile,"'\n") recoverFromFile(scenario$recoveryFile) # We call checkScenario again to fix any inconsistencies in the recovered # data. # FIXME: Do not call checkScenario earlier and instead do the minimum to check recoveryFile. scenario <- checkScenario(scenario) firstRace <- FALSE stopParallel() startParallel(scenario) on.exit(stopParallel(), add = TRUE) } else { # Do not recover firstRace <- TRUE scenario <- irace.init(scenario) forbiddenExps <- scenario$forbiddenExps # Set options controlling debug level. # FIXME: This should be the other way around, the options set the debugLevel. options(.race.debug.level = debugLevel) options(.irace.debug.level = debugLevel) # Create a data frame of all configurations ever generated. allConfigurations <- allConfigurationsInit(scenario, parameters) nbUserConfigurations <- nrow(allConfigurations) # To save the logs iraceResults <- list( scenario = scenario, irace.version = irace.version, parameters = parameters, allElites = list(), experiments = matrix(nrow = 0, ncol = 0), experimentLog = matrix(nrow = 0, ncol = 5, dimnames = list(NULL, c("iteration", "instance", "configuration", "time", "bound"))) ) model <- NULL nbConfigurations <- 0 eliteConfigurations <- data.frame(stringsAsFactors=FALSE) nbIterations <- ifelse (scenario$nbIterations == 0, computeNbIterations(parameters$nbVariable), scenario$nbIterations) nbIterations <- floor(nbIterations) minSurvival <- ifelse (scenario$minNbSurvival == 0, computeTerminationOfRace(parameters$nbVariable), scenario$minNbSurvival) minSurvival <- floor(minSurvival) # Generate initial instance + seed list .irace$instancesList <- addInstances(scenario, NULL, if (scenario$maxExperiments != 0) ceiling(scenario$maxExperiments / minSurvival) else max (scenario$firstTest, length(scenario$instances))) indexIteration <- 1 experimentsUsedSoFar <- 0 timeUsed <- 0 boundEstimate <- NA rejectedIDs <- c() startParallel(scenario) on.exit(stopParallel(), add = TRUE) if (scenario$maxTime == 0) { remainingBudget <- scenario$maxExperiments } else { ## Estimate time when maxTime is defined. # Get the number of instances to be used. ## IMPORTANT: This is firstTest because these configurations will be ## considered elite later, thus preserved up to firstTest, which is ## fine. If a larger number of instances is used, it would prevent ## discarding these configurations. ninstances <- scenario$firstTest estimationTime <- ceiling(scenario$maxTime * scenario$budgetEstimation) irace.note("Estimating execution time using ", 100 * scenario$budgetEstimation, "% of ", scenario$maxTime, " = ", estimationTime, "\n") # Estimate the number of configurations to be used nconfigurations <- max(2, floor(scenario$parallel / ninstances)) next.configuration <- 1 while (TRUE) { # Sample new configurations if needed if (nrow(allConfigurations) < nconfigurations) { newConfigurations <- sampleUniform(parameters, nconfigurations - nrow(allConfigurations), digits = scenario$digits, forbidden = forbiddenExps, repair = scenario$repairConfiguration) newConfigurations <- cbind (.ID. = max(0, allConfigurations$.ID.) + 1:nrow(newConfigurations), newConfigurations) allConfigurations <- rbind(allConfigurations, newConfigurations) rownames(allConfigurations) <- allConfigurations$.ID. } # Execute tests output <- do.experiments(configurations = allConfigurations[next.configuration:nconfigurations, ], ninstances = ninstances, scenario = scenario, parameters = parameters) iraceResults$experimentLog <- rbind(iraceResults$experimentLog, # These experiments are assigned iteration 0 cbind(iteration=0L, output$experimentLog)) iraceResults$experiments <- merge.matrix (iraceResults$experiments, output$experiments) rownames(iraceResults$experiments) <- 1:nrow(iraceResults$experiments) rejectedIDs <- c(rejectedIDs, output$rejectedIDs) iraceResults$rejectedConfigurations <- rejectedIDs forbiddenExps <- c(forbiddenExps, buildForbiddenExp(configurations = allConfigurations[ allConfigurations$.ID. %in% output$rejectedIDs, , drop = FALSE], parameters = parameters)) # For the used time, we count the time reported in all configurations # including rejected ones. timeUsed <- sum(timeUsed, output$experimentLog[, "time"], na.rm = TRUE) # User should return time zero for rejectedIDs. boundEstimate <- mean(iraceResults$experimentLog[, "time"], na.rm = TRUE) if (boundEstimate <= 0) boundEstimate <- if (!is.null(scenario$boundMax)) scenario$boundMax else 1.0 next.configuration <- nconfigurations + 1L # Calculate how many new configurations: # 1. We do not want to overrun estimationTime new.conf <- floor(((estimationTime - timeUsed) / boundEstimate) / ninstances) # 2. But there is no point in executing more configurations than those # that we can execute in parallel. new.conf <- min(new.conf, max(1, floor(scenario$parallel / ninstances))) if (timeUsed >= estimationTime || new.conf == 0 || nconfigurations == 1024) { break } else { nconfigurations <- min(1024, nconfigurations + new.conf) } } # end of while(TRUE) if (length(rejectedIDs) > 0) { irace.note ("Immediately rejected configurations: ", paste0(rejectedIDs, collapse = ", ") , "\n") } # Update budget remainingBudget <- round((scenario$maxTime - timeUsed) / boundEstimate) experimentsUsedSoFar <- experimentsUsedSoFar + nrow(iraceResults$experimentLog) eliteConfigurations <- allConfigurations[allConfigurations$.ID. %!in% rejectedIDs, ,drop = FALSE] # Without elitist, the racing does not re-use the results computed during # the estimation. This means that the time used during estimation needs # to be spent again during racing, thus leaving less time for racing. We # want to avoid having less time for racing, and this is an # implementation detail, thus we assume that the time was not actually # wasted. if (!scenario$elitist) timeUsed <- 0 irace.note("Estimated execution time is ", boundEstimate, " based on ", next.configuration - 1, " configurations and ", ninstances," instances. Used time: ", timeUsed, ", remaining time: ", (scenario$maxTime - timeUsed), ", remaining budget (experiments): ", remainingBudget, "\n") } # end of time estimation # Compute the total initial budget, that is, the maximum number of # experiments that we can perform. currentBudget <- ifelse (scenario$nbExperimentsPerIteration == 0, computeComputationalBudget(remainingBudget, indexIteration, nbIterations), scenario$nbExperimentsPerIteration) # Check that the budget is enough, for the time estimation case we reduce # the number of iterations. warn_msg <- NULL while (!checkMinimumBudget(scenario, remainingBudget, minSurvival, nbIterations, boundEstimate, timeUsed)) { if (is.null(warn_msg)) warn_msg <- paste0("with the current settings and estimated time per run (", boundEstimate, ") irace will not have enough budget to execute the minimum", " number of iterations (", nbIterations, "). ", "Execution will continue by assuming that the estimated time", " is too high and reducing the minimum number of iterations,", " however, if the estimation was correct or too low,", " results might not be better than random sampling.\n") nbIterations <- nbIterations - 1 } if (!is.null(warn_msg)) irace.warning(warn_msg) } #end of do not recover catInfo("Initialization\n", if (scenario$elitist) paste0("# Elitist race\n", "# Elitist new instances: ", scenario$elitistNewInstances, "\n", "# Elitist limit: ", scenario$elitistLimit, "\n") else paste0("# Non-elitist race\n"), "# nbIterations: ", nbIterations, "\n", "# minNbSurvival: ", minSurvival, "\n", "# nbParameters: ", parameters$nbVariable, "\n", "# seed: ", scenario$seed, "\n", "# confidence level: ", scenario$confidence, "\n", "# budget: ", remainingBudget, "\n", if (scenario$maxTime == 0) "" else paste0("# time budget: ", scenario$maxTime - timeUsed, "\n"), "# mu: ", max(scenario$mu, scenario$firstTest), "\n", "# deterministic: ", scenario$deterministic, "\n", if (scenario$capping) paste0("# capping: ", scenario$cappingType, "\n", "# type bound: ", scenario$boundType, "\n", "# boundMax: ", scenario$boundMax, "\n", "# par bound: ", scenario$boundPar, "\n", "# bound digits: ", scenario$boundDigits, "\n") else if (!is.null(scenario$boundMax)) paste0("# boundMax: ", scenario$boundMax, "\n"), verbose = FALSE) while (TRUE) { # Recovery info iraceResults$state <- list(.Random.seed = get(".Random.seed", .GlobalEnv), .irace = .irace, currentBudget = currentBudget, debugLevel = debugLevel, eliteConfigurations = eliteConfigurations, experimentsUsedSoFar = experimentsUsedSoFar, indexIteration = indexIteration, minSurvival = minSurvival, model = model, nbConfigurations = nbConfigurations, nbIterations = nbIterations, remainingBudget = remainingBudget, timeUsed = timeUsed, boundEstimate = boundEstimate, rejectedIDs = rejectedIDs, forbiddenExps = forbiddenExps, completed = "Incomplete") ## Save to the log file iraceResults$allConfigurations <- allConfigurations irace_save_logfile(iraceResults, scenario) # Consistency checks irace.assert(nrow(iraceResults$experimentLog) == experimentsUsedSoFar) # With elitist=0 we may re-run the same configuration on the same (instance,seed) pair # FIXME: This assert is failing when sampleInstances=FALSE because we are re-executing again an instance that was executed at the start of the race. if (FALSE && scenario$elitist) irace.assert(sum(!is.na(iraceResults$experiments)) == experimentsUsedSoFar) if (remainingBudget <= 0) { catInfo("Stopped because budget is exhausted") return(irace_finish(iraceResults, scenario, reason = "Budget exhausted")) } if (scenario$maxTime > 0 && timeUsed >= scenario$maxTime) { catInfo("Stopped because time budget is exhausted") return(irace_finish(iraceResults, scenario, reason = "Time budget exhausted")) } if (indexIteration > nbIterations) { if (scenario$nbIterations == 0) { nbIterations <- indexIteration } else { if (debugLevel >= 1) { catInfo("Limit of iterations reached", verbose = FALSE) } return(irace_finish(iraceResults, scenario, reason = "Limit of iterations reached")) } } # Compute the current budget (nb of experiments for this iteration), # or take the value given as parameter. currentBudget <- ifelse (scenario$nbExperimentsPerIteration == 0, computeComputationalBudget(remainingBudget, indexIteration, nbIterations), scenario$nbExperimentsPerIteration) # Compute the number of configurations for this race. if (scenario$elitist && !firstRace) { nbConfigurations <- computeNbConfigurations(currentBudget, indexIteration, firstTest = max(scenario$mu, scenario$firstTest), eachTest = scenario$eachTest, nElites = nrow(eliteConfigurations), nOldInstances = nrow(iraceResults$experiments), newInstances = scenario$elitistNewInstances) } else { nbConfigurations <- computeNbConfigurations(currentBudget, indexIteration, firstTest = max(scenario$mu, scenario$firstTest), eachTest = scenario$eachTest, nElites = 0, nOldInstances = 0, newInstances = 0) } # If a value was given as a parameter, then this value limits the maximum, # but if we have budget only for less than this, then we have run out of # budget. if (scenario$nbConfigurations > 0) { if (scenario$nbConfigurations <= nbConfigurations) { nbConfigurations <- scenario$nbConfigurations } else if (currentBudget < remainingBudget) { # We skip one iteration catInfo("Not enough budget for this iteration, ", "skipping to the next one.") indexIteration <- indexIteration + 1 next } else { catInfo("Stopped because ", "there is not enough budget to enforce the value of nbConfigurations.") return(irace_finish(iraceResults, scenario, reason = "Not enough budget to enforce the value of nbConfigurations")) } } # Stop if the number of configurations to test is NOT larger than the minimum. if (nbConfigurations <= minSurvival) { catInfo("Stopped because there is not enough budget left to race more than ", "the minimum (", minSurvival,")\n", "# You may either increase the budget or set 'minNbSurvival' to a lower value") return(irace_finish(iraceResults, scenario, reason = "Not enough budget to race more than the minimum configurations")) } # If we have too many eliteConfigurations, reduce their number. This can # happen before the first race due to the initial budget estimation. if (firstRace) { if (nbConfigurations < nrow(eliteConfigurations)) { eliteRanks <- overall.ranks(iraceResults$experiments, stat.test = scenario$testType) eliteConfigurations <- eliteConfigurations[order(eliteRanks), ] eliteConfigurations <- eliteConfigurations[1:nbConfigurations, ] } } else if (nbConfigurations <= nrow(eliteConfigurations)) { # Stop if the number of configurations to produce is not greater than # the number of elites. catInfo("Stopped because ", "there is not enough budget left to race newly sampled configurations") #(number of elites + 1) * (mu + min(5, indexIteration)) > remainingBudget" return(irace_finish(iraceResults, scenario, reason = "Not enough budget left to race newly sampled configurations")) } if (scenario$elitist) { # The non-elite have to run up to the first test. The elites consume # budget at most up to the new instances. if ((nbConfigurations - nrow(eliteConfigurations)) * max(scenario$mu, scenario$firstTest) + nrow(eliteConfigurations) * min(scenario$elitistNewInstances, max(scenario$mu, scenario$firstTest)) > currentBudget) { catInfo("Stopped because there is not enough budget left to race all configurations up to the first test (or mu)") return(irace_finish(iraceResults, scenario, reason = "Not enough budget to race all configurations up to the first test (or mu)")) } } else if (nbConfigurations * max(scenario$mu, scenario$firstTest) > currentBudget) { catInfo("Stopped because there is not enough budget left to race all configurations up to the first test (or mu)") return(irace_finish(iraceResults, scenario, reason = "Not enough budget to race all configurations up to the first test (or mu)")) } catInfo("Iteration ", indexIteration, " of ", nbIterations, "\n", "# experimentsUsedSoFar: ", experimentsUsedSoFar, "\n", if (scenario$maxTime == 0) "" else paste0("# timeUsed: ", timeUsed, "\n", "# boundEstimate: ", boundEstimate, "\n"), "# remainingBudget: ", remainingBudget, "\n", "# currentBudget: ", currentBudget, "\n", "# nbConfigurations: ", nbConfigurations, verbose = FALSE) iraceResults$softRestart[indexIteration] <- FALSE # Sample for the first time. if (firstRace) { # If we need more configurations, sample uniformly. nbNewConfigurations <- nbConfigurations - sum(allConfigurations$.ID. %!in% rejectedIDs) if (nbNewConfigurations > 0) { # Sample new configurations. if (debugLevel >= 1) { catInfo("Sample ", nbNewConfigurations, " configurations from uniform distribution", verbose = FALSE) } newConfigurations <- sampleUniform(parameters, nbNewConfigurations, digits = scenario$digits, forbidden = forbiddenExps, repair = scenario$repairConfiguration) newConfigurations <- cbind (.ID. = max(0, allConfigurations$.ID.) + 1:nrow(newConfigurations), newConfigurations) allConfigurations <- rbind(allConfigurations, newConfigurations) rownames(allConfigurations) <- allConfigurations$.ID. raceConfigurations <- allConfigurations[allConfigurations$.ID. %!in% rejectedIDs, , drop = FALSE] } else if (nbNewConfigurations <= 0) { # We let the user know that not all configurations will be used. if (nbUserConfigurations > nbConfigurations) { catInfo("Only ", nbConfigurations, " from the initial configurations will be used", verbose = FALSE) } # This is made only in case that the number of configurations used in # the time estimation is more than needed. if (nrow(eliteConfigurations) == nbConfigurations) { raceConfigurations <- eliteConfigurations } else { raceConfigurations <- allConfigurations[allConfigurations$.ID. %!in% rejectedIDs, , drop = FALSE] raceConfigurations <- raceConfigurations[1:nbConfigurations,] } } # end of indexIteration == 1 } else { # How many new configurations should be sampled? nbNewConfigurations <- nbConfigurations - nrow(eliteConfigurations) # Update the model based on elites configurations if (debugLevel >= 1) irace.note("Update model\n") model <- updateModel(parameters, eliteConfigurations, model, indexIteration, nbIterations, nbNewConfigurations, scenario) if (debugLevel >= 2) printModel (model) if (debugLevel >= 1) irace.note("Sample ", nbNewConfigurations, " configurations from model\n") newConfigurations <- sampleModel(parameters, eliteConfigurations, model, nbNewConfigurations, digits = scenario$digits, forbidden = forbiddenExps, repair = scenario$repairConfiguration) # Set ID of the new configurations. newConfigurations <- cbind (.ID. = max(0, allConfigurations$.ID.) + 1:nrow(newConfigurations), newConfigurations) raceConfigurations <- rbind(eliteConfigurations[, colnames(newConfigurations)], newConfigurations) rownames(raceConfigurations) <- raceConfigurations$.ID. if (scenario$softRestart) { # Rprof("profile.out") tmp.ids <- similarConfigurations (raceConfigurations, parameters, threshold = scenario$softRestartThreshold) # Rprof(NULL) if (!is.null(tmp.ids)) { if (debugLevel >= 1) irace.note("Soft restart: ", paste(collapse = " ", tmp.ids), " !\n") model <- restartConfigurations (raceConfigurations, tmp.ids, model, parameters, nbNewConfigurations, scenario$digits) iraceResults$softRestart[indexIteration] <- TRUE ## FIXME: What is this for? # iraceResults$model$afterSR[[indexIteration]] <- model if (debugLevel >= 2) { printModel (model) } # Re-sample after restart like above #cat("# ", format(Sys.time(), usetz=TRUE), " sampleModel()\n") newConfigurations <- sampleModel(parameters, eliteConfigurations, model, nbNewConfigurations, digits = scenario$digits, forbidden = forbiddenExps, repair = scenario$repairConfiguration) #cat("# ", format(Sys.time(), usetz=TRUE), " sampleModel() DONE\n") # Set ID of the new configurations. newConfigurations <- cbind (.ID. = max(0, allConfigurations$.ID.) + 1:nrow(newConfigurations), newConfigurations) raceConfigurations <- rbind(eliteConfigurations[, colnames(newConfigurations)], newConfigurations) rownames(raceConfigurations) <- raceConfigurations$.ID. } } # Append these configurations to the global table. allConfigurations <- rbind(allConfigurations, newConfigurations) rownames(allConfigurations) <- allConfigurations$.ID. } if (debugLevel >= 2) { irace.note("Configurations for the race n ", indexIteration, " (elite configurations listed first, then new configurations):\n") configurations.print(raceConfigurations, metadata = TRUE) } # Get data from previous elite tests if (scenario$elitist && nrow(eliteConfigurations) > 0) { elite.data <- list() elite.data[["experiments"]] <- iraceResults$experiments[, as.character(eliteConfigurations[,".ID."]), drop=FALSE] if (scenario$capping) elite.data[["time"]] <- generateTimeMatrix(elites = eliteConfigurations, experimentLog = iraceResults$experimentLog) } else elite.data <- NULL .irace$next.instance <- max(nrow(iraceResults$experiments), 0) + 1 # Add instances if needed # Calculate budget needed for old instances assuming non elitist irace if ((nrow(.irace$instancesList) - (.irace$next.instance - 1)) < ceiling(remainingBudget / minSurvival)) { .irace$instancesList <- addInstances(scenario, .irace$instancesList, ceiling(remainingBudget/minSurvival)) } if (debugLevel >= 1) irace.note("Launch race\n") raceResults <- race (scenario = scenario, configurations = raceConfigurations, parameters = parameters, maxExp = currentBudget, minSurvival = minSurvival, elite.data = elite.data, elitistNewInstances = if (firstRace) 0 else scenario$elitistNewInstances) # Update experiments # LESLIE: Maybe we can think is make iraceResults an environment, so these values # can be updated in the race function. # We add indexIteration as an additional column. iraceResults$experimentLog <- rbind(iraceResults$experimentLog, cbind(rep(indexIteration, nrow(raceResults$experimentLog)), raceResults$experimentLog)) # Merge new results. iraceResults$experiments <- merge.matrix (iraceResults$experiments, raceResults$experiments) if (length(raceResults$rejectedIDs) > 0) { rejectedIDs <- c(rejectedIDs, raceResults$rejectedIDs) iraceResults$rejectedConfigurations <- rejectedIDs forbiddenExps <- c(forbiddenExps, buildForbiddenExp( configurations = allConfigurations[ allConfigurations$.ID. %in% raceResults$rejectedIDs, , drop = FALSE], parameters = parameters)) } experimentsUsedSoFar <- experimentsUsedSoFar + raceResults$experimentsUsed # Update remaining budget. if (scenario$maxTime > 0) { timeUsed <- sum(timeUsed, raceResults$experimentLog[, "time"], na.rm=TRUE) boundEstimate <- mean(iraceResults$experimentLog[, "time"], na.rm=TRUE) remainingBudget <- round((scenario$maxTime - timeUsed) / boundEstimate) } else { remainingBudget <- remainingBudget - raceResults$experimentsUsed } if (debugLevel >= 3) { irace.note("Results for the race of iteration ", indexIteration, " (from best to worst, according to the ", test.type.order.str(scenario$testType), "):\n") configurations.print (raceResults$configurations, metadata = TRUE) } if (debugLevel >= 1) { irace.note("Extracting elites\n") } # FIXME: Since we only actually keep the alive ones, we don't need # to carry around rejected ones in raceResults$configurations. This # would reduce overhead. eliteConfigurations <- extractElites(scenario, parameters, raceResults$configurations, min(raceResults$nbAlive, minSurvival)) irace.note("Elite configurations (first number is the configuration ID;", " listed from best to worst according to the ", test.type.order.str(scenario$testType), "):\n") if (!quiet) configurations.print(eliteConfigurations, metadata = debugLevel >= 1) iraceResults$iterationElites <- c(iraceResults$iterationElites, eliteConfigurations$.ID.[1]) iraceResults$allElites[[indexIteration]] <- eliteConfigurations$.ID. if (firstRace) { if (debugLevel >= 1) { irace.note("Initialise model\n") } model <- initialiseModel(parameters, eliteConfigurations, scenario$digits) } if (debugLevel >= 1) { irace.note("End of iteration ", indexIteration, "\n") } if (debugLevel >= 3) { irace.note("All configurations (sampling order):\n") configurations.print(allConfigurations, metadata = TRUE) } indexIteration <- indexIteration + 1 firstRace <- FALSE if (scenario$debugLevel >= 3) { irace.note ("Memory used in irace():\n") irace.print.memUsed() } } irace.internal.error("This code is actually never executed because we return above") } irace/R/tnorm.R0000644000176200001440000001206214157104332013021 0ustar liggesusers## The naive method would be: ## ## rtnorm.naive <- function(n, mean = 0, sd = 1, lower, upper) ## { ## lower <- (lower - mean) / sd ## Algorithm works on mean 0, sd 1 scale ## upper <- (upper - mean) / sd ## x <- rnorm(n) ## repeat { ## w <- which( x > upper | x < lower) ## if (length(w) == 0) break ## x[w] <- rnorm(length(w)) ## } ## return(x * sd + mean) ## } ## ## There is also https://cran.r-project.org/web/packages/truncnorm/ ## Function to sample according to a truncated normal distribution ## This function comes from the R package 'msm' maintained by ## Christopher Jackson # # Package: msm # Version: 1.2 # Date: 2013-05-14 # Title: Multi-state Markov and hidden Markov models in continuous time # Author: Christopher Jackson # Maintainer: Christopher Jackson # Description: Functions for fitting general continuous-time Markov and # hidden Markov multi-state models to longitudinal data. A # variety of observation schemes are supported, including # processes observed at arbitrary times (panel data), # continuously-observed processes, and censored states. Both # Markov transition rates and the hidden Markov output process # can be modelled in terms of covariates, which may be constant # or piecewise-constant in time. # License: GPL (>= 2) # Repository: CRAN ## Rejection sampling algorithm by Robert (Stat. Comp (1995), 5, 121-5) ## for simulating from the truncated normal distribution. rtnorm <- function (n, mean = 0, sd = 1, lower = -Inf, upper = Inf) { if (length(n) > 1) n <- length(n) mean <- rep_len(mean, n) sd <- rep_len(sd, n) lower <- rep_len(lower, n) upper <- rep_len(upper, n) lower <- (lower - mean) / sd ## Algorithm works on mean 0, sd 1 scale upper <- (upper - mean) / sd ind <- seq_len(n) ret <- numeric(n) ## Different algorithms depending on where upper/lower limits lie. alg <- ifelse( lower > upper, -1,# return NaN if lower > upper ifelse( ((lower < 0 & upper == Inf) | (lower == -Inf & upper > 0) | (is.finite(lower) & is.finite(upper) & (lower < 0) & (upper > 0) & (upper-lower > sqrt(2*pi))) ), 0, # standard "simulate from normal and reject if outside limits" method. Use if bounds are wide. FIXME HSOULD BE ifelse( (lower >= 0 & (upper > lower + 2*sqrt(exp(1)) / (lower + sqrt(lower^2 + 4)) * exp((lower*2 - lower*sqrt(lower^2 + 4)) / 4))), 1, # rejection sampling with exponential proposal. Use if lower >> mean ifelse(upper <= 0 & (-lower > -upper + 2*sqrt(exp(1)) / (-upper + sqrt(upper^2 + 4)) * exp((upper*2 - -upper*sqrt(upper^2 + 4)) / 4)), 2, # rejection sampling with exponential proposal. Use if upper << mean. 3)))) # rejection sampling with uniform proposal. Use if bounds are narrow and central. ind.nan <- ind[alg==-1]; ind.no <- ind[alg==0]; ind.expl <- ind[alg==1]; ind.expu <- ind[alg==2]; ind.u <- ind[alg==3] ret[ind.nan] <- NaN while (length(ind.no) > 0) { y <- rnorm(length(ind.no)) done <- which(y >= lower[ind.no] & y <= upper[ind.no]) ret[ind.no[done]] <- y[done] ind.no <- setdiff(ind.no, ind.no[done]) } stopifnot(length(ind.no) == 0) while (length(ind.expl) > 0) { a <- (lower[ind.expl] + sqrt(lower[ind.expl]^2 + 4)) / 2 z <- rexp(length(ind.expl), a) + lower[ind.expl] u <- runif(length(ind.expl)) done <- which((u <= exp(-(z - a)^2 / 2)) & (z <= upper[ind.expl])) ret[ind.expl[done]] <- z[done] ind.expl <- setdiff(ind.expl, ind.expl[done]) } stopifnot(length(ind.expl) == 0) while (length(ind.expu) > 0) { a <- (-upper[ind.expu] + sqrt(upper[ind.expu]^2 +4)) / 2 z <- rexp(length(ind.expu), a) - upper[ind.expu] u <- runif(length(ind.expu)) done <- which((u <= exp(-(z - a)^2 / 2)) & (z <= -lower[ind.expu])) ret[ind.expu[done]] <- -z[done] ind.expu <- setdiff(ind.expu, ind.expu[done]) } stopifnot(length(ind.expu) == 0) while (length(ind.u) > 0) { z <- runif(length(ind.u), lower[ind.u], upper[ind.u]) rho <- ifelse(lower[ind.u] > 0, exp((lower[ind.u]^2 - z^2) / 2), ifelse(upper[ind.u] < 0, exp((upper[ind.u]^2 - z^2) / 2), exp(-z^2/2))) u <- runif(length(ind.u)) done <- which(u <= rho) ret[ind.u[done]] <- z[done] ind.u <- setdiff(ind.u, ind.u[done]) } stopifnot(length(ind.u) == 0) ret*sd + mean } irace/R/testing.R0000644000176200001440000000724114240240445013341 0ustar liggesusers#' Execute the given configurations on the testing instances specified in the #' scenario #' #' @template arg_configurations #' @template arg_scenario #' @template arg_parameters #' #' @return A list with the following elements: #' \describe{ #' \item{\code{experiments}}{Experiments results.} #' \item{\code{seeds}}{Array of the instance seeds used in the experiments.} #' } #' #' @details A test instance set must be provided through `scenario[["testInstances"]]`. #' #' @seealso #' [testing_fromlog()] #' #' @author Manuel López-Ibáñez #' @export testConfigurations <- function(configurations, scenario, parameters) { # We need to set up a default scenario (and repeat all checks) in case # we are called directly instead of being called after executing irace. scenario <- checkScenario(scenario) testInstances <- scenario[["testInstances"]] instances.ID <- names(testInstances) if (length(testInstances) == 0) irace.error("No test instances given") if (is.null(instances.ID)) irace.error("testInstances must have names") # 2147483647 is the maximum value for a 32-bit signed integer. # We use replace = TRUE, because replace = FALSE allocates memory for each possible number. ## FIXME: scenario[["testInstances"]] and scenario$instances behave differently, ## we should unify them so that the seeds are also saved in scenario. instanceSeed <- sample.int(2147483647, size = length(testInstances), replace = TRUE) names(instanceSeed) <- instances.ID values <- removeConfigurationsMetaData(configurations) values <- values[, parameters$names, drop = FALSE] switches <- parameters$switches[parameters$names] bounds <- rep(scenario$boundMax, nrow(configurations)) # If there is no ID (e.g., after using readConfigurations), then add it. if (! (".ID." %in% colnames(configurations))) { configurations$.ID. <- 1:nrow(configurations) } # Create experiment list experiments <- createExperimentList(configurations, parameters, testInstances, instances.ID, instanceSeed, scenario, bounds) startParallel(scenario) on.exit(stopParallel()) if (scenario$debugLevel >= 3) { irace.note ("Memory used before execute.experiments in testConfigurations():\n") irace.print.memUsed() } target.output <- execute.experiments (experiments, scenario) # targetEvaluator may be NULL. If so, target.output must # contain the right output already. if (!is.null(scenario$targetEvaluator)) target.output <- execute.evaluator (experiments, scenario, target.output, configurations$.ID.) # FIXME: It would be much faster to get convert target.output$cost to a # vector, then initialize the matrix with the vector. testResults <- matrix(NA, ncol = nrow(configurations), nrow = length(testInstances), # dimnames = list(rownames, colnames) dimnames = list (instances.ID, configurations$.ID.)) # FIXME: It would be much faster to get a vector cost, applyPAR to it, then # assign it. for (i in seq_along(experiments)) { cost <- target.output[[i]]$cost if (scenario$capping) cost <- applyPAR(cost, boundMax = scenario$boundMax, boundPar = scenario$boundPar) testResults[rownames(testResults) == experiments[[i]]$id.instance, colnames(testResults) == experiments[[i]]$id.configuration] <- cost } if (scenario$debugLevel >= 3) { irace.note ("Memory used at the end of testConfigurations():\n") irace.print.memUsed() } ## FIXME: Shouldn't we record these experiments in experimentLog ? return(list(experiments = testResults, seeds = instanceSeed)) } irace/R/cluster.R0000644000176200001440000001157714252645421013362 0ustar liggesusers### Submit/wait for jobs in batch clusters. sge.job.finished <- function(jobid) system (paste0("qstat -j ", jobid), ignore.stdout = TRUE, ignore.stderr = TRUE, intern = FALSE, wait = TRUE) pbs.job.finished <- function(jobid) system (paste0("qstat ", jobid), ignore.stdout = TRUE, ignore.stderr = TRUE, intern = FALSE, wait = TRUE) torque.job.finished <- function(jobid) { output <- suppressWarnings(system2("qstat", jobid, stdout = TRUE, stderr = TRUE)) ## 1. If the return code of qstat in not 0, then no such job is in the queue ## anymore. That means that the job was (assumption) successfully submitted, ## executed and completed. Then, for some time the job id was marked as ## completed in the jobs queue and the job status removed from the queue list ## after some time. No job completed status of successfully submitted and ## completed jobs in the queue list happens if a small task was executed ## first and a task of some hours execution time finishes much later. The job ## status of the small task will not be in the queue anymore after some ## minutes. So: If no job id is in the queue list anymore and torq's qstat ## returns an error (return code > 0), then the job has been successfully ## executed (if it has been started successfully before). if (!is.null(attr(output, "status"))) return(TRUE) # 2. If qstat returns OK (return code ==0), then one has to parse qstat's # output. If the 5th token in the last line is a 'C', then the job has # terminated and its output files can be processed. Otherwise the job is not # completed (queued, running, exiting...) any(grepl(paste0(jobid, ".*\\sC\\s"), output)) } slurm.job.finished <- function(jobid) { output <- suppressWarnings(system2("squeue", c("-j", jobid, "--noheader"), stdout = TRUE, stderr = TRUE)) # If the above returns non-zero, either the job terminated or it never # existed. if (!is.null(attr(output, "status"))) return(TRUE) # If may return zero, but the job is not in the system anymore because it # completed. This is different from the Torque case. !any(grepl(paste0("\\s", jobid, "\\s"), output)) } htcondor.job.finished <- function(jobid) { output <- suppressWarnings(system2("condor_q", jobid, stdout = TRUE, stderr = TRUE)) # Check if job is still in the queue, otherwise it is considered finished !any(grepl(paste0("ID:\\s", jobid), output)) } ## Launch a job with qsub and return its jobID. This function does not ## call qsub directly, but instead targetRunner should be a script that ## invokes qsub and returns a jobID. target.runner.qsub <- function(experiment, scenario) { debugLevel <- scenario$debugLevel res <- run_target_runner(experiment, scenario) cmd <- res$cmd output <- res$output args <- res$args jobID <- NULL outputRaw <- output$output err.msg <- output$error if (is.null(err.msg)) { # We cannot use parse.output because that tries to convert to numeric. if (debugLevel >= 2) { cat (outputRaw, sep = "\n") } # Initialize output as raw. If it is empty stays like this. # strsplit crashes if outputRaw == character(0) if (length(outputRaw) > 0) { jobID <- strsplit(trim(outputRaw), "[[:space:]]+")[[1]] } if (length(jobID) != 1) { err.msg <- paste0("The output of targetRunner should be only the jobID!") jobID <- NULL } } list(jobID = jobID, error = err.msg, outputRaw = outputRaw, call = paste(cmd, args)) } cluster.lapply <- function(X, scenario, poll.time = 2) { debugLevel <- scenario$debugLevel cluster.job.finished <- switch(scenario$batchmode, sge = sge.job.finished, pbs = pbs.job.finished, torque = torque.job.finished, slurm = slurm.job.finished, htcondor = htcondor.job.finished, irace.error ("Invalid value of scenario$batchmode = ", scenario$batchmode)) # Parallel controls how many jobs we send at once. Some clusters have low # limits. ## FIXME: It would be better to submit up to the limit, then one by one as jobs finish. chunksize <- scenario$parallel if (chunksize < 0) { chunksize <- length(X) } chunks <- split(X, ceiling(seq_along(X) / chunksize)) for (chunk in chunks) { if (debugLevel >= 1) { irace.note ("Sending ", length(chunk), " / ", length(X), " jobs\n") } output <- lapply(chunk, exec.target.runner, scenario = scenario, target.runner = target.runner.qsub) jobIDs <- sapply(output, "[[", "jobID") ## Wait for cluster jobs to finish. if (length(jobIDs) > 0 && debugLevel >= 1) { irace.note("Waiting for jobs ('.' == ", poll.time, " s) ") } for (jobID in jobIDs) { while (!cluster.job.finished(jobID)) { if (debugLevel >= 1) { cat(".") } Sys.sleep(poll.time) } if (debugLevel >= 1) { cat("\n") irace.note ("DONE (", jobID, ")\n") } } } output } irace/R/readParameters.R0000644000176200001440000007024014315060133014617 0ustar liggesusers#' Reads the parameters to be tuned by \pkg{irace} from a file or from a #' character string. #' #' @param file (`character(1)`) \cr Filename containing the definitions of #' the parameters to be tuned. #' @param digits The number of decimal places to be considered for the real #' parameters. #' @template arg_debuglevel #' @template arg_text #' #' @return A list containing the definitions of the parameters read. The list is #' structured as follows: #' \describe{ #' \item{`names`}{Vector that contains the names of the parameters.} #' \item{`types`}{Vector that contains the type of each parameter 'i', 'c', 'r', 'o'. #' Numerical parameters can be sampled in a log-scale with 'i,log' and 'r,log' #' (no spaces).} #' \item{`switches`}{Vector that contains the switches to be used for the #' parameters on the command line.} #' \item{`domain`}{List of vectors, where each vector may contain two #' values (minimum, maximum) for real and integer parameters, or #' possibly more for categorical parameters.} #' \item{`conditions`}{List of R logical expressions, with variables #' corresponding to parameter names.} #' \item{`isFixed`}{Logical vector that specifies which parameter is fixed #' and, thus, it does not need to be tuned.} #' \item{`nbParameters`}{An integer, the total number of parameters.} #' \item{`nbFixed`}{An integer, the number of parameters with a fixed value.} #' \item{`nbVariable`}{Number of variable (to be tuned) parameters.} #' \item{`depends`}{List of character vectors, each vector specifies #' which parameters depend on this one.} #' \item{`isDependent`}{Logical vector that specifies which parameter has #' a dependent domain.} #' } #' #' @details Either `file` or `text` must be given. If `file` is given, the #' parameters are read from the file `file`. If `text` is given instead, #' the parameters are read directly from the `text` character string. #' In both cases, the parameters must be given (in `text` or in the file #' whose name is `file`) in the expected form. See the documentation #' for details. If none of these parameters is given, \pkg{irace} #' will stop with an error. #' #' A fixed parameter is a parameter that should not be sampled but #' instead should be always set to the only value of its domain. In this #' function we set isFixed to TRUE only if the parameter is a categorical #' and has only one possible value. If it is an integer and the minimum #' and maximum are equal, or it is a real and the minimum and maximum #' values satisfy `round(minimum, digits) == round(maximum, digits)`, #' then the parameter description is rejected as invalid to identify #' potential user errors. #' #' @examples #' ## Read the parameters directly from text #' parameters.table <- ' #' # name switch type values [conditions (using R syntax)] #' algorithm "--" c (as,mmas,eas,ras,acs) #' localsearch "--localsearch " c (0, 1, 2, 3) #' alpha "--alpha " r (0.00, 5.00) #' beta "--beta " r (0.00, 10.00) #' rho "--rho " r (0.01, 1.00) #' ants "--ants " i,log (5, 100) #' q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" #' rasrank "--rasranks " i (1, "min(ants, 10)") | algorithm == "ras" #' elitistants "--elitistants " i (1, ants) | algorithm == "eas" #' nnls "--nnls " i (5, 50) | localsearch %in% c(1,2,3) #' dlb "--dlb " c (0, 1) | localsearch %in% c(1,2,3) #' ' #' parameters <- readParameters(text=parameters.table) #' str(parameters) #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export readParameters <- function (file, digits = 4, debugLevel = 0, text) { if (missing(file) && !missing(text)) { filename <- strcat("text=", deparse(substitute(text))) file <- textConnection(text) on.exit(close(file)) } else if (is.character(file)) { filename <- file file.check (file, readable = TRUE, text = "readParameter: parameter file") } else { irace.error("'file' must be a character string") } field.match <- function (line, pattern, delimited = FALSE, sep = "[[:space:]]") { #cat ("pattern:", pattern, "\n") positions <- lapply(1:length(pattern), function(x) regexpr (paste0("^", pattern[x], sep), line)) if (all(sapply(positions, "[[", 1) == -1)) { #cat("no match: NULL\n") return (list(match = NULL, line = line)) } pos.matched.list <- lapply(1:length(pattern), function(x) regexpr (paste0("^", pattern[x]), line)) #cat("pos.matched:", pos.matched, "\n") if (all(sapply(pos.matched.list, "[[", 1) == -1)) { #cat(line) return (list(match = NULL, line = line)) } position <- which(sapply(pos.matched.list, `[[`,1) != -1) if (length(position) > 1) { position <- position[1] } pos.matched <- pos.matched.list[[position]] delimited <- as.integer(delimited) match <- substr(line, pos.matched[1] + delimited, attr(pos.matched, "match.length") - delimited) #cat("match:",match, "\n") line <- substr (line, pos.matched[1] + attr(pos.matched, "match.length"), nchar(line)) line <- trim.leading (line) #cat(line) return (list(match = match, line = line)) } # FIXME: Make quotes mandatory for categorical and ordered parameters. string2vector <- function(str) { v <- c() str <- trim(str) #cat("string2vector:", str, "\n") while (nchar (str)) { result <- field.match (str, "\"[^\"]*\"", delimited = TRUE, sep="") #cat("result.match: ", result$match,"\n") if (is.null (result$match)) { result <- field.match (str, "[^,]+", sep="") #cat("result.match: ", result$match,"\n") } v <- c(v, result$match) #print(v) str <- sub(",[[:space:]]*", "", result$line) #print(str) } return (v) } # Determine if a parameter is fixed. isFixed <- function (type, domain) { type <- as.character(type) if (type == "i" || type == "r") { return (domain[[1]] == domain[[2]]) } else if (type == "c" || type == "o") { return (length(domain) == 1) } } # ************************************************************************* # Subordinate parameter: ordering of the parameters according to # conditions hierarchy # * The conditions hierarchy is an acyclic directed graph. # Function treeLevel() computes an order on vertex s.t: # level(A) > level(B) <=> There is an arc A ---> B # (A depends on B to be activated) # * If a cycle is detected, execution is stopped # * If a parameter depends on another one not defined, execution is stopped treeLevel <- function(paramName, varsTree, rootParam = paramName) { # The last parameter is used to record the root parameter of the # recursive call in order to detect the presence of cycles. vars <- varsTree[[paramName]] if (length(vars) == 0) return (1) # This parameter does not have conditions # This parameter has some conditions # Recursive call: level <- MAX( level(m) : m in children ) maxChildLevel <- 0 for (child in vars) { # The following line detects cycles if (child == rootParam) irace.error("A cycle detected in subordinate parameters! ", "Check definition of conditions and/or dependent domains.\n", "One parameter of this cycle is '", rootParam, "'") # The following line detects a missing definition if (child %!in% names(varsTree)) irace.error("A parameter definition is missing! ", "Check definition of parameters.\n", "Parameter '", paramName, "' depends on '", child, "' which is not defined.") level <- treeLevel(child, varsTree, rootParam) if (level > maxChildLevel) maxChildLevel <- level } level <- maxChildLevel + 1 return (level) } errReadParameters <- function(filename, line, context, ...) { if (!is.null (context)) { context <- paste0(" when reading: \"", context, "\"") } irace.error (paste0 (...), " at ", filename, ", line ", line, context) } transform.domain <- function(transf, domain, type) { if (transf == "") return(transf) # We do not support transformation of dependent parameters, yet # TODO: think about dependent domain transfomation if (is.expression(domain)) irace.error("Parameter domain transformations are not yet available for", " dependent parameter domains.") lower <- domain[1] upper <- domain[2] if (transf == "log") { # Reject log if domain contains zero or negative values if (any(domain <= 0)) return(NULL) trLower <- log(lower) # +1 to adjust before floor() trUpper <- if (type == "i") log(upper + 1) else log(upper) irace.assert(is.finite(trLower)) irace.assert(is.finite(trUpper)) attr(transf, "lower") <- trLower attr(transf, "upper") <- trUpper return(transf) } irace.internal.error("unrecognized transformation type '", transf, "'") } # Checks that variables in the expressions are within # the parameters names. check_parameter_dependencies <- function (parameters) { for (p in names(Filter(length, parameters$depends))) { vars <- parameters$depends[[p]] flag <- vars %in% parameters$names if (!all(flag)) { irace.error ("Domain (", paste0(parameters$domain[[p]], collapse=", "), ") of parameter '", p, "' is not valid: '", paste0(vars[!flag], collapse=", "), "' cannot be found in the scenario parameters: ", paste0(parameters$names, collapse=" , ")," .") } flag <- parameters$types[vars] %in% c("i", "r") if (!all(flag)) { irace.error ("Domain of parameter '", p, "' depends on non-numerical", " parameters: ", paste0(vars[!flag], collapse=", "), " .") } # Supported operations for dependent domains allowed.fx <- c("+", "-", "*", "/", "%%", "min", "max", "round", "floor", "ceiling", "trunc") fx <- setdiff(all.names(parameters$domain[[p]], unique=TRUE), all.vars(parameters$domain[[p]], unique=TRUE)) flag <- fx %in% allowed.fx if (!all(flag)) { irace.error ("Domain of parameter '", p, "' uses function(s) ", "not yet supported by irace: ", paste0(fx[!flag], collapse=", "), " .") } } return(TRUE) } parameters <- list(names = character(0), types = character(0), switches = character(0), domain = list(), conditions = list(), isFixed = logical(0), # FIXME: This has to be a list because we assign # attributes to elements. transform = list(), isDependent = logical(0)) conditions <- list() lines <- readLines(con = file) nbLines <- 0 count <- 0 for (line in lines) { nbLines <- nbLines + 1 # Delete comments line <- trim(sub("#.*$", "", line)) if (nchar(line) == 0) { next } ## Match param.name (unquoted alphanumeric string) result <- field.match (line, "[._[:alnum:]]+") param.name <- result$match line <- result$line if (is.null (result$match)) { errReadParameters (filename, nbLines, line, "parameter name must be alphanumeric") } if (param.name %in% parameters$names) { errReadParameters (filename, nbLines, NULL, "duplicated parameter name '", param.name, "'") } ## Match param.switch (quoted string) result <- field.match (line, "\"[^\"]*\"", delimited = TRUE) param.switch <- result$match line <- result$line if (is.null (result$match)) { errReadParameters (filename, nbLines, line, "parameter switch must be a double-quoted string") } ## Match param.type (longer matches must precede shorter ones) result <- field.match (line, c("i,log", "r,log", "c","i","r","o")) param.type <- result$match line <- result$line if (is.null (result$match)) { errReadParameters ( filename, nbLines, line, "parameter type must be a single character in {'c','i','r','o'}, ", "with 'i', 'r' optionally followed by ',log' (no spaces in between) ", "to sample using a logarithmic scale") } else if (param.type == "i,log") { param.type <- "i" param.transform <- "log" } else if (param.type == "r,log") { param.type <- "r" param.transform <- "log" } else { param.transform <- "" } ## Match param.value (delimited by parenthesis) # Regexp to detect dependent domains of the type ("min(p1)", 100) result <- field.match (line, "\\([^|]+\\)", delimited = TRUE, sep = "") param.value <- result$match line <- result$line if (is.null (param.value)) { errReadParameters (filename, nbLines, line, "Allowed values must be a list within parenthesis") } # For numerical parameters domains could be dependent # thus, we keep the string values in a variable # for example (10, param1+2) param.value.str <- string2vector(param.value) if (param.type %in% c("r","i")) { # For dependent domains param.value will be NA (we will parse # it later) param.value <- suppressWarnings(as.numeric(param.value.str)) if (length(param.value) != 2) { errReadParameters (filename, nbLines, NULL, "incorrect numeric range (", result$match, ") for parameter '", param.name, "'") } if (param.type == "r") { # FIXME: Given (0.01,0.99) and digits=1, this produces (0, 1), which is # probably not what the user wants. param.value <- round(param.value, digits = digits) } else if (param.type == "i" && any(!is.wholenumber(param.value[!is.na(param.value)]))) { errReadParameters (filename, nbLines, NULL, "for parameter type 'i' values must be integers (", result$match, ") for parameter '", param.name, "'") } # Time to parse dependent domains or check values if (any(is.na(param.value))) { try(param.value[is.na(param.value)] <- parse(text=param.value.str[is.na(param.value)])) } else if (param.value[1] >= param.value[2]) { errReadParameters (filename, nbLines, NULL, "lower bound must be smaller than upper bound in numeric range (", result$match, ") for parameter '", param.name, "'") } param.transform <- transform.domain(param.transform, param.value, param.type) if (is.null(param.transform)) { errReadParameters (filename, nbLines, NULL, "The domain of parameter '", param.name, "' of type 'log' cannot contain zero") } } else { param.value <- param.value.str if (anyDuplicated(param.value)) { dups <- duplicated(param.value) errReadParameters (filename, nbLines, NULL, "duplicated values (", paste0('\"', param.value[dups], "\"", collapse = ', '), ") for parameter '", param.name, "'") } } count <- count + 1 parameters$names[count] <- param.name parameters$switches[count] <- param.switch parameters$types[count] <- param.type parameters$domain[[count]] <- param.value parameters$transform[[count]] <- param.transform parameters$isFixed[count] <- isFixed(type = param.type, domain = parameters$domain[[count]]) # Reject non-categorical fixed parameters. They are often the # result of a user error. if (parameters$isFixed[[count]]) { if (param.type == "i") { errReadParameters (filename, nbLines, NULL, "lower and upper bounds are the same in numeric range (", param.value[1], ", ", param.value[2], ") for parameter '", param.name, "'") } else if (param.type == "r") { errReadParameters (filename, nbLines, NULL, "given digits=", digits, ", lower and upper bounds are the same in numeric range (", param.value[1], ", ", param.value[2], ") for parameter '", param.name, "'") } } ## Match start of conditions result <- field.match (line, "\\|", sep="") line <- result$line if (!is.null(result$match) && nchar(result$match)) { result <- field.match (line, ".*$", sep="") if (is.null(result$match) || !nchar(result$match)) errReadParameters (filename, nbLines, line, "expected condition after '|'") # FIXME: Provide a better error for invalid conditions like "a 2 0" conditions[[param.name]] <- NA # keep.source = FALSE avoids adding useless attributes. try(conditions[[param.name]] <- parse(text=result$match, keep.source = FALSE)) if (!is.expression (conditions[[param.name]])) errReadParameters (filename, nbLines, line, "invalid condition after '|'") line <- result$line } else if (!is.null(result$line) && nchar(result$line)) { errReadParameters (filename, nbLines, line, "expected '|' before condition") } else { conditions[[param.name]] <- TRUE } # ***************************************************************** } # end loop on lines # Check that we have read at least one parameter if (count == 0) { irace.error("No parameter definition found: ", "check that the parameter file is not empty") } # Generate dependency flag # FIXME: check if we really need this vector parameters$isDependent <- sapply(parameters$domain, is.expression) names(parameters$types) <- names(parameters$switches) <- names(parameters$domain) <- names(parameters$isFixed) <- names(parameters$transform) <- names(parameters$isDependent) <- parameters$names # Obtain the variables in each condition ## FIXME: In R 3.2, all.vars does not work with byte-compiled expressions, ## thus we do not byte-compile them; but we could use ## all.vars(.Internal(disassemble(condition))[[3]][[1]]) ## LESLIE: should we make then an all.vars in utils.R so we can ## use it without problems? parameters$depends <- lapply(parameters$domain, all.vars) # Check that dependencies are ok check_parameter_dependencies(parameters) # Merge dependencies and conditions parameters$depends <- Map(c, parameters$depends, lapply(conditions, all.vars)) parameters$depends <- lapply(parameters$depends, unique) # Sort parameters in 'conditions' in the proper order according to # conditions hierarchyLevel <- sapply(parameters$names, treeLevel, varsTree = parameters$depends) parameters$hierarchy <- hierarchyLevel parameters$conditions <- conditions[order(hierarchyLevel)] names(parameters$hierarchy) <- parameters$names # Print the hierarchy vector: if (debugLevel >= 1) { cat ("# --- Parameters Hierarchy ---\n") print(data.frame(Parameter = paste0(names(parameters$hierarchy)), Level = parameters$hierarchy, "Depends on" = sapply(parameters$depends, paste0, collapse=", "), row.names=NULL)) cat("\n# ------------------------\n") } irace.assert(length(parameters$conditions) == length(parameters$names)) parameters$nbParameters <- length(parameters$names) parameters$nbFixed <- sum(parameters$isFixed == TRUE) parameters$nbVariable <- sum(parameters$isFixed == FALSE) if (debugLevel >= 2) { print(parameters, digits = 15) irace.note("Parameters have been read\n") } parameters } #' Read parameters in PCS (AClib) format and write them in irace format. #' #' @param file (`character(1)`) \cr Filename containing the definitions of #' the parameters to be tuned. #' @param digits The number of decimal places to be considered for the real #' parameters. #' @template arg_debuglevel #' @template arg_text #' #' @return A string representing the parameters in irace format. #' #' @details Either `file` or `text` must be given. If `file` is given, the #' parameters are read from the file `file`. If `text` is given instead, #' the parameters are read directly from the `text` character string. #' In both cases, the parameters must be given (in `text` or in the file #' whose name is `file`) in the expected form. See the documentation #' for details. If none of these parameters is given, \pkg{irace} #' will stop with an error. #' #' **FIXME:** Forbidden configurations, default configuration and transformations ("log") are currently ignored. See #' #' @references #' Frank Hutter, Manuel López-Ibáñez, Chris Fawcett, Marius Thomas Lindauer, Holger H. Hoos, Kevin Leyton-Brown, and Thomas Stützle. **AClib: A Benchmark Library for Algorithm Configuration**. In P. M. Pardalos, M. G. C. Resende, C. Vogiatzis, and J. L. Walteros, editors, _Learning and Intelligent Optimization, 8th International Conference, LION 8_, volume 8426 of Lecture Notes in Computer Science, pages 36–40. Springer, Heidelberg, 2014. #' #' @examples #' ## Read the parameters directly from text #' pcs_table <- ' #' # name domain #' algorithm {as,mmas,eas,ras,acs}[as] #' localsearch {0, 1, 2, 3}[0] #' alpha [0.00, 5.00][1] #' beta [0.00, 10.00][1] #' rho [0.01, 1.00][0.95] #' ants [5, 100][10]i #' q0 [0.0, 1.0][0] #' rasrank [1, 100][1]i #' elitistants [1, 750][1]i #' nnls [5, 50][5]i #' dlb {0, 1}[1] #' Conditionals: #' q0 | algorithm in {acs} #' rasrank | algorithm in {ras} #' elitistants | algorithm in {eas} #' nnls | localsearch in {1,2,3} #' dlb | localsearch in {1,2,3} #' ' #' parameters_table <- read_pcs_file(text=pcs_table) #' cat(parameters_table) #' parameters <- readParameters(text=parameters_table) #' str(parameters) #' #' @author Manuel López-Ibáñez #' @export read_pcs_file <- function(file, digits = 4, debugLevel = 0, text) { if (missing(file) && !missing(text)) { filename <- strcat("text=", deparse(substitute(text))) file <- textConnection(text) on.exit(close(file)) } else if (is.character(file)) { filename <- file file.check (file, readable = TRUE, text = "read_pcs_file: parameter file") } else { irace.error("'file' must be a character string") } lines <- readLines(con = file) handle_conditionals <- FALSE conditions <- list() for (k in seq_along(lines)) { if (grepl("Conditionals:", lines[k])) { handle_conditionals <- TRUE lines[k] <- "" } else if (handle_conditionals) { matches <- regmatches(lines[k], regexec("^[[:space:]]*([^[:space:]]+)[[:space:]]+\\|[[:space:]]+(.+)$", lines[k], perl=TRUE))[[1]] if (length(matches) > 0) { lines[k] <- "" conditions[[matches[2]]] <- matches[3] } } } parse_pcs_condition <- function(x, types) { if (is.null(x)) return ("") matches <- regmatches(x, regexec("([^[:space:]]+)[[:space:]]+in[[:space:]]+\\{([^}]+)\\}[[:space:]]*$", x, perl=TRUE))[[1]] if (length(matches) == 0) irace.error("unknown condition ", x) type <- types[[matches[2]]] if (is.null(type)) irace.error("unknown type for ", matches[2], " in condition: ", x) cond <- matches[3] if (type %in% c("c", "o")) cond <- paste0('"', strsplit(cond, ",[[:space:]]*")[[1]], '"', collapse=',') # FIXME: Use "==" if there is only one element in cond. return(paste0(" | ", matches[2], " %in% c(", cond, ")")) } param_types <- list() param_domains <- list() param_comments <- list() for (line in lines) { if (grepl("^[[:space:]]*#", line) || grepl("^[[:space:]]*$", line)) next # match a parameter matches <- regmatches(line, regexec("^[[:space:]]*([^[:space:]]+)[[:space:]]+\\[([^,]+),[[:space:]]*([^]]+)\\][[:space:]]*\\[[^]]+\\](i?l?i?)(.*)$", line, perl=TRUE))[[1]] if (length(matches) > 0) { param_name <- matches[2] param_type <- paste0(if(grepl("i", matches[5], fixed=TRUE)) "i" else "r", if(grepl("l", matches[5], fixed=TRUE)) ",log" else "") param_types[[param_name]] <- param_type param_domains[[param_name]] <- paste0("(", matches[3], ", ", matches[4], ")") param_comments[[param_name]] <- matches[6] next } matches <- regmatches(line, regexec("^[[:space:]]*([^[:space:]]+)[[:space:]]+\\{([^}]+)\\}[[:space:]]*\\[[^]]+\\](.*)$", line, perl=TRUE))[[1]] if (length(matches) > 0) { param_name <- matches[2] param_type <- "c" param_types[[param_name]] <- param_type param_types[[param_name]] <- param_type param_domains[[param_name]] <- paste0("(", matches[3], ")") param_comments[[param_name]] <- matches[4] next } } output <- "" for (line in lines) { if (grepl("^[[:space:]]*#", line) || grepl("^[[:space:]]*$", line)) { output <- paste0(output, line, "\n") next } # match a parameter matches <- regmatches(line, regexec("^[[:space:]]*([^[:space:]]+)[[:space:]]+", line, perl=TRUE))[[1]] if (length(matches) > 0) { param_name <- matches[2] cond <- parse_pcs_condition(conditions[[param_name]], param_types) output <- paste0(output, sprintf('%s "%s" %s %s%s%s\n', param_name, param_name, param_types[[param_name]], param_domains[[param_name]], cond, param_comments[[param_name]])) next } irace.error("unrecognized line: ", line) } output } #' checkParameters #' #' FIXME: This is incomplete, for now we only repair inputs from previous irace #' versions. #' #' @template arg_parameters #' @export checkParameters <- function(parameters) { if (is.null(parameters$isDependent)) { parameters$isDependent <- sapply(parameters$domain, is.expression) names(parameters$isDependent) <- parameters$names } parameters } #' Print parameter space in the textual format accepted by irace. #' #' FIXME: Dependent parameter bounds are not supported yet. #' #' @param params (`list()`) Parameter object stored in `irace.Rdata` or read with `irace::readParameters()`. #' #' @param digits (`integer()`) The desired number of digits after the decimal point for real-valued parameters. Default is 15, but it should be the value in `scenario$digits`. #' #' @examples #' parameters.table <- ' #' # name switch type values [conditions (using R syntax)] #' algorithm "--" c (as,mmas,eas,ras,acs) #' localsearch "--localsearch " c (0, 1, 2, 3) #' ants "--ants " i,log (5, 100) #' q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" #' nnls "--nnls " i (5, 50) | localsearch %in% c(1,2,3) #' ' #' parameters <- readParameters(text=parameters.table) #' printParameters(parameters) #' @export printParameters <- function(params, digits = 15L) { names_len <- max(nchar(params$names)) switches_len <- max(nchar(params$switches)) + 2 for (name in params$names) { switch <- paste0('"', params$switches[[name]], '"') type <- params$types[[name]] transf <- params$transform[[name]] domain <- params$domain[[name]] if (type == "r") domain <- formatC(domain, digits=digits, format="f", drop0trailing=TRUE) domain <- paste0('(', paste0(domain, collapse=","), ')') condition <- params$conditions[[name]] condition <- if (isTRUE(condition)) "" else paste0(" | ", condition) if (!is.null(transf) && transf != "") type <- paste0(type, ",", transf) cat(sprintf('%*s %*s %s %-15s%s\n', -names_len, name, -switches_len, switch, type, domain, condition)) } } irace/R/race.R0000644000176200001440000015246614321266642012617 0ustar liggesusers# ---------------------------------------- -*- mode: r; mode: font-lock -*- # # race.R Racing methods for the selection of the best # # ------------------------------------------------------------------------- # # ========================================================================= # # Racing methods for the selection of the best # # ------------------------------------------------------------------------- # # Copyright (C) 2003 Mauro Birattari # # ========================================================================= # # This program is free software; you can redistribute it and/or modify it # # under the terms of the GNU General Public License as published by the # # Free Software Foundation; either version 2 of the License, or (at your # # option) any later version. # # # # This program is distributed in the hope that it will be useful, but # # WITHOUT ANY WARRANTY; without even the implied warranty of # # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # # General Public License for more details. # # # # You should have received a copy of the GNU General Public License along # # with this program; if not, write to the Free Software Foundation, Inc., # # 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. # # ========================================================================= # # ========================================================================= # # Mauro BIRATTARI # # IRIDIA - ULB, CP 194/6 # # Av. F. D. Roosevelt 50 mbiro@ulb.ac.be # # 1050 Brussels, Belgium http://iridia.ulb.ac.be/~mbiro # # ========================================================================= # # $Id: race.R,v 1.54 2005/03/30 12:40:42 mbiro Exp $ # createExperimentList <- function(configurations, parameters, instances, instances.ID, seeds, scenario, bounds = NULL) { instances <- instances[instances.ID] experiments <- vector("list", nrow(configurations) * length(instances)) configurations.ID <- configurations[, ".ID."] pnames <- parameters$names values <- configurations[, pnames, drop = FALSE] switches <- parameters$switches[pnames] # There must be a bound for each configuration. # FIXME: There must be a bound for each configuration AND each instance. irace.assert(is.null(bounds) || length(bounds) == nrow(configurations)) bounds <- rep(bounds, length(instances)) count <- 1 for (i in seq_len(nrow(configurations))) { values_i <- values[i, , drop = FALSE] for (j in seq_along(instances)) { experiments[[count]] <- list (id.configuration = configurations.ID[i], id.instance = instances.ID[j], seed = seeds[j], configuration = values_i, # mlr uses lists of objects as instances. instance = instances[[j]], bound = bounds[count], switches = switches) count <- count + 1 } } return(experiments) } ## Executes a list of configurations in a particular instance ## configurations: description having the id of the configuration ## instance.idx: index of the instance,seed pair in .irace$instancesList ## bounds: execution bounds (if needed). ## which.alive: index of the configurations that are still alive ## which.exe: index of the alive configurations that should be executed race.wrapper <- function(configurations, instance.idx, bounds = NULL, # FIXME: we actually only need which.exps, not # which.alive nor which.exe which.alive, which.exe, parameters, scenario) { irace.assert (parameters$nbVariable > 0) irace.assert (length(parameters$names) == parameters$nbParameters) # FIXME: Accessing 'seed' and 'instance' should be moved to createExperimentList. seed <- .irace$instancesList[instance.idx, "seed"] id.instance <- .irace$instancesList[instance.idx, "instance"] # Experiment list to execute experiments <- createExperimentList(configurations, parameters, instances = scenario$instances, instances.ID = id.instance, seeds = seed, scenario, bounds = bounds) target.output <- vector("list", length(experiments)) # Execute commands if (length(which.exe) > 0) { # which.exe values are within 1:nbConfigurations, whereas experiments # indices are within 1:length(which.alive). The following line converts # from one to the other. which.exps <- which(which.alive %in% which.exe) irace.assert(length(which.exps) == length(which.exe)) target.output[which.exps] <- execute.experiments (experiments[which.exps], scenario) } # targetEvaluator may be NULL. If so, target.output must # contain the right output already. # Otherwise, targetEvaluator always re-evaluates. if (!is.null(scenario$targetEvaluator)) target.output <- execute.evaluator (experiments, scenario, target.output, configurations[, ".ID."]) target.output } aux2.friedman <- function(y, I, alive, conf.level = 0.95) { dropped.any <- FALSE n <- nrow(y) k <- length(I) r <- t(apply(y[,I], 1L, rank)) R <- colSums(r) o <- order(R) best <- I[o[1]] TIES <- tapply(r, row(r), table) STATISTIC <- ((12 * sum((R - n * (k + 1) / 2)^2)) / (n * k * (k + 1) - (sum(unlist(lapply(TIES, function (u) {u^3 - u}))) / (k - 1)))) PARAMETER <- k - 1 PVAL <- pchisq(STATISTIC, PARAMETER, lower.tail = FALSE) #names(STATISTIC) <- "Friedman chi-squared" #names(PARAMETER) <- "df" alpha <- 1 - conf.level if (!is.nan(PVAL) && PVAL < alpha) { # This formula for multiple comparisons comes from Conover, "Practical # Nonparametric Statistics", 1999, pages 369-371. A <- sum(as.vector(r)^2) t <- qt(1 - alpha / 2, df = (n - 1) * (k - 1)) * (2 * (n * A - sum(R^2)) / ((n - 1) * (k - 1)))^(1 / 2) J <- best for (j in 2:k) { if (abs(R[o[j]] - R[o[1]]) > t) { break } else { J <- c(J, I[o[j]]) } } alive[-J] <- FALSE dropped.any <- TRUE } irace.assert(I[which.min(R)] == best) return(list(best = best, ranks = R, alive = alive, dropped.any = dropped.any, p.value = PVAL)) } aux_friedman <- function(results, alive, which.alive, conf.level) { no.alive <- length(which.alive) if (no.alive == 2) { best <- NULL ranks <- NULL dropped.any <- TRUE PVAL <- 0 # If only 2 configurations are left, switch to Wilcoxon V1 <- results[, which.alive[1]] V2 <- results[, which.alive[2]] # Avoid the test if the answer is obvious if (all(V1 <= V2)) { ranks <- c(1,2) } else if (all(V2 <= V1)) { ranks <- c(2,1) } else { res <- wilcox.test(V1, V2, paired = TRUE, conf.int = TRUE) PVAL <- res$p.value irace.assert(!is.nan(PVAL) & !is.na(PVAL)) if (PVAL >= 1 - conf.level) dropped.any <- FALSE # We use the pseudo median estimated by the test. ranks <- if (res$estimate <= 0) c(1,2) else c(2,1) } best <- which.alive[ranks[1]] if (dropped.any) alive[which.alive[ranks[2]]] <- FALSE irace.assert(which.alive[which.min(ranks)] == best) return(list(best = best, ranks = ranks, alive = alive, dropped.any = dropped.any, p.value = PVAL)) } else { # If more then 2 configurations are left, use Friedman return (aux2.friedman(results, which.alive, alive, conf.level = conf.level)) } } aux.one_ttest <- function(results, alive, which.alive, conf.level, adjust = c("none","bonferroni","holm")) { adjust <- match.arg(adjust) irace.assert(sum(alive) == length(which.alive)) results <- results[, which.alive] means <- colMeans(results) best <- which.min(means) mean_best <- means[best] pvals <- sapply(means, function(x) as.numeric(isTRUE( all.equal.numeric(mean_best[[1]], x[[1]], check.attributes = FALSE)))) results_best <- results[, best] var_best <- var(results_best) which_test <- which(pvals < 1.0) for (j in which_test) { PVAL <- pvals[j] if (PVAL == 1.0) next results_j <- results[, j] # t.test may fail if the data in each group is almost constant. Hence, we # surround the call in a try() and we initialize p with 1 if the means are # equal or zero if they are different if (min(var(results_best), var(results_j)) < 10 * .Machine$double.eps) next # The t.test may fail if the data are not normal despite one configuration # clearly dominating the other. if (all(results_best <= results_j)) next try(PVAL <- t.test(results_best, results_j, alternative = "less", paired = TRUE)$p.value) irace.assert(!is.nan(PVAL) & !is.na(PVAL)) pvals[j] <- PVAL } pvals <- p.adjust(pvals, method = adjust) dropj <- which.alive[pvals < 1.0 - conf.level] dropped_any <- length(dropj) > 0 irace.assert(all(alive[dropj])) alive[dropj] <- FALSE return(list(best = which.alive[best], ranks = means, alive = alive, dropped.any = dropped_any, p.value = min(pvals))) } aux.ttest <- function(results, alive, which.alive, conf.level, adjust = c("none","bonferroni","holm")) { adjust <- match.arg(adjust) irace.assert(sum(alive) == length(which.alive)) results <- results[, which.alive] means <- colMeans(results) # FIXME: break ties using median or ranks? best <- which.min(means) mean_best <- means[best] pvals <- sapply(means, function(x) as.numeric(isTRUE( all.equal.numeric(mean_best[[1]], x[[1]], check.attributes = FALSE)))) results_best <- results[, best] var_best <- var(results_best) which_test <- which(pvals < 1.0) for (j in which_test) { PVAL <- pvals[j] if (PVAL == 1.0) next results_j <- results[, j] # t.test may fail if the data in each group is almost constant. Hence, we # surround the call in a try() and we initialize p with 1 if the means are # equal or zero if they are different if (min(var(results_best), var(results_j)) < 10 * .Machine$double.eps) next # The t.test may fail if the data are not normal despite one configuration # clearly dominating the other. if (all(results_best <= results_j)) next try(PVAL <- t.test(results_best, results_j, paired = TRUE)$p.value) irace.assert(!is.nan(PVAL) & !is.na(PVAL)) pvals[j] <- PVAL } pvals <- p.adjust(pvals, method = adjust) dropj <- which.alive[pvals < 1.0 - conf.level] dropped_any <- length(dropj) > 0 irace.assert(all(alive[dropj])) alive[dropj] <- FALSE return(list(best = which.alive[best], ranks = means, alive = alive, dropped.any = dropped_any, p.value = min(pvals))) } no_elitrace.init.instances <- function(deterministic, max_instances) { # if next.instance == 1 then this is the first iteration. # If deterministic consider all (do not resample). if (.irace$next.instance == 1 || deterministic) return(1:max_instances) irace.assert(.irace$next.instance < max_instances) return(.irace$next.instance : max_instances) } elitrace.init.instances <- function(race.env, deterministic, max_instances, sampleInstances) { # if next.instance == 1 then this is the first iteration. if (.irace$next.instance == 1) return(1:max_instances) # Consider all new.instances <- NULL last.new <- .irace$next.instance + race.env$elitistNewInstances - 1 # Do we need to add new instances? if (race.env$elitistNewInstances > 0) { if (last.new > max_instances) { # This may happen if the scenario is deterministic and we would need # more instances than what we have. irace.assert(deterministic) if (.irace$next.instance <= max_instances) { # Add all instances that we have not seen yet as new ones. last.new <- max_instances new.instances <- .irace$next.instance : last.new } # else new.instances remains NULL and last.new remains > number of instances. # We need to update this because the value is used below and now there # may be fewer than expected, even zero. race.env$elitistNewInstances <- length(new.instances) } else { new.instances <- .irace$next.instance : last.new } } future.instances <- NULL if ((last.new + 1) <= max_instances) { future.instances <- (last.new + 1) : max_instances } # new.instances + past.instances + future.instances # FIXME: we should sample taking into account the block-size, so we sample blocks, not instances. past_instances <- if (sampleInstances) sample.int(.irace$next.instance - 1) else 1:.irace$next.instance c(new.instances, past_instances, future.instances) } table_hline <- function(widths) { s <- "+" for(w in widths) { s <- paste0(s, strrep("-", w), "+") } return(paste0(s, "\n")) } table_sprint <- function(text, widths) { s <- "|" for (i in seq_along(text)) { s <- paste0(s, sprintf("%*s", widths[i], text[i]), "|") } return(paste0(s, "\n")) } .nocap_table_fields_width <- c(1, 11, 11, 11, 16, 11, 8, 5, 4, 6) .nocap_colum_names <- c(" ", "Instance", "Alive", "Best", "Mean best", "Exp so far", "W time", "rho", "KenW", "Qvar") .capping_table_fields_width <- c(1, 11, 8, 11, 11, 16, 11, 8, 5, 4, 6) .capping_colum_names <- c(" ", "Instance", "Bound", "Alive", "Best", "Mean best", "Exp so far", "W time", "rho", "KenW", "Qvar") capping_hline <- table_hline(.capping_table_fields_width) capping_header <- table_sprint(.capping_colum_names, .capping_table_fields_width) nocap_hline <- table_hline(.nocap_table_fields_width) nocap_header <- table_sprint(.nocap_colum_names, .nocap_table_fields_width) # FIXME: Depending on capping here is ugly. We should simply set-up the correct printing functions at the start of race(). race.print.header <- function(capping) { cat(sep = "", "# Markers: x No test is performed. c Configurations are discarded only due to capping. - The test is performed and some configurations are discarded. = The test is performed but no configuration is discarded. ! The test is performed and configurations could be discarded but elite configurations are preserved. . All alive configurations are elite and nothing is discarded\n\n") if (capping) cat(sep = "", capping_hline, capping_header, capping_hline) else cat(sep = "", nocap_hline, nocap_header, nocap_hline) } race.print.task <- function(res.symb, Results, instance, current.task, alive, id.best, best, experimentsUsed, start.time, bound, capping) { elapsed_wctime_str <- function(now, start) { if (now <= start) return("00:00:00") elapsed <- difftime(now, start, units = "secs") # FIXME: Maybe better and faster if we only print seconds? format(.POSIXct(elapsed, tz="GMT"), "%H:%M:%S") } # FIXME: This is the mean of the best, but perhaps it should # be the sum of ranks in the case of test == friedman? mean_best <- mean(Results[, best]) time_str <- elapsed_wctime_str(Sys.time(), start.time) cat(sprintf("|%s|%11d|", res.symb, instance)) if (capping) { if (is.null(bound)) cat(" NA|") else cat(sprintf("%8.2f|", bound)) } cat(sprintf(paste0("%11d|%11d|", .irace.format.perf, "|%11d|%s"), sum(alive), id.best, mean_best, experimentsUsed, time_str)) if (current.task > 1 && sum(alive) > 1) { conc <- concordance(Results[1:current.task, alive, drop = FALSE]) qvar <- dataVariance(Results[1:current.task, alive, drop = FALSE]) # FIXME: We would like to use %+#4.2f but this causes problems with # https://github.com/oracle/fastr/issues/191 cat(sprintf("|%+4.2f|%.2f|%.4f|\n", conc$spearman.rho, conc$kendall.w, qvar)) } else { cat("| NA| NA| NA|\n") } } race.print.footer <- function(bestconf, mean.best, break.msg, debug.level, capping = FALSE) { cat(sep = "", if (capping) capping_hline else nocap_hline, if (debug.level >= 1) paste0("# Stopped because ", break.msg, "\n"), sprintf("Best-so-far configuration: %11d", bestconf[1, ".ID."]), " mean value: ", sprintf(.irace.format.perf, mean.best), "\n", "Description of the best-so-far configuration:\n") configurations.print(bestconf, metadata = TRUE) cat("\n") } ## This function calculates an execution bound ## data: matrix columns as configurations and rows instances ## type: ## median: bound based on the configuration's mean median ## mean: bound based on the configuration's mean mean ## worst: bound based on the worst configuration's mean ## best: bound based on the best configurations's mean executionBound <- function(data, type = "median") { irace.assert (ncol(data) >= 1) if (ncol(data) == 1) { return (mean(data[,1], na.rm = TRUE)) } # This should never happen because the data used to obtain the execution # bound should be complete, that is, the bounding configurations should have # been executed on all previous instances. irace.assert (all(!is.na(data))) bound <- switch (type, median = median(colMeans(data)), mean = mean(colMeans(data)), worst = max(colMeans(data)), # default: min(colMeans(data))) return (bound) } ## This function calculates an execution bound per instance ## data: array of the execution times on the current instance ## type: ## median: bound based on the median time ## mean: bound based on the mean time ## worst: bound based on the worst time ## best: bound based on the best time instanceBound <- function(data, type="median") { irace.assert (all(!is.na(data))) bound <- switch (type, median = median(data, na.rm=TRUE), mean = mean(data, na.rm=TRUE), worst = max(data, na.rm=TRUE), # default: min(data, na.rm=TRUE)) return (bound) } ## This function returns survivors obtained after applying the dominance elimination ## criterion. ## results: matrix of experiments results (all configurations) ## elites: index of elite configurations ## alive: bolean array of alive configurations ## eps: constant added to the bound to account for the measuring precision dom.elim <- function(results, elites, alive, scenario, minSurvival, eps = 1e-5) { irace.assert(sum(alive) >= minSurvival) which.alive <- which(alive) cmeans <- colMeans(results[, alive, drop = FALSE]) irace.assert(!all(is.na(cmeans))) # Only NA values when calculating mean for dominance. # When there are no protected elites left, select the best configurations to # calculate the bounds. This is quite aggressive and another alternative # would be to disable dom.elim when elites == 0. if (length(elites) == 0) { # In the case we have only two alive configurations only one can be elite. if (sum(alive) <= 2) elites <- which.alive[which.min(cmeans)] else elites <- which.alive[order(cmeans, na.last = TRUE, decreasing = FALSE)[1:minSurvival]] } bound <- executionBound(results[, elites, drop = FALSE], type = scenario$cappingType) alive[which.alive] <- ((bound + eps) >= cmeans) return (alive) } ## This function applies PARX (X=boundPar) to all experiments ## that exceed the maximum execution time (boundMax) applyPAR <- function(results, boundMax, boundPar) { # We do not want to change Inf or -Inf because those represent rejection. if (boundPar != 1) results[is.finite(results) & results >= boundMax] <- boundMax * boundPar return(results) } ## This function calculates the execution time allowed for the executions ## of configurations based on previous execution times. ## It returns a list with two elements: ## * elite.bound : value (ei. mean execution time or per instance execution time) ## used to calculate the maximimum exeuction time (final.bounds) for each configuration. ## * final.bounds[i] : maximum execution time for candidate i on the current instance. final.execution.bound <- function(experimentsTime, elites, no.configurations, current.task, which.exe, scenario) { minMeasurableTime <- scenario$minMeasurableTime boundMax <- scenario$boundMax # FIXME: should we use an adjusted boundMax final.bounds <- rep(boundMax, no.configurations) total.time <- current.task * boundMax elite.bound <- boundMax # Elite candidates can have NA values due to the rejection if (length(elites) > 0) elites <- elites[!is.na(experimentsTime[current.task,elites])] # Only apply bounds when there is previous data if (length(elites) > 0 && length(which.exe) > 0) { # The elite membership is updated before calling this function to know # the configurations used for calculating the bounds we need to do this. # Note that some of these configurations could be not elite anymore for the # elimination phase, given that all their evaluations have been used. if (scenario$boundType == "instance") { elite.bound <- instanceBound(experimentsTime[current.task, elites], type = scenario$cappingType) final.bounds[which.exe] <- min(elite.bound + minMeasurableTime, boundMax) final.bounds[which.exe] <- ceiling.digits(final.bounds[which.exe], scenario$boundDigits) } else { elite.bound <- executionBound(experimentsTime[1:current.task, elites, drop = FALSE], type = scenario$cappingType) elite.bound <- min(elite.bound, boundMax) # FIXME: This minMeasurableTime should be a scenario setting and it # should be the same value that we use in check.output.target.runner total.time <- (current.task * elite.bound) + minMeasurableTime time.left <- total.time - colSums(experimentsTime[1:current.task, which.exe, drop = FALSE], na.rm = TRUE) final.bounds[which.exe] <- sapply(time.left, min, boundMax) # We round up the bounds up to the specified number of digits. This may # be necessary if the target-algorithm does not support higher precision. final.bounds[which.exe] <- ceiling.digits(final.bounds[which.exe], scenario$boundDigits) # There are cases in which a small negative budget is used. For example: # assuming candidates 1 and 2 are elite maxbound=80 # executionTime <- matrix(c(0.010,0.0170,0.010, 24,28,27, 0.010,0.017,NA), # byrow=TRUE, ncol=3, nrow=3, dimnames=list(c(1,2,3),c(1,2,3))) # current.task <- 3; boundDigits=0 # elite.bound <- irace:::executionBound(executionTime[1:current.task,1:2], "median") # total.time <- elite.bound * current.task + 0.01 # time.left <- total.time - colSums(executionTime[1:current.task,3,drop=FALSE], na.rm=TRUE) # We set the execution time to the elite.bound this should be enough # to eliminate a bad candidate for the next task. final.bounds[final.bounds <= 0] <- elite.bound irace.assert(all(final.bounds > 0)) } } return(list(final.bounds = final.bounds, elite.bound = elite.bound)) } # Recompute the best as follows. Given two configurations, the one evaluated # on more instances is ranked better. Otherwise, break ties according to the # criteria of the stat test. overall.ranks <- function(x, stat.test) { if (ncol(x) == 1) return(1) ninstances <- colSums(!is.na(x)) uniq.ninstances <- sort(unique(ninstances), decreasing = TRUE) last.r <- 0 ranks <- rep(Inf, ncol(x)) # Iterate from the largest to the lowest number of instances. for (k in uniq.ninstances) { confs <- which(ninstances == k) irace.assert(all(is.infinite(ranks[confs]))) r <- 1 if (length(confs) > 1) { # Select only non-NA rows y <- x[, confs, drop = FALSE] y <- y[complete.cases(y), , drop = FALSE] irace.assert(!any(is.na(y))) if (stat.test == "friedman") { r <- colSums(t(apply(y, 1L, rank))) } else { r <- rank(colMeans(y)) } } r <- r + last.r last.r <- max(r) ranks[confs] <- r } return(ranks) } # Remove one elite count from every configuration not executed. update.is.elite <- function(is.elite, which.exe) { which.notexecuted <- setdiff(which(is.elite > 0), which.exe) is.elite[which.notexecuted] <- is.elite[which.notexecuted] - 1 irace.assert (all(is.elite >= 0)) return(is.elite) } update.elite.safe <- function(Results, is.elite) { if (any(is.elite > 0L)) return(max(which(apply(!is.na(Results[, is.elite > 0, drop=FALSE]), 1, any)))) # All elites rejected. return(0L) } race <- function(maxExp = 0, minSurvival = 1, elite.data = NULL, configurations, parameters, scenario, elitistNewInstances) { race.env <- new.env(parent = emptyenv()) # FIXME: We should take this from scenario. However, this value should be # zero for the first iteration. ## FIXME2: Probably, instead of this, we should keep elite.safe in the race.env. race.env$elitistNewInstances <- elitistNewInstances stat.test <- scenario$testType conf.level <- scenario$confidence first.test <- scenario$firstTest each.test <- scenario$eachTest elitist <- scenario$elitist capping <- scenario$capping quiet <- scenario$quiet no.configurations <- nrow(configurations) experimentLog <- matrix(nrow = 0, ncol = 4, dimnames = list(NULL, c("instance", "configuration", "time", "bound"))) alive <- rep(TRUE, no.configurations) is.rejected <- rep(FALSE, no.configurations) ## FIXME: Remove argument checking. This must have been done by the caller. # Check argument: maxExp if (!missing(maxExp) && (!is.numeric(maxExp) || length(maxExp)!=1 || !is.finite(maxExp))) stop("maxExp must be an single number") maxExp <- ifelse(maxExp>0,maxExp,0) maxExp <- floor(maxExp) if (maxExp && no.configurations > maxExp) irace.error("Max number of experiments is smaller than number of configurations") if (no.configurations <= minSurvival) { irace.error("Not enough configurations (", no.configurations, ") for a race (minSurvival=", minSurvival, ")") } # Check argument: conf.level if (!missing(conf.level) && (!is.numeric(conf.level) || length(conf.level)!=1 || !is.finite(conf.level) || conf.level < 0 || conf.level > 1)) stop("conf.level must be a single number between 0 and 1") # Create the instance list according to the algorithm selected if (elitist) race.instances <- elitrace.init.instances(race.env, scenario$deterministic, max_instances = nrow(.irace$instancesList), sampleInstances = scenario$sampleInstances) else race.instances <- no_elitrace.init.instances(scenario$deterministic, max_instances = nrow(.irace$instancesList)) no.tasks <- length(race.instances) # Initialize some variables... experimentsUsed <- 0L # is.elite[i] : number of instances to be seen in this race on which i has # been previously evaluated. is.elite <- rep(0L, no.configurations) if (is.null(elite.data)) { elite.safe <- 0L elite.instances.ID <- NULL } else { irace.assert(.irace$next.instance - 1 == nrow(elite.data$experiments)) # There must be a non-NA entry for each instance. irace.assert(all(apply(!is.na(elite.data$experiments), 1, any)), eval.after = { print(elite.data$experiments)}) # There must be a non-NA entry for each configuration. irace.assert(all(apply(!is.na(elite.data$experiments), 2, any))) # elite.safe: maximum instance number for which any configuration may be # considered elite. After evaluating this instance, no configuration is # elite. elite.safe <- race.env$elitistNewInstances + nrow(elite.data[["experiments"]]) elite.instances.ID <- as.character(race.instances[1:elite.safe]) } configurations.ID <- as.character(configurations[, ".ID."]) Results <- matrix(NA, nrow = elite.safe, ncol = no.configurations, dimnames = list(elite.instances.ID, configurations.ID)) if (capping) experimentsTime <- matrix(NA, nrow = elite.safe, ncol = no.configurations, dimnames = list(elite.instances.ID, configurations.ID)) if (! is.null(elite.data)) { Results[rownames(elite.data[["experiments"]]), colnames(elite.data[["experiments"]])] <- elite.data[["experiments"]] if (capping) { experimentsTime[rownames(elite.data[["time"]]), colnames(elite.data[["time"]])] <- elite.data[["time"]] } # Preliminary execution of elite configurations to calculate # the execution bound of initial configurations (capping only). if (capping && elitistNewInstances != 0) { # FIXME: This should go into its own function. n.elite <- ncol(elite.data[["experiments"]]) which.elites <- which(rep(TRUE, n.elite)) irace.note("Preliminary execution of ", n.elite, " elite configuration(s) over ", elitistNewInstances, " instance(s).\n") for (k in 1:elitistNewInstances) { output <- race.wrapper (configurations = configurations[which.elites, , drop = FALSE], instance.idx = race.instances[k], bounds = rep(scenario$boundMax, n.elite), which.alive = which.elites, which.exe = which.elites, parameters = parameters, scenario = scenario) # Extract results # FIXME: check what would happen in case of having the target evaluator # MANUEL: Note how similar is this to what we do in do.experiments(), # perhaps we can create a function that takes output and experimentLog # and returns experimentLog. # LESLIE: Yes you are right, Ill do it once we figure out the rest! vcost <- unlist(lapply(output, "[[", "cost")) irace.assert(length(vcost) == n.elite) vcost <- applyPAR(vcost, boundMax = scenario$boundMax, boundPar = scenario$boundPar) Results[k, 1:n.elite] <- vcost vtimes <- unlist(lapply(output, "[[", "time")) irace.assert(length(vtimes) == n.elite) experimentsTime[k, which.elites] <- vtimes experimentLog <- rbind(experimentLog, cbind(race.instances[k], configurations[which.elites, ".ID."], vtimes, scenario$boundMax)) experimentsUsed <- experimentsUsed + n.elite # We remove elite configurations that are rejected given that # is not possible to calculate the bounds. rejected <- is.infinite(Results[k, which.elites]) is.rejected[which.elites] <- rejected which.elites <- which.elites[!rejected] n.elite <- length(which.elites) # If all elite are eliminated we stop execution of initial instances. if (n.elite == 0L) { irace.note ("All elite configurations are rejected. Execution of non-elites will be not bounded.\n") break } } if (any(is.rejected)) { irace.note ("Immediately rejected configurations: ", paste0(configurations[is.rejected, ".ID."], collapse = ", ") , "\n") alive[is.rejected] <- FALSE # Calculate the maximum instance that has any non-NA value. # FIXME: Use update.elite.safe() if (n.elite > 0L) elite.safe <- max(which(apply(!is.na(Results[, which.elites, drop=FALSE]), 1, any))) else elite.safe <- 0L } } # Compute the elite membership. is.elite <- colSums(!is.na(Results)) # Remove rejected configurations. is.elite[is.rejected] <- 0L } best <- 0 race.ranks <- c() no.elimination <- 0 # number of tasks without elimination. if (!quiet) race.print.header(capping) # Test that all instances that have been previously seen have been evaluated # by at least one configuration. all_elite_instances_evaluated <- function() { if (!elitist) return(TRUE) return(all(apply(!is.na(Results[ as.character(seq_len(.irace$next.instance - 1)), alive, drop=FALSE]), 1, any))) } # Start main loop break.msg <- NULL for (current.task in seq_len(no.tasks)) { which.alive <- which(alive) nbAlive <- length(which.alive) which.exe <- which.alive if (elitist && any(is.elite > 0)) { # Filter configurations that do not need to be executed (elites). # This is valid only for previous iteration instances. irace.assert(current.task <= elite.safe) # Execute everything that is alive and not yet executed. which.exe <- which(alive & is.na(Results[current.task, ])) if (length(which.exe) == 0) { is.elite <- update.is.elite(is.elite, which.exe) # LESLIE: This is the case in which there are only elite configurations alive # and we are still in the previous instances execution, but we can still # continue with the race. (This is only possible because the early termination # criterion is disabled) ## MANUEL: So what is the reason to not immediately terminate here? Is ## there a reason to continue? if (!quiet) race.print.task(".", Results[1:current.task, , drop = FALSE], race.instances[current.task], current.task, alive = alive, configurations[best, ".ID."], best = best, experimentsUsed, Sys.time(), bound = NA, capping) next } } # LESLIE: FIXME: Stopping deactivated by Thomas suggestion. Remove second # condition to restore. # LESLIE: Should we keep the early termination disabled? The difference between keeping it or # not is that elite configurations could be eliminated later # LESLIE: I think we should remove this ## We continue running if (1) we have not reached the first.test or (2) ## there are instances previously seen that have not been evaluated on any ## alive configuration. if (current.task > first.test) { #if ((current.task > first.test && !capping) # MANUEL: This is new and I'm not sure what it does. # LESLIE: When using capping, we dont finish any race until all # previous instances have been executed (this makes sure that all non-elite # configurations execute all the previous instances) # || (capping && (current.task > elite.safe))) { # MANUEL: How is this even possible? # LESLIE: It can be that the capping eliminate all but one configuration # (which should be an elite one) after we finish the new instances to be evaluated, # we allow the race to be finished. Maybe it wold be better: sum(is.elite) == sum(alive) # instead of nbAlive == 1, # LESLIE:Removing this because now is ponitless because of the elite candidates previos # execution # || (current.task > elitistNewInstances && nbAlive == 1)))) { # We always stop when we have less configurations than required. if (nbAlive <= minSurvival && all_elite_instances_evaluated()) { # Stop race if we have less or equal than the minimum number of # configurations. break.msg <- paste0("number of alive configurations (", nbAlive, ") <= minimum number of configurations (", minSurvival, ")") break } # If we just did a test, check that we have enough budget to reach the # next test. if (maxExp && ( (current.task - 1) %% each.test) == 0 && experimentsUsed + length(which.exe) * each.test > maxExp && all_elite_instances_evaluated()) { break.msg <- paste0("experiments for next test (", experimentsUsed + length(which.exe) * each.test, ") > max experiments (", maxExp, ")") break } } if (elitist) { if (scenario$elitistLimit != 0 && no.elimination >= scenario$elitistLimit && all_elite_instances_evaluated()) { break.msg <- paste0("tests without elimination (", no.elimination, ") >= elitistLimit (", scenario$elitistLimit, ")") break } ## This is not needed anymore... # else if (current.task > initial.tests && nbAlive <= minSurvival) { # # We can stop the race ONLY when we pass the elite.safe # # this is because how we are recovering the data from # # previous runs (based on iteration). # break.msg <- paste0("number of alive configurations (", nbAlive, # ") less or equal than minimum number (", # minSurvival, ")") # break # } } if (nrow(Results) < current.task) { Results <- rbind(Results, rep(NA, ncol(Results))) rownames(Results) <- race.instances[1:nrow(Results)] if (capping) { experimentsTime <- rbind(experimentsTime, rep(NA, ncol(experimentsTime))) rownames(experimentsTime) <- race.instances[1:nrow(experimentsTime)] } } start.time <- Sys.time() # Execution bounds calculation (capping only) final.bounds <- elite.bound <- NULL # Calculate bounds for executing if needed. which.elite.exe <- intersect(which.exe, which(is.elite > 0)) irace.assert(setequal(which.elite.exe, which(is.elite & is.na(Results[current.task,])))) if (capping) { # Pre-execute elite configurations that are not yet executed in the current instance. if (length(which.elite.exe)) { # FIXME: This should go into its own function output <- race.wrapper (configurations = configurations[which.elite.exe, , drop = FALSE], instance.idx = race.instances[current.task], bounds = rep(scenario$boundMax, length(which.elite.exe)), # MANUEL: How does this work for target-evaluator? # We are telling race.wrapper that only some elites are alive! which.alive = which.elite.exe, which.exe = which.elite.exe, parameters = parameters, scenario = scenario) # Extract results vcost <- unlist(lapply(output, "[[", "cost")) irace.assert(length(vcost) == length(which.elite.exe)) vcost <- applyPAR(vcost, boundMax = scenario$boundMax, boundPar = scenario$boundPar) Results[current.task, which.elite.exe] <- vcost vtimes <- unlist(lapply(output, "[[", "time")) irace.assert(length(vtimes) == length(which.elite.exe)) experimentsTime[current.task, which.elite.exe] <- vtimes experimentLog <- rbind(experimentLog, cbind(race.instances[current.task], configurations[which.elite.exe, ".ID."], vtimes, scenario$boundMax)) experimentsUsed <- experimentsUsed + length(which.elite.exe) # We remove elite configurations that are rejected given that # is not possible to calculate the bounds rejected <- is.infinite(Results[current.task, which.elite.exe]) if (any(rejected)) { irace.note ("Immediately rejected configurations: ", paste0(configurations[which.elite.exe[rejected], ".ID."], collapse = ", ") , "\n") is.rejected[which.elite.exe] <- rejected is.elite [is.rejected] <- 0 alive[which.elite.exe] <- !rejected if (!any(alive)) irace.error("All configurations have been immediately rejected (all of them returned Inf) !") which.alive <- which(alive) nbAlive <- length(which.alive) elite.safe <- update.elite.safe(Results, is.elite) } which.exe <- setdiff(which.exe, which.elite.exe) # FIXME: There is similar code above. if (length(which.exe) == 0L) { is.elite <- update.is.elite(is.elite, which.elite.exe) if (!quiet) race.print.task(".", Results[1:current.task, , drop = FALSE], race.instances[current.task], current.task, alive = alive, configurations[best, ".ID."], best = best, experimentsUsed, start.time, bound = NA, capping) next } } all.bounds <- final.execution.bound(experimentsTime, elites = which(is.elite > 0), no.configurations, current.task, which.exe, scenario) final.bounds <- all.bounds$final.bounds elite.bound <- all.bounds$elite.bound } else { final.bounds <- rep(scenario$boundMax, no.configurations) } # Execute experiments output <- race.wrapper (configurations = configurations[which.alive, , drop = FALSE], instance.idx = race.instances[current.task], # FIXME: Why are we computing bounds for configurations that are dead? # Also, do we use the final.bounds of which.alive or only the ones of which.exe? bounds = final.bounds[which.alive], which.alive = which.alive, which.exe = which.exe, parameters = parameters, scenario = scenario) # Extract results vcost <- unlist(lapply(output, "[[", "cost")) # If the experiment was executed or target.evaluator exists # then the result is in the output. ## Currently, targetEvaluator always re-evaluates, which implies that the ## value may change. We do this to allow online normalization. which.exps <- if (is.null(scenario$targetEvaluator)) which.exe else which.alive irace.assert(length(vcost) == length(which.exps)) # Set max execution bound to timed out executions which have execution # times smaller than boundMax and implement parX if required if (capping) { vcost <- applyPAR(vcost, boundMax = scenario$boundMax, boundPar = scenario$boundPar) if (scenario$boundAsTimeout) vcost[(vcost >= final.bounds[which.exps]) & (vcost < scenario$boundMax)] <- scenario$boundMax } Results[current.task, which.exps] <- vcost # Output is not indexed in the same way as configurations. which.exps <- which(which.alive %in% which.exe) irace.assert(length(which.exps) == length(which.exe)) vtimes <- unlist(lapply(output[which.exps], "[[", "time")) irace.assert(length(vtimes) == length(which.exe)) if (capping) { # Correct higher execution times. experimentsTime[current.task, which.exps] <- pmin(vtimes, final.bounds[which.exps]) } experimentLog <- rbind(experimentLog, cbind(race.instances[current.task], configurations[which.exe, ".ID."], vtimes, if (is.null(final.bounds)) NA else final.bounds[which.exe])) experimentsUsed <- experimentsUsed + length(which.exe) # We update the elites that have been executed. is.elite <- update.is.elite(is.elite, which.elite.exe) ## Drop bad configurations. ## Infinite values denote immediate rejection of a configuration. rejected <- is.infinite(Results[current.task, which.exe]) if (any(rejected)) { irace.note ("Immediately rejected configurations: ", paste0(configurations[which.exe[rejected], ".ID."], collapse = ", ") , "\n") is.rejected[which.exe] <- rejected is.elite[is.rejected] <- 0 alive[is.rejected] <- FALSE if (!any(alive)) irace.error("All configurations have been immediately rejected (all of them returned Inf) !") which.alive <- which(alive) nbAlive <- length(which.alive) # FIXME: Should we stop if (nbAlive <= minSurvival) ??? elite.safe <- update.elite.safe(Results, is.elite) } irace.assert(!anyNA(Results[1:current.task, alive, drop=FALSE])) irace.assert(!any(is.infinite(Results[, alive, drop=FALSE]))) # Variables required to produce output of elimination test. cap.done <- FALSE #if dominance elimination was performed test.done <- FALSE #if statistical test elimination was performed cap.dropped <- FALSE #if capping has drop any configuration test.dropped <- FALSE #if any candidates has been eliminated by testing cap.alive <- test.alive <- alive ## Dominance elimination (Capping only). # The second condition can be false if we eliminated via immediate # rejection if (capping && sum(alive) > minSurvival) { irace.assert(!any(is.elite > 0) == (current.task >= elite.safe)) cap.alive <- dom.elim(Results[1:current.task, , drop = FALSE], # Get current elite configurations elites = which(is.elite > 0), alive, scenario, minSurvival) cap.dropped <- sum(alive) > sum(cap.alive) cap.done <- TRUE } # We assume that first.test is a multiple of each.test. In any # case, this will only do the first test after the first multiple # of each.test that is larger than first.test. if (current.task >= first.test && (current.task %% each.test) == 0 && nbAlive > 1L) { irace.assert(sum(alive) == nbAlive) test.res <- switch(stat.test, friedman = aux_friedman(Results[1:current.task, ], alive, which.alive, conf.level), t.none = aux.ttest(Results[1:current.task, ], alive, which.alive, conf.level, adjust = "none"), t.holm = aux.ttest(Results[1:current.task, ], alive, which.alive, conf.level, adjust = "holm"), t.bonferroni = aux.ttest(Results[1:current.task, ], alive, which.alive, conf.level, adjust = "bonferroni")) best <- test.res$best race.ranks <- test.res$ranks test.alive <- test.res$alive test.dropped <- sum(alive) > sum(test.alive) test.done <- TRUE } # Merge the result of both eliminations. prev.sum.alive <- sum(alive) alive <- cap.alive & test.alive # Handle elites when elimination is performed. The elite configurations # can be removed only when they have no more previously-executed instances. irace.assert(!any(is.elite > 0) == (current.task >= elite.safe)) if (!is.null(elite.data) && any(is.elite > 0)) { irace.assert (length(alive) == length(is.elite)) alive <- alive | (is.elite > 0) } # It may happen that the capping and the test eliminate together all # configurations. In that case, we only trust the capping elimination. if (capping && !any(alive)) { if (scenario$debugLevel >= 2) { irace.warning("Elimination tests have eliminated all configurations, keeping the capping results.\n") irace.note("Alive according to capping:", which(cap.alive), "\n") irace.note("Alive according to test:", which(test.alive), "\n") } alive <- cap.alive } # Output the result of the elimination test res.symb <- if (cap.dropped && !test.dropped && prev.sum.alive != sum(alive)) { "c" # Removed just by capping } else if (cap.dropped || test.dropped) { if (prev.sum.alive != sum(alive)) "-" else "!" } else if (cap.done || test.done) "=" else "x" # Rank alive configurations: order all configurations (eliminated or not) # LESLIE: we have to make the ranking outside: we can have configurations eliminated by capping # that are not eliminated by the test. # MANUEL: I don't understand the above comment. if (length(which.alive) == 1) { race.ranks <- 1 best <- which.alive } else { tmpResults <- Results[1:current.task, which.alive, drop = FALSE] irace.assert(!any(is.na(tmpResults))) if (stat.test == "friedman") { race.ranks <- colSums(t(apply(tmpResults, 1, rank))) } else { race.ranks <- colMeans(tmpResults) } # which.min returns only the first minimum. best <- which.alive[which.min(race.ranks)] } irace.assert(best == which.alive[order(race.ranks)][1]) irace.assert(length(race.ranks) == length(which.alive)) prev.alive <- which.alive which.alive <- which(alive) # Remove the ranks of those that are not alive anymore race.ranks <- race.ranks[which.alive] irace.assert(length(race.ranks) == sum(alive)) if (!quiet) race.print.task(res.symb, Results[1:current.task, , drop = FALSE], race.instances[current.task], current.task, alive = alive, configurations[best, ".ID."], best = best, experimentsUsed, start.time, bound = elite.bound, capping) if (elitist) { # Compute number of statistical tests without eliminations. irace.assert(!any(is.elite > 0) == (current.task >= elite.safe)) if (!any(is.elite > 0) && current.task > first.test && (current.task %% each.test) == 0) { if (length(which.alive) == length(prev.alive)) { no.elimination <- no.elimination + 1 } else { no.elimination <- 0 } } } } if (is.null(break.msg)) break.msg <- paste0("all instances (", no.tasks, ") evaluated") # Adding this given that when ncandidates = minsurvival+1 # and there one elite configuration that gets discarded in the new instances # execution the race is finished with no executions. # FIXME: we should handle this better, maybe allowing irace to handle no elite # in irace() # MANUEL: Leslie, how can we reach this error in normal circumstances? # Can we handle this better? if (current.task == 1 && !any(is.elite > 0)) irace.error ("Maximum number configurations immediately rejected reached!") # All instances that are not new in this race must have been evaluated by at # least one configuration. irace.assert(all_elite_instances_evaluated(), eval.after = { print(Results[,alive, drop=FALSE])}) # If we stop the loop before we see all new instances, there may be new # instances that have not been executed by any configuration. Results <- Results[apply(!is.na(Results), 1, any), , drop = FALSE] race.ranks <- overall.ranks(Results[, alive, drop = FALSE], stat.test = stat.test) best <- which.alive[which.min(race.ranks)] if (!quiet) race.print.footer(bestconf = configurations[best, , drop = FALSE], # FIXME: This is the mean of the best, but perhaps it # should be the sum of ranks in the case of test == # friedman? mean.best = mean(Results[, best]), break.msg = break.msg, debug.level = scenario$debugLevel, capping = capping) nbAlive <- sum(alive) configurations$.ALIVE. <- as.logical(alive) # Assign the proper ranks in the configurations data.frame. configurations$.RANK. <- Inf configurations[which.alive, ".RANK."] <- race.ranks # Now we can sort the data.frame by the rank. configurations <- configurations[order(as.numeric(configurations[, ".RANK."])), ] # Consistency check. irace.assert (all(configurations[1:nbAlive, ".ALIVE."])) if (nbAlive < nrow(configurations)) irace.assert(!any(configurations[(nbAlive + 1):nrow(configurations), ".ALIVE."])) if (scenario$debugLevel >= 3) { irace.note ("Memory used in race():\n") irace.print.memUsed() } # nrow(Results) may be smaller, equal or larger than current.task. irace.assert(nrow(experimentLog) == experimentsUsed) return(list(experiments = Results, experimentLog = experimentLog, experimentsUsed = experimentsUsed, nbAlive = nbAlive, configurations = configurations, rejectedIDs = configurations[is.rejected, ".ID."])) } irace/R/model.R0000644000176200001440000001665214157104332012773 0ustar liggesusers#################################################### ## INITIALISE AND UPDATE THE MODEL #################################################### ## Initialisation of the model after the first iteration ## # IN: configurations matrix, parameters datastructure ## # OUTPUT: A list of list of vectors. The higher-level list contains # one element per categorical parameter. Each categorical parameter # contains a list of vector. This list contains elements which are the # .ID. of the configuration. initialiseModel <- function (parameters, configurations, digits) { model <- list() nbConfigurations <- nrow(configurations) for (currentParameter in parameters$names[!parameters$isFixed]) { type <- parameters$types[[currentParameter]] nbValues <- length(parameters$domain[[currentParameter]]) if (type == "c") { value <- rep((1 / nbValues), nbValues) } else if (type %in% c("i","r")) { value <- init.model.numeric(currentParameter, parameters) } else { irace.assert(type == "o") value <- (nbValues - 1) / 2 } param <- list() for (indexConfig in seq_len(nbConfigurations)) { idCurrentConfig <- as.character(configurations[indexConfig, ".ID."]) # Assign current parameter value to model if (type %in% c("i","r")) { value[2] <- configurations[indexConfig, currentParameter] } param[[idCurrentConfig]] <- value } model[[currentParameter]] <- param } return (model) } ## FIXME (MANUEL): This function needs a description. ## Update the model updateModel <- function (parameters, eliteConfigurations, oldModel, indexIteration, nbIterations, nbNewConfigurations, scenario) { newModel <- list() for (idxConfiguration in seq_len(nrow(eliteConfigurations))) { idCurrentConfiguration <- eliteConfigurations[idxConfiguration, ".ID."] idCurrentConfiguration <- as.character(idCurrentConfiguration) for (currentParameter in parameters$names[!parameters$isFixed]) { type <- parameters$types[[currentParameter]] ## If the elite is older than the current iteration, it has ## its own model that has evolved with time. If the elite is ## new (generated in the current iteration), it does not have ## any, and we have to copy the one from its parent. The ## condition of the IF statement is for checking wether the ## configuration already has its model or not. # FIXME: FIX character IDs, they should be numeric! if (idCurrentConfiguration %in% names(oldModel[[currentParameter]])) { # cat("This configuration has already an entry, to be updated\n") probVector <- oldModel[[currentParameter]][[idCurrentConfiguration]] } else { # cat("This configuration does not have any entry, copy the parent one\n") idParent <- eliteConfigurations[idxConfiguration, ".PARENT."] irace.assert(as.integer(idParent) < as.integer(idCurrentConfiguration)) idParent <- as.character(idParent) # cat("The parent found is ", idParent, "\n") probVector <- oldModel[[currentParameter]][[idParent]] # Change the current parameter value of the model if (type %in% c("i", "r") && !is.na(eliteConfigurations[idCurrentConfiguration,currentParameter])) probVector[2] <- eliteConfigurations[idCurrentConfiguration,currentParameter] } # cat("probVector: ", probVector) if (type == "c") { actualValue <- eliteConfigurations[idxConfiguration, currentParameter] if (is.na(actualValue)) { # cat ("NA found, don't change the prob vector") } else { possibleValues <- parameters$domain[[currentParameter]] # Decrease first all values in the vector: probVector <- probVector * (1 - ((indexIteration - 1) / nbIterations)) # cat("new probVector after decrease: ", probVector) # Find the value that has been "chosen" to increase its probability. indexValue <- which (possibleValues == actualValue) probVector[indexValue] <- (probVector[indexValue] + ((indexIteration - 1) / nbIterations)) # cat("The value found for the configuration n.", # idxConfiguration, "(ID=", # idCurrentConfiguration, ") is the ", indexValue, # "th.\n") # Prevent probabilities from growing too much. if (scenario$elitist) { probVector <- probVector / sum(probVector) probMax <- 0.2^(1 / parameters$nbVariable) probVector <- pmin(probVector, probMax) } # Normalize probabilities. probVector <- probVector / sum(probVector) #print("newProbVector after increase: ") #print(newVector) } } else { irace.assert(type %in% c("i", "r", "o")) probVector[1] <- probVector[1] * ((1 / nbNewConfigurations)^(1 / parameters$nbVariable)) } newModel[[currentParameter]][[idCurrentConfiguration]] <- probVector } } return (newModel) } printModel <- function (model) { cat("# Model:\n") print(model) } restartConfigurations <- function (configurations, restart.ids, model, parameters, nbConfigurations, digits) { #print(configurations) tmp.ids <- c() for (param in parameters$names[!parameters$isFixed]) { for (id in restart.ids) { if (!(id %in% names(model[[param]]))) { id <- configurations[configurations$.ID. == id, ".PARENT."] } tmp.ids <- c(tmp.ids, id) } } restart.ids <- unique(tmp.ids) #print(restart.ids) for (param in parameters$names[!parameters$isFixed]) { type <- parameters$types[[param]] for (id in restart.ids) { id <- as.character(id) irace.assert (id %in% names(model[[param]])) if (type == "c") { probVector <- model[[param]][[id]] probVector <- 0.9 * probVector + 0.1 * max(probVector) model[[param]][[id]] <- probVector / sum(probVector) } else { if (type == "i" || type == "r") { value <- c(init.model.numeric(param, parameters), # We keep the value of the configuration as last known configurations[id, param]) } else { irace.assert(type == "o") value <- (length(parameters$domain[[param]]) - 1) / 2 } # Bring back the value 2 iterations or to the second iteration value. stdev <- model[[param]][[id]][1] model[[param]][[id]][1] <- min(stdev * (nbConfigurations^(2 / parameters$nbVariable)), value[1] * ((1 / nbConfigurations)^(1 / parameters$nbVariable))) } } } return (model) } # Initialise model in case of numerical variables. # it retuns an array size 2, first number indicates the # standard deviation and second the last known value (initially NA) init.model.numeric <- function(param, parameters) { # Dependent parameters define the standard deviation as # a portion of the size of the domain interval. In this case, # 0.5 indicates half of the interval, equivalent to # (domain[2] - domain[1]) * 0.5 if (parameters$isDependent[[param]]) { return(0.5) } transf <- parameters$transform[[param]] if (transf == "log") { domain <- c(0,1) } else { domain <- parameters$domain[[param]] } value <- (domain[2] - domain[1]) / 2.0 irace.assert(is.finite(value)) return(value) } irace/R/timer.R0000644000176200001440000000061114252650732013005 0ustar liggesusersTimer <- R6::R6Class("Timer", cloneable = FALSE, list( start = NULL, initialize = function() { self$start <- proc.time() self }, elapsed = function() { x <- proc.time() - self$start if (!is.na(x[4L])) x[1L] <- x[1L] + x[4L] if (!is.na(x[5L])) x[2L] <- x[2L] + x[5L] x <- x[1L:3L] names(x) <- c("user", "system", "wallclock") x }) ) irace/R/ablation.R0000644000176200001440000006147714321300675013472 0ustar liggesusers.ablation.params.def <- read.table(header=TRUE, stringsAsFactors = FALSE, text=" name type short long default description iraceResults p -l --log-file NA 'Path to the (.Rdata) file created by irace from which the \"iraceResults\" object will be loaded.' src i -S --src 1 'Source configuration ID.' target i -T --target NA 'Target configuration ID. By default the best configuration found by irace.' ab.params s -P --params '' 'Specific parameter names to be used for the ablation (separated with commas). By default use all' type s -t --type 'full' 'Type of ablation to perform: \"full\" will execute each configuration on all \"--n-instances\" to determine the best-performing one; \"racing\" will apply racing to find the best configurations.' n_instances i -n --n-instances 1 'Number of instances used in \"full\" ablation will be n_instances * scenario$firstTest.' seed i '' --seed 1234567 'Integer value to use as seed for the random number generation.' ablationLogFile p -o --output-file 'log-ablation.Rdata' 'Log file to save the ablation log. If \"\", the results are not saved to a file.' plot s -p --plot '' 'Output filename (.pdf) for the plot. If not given, no plot is created.' plot_type s -O --plot-type 'mean' 'Type of plot. Supported values are \"mean\" and \"boxplot\".' old_path p '' --old-path NA 'Old path found in the log-file (.Rdata) given as input to be replaced by --new-path.' new_path p '' --new-path NA 'New path to replace the path found in the log-file (.Rdata) given as input.' execDir p -e --exec-dir NA 'Directory where the target runner will be run.' scenarioFile p -s --scenario NA 'Scenario file to override the scenario given in the log-file (.Rdata)' parallel i '' --parallel NA 'Number of calls to targetRunner to execute in parallel. Values 0 or 1 mean no parallelization.' ") cat_ablation_license <- function() { ablation_license <- '#------------------------------------------------------------------------------ # ablation: An implementation in R of Ablation Analysis # Version: __VERSION__ # Copyright (C) 2020--2022 # Manuel Lopez-Ibanez # Leslie Perez Caceres # # This is free software, and you are welcome to redistribute it under certain # conditions. See the GNU General Public License for details. There is NO # WARRANTY; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. #------------------------------------------------------------------------------ ' cat(sub("__VERSION__", irace.version, ablation_license, fixed=TRUE)) } #' Launch ablation with command-line options. #' #' Launch [ablation()] with the same command-line options as the command-line #' executable (`ablation.exe` in Windows). #' #' @param argv (`character()`) \cr The arguments #' provided on the R command line as a character vector, e.g., #' `c("-i", "irace.Rdata", "--src", 1)`. #' #' @details The function reads the parameters given on the command line #' used to invoke R, launches [ablation()] and possibly [plotAblation()]. #' #' List of command-line options: #' ```{r echo=FALSE,comment=NA} #' cmdline_usage(.ablation.params.def) #' ``` #' @template ret_ablog #' @examples #' # ablation_cmdline("--help") #' #' @author Manuel López-Ibáñez #' @concept running #' @export ablation_cmdline <- function(argv = commandArgs(trailingOnly = TRUE)) { op <- options(width = 9999L) # Do not wrap the output. on.exit(options(op), add = TRUE) cat_ablation_license() cat ("# installed at: ", system.file(package="irace"), "\n", "# called with: ", paste(argv, collapse = " "), "\n", sep = "") parser <- CommandArgsParser$new(argv = argv, argsdef = .ablation.params.def) if (!is.null(parser$readArg (short = "-h", long = "--help"))) { parser$cmdline_usage() return(invisible(NULL)) } if (!is.null(parser$readArg (short = "-v", long = "--version"))) { print(citation(package="irace")) return(invisible(NULL)) } params <- parser$readAll() # TODO: Send the other options to the irace command-line parser so the user # can override options in scenario via the command-line. if (length(parser$argv) > 0) stop("Unknown command-line options: ", paste(parser$argv, collapse = " ")) if (is.null(params$iraceResults)) { irace.error("You must provide the path to the '.Rdata' file that contains the 'iraceResults' object generated by irace.") return(invisible(NULL)) } iraceResults <- read_logfile(params$iraceResults) if (is.null(params$old_path) != is.null(params$new_path)) { irace.error("To update paths you must provide both --old-path and --new-path.") return(invisible(NULL)) } else if (!is.null(params$old_path)) { iraceResults$scenario <- scenario_update_paths(iraceResults$scenario, params$old_path, params$new_path) } if (!is.null(params$scenarioFile)) { scenario <- readScenario(params$scenarioFile) } if (is_null_or_empty_or_na(trim(params$ablationLogFile))) { params$ablationLogFile <- NULL } for (p in c("execDir", "parallel")) { if (!is.null(params[[p]])) scenario[[p]] <- params[[p]] } if (!is.null(params$ablationLogFile)) params$ablationLogFile <- path_rel2abs(params$ablationLogFile) if (!is.null(params$ab.params)) params$ab.params <- sapply(strsplit(params$ab.params, ",")[[1]], trimws, USE.NAMES=FALSE) ablog <- do.call(ablation, args = c(list(iraceResults = iraceResults, src = params$src, target = params$target, ab.params = params$ab.params, type = params$type, n_instances = params$n_instances, seed = params$seed, ablationLogFile = params$ablationLogFile), scenario)) if (!is.null(params[["plot"]]) || base::interactive()) { plotAblation(ablog, pdf.file = params[["plot"]], type = params[["plot_type"]]) } invisible(ablog) } ## This function fixes dependent parameters when a parameter value has been ## changed. fixDependenciesWithReference <- function(configuration, ref.configuration, parameters) { # Search parameters that need a value changed <- c() for (pname in names(which(!parameters[["isFixed"]]))) { # If dependent parameter has been activated, set the value of the reference. if (is.na(configuration[,pname]) && conditionsSatisfied(parameters, configuration, pname)) { if (!is.null(ref.configuration)) { configuration[,pname] <- ref.configuration[pname] } changed <- c(changed, pname) # MANUEL: Why do we need to recurse here? aux <- fixDependenciesWithReference(configuration=configuration, ref.configuration=ref.configuration, parameters) changed <- c(changed, aux$changed) configuration <- aux$configuration } } final <- list(configuration=configuration, changed=changed) return(final) } ## Function that generates the configurations of the ablation path ## between initial.configuration and final.configuration. ## parameters can be selected by specifying them in para.names. generateAblation <- function(initial.configuration, final.configuration, parameters, param.names = NULL) { if (is.null(param.names)) param.names <- parameters[["names"]] # Only change non-fixed param.names <- param.names[!parameters[["isFixed"]][param.names]] configurations <- NULL changed.params <- list() for (pname in param.names) { # Check if parameter is active. if (!conditionsSatisfied(parameters, initial.configuration, pname)) next # Check value is different in the initial and final configuration and if # so, change the value. if (initial.configuration[, pname] == final.configuration[, pname]) next new.configuration <- initial.configuration new.configuration[, pname]<- final.configuration[,pname] # Set newly activated parameters if needed. aux <- fixDependenciesWithReference(new.configuration, final.configuration, parameters) new.configuration <- aux[["configuration"]] changed.params[[length(changed.params) + 1]] <- c(pname, aux[["changed"]]) new.configuration[, ".PARENT."] <- initial.configuration$.ID. configurations <- rbind.data.frame(configurations, new.configuration) } rownames(configurations) <- NULL list(configurations=configurations, changed.params=changed.params) } report_duplicated_results <- function(experiments, configurations) { x <- t(experiments) x <- x[duplicated(x) | duplicated(x, fromLast = TRUE), , drop=FALSE] if (nrow(x) == 0L) return(NULL) # No duplicates dups <- split(rownames(x), apply(x, 1, paste0, collapse="")) names(dups) <- NULL for (g in dups) { cat("Warning: The following configuration produced the same results (the different parameters had not effect):\n") print(configurations[configurations$.ID. %in% g, , drop=FALSE]) } dups } #' Performs ablation between two configurations (from source to target). #' #' @description Ablation is a method for analyzing the differences between two configurations. #' #' @template arg_iraceresults #' @param src,target Source and target configuration IDs. By default, the first configuration ever evaluated (ID 1) is used as `src` and the best configuration found by irace is used as target. #' @param ab.params Specific parameter names to be used for the ablation. They must be in `parameters$names`. By default, use all parameters. #' @param type Type of ablation to perform: `"full"` will execute each configuration on all `n_instances` to determine the best-performing one; `"racing"` will apply racing to find the best configurations. #' @param n_instances (`integer(1)`) Number of instances used in `"full"` ablation will be `n_instances * scenario$firstTest`. #' @param seed (`integer(1)`) Integer value to use as seed for the random number generation. #' @param ablationLogFile (`character(1)`) Log file to save the ablation log. If `NULL`, the results are not saved to a file. #' @param ... Further arguments to override scenario settings, e.g., `debugLevel`, `parallel`, etc. #' #' @references #' C. Fawcett and H. H. Hoos. Analysing differences between algorithm #' configurations through ablation. Journal of Heuristics, 22(4):431–458, 2016. #' #' @template ret_ablog #' @seealso [plotAblation()] #' @examples #' \donttest{ #' logfile <- system.file(package="irace", "exdata", "sann.rda") #' # Execute ablation between the first and the best configuration found by irace. #' ablog <- ablation(logfile, ablationLogFile = NULL) #' plotAblation(ablog) #' # Execute ablation between two selected configurations, and selecting only a #' # subset of parameters, directly reading the setup from the irace log file. #' ablog <- ablation(logfile, src = 1, target = 10, #' ab.params = c("temp"), ablationLogFile = NULL) #' plotAblation(ablog) #' } #' #' @author Leslie Pérez Cáceres and Manuel López-Ibáñez #' @export ablation <- function(iraceResults, src = 1L, target = NULL, ab.params = NULL, type = c("full", "racing"), n_instances = 1L, seed = 1234567, ablationLogFile = "log-ablation.Rdata", ...) { # Input check if (missing(iraceResults) || is.null(iraceResults)) stop("You must provide an 'iraceResults' object generated by irace or the path to the '.Rdata' file that contains this object.") type <- match.arg(type) if (n_instances != 1L && type == "racing") stop("'n_instances' has no effect when type == 'racing'") if (!is.null(ablationLogFile)) file.check(ablationLogFile, writeable = TRUE, text = 'logFile') save_ablog <- function(complete) { ablog <- list(changes = changes, configurations = all_configurations, experiments = results, instances = instances, scenario = scenario, trajectory = trajectory, best = best.configuration, complete = complete) if (!is.null(ablationLogFile)) save(ablog, file = ablationLogFile, version = 2) ablog } # FIXME: The previous seed needs to be saved and restored at the end. set.seed(seed) # Load the data of the log file iraceResults <- read_logfile(iraceResults) if (is.null(target)) target <- iraceResults$iterationElites[length(iraceResults$iterationElites)] irace.note ("Starting ablation from ", src, " to ", target, "\n# Seed: ", seed, "\n") if (src %!in% iraceResults$allConfigurations$.ID.) stop("Source configuration ID (", src, ") cannot be found") if (target %!in% iraceResults$allConfigurations$.ID.) stop("Target configuration ID (", target, ") cannot be found") src.configuration <- iraceResults$allConfigurations[src, , drop = FALSE] target.configuration <- iraceResults$allConfigurations[target, , drop = FALSE] parameters <- iraceResults$parameters scenario <- iraceResults$scenario scenario_args <- list(...) if (length(scenario_args) > 0L) { unknown_scenario_args <- setdiff(names(scenario_args), names(scenario)) if (length(unknown_scenario_args) > 0L) irace.error("Unknown scenario settings given: ", paste0(unknown_scenario_args, collapse=", ")) scenario <- modifyList(scenario, scenario_args) } scenario$logFile <- "" scenario <- checkScenario (scenario) startParallel(scenario) on.exit(stopParallel(), add = TRUE) n_instances <- if (type == "racing") length(scenario$instances) else n_instances * scenario$firstTest instances <- generateInstances(scenario, n_instances) .irace$instancesList <- instances # Select the parameters used for ablation if (is.null(ab.params)) { ab.params <- parameters$names } else if (!all(ab.params %in% parameters$names)) { irace.error("Some of the parameters provided (", paste0(setdiff(ab.params, parameters$names), collapse=", "), ") are not defined in the parameter space.") } cat("# Source configuration (row number is ID):\n") configurations.print(src.configuration) cat("# Target configuration (row number is ID):\n") configurations.print(target.configuration) # Select parameters that are different in both configurations neq.params <- which(src.configuration[,ab.params] != target.configuration[,ab.params]) if (length(neq.params) < 1) irace.error("Candidates are equal considering the parameters selected\n") param.names <- colnames(src.configuration[,ab.params])[neq.params] # FIXME: Do we really need to override the ID? src.configuration$.ID. <- best.id <- 1 best.configuration <- all_configurations <- src.configuration # Execute source and target configurations. ## FIXME: We may already have these experiments in the logFile! experiments <- createExperimentList(configurations = rbind(src.configuration, target.configuration), parameters = parameters, instances = scenario$instances, instances.ID = instances[, "instance"], seeds = instances[, "seed"], scenario = scenario, bounds = scenario$boundMax) # Define variables needed trajectory <- 1 names(trajectory) <- "source" # FIXME: changes should only store the changed parameters. changes <- list() irace.note("Executing source and target configurations on the given instances (", nrow(instances), ")...\n") target.output <- execute.experiments(experiments, scenario) if (!is.null(scenario$targetEvaluator)) target.output <- execute.evaluator (experiments, scenario, target.output, src.configuration) # Save results output <- unlist(lapply(target.output, "[[", "cost")) results <- matrix(NA, ncol = 1, nrow = nrow(instances), dimnames = list(seq(1,nrow(instances)), 1)) results[,1] <- output[1:nrow(instances)] lastres <- output[(nrow(instances)+1):(2 * nrow(instances))] step <- 1 ablog <- save_ablog(complete = FALSE) while (length(param.names) > 1) { # Generate ablation configurations cat("# Generating configurations (row number is ID):", param.names,"\n") ab.aux <- generateAblation(best.configuration, target.configuration, parameters, param.names) aconfigurations <- ab.aux$configurations if (is.null(aconfigurations)) { cat("# Stopping ablation, no parameter change possible.\n") break } ## FIXME: We may already have these configurations in the logFile! # New configurations ids ## FIXME: These should be generated with respect to the logFile to make ## sure we don't have duplicate IDs. aconfigurations[,".ID."] <- seq(max(all_configurations$.ID.) + 1, max(all_configurations$.ID.) + nrow(aconfigurations)) configurations.print(aconfigurations, metadata = FALSE) all_configurations <- rbind(all_configurations, aconfigurations) # Set variables for the racing procedure if (scenario$capping) { # For using capping we must set elite data elite.data <- list(experiments = results[,best.configuration$.ID., drop=FALSE]) race.conf <- rbind(best.configuration, aconfigurations) .irace$next.instance <- nrow(instances) + 1 } else { #LESLIE: for now we apply the non-elitis irace when type=="racing" # we should define what is the standard elite.data <- NULL race.conf <- aconfigurations scenario$elitist <- FALSE .irace$next.instance <- 1 } irace.note("Ablation (", type, ") of ", nrow(aconfigurations), " configurations on ", nrow(instances), " instances.\n") # Force the race to see all instances in "full" mode if (type == "full") scenario$firstTest <- nrow(instances) race.output <- race(maxExp = nrow(aconfigurations) * nrow(instances), minSurvival = 1, elite.data = elite.data, configurations = race.conf, parameters = parameters, scenario = scenario, elitistNewInstances = 0) results <- merge.matrix (results, race.output$experiments) # Save log ablog <- save_ablog(complete = FALSE) # Get the best configuration based on the criterion of irace # MANUEL: Doesn't race.output already give you all this info??? cranks <- overall.ranks(results[,aconfigurations$.ID.,drop=FALSE], scenario$testType) best_id <- which.min(cranks)[1] # cand.mean <- colMeans(results[,aconfigurations$.ID.,drop=FALSE], na.rm=TRUE) changes[[step]] <- ab.aux$changed.params best.change <- changes[[step]][[best_id]] trajectory <- c(trajectory, aconfigurations[best_id, ".ID."]) # Report best # FIXME: This ID does not actually match the configuration ID # The race already reports the best. cat("# Best changed parameters:\n") for (i in seq_along(best.change)) { cat("#", best.change[i], ":", best.configuration[,best.change[i]], "->", aconfigurations[best_id, best.change[i]], "\n") } best.configuration <- aconfigurations[best_id,,drop=FALSE] best.id <- best.configuration$.ID. param.names <- param.names[!(param.names %in% best.change)] step <- step + 1 } # Add last configuration and its results # FIXME: This may be overriding the ID of an existing configuration!!! target.configuration$.ID. <- max(all_configurations$.ID.) + 1 all_configurations <- rbind(all_configurations, target.configuration) results <- cbind(results, matrix(lastres, ncol = 1, dimnames=list(seq(1, nrow(instances)), target.configuration$.ID.))) trajectory <- c(trajectory, target.configuration$.ID.) # Get the overall best cranks <- overall.ranks(results[,trajectory, drop=FALSE], scenario$testType) best_id <- which.min(cranks)[1] ## FIXME: At this point, the rownames of all_configurations does not match ## all_configurations$.ID. That is confusing and a potential source of ## bugs. Instead of fixing it here, we should not generate the discrepancy ## ever. best.configuration <- all_configurations[trajectory[best_id],,drop=FALSE] irace.note("Final best configuration:\n") configurations.print(best.configuration) # Check for duplicated results: report_duplicated_results(results, all_configurations) # LESLIE: If we use racing we can have a matrix of results that is not # complete, how should we do the plots? # MANUEL: Do not plot anything that was discarded save_ablog(complete = TRUE) } ablation.labels <- function(trajectory, configurations) { configurations <- removeConfigurationsMetaData(configurations[trajectory, , drop = FALSE]) labels <- names(trajectory) last <- configurations[1, , drop = FALSE] param.names <- colnames(last) for (i in 2:length(trajectory)) { current <- configurations[i, , drop = FALSE] # select everything that is NOT NA now and was different or NA before. select <- !is.na(current) & (is.na(last) | (current != last)) irace.assert(!anyNA(select)) labels[i] <- paste0(param.names[select], "=", current[, select], collapse = "\n") last <- current } return(labels) } #' Create plot from an ablation log #' #' @param ablog (`list()`|`character(1)`) Ablation log object returned by [ablation()]. Alternatively, the path to an `.Rdata` file, e.g., `"log-ablation.Rdata"`, from which the object will be loaded. #' @param pdf.file Output filename. #' @param pdf.width Width provided to create the pdf file. #' @param type Type of plot. Supported values are `"mean"` and `"boxplot"`. #' @param mar Vector with the margins for the ablation plot. #' @param ylab Label of y-axis. #' @param ylim Numeric vector of length 2, giving the y coordinates ranges. #' @param ... Further graphical parameters may also be supplied as #' arguments. See [graphics::plot.default()]. #' #' @author Leslie Pérez Cáceres and Manuel López-Ibáñez #' @seealso [ablation()] #' @examples #' logfile <- file.path(system.file(package="irace"), "exdata", "log-ablation.Rdata") #' plotAblation(ablog = logfile) #' @export plotAblation <- function (ablog, pdf.file = NULL, pdf.width = 20, type = c("mean", "boxplot"), mar = par("mar"), ylab = "Mean configuration cost", ylim = NULL, ...) { type <- match.arg(type) if (missing(ablog) || is.null(ablog)) { irace.error("You must provide an 'ablog' object generated by ablation() or the path to the '.Rdata' file that contains this object.") } ablog <- read_logfile(ablog, name = "ablog") if (!ablog$complete) stop("The ablog shows that the ablation procedure did not complete cleanly and only contains partial information") if (!is.null(pdf.file)) { if (!is.file.extension(pdf.file, ".pdf")) pdf.file <- paste0(pdf.file, ".pdf") cat("Creating PDF file '", pdf.file, "'\n", sep="") pdf(file = pdf.file, width = pdf.width, title = paste0("Ablation plot: ", pdf.file)) on.exit(dev.off(), add = TRUE) } trajectory <- ablog$trajectory configurations <- ablog$configurations # Generate labels # FIXME: allow overriding these labels. labels <- ablation.labels(trajectory, configurations) inches_to_lines <- (par("mar") / par("mai"))[1] lab.width <- max(strwidth(labels, units = "inches")) * inches_to_lines old.par <- par(mar = c(lab.width + 2.1, 4.1, 0.1, 0.1), cex.axis = 1) if (!is.null(pdf.file)) on.exit(par(old.par), add = TRUE) experiments <- ablog$experiments # FIXME: We could also show the other alternatives at each step not just the # one selected. See Leonardo's thesis. if (type == "boxplot") { bx <- boxplot(experiments[, trajectory], plot=FALSE) if (is.null(ylim)) { ylim <- range(bx$stats[is.finite(bx$stats)], bx$out[is.finite(bx$out)], bx$conf[is.finite(bx$conf)]) } } costs.avg <- colMeans(experiments[, trajectory]) plot(costs.avg, xaxt = "n", xlab = NA, ylab = ylab, ylim = ylim, type = "b", pch = 19, ..., panel.first = { grid(nx = NA, ny = NULL, lwd = 2); abline(h = c(costs.avg[1], tail(costs.avg, n = 1)), col = "lightgray", lty = "dotted", lwd = 2) }) axis(1, at = 1:length(costs.avg), labels = labels, las = 3) if (type == "boxplot") { bxp(bx, show.names = FALSE, add = TRUE) } } irace/R/generation.R0000644000176200001440000003701114324004557014022 0ustar liggesusers####################################### ## GENERATE CONFIGURATIONS ####################################### ## When called with an unconditional parameter, it ## must return TRUE conditionsSatisfied <- function (parameters, partialConfiguration, paramName) { condition <- parameters$conditions[[paramName]] # If there is no condition, do not waste time evaluating it. if (isTRUE(condition)) return(TRUE) v <- eval(condition, as.list(partialConfiguration)) # Return TRUE if TRUE, FALSE if FALSE or NA ## FIXME: If we byte-compile the condition, then we should incorporate the ## following into the condition directly. See readForbiddenFile. v <- !is.na(v) && v return(v) } new_empty_configuration <- function(parameters) { newConfigurationsColnames <- c(names(parameters$conditions), ".PARENT.") return(setNames(as.list(rep(NA, length(newConfigurationsColnames))), newConfigurationsColnames)) } get.fixed.value <- function(param, parameters) { value <- parameters$domain[[param]][1] type <- parameters$types[[param]] if (type == "i") { return (as.integer(value)) } else if (type == "c" || type == "o") { return (value) } else { irace.assert (type == "r") return (as.double(value)) } } ## Calculates the parameter bounds when parameters domain is dependent getDependentBound <- function(parameters, param, configuration) { values <- parameters$domain[[param]] if (is.expression(values)) { # Depends contains parameters that enable param and parameters that define # its domain. If this is a partial configuration, we need only the latter. # Use names() here in case the configuration is simply a list. deps <- intersect(names(configuration), parameters$depends[[param]]) # If it depends on a parameter that is disabled, then this is disabled. if (anyNA(configuration[deps])) return(NA) values <- sapply(values, eval, configuration) irace.assert(all(is.finite(values))) # Value gets truncated (defined from robotics initial requirements) if (parameters$types[param] == "i") values <- as.integer(values) if (values[1] > values[2]) { irace.error ("Invalid domain (", paste0(values, collapse=", "), ") generated for parameter '", param, "' that depends on parameters (", paste0(parameters$depends[[param]], collapse=", "), "). This is NOT a bug in irace. Check the definition of these parameters.") } } return(values) } ### Uniform sampling for the initial generation sampleUniform <- function (parameters, nbConfigurations, digits, forbidden = NULL, repair = NULL) { if (is.null(repair)) repair <- function(c, p, d) c namesParameters <- names(parameters$conditions) newConfigurations <- as.data.frame(matrix(nrow = nbConfigurations, ncol = length(namesParameters) + 1, dimnames = list(NULL, c(namesParameters, ".PARENT.")) ), stringsAsFactors=FALSE) empty_configuration <- new_empty_configuration(parameters) for (idxConfiguration in seq_len(nbConfigurations)) { forbidden.retries <- 0 while (forbidden.retries < 100) { configuration <- empty_configuration for (p in seq_along(namesParameters)) { currentParameter <- namesParameters[p] if (!conditionsSatisfied(parameters, configuration, currentParameter)) { configuration[[p]] <- NA next } # FIXME: We must be careful because parameters$types does not have the # same order as namesParameters, because we sample in the order of the # conditions. currentType <- parameters$types[[currentParameter]] if (isFixed(currentParameter, parameters)) { # We don't even need to sample, there is only one possible value ! newVal <- get.fixed.value (currentParameter, parameters) # The parameter is not a fixed and should be sampled } else if (currentType %in% c("i","r")) { domain <- getDependentBound(parameters, currentParameter, configuration) newVal <- sample_unif(currentType, domain, transf = parameters$transform[[currentParameter]], digits) } else { irace.assert(currentType %in% c("c","o")) possibleValues <- parameters$domain[[currentParameter]] newVal <- sample(possibleValues, 1) } configuration[[p]] <- newVal } configuration <- as.data.frame(configuration, stringsAsFactors=FALSE) configuration <- repair(configuration, parameters, digits) if (is.null(forbidden) || nrow(checkForbidden(configuration, forbidden)) == 1) { newConfigurations[idxConfiguration,] <- configuration break } forbidden.retries <- forbidden.retries + 1 } if (forbidden.retries >= 100) { irace.error("irace tried 100 times to sample from the model a configuration not forbidden without success, perhaps your constraints are too strict?") } } return (newConfigurations) } # To be called the first time before the second race (with indexIter = # 2) Nb configurations is the number of configurations at the end # included the elite ones obtained from the previous iteration sampleModel <- function (parameters, eliteConfigurations, model, nbNewConfigurations, digits, forbidden = NULL, repair = NULL) { if (is.null(repair)) repair <- function(c, p, d) c if (nbNewConfigurations <= 0) { irace.error ("The number of configurations to generate appears to be negative or zero.") } namesParameters <- names(parameters$conditions) newConfigurations <- as.data.frame(matrix(nrow = nbNewConfigurations, ncol = length(namesParameters) + 1, dimnames = list(NULL, c(namesParameters, ".PARENT.")) ), stringsAsFactors=FALSE) empty_configuration <- new_empty_configuration(parameters) for (idxConfiguration in seq_len(nbNewConfigurations)) { forbidden.retries <- 0 while (forbidden.retries < 100) { # Choose the elite which will be the parent. indexEliteParent <- sample.int (n = nrow(eliteConfigurations), size = 1, prob = eliteConfigurations[[".WEIGHT."]]) eliteParent <- eliteConfigurations[indexEliteParent, ] idEliteParent <- eliteParent[[".ID."]] configuration <- empty_configuration configuration[[".PARENT."]] <- idEliteParent # Sample a value for every parameter of the new configuration. for (p in seq_along(namesParameters)) { # FIXME: We must be careful because parameters$types does not # have the same order as parameters$conditions. Ideally, we # should fix this or make it impossible to confuse them. currentParameter <- namesParameters[p] currentType <- parameters$types[[currentParameter]] if (!conditionsSatisfied(parameters, configuration, currentParameter)) { # Some conditions are unsatisfied. # Should be useless, NA is (always?) assigned when matrix created newVal <- NA } else if (isFixed(currentParameter, parameters)) { # We don't even need to sample, there is only one possible value ! newVal <- get.fixed.value (currentParameter, parameters) # The parameter is not a fixed and should be sampled } else if (currentType %in% c("i", "r")) { domain <- getDependentBound(parameters, currentParameter, configuration) mean <- as.numeric(eliteParent[currentParameter]) # If there is not value we obtain it from the model or the mean obtained is # not in the current domain, this can happen when using dependent domains if (is.na(mean)) mean <- model[[currentParameter]][[as.character(idEliteParent)]][2] if (is.na(mean) || !inNumericDomain(mean, domain)) { # The elite parent does not have any value for this parameter, # let's sample uniformly. newVal <- sample_unif(currentType, domain, transf = parameters$transform[[currentParameter]], digits) } else { stdDev <- model[[currentParameter]][[as.character(idEliteParent)]][1] # If parameters are dependent standard deviation must be computed # based on the current domain if (parameters$isDependent[currentParameter]) { # Conditions should be satisfied for the parameter, thus domain cannot be NA stdDev <- (domain[2] - domain[1]) * stdDev } newVal <- sample_norm(mean, stdDev, currentType, domain, transf = parameters$transform[[currentParameter]], digits) } } else if (currentType == "o") { possibleValues <- paramDomain(currentParameter, parameters) value <- eliteParent[currentParameter] if (is.na(value)) { # The elite parent does not have any value for this # parameter, let's sample uniformly ## FIXME: We should save the last used parameter in the model and use it here. newVal <- sample(possibleValues, 1) } else { # Find the position within the vector of possible # values to determine the equivalent integer. mean <- match(value, possibleValues) # Return index of value in array stdDev <- model[[currentParameter]][[as.character(idEliteParent)]] # Sample with truncated normal distribution as an integer. # See sample_norm() for an explanation. newValAsInt <- floor(rtnorm(1, mean + 0.5, stdDev, lower = 1, upper = length(possibleValues) + 1L)) # The probability of this happening is very small, but it can happen. if (newValAsInt == length(possibleValues) + 1L) newValAsInt <- length(possibleValues) irace.assert(newValAsInt >= 1L && newValAsInt <= length(possibleValues)) # Get back to categorical values, find the one corresponding to the # newVal newVal <- possibleValues[newValAsInt] } } else if (currentType == "c") { # FIXME: Why is idEliteParent character? # FIXME: Why the model is ? It makes more sense to be . probVector <- model[[currentParameter]][[as.character(idEliteParent)]] possibleValues <- paramDomain(currentParameter, parameters) newVal <- sample(x = possibleValues, size = 1, prob = probVector) } else { irace.internal.error("Unexpected condition in sampleModel") } configuration[[p]] <- newVal } configuration <- as.data.frame(configuration, stringsAsFactors = FALSE) configuration <- repair(configuration, parameters, digits) if (is.null(forbidden) || nrow(checkForbidden(configuration, forbidden)) == 1) { newConfigurations[idxConfiguration,] <- configuration break } forbidden.retries <- forbidden.retries + 1 } if (forbidden.retries >= 100) { irace.error("irace tried 100 times to sample from the model a configuration not forbidden without success, perhaps your constraints are too strict?") } } return (newConfigurations) } transform.from.log <- function(x, transf, lowerBound, upperBound) { trLower <- attr(transf, "lower") trUpper <- attr(transf, "upper") x <- exp(trLower + (trUpper - trLower) * x) return(x) } transform.to.log <- function(x, transf, lowerBound, upperBound) { trLower <- attr(transf, "lower") trUpper <- attr(transf, "upper") return((log(x) - trLower)/(trUpper - trLower)) } ## How to sample integer values? # # The problem: If we have an integer with domain [1,3] and we sample a real value # and round, then there are more chances of getting 2 than 1 or 3: # [1, 1,5) -> 1 # [1.5, 2,5) -> 2 # [2.5, 3) -> 3 # # The solution: Sample in [lowerbound, upperbound + 1], that is, [1, 4], then floor(): # [1, 2) -> 1 # [2, 3) -> 2 # [3, 4) -> 3 # # Why floor() and not trunc()? # Because trunc(-1.5) -> -1, while floor(-1.5) -> -2, so for a domain [-3,-1]: # # [-3, -2) -> -3 # [-2, -1) -> -2 # [-1, 0) -> -1 # # Issue 1: We can sample 4 (upperbound + 1). In that case, we return 3. # # Issue 2: When sampling from a truncated normal distribution, the extremes are # not symmetric. # # nsamples <- 100000 # table(floor(rtnorm(nsamples, mean=1, sd=1, lower=1,upper=4)))/nsamples # table(floor(rtnorm(nsamples, mean=3, sd=1, lower=1,upper=4)))/nsamples # # To make them symmetric, we translate by 0.5, so that the mean is at the # actual center of the interval that will produce the same value after # truncation, e.g., given an integer value of 1, then mean=1.5, which is at the # center of [1,2). # # nsamples <- 100000 # table(floor(rtnorm(nsamples, mean=1.5, sd=1, lower=1,upper=4)))/nsamples # table(floor(rtnorm(nsamples, mean=3.5, sd=1, lower=1,upper=4)))/nsamples # # The above reasoning also works for log-transformed domains, because # floor() happens in the original domain, not in the log-transformed one, # except for the case of log-transformed negative domains, where we have to # translate by -0.5. # numeric_value_round <- function(type, value, lowerBound, upperBound, digits) { irace.assert(is.finite(value)) if (type == "i") { value <- floor(value) # The probability of this happening is very small, but it could happen. if (value == upperBound + 1L) value <- upperBound } else value <- round(value, digits) irace.assert(value >= lowerBound && value <= upperBound) return (value) } # Sample value for a numerical parameter. sample_unif <- function(type, domain, transf, digits) { # Dependent domains could be not available because of inactivity of parameters # on which they are depedent. In this case, the dependent parameter becomes # not active and we return NA. if (anyNA(domain)) return(NA) lowerBound <- domain[1] upperBound <- domain[2] if (type == "i") { # +1 for correct rounding before floor() upperBound <- 1L + upperBound } if (transf == "log") { value <- runif(1, min = 0, max = 1) value <- transform.from.log(value, transf, lowerBound, upperBound) } else { value <- runif(1, min = lowerBound, max = upperBound) } # We use original upperBound, not the +1L for 'i'. value <- numeric_value_round(type, value, lowerBound, upperBound = domain[2], digits) return(value) } sample_norm <- function(mean, sd, type, domain, transf, digits) { # Dependent domains could be not available because of inactivity of parameters # on which they are depedent. In this case, the dependent parameter becomes # not active and we return NA. if (anyNA(domain)) return(NA) lowerBound <- domain[1] upperBound <- domain[2] if (type == "i") { upperBound <- 1L + upperBound # Because negative domains are log-transformed to positive domains. mean <- mean + 0.5 } if (transf == "log") { trMean <- transform.to.log(mean, transf, lowerBound, upperBound) value <- rtnorm(1, trMean, sd, lower = 0, upper = 1) value <- transform.from.log(value, transf, lowerBound, upperBound) } else { value <- rtnorm(1, mean, sd, lowerBound, upperBound) } # We use original upperBound, not the +1L for 'i'. value <- numeric_value_round(type, value, lowerBound, upperBound = domain[2], digits) return(value) } irace/R/race-wrapper.R0000644000176200001440000006404514321266642014270 0ustar liggesusers# FIXME: This is needed because race.R is not divided in two-stages # run/evaluate like irace is, so there is no way to communicate data # from the first stage to the second. # # FIXME: In fact, we should use this trick also in irace.R to avoid # pass-by-copy-on-write of huge matrices and data.frames and instead # pass-by-reference an environment containing those. .irace <- new.env(parent = emptyenv()) #' Generate a command-line representation of a configuration #' #' @description `buildCommandLine` receives two vectors, one containing #' the values of the parameters, the other containing the switches of the #' parameters. It builds a string with the switches and the values that can #' be used as a command line to call the program to be tuned, thus generating #' one candidate configuration. #' #' #' @param values A vector containing the value of each parameter for the #' candidate configuration. #' @param switches A vector containing the switches of each paramter (in an #' order that corresponds to the values vector). #' #' @return A string concatenating each element of `switches` and #' `values` for all parameters with a space between each pair of #' parameters (but none between the switches and the corresponding values). #' #' @examples #' switches <- c("--switch1 ", "--switch2 ") #' values <- c("value_1", "value_2") #' buildCommandLine (values, switches) #' ## Build a command-line from the results produced by a previous run of irace. #' # First, load the data produced by irace. #' irace.logfile <- file.path(system.file(package="irace"), #' "exdata", "irace-acotsp.Rdata") #' load(irace.logfile) #' allConfigurations <- iraceResults$allConfigurations #' parameters <- iraceResults$parameters #' apply(allConfigurations[1:10, unlist(parameters$names)], 1, buildCommandLine, #' unlist(parameters$switches)) #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export buildCommandLine <- function(values, switches) { irace.assert(length(values) == length(switches)) command <- "" # FIXME: This probably can be implemented faster with apply() and # paste(collapse=" "). But how to get the index i in that case? for (i in seq_along(values)) { value <- values[i] if (!is.na(value)) { command <- paste0(command, " ", switches[i], format(value, digits=15, scientific=FALSE)) } } return(command) } # This function tries to convert a, possibly empty, character vector into a # numeric vector. parse.output <- function(outputRaw, verbose) { if (verbose) { cat (outputRaw, sep = "\n") } # Initialize output as raw. If it is empty stays like this. output <- outputRaw # strsplit crashes if outputRaw == character(0) if (length(outputRaw) > 0) { outputRaw <- paste0(outputRaw, collapse = "\n") output <- strsplit(trim(outputRaw), "[[:space:]]+")[[1]] } # suppressWarnings to avoid messages about NAs introduced by coercion return(suppressWarnings (as.numeric (output))) } target.error <- function(err.msg, output, scenario, target.runner.call, target.evaluator.call = NULL) { if (!is.null(target.evaluator.call)) { err.msg <- paste0(err.msg, "\n", .msg.prefix, "The call to targetEvaluator was:\n", target.evaluator.call) } if (!is.null(target.runner.call)) { err.msg <- paste0(err.msg, "\n", .msg.prefix, "The call to targetRunner was:\n", target.runner.call) } if (is.null(output$outputRaw)) { # Message for a function call. # FIXME: Ideally, we should print the list as R would print it. output$outputRaw <- toString(output) advice.txt <- paste0( "This is not a bug in irace, but means that something failed in ", "a call to the targetRunner or targetEvaluator functions provided by the user.", " Please check those functions carefully.") } else { # Message for an external script. advice.txt <- paste0( "This is not a bug in irace, but means that something failed when", " running the command(s) above or they were terminated before completion.", " Try to run the command(s) above from the execution directory '", scenario$execDir, "' to investigate the issue. See also Appendix B (targetRunner troubleshooting checklist) of the User Guide (https://cran.r-project.org/package=irace/vignettes/irace-package.pdf).") } irace.error(err.msg, "\n", .msg.prefix, "The output was:\n", paste(output$outputRaw, collapse = "\n"), "\n", .msg.prefix, advice.txt) } check.output.target.evaluator <- function (output, scenario, target.runner.call = NULL) { if (!is.list(output)) { target.error ("The output of targetEvaluator must be a list", list(), scenario, target.runner.call = target.runner.call) return(NULL) } err.msg <- output$error if (is.null(err.msg)) { if (is.null(output$cost)) { err.msg <- "The output of targetEvaluator must contain 'cost'!" } else if (is.na.nowarn (output$cost)) { err.msg <- "The output of targetEvaluator is not numeric!" } if (scenario$batchmode != 0 && scenario$maxTime > 0) { if (is.null (output$time)) { err.msg <- "When batchmode != 0 and maxTime > 0, the output of targetEvaluator must be two numbers 'cost time'!" } else if (is.na.nowarn(output$time)) { err.msg <- "The time returned by targetEvaluator is not numeric!" } else if (is.infinite(output$time)) { err.msg <- "The time returned by targetEvaluator is not finite!" } } } if (!is.null(err.msg)) { target.error (err.msg, output, scenario, target.runner.call = target.runner.call, target.evaluator.call = output$call) } } exec.target.evaluator <- function (experiment, num.configurations, all.conf.id, scenario, target.runner.call) { output <- .irace$target.evaluator(experiment, num.configurations, all.conf.id, scenario, target.runner.call) check.output.target.evaluator (output, scenario, target.runner.call = target.runner.call) # Fix too small time. output$time <- if (is.null(output$time)) NA else max(output$time, scenario$minMeasurableTime) return (output) } #' target.evaluator.default #' #' `target.evaluator.default` is the default `targetEvaluator` function that is #' invoked if `targetEvaluator` is a string (by default #' `targetEvaluator` is `NULL` and this function is not invoked). You can use it as #' an advanced example of how to create your own `targetEvaluator` function. #' #' @param experiment A list describing the experiment. It contains at least: #' \describe{ #' \item{`id.configuration`}{An alphanumeric string that uniquely identifies a configuration;} #' \item{`id.instance`}{An alphanumeric string that uniquely identifies an instance;} #' \item{`seed`}{Seed for the random number generator to be used for #' this evaluation, ignore the seed for deterministic algorithms;} #' \item{`instance`}{String giving the instance to be used for this evaluation;} #' \item{`bound`}{(only when `capping` is enabled) Time bound for the execution;} #' \item{`configuration`}{1-row data frame with a column per parameter #' name;} #' \item{`switches`}{Vector of parameter switches (labels) in the order #' of parameters used in `configuration`.} #' } #' @param num.configurations Number of configurations alive in the race. #' @param all.conf.id Vector of configuration IDs of the alive configurations. #' @template arg_scenario #' @param target.runner.call String describing the call to `targetRunner` that #' corresponds to this call to `targetEvaluator`. This is used for #' providing extra information to the user, for example, in case #' `targetEvaluator` fails. #' #' @return The function `targetEvaluator` must return a list with one element #' `"cost"`, the numerical value corresponding to the cost measure of the #' given configuration on the given instance. #' #' The return list may also contain the following optional elements that are used #' by \pkg{irace} for reporting errors in `targetEvaluator`: #' \describe{ #' \item{`error`}{is a string used to report an error;} #' \item{`outputRaw`}{is a string used to report the raw output of calls to #' an external program or function;} #' \item{`call`}{is a string used to report how `targetRunner` called #' an external program or function.} #' } #' #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export target.evaluator.default <- function(experiment, num.configurations, all.conf.id, scenario, target.runner.call) { configuration.id <- experiment$id.configuration instance.id <- experiment$id.instance seed <- experiment$seed instance <- experiment$instance debugLevel <- scenario$debugLevel targetEvaluator <- scenario$targetEvaluator if (as.logical(file.access(targetEvaluator, mode = 1))) { irace.error ("targetEvaluator", shQuote(targetEvaluator), "cannot be found or is not executable!\n") } cwd <- setwd (scenario$execDir) args <- c(configuration.id, instance.id, seed, instance, num.configurations, all.conf.id) output <- runcommand(targetEvaluator, args, configuration.id, debugLevel) setwd (cwd) cost <- time <- NULL err.msg <- output$error if (is.null(err.msg)) { v.output <- parse.output(output$output, verbose = (scenario$debugLevel >= 2)) if (length(v.output) > 2) { err.msg <- paste0("The output of targetEvaluator should not be more than two numbers!") } else if (length(v.output) == 0) { err.msg <- paste0("The output of targetEvaluator must be at least one number 'cost'!") } else if (length(v.output) == 1) { cost <- v.output[1] } else if (length(v.output) == 2) { cost <- v.output[1] time <- v.output[2] } } return(list(cost = cost, time = time, error = err.msg, outputRaw = output$output, call = paste(targetEvaluator, args, collapse=" "))) } check.output.target.runner <- function (output, scenario) { if (!is.list(output)) { output <- list() err.msg <- "The output of targetRunner must be a list" target.error (err.msg, output, scenario, target.runner.call = NULL) return(output) } err.msg <- output$error if (is.null(err.msg)) { if (!is.null (output$cost)) { if (is.na.or.empty(output$cost)) { err.msg <- "The cost returned by targetRunner is not numeric!" } } if (!is.null (output$time)) { if (is.na.or.empty(output$time)) { err.msg <- paste0("The time returned by targetRunner is not numeric!") } else if (is.infinite(output$time)) { err.msg <- paste0("The time returned by targetRunner is not finite!") } } } if (is.null(err.msg)) { # When targetEvaluator is provided, targetRunner must return only the time. if (!is.null(scenario$targetEvaluator)) { # unless using batchmode, in that case targetRunner returns neither the # time nor the cost. if (scenario$batchmode != 0) { if (!is.null(output$time) || !is.null(output$cost)) { err.msg <- "When batchmode != 0, the output of targetRunner must not contain a cost nor a time!" } } else if (scenario$maxTime > 0 && is.null(output$time)) { err.msg <- "The output of targetRunner must be one number 'time'!" } else if (!is.null(output$cost)) { err.msg <- "The output of targetRunner must be empty or just one number 'time'!" } } else if (scenario$maxTime > 0 && (is.null (output$cost) || is.null(output$time))) { err.msg <- "The output of targetRunner must be two numbers 'cost time'!" } else if (scenario$maxExperiments > 0 && is.null (output$cost)) { err.msg <- "The output of targetRunner must be one number 'cost'!" } else if (!is.null(output$time) && output$time < 0) { err.msg <- paste0("The value of time returned by targetRunner cannot be negative (", output$time, ")!") } } if (!is.null(err.msg)) { target.error (err.msg, output, scenario, target.runner.call = output$call) } # Fix too small time. output$time <- if (is.null(output$time)) NA else max(output$time, scenario$minMeasurableTime) return (output) } # This function invokes target.runner. When used on a remote node by Rmpi, # environments do not seem to be shared and the default value is evaluated too # late, thus we have to pass .irace$target.runner explicitly. exec.target.runner <- function(experiment, scenario, target.runner) { doit <- function(experiment, scenario) { x <- target.runner(experiment, scenario) return (check.output.target.runner(x, scenario)) } retries <- scenario$targetRunnerRetries while (retries > 0) { output <- try (doit(experiment, scenario)) if (!inherits(output, "try-error") && is.null(output$error)) { return (output) } irace.note("Retrying (", retries, " left).\n") retries <- retries - 1 } output <- doit(experiment, scenario) return (output) } parse.aclib.output <- function(outputRaw) { outputRaw <- paste0(outputRaw, collapse = "\n") text <- regmatches(outputRaw, regexec("Result of this algorithm run:\\s*\\{(.+)\\}\\s*\n", outputRaw))[[1]][2] aclib.match <- function(text, key, value) { pattern <- paste0('"', key, '":\\s*', value) return(regmatches(text, regexec(pattern, text))[[1]][2]) } cost <- runtime <- error <- NULL # AClib wrappers print: # Result of this algorithm run: {"status": "SUCCESS", "cost": cost, "runtime": time } # FIXME: This is not very robust. If we are going to be using jsonlite, then we can simply do: # jsonlite::fromJSON('{\"misc\": \"\", \"runtime\": 164.14, \"status\": \"SUCCESS\", \"cost\": \"0.121340\"}') status <- aclib.match(text, "status", '"([^"]+)"') if (!is.character(status)) { error <- paste0("Not valid AClib output") } else if (status %in% c("SUCCESS", "TIMEOUT")) { cost <- aclib.match(text, "cost", "([^[:space:],}]+)") cost <- suppressWarnings(as.numeric(cost)) runtime <- aclib.match(text, "runtime", "([^[:space:],}]+)") runtime <- suppressWarnings(as.numeric(runtime)) if (is.null.or.na(cost) && is.null.or.na(runtime)) error <- paste0("Not valid cost or runtime in AClib output") } else if (status %in% c("CRASHED", "ABORT")) { # FIXME: Implement ABORT semantics of fatal error error <- paste0("targetRunner returned status (", status, ")") } else { error <- paste0("Not valid AClib output status (", status, ")") } return(list(status = status, cost = cost, time = runtime, error = error)) } target.runner.aclib <- function(experiment, scenario) { debugLevel <- scenario$debugLevel res <- run_target_runner(experiment, scenario) cmd <- res$cmd output <- res$output args <- res$args err.msg <- output$error if (is.null(err.msg)) { return(c(parse.aclib.output(output$output), list(outputRaw = output$output, call = paste(cmd, args)))) } list(cost = NULL, time = NULL, error = err.msg, outputRaw = output$output, call = paste(cmd, args)) } check_launcher_args <- function(targetRunnerLauncherArgs) { if (!grepl("{targetRunner}", targetRunnerLauncherArgs, fixed=TRUE)) irace.error("targetRunnerLauncherArgs '", targetRunnerLauncherArgs, "' must contain '{targetRunner}'") if (!grepl("{targetRunnerArgs}", targetRunnerLauncherArgs, fixed=TRUE)) irace.error("targetRunnerLauncherArgs '", targetRunnerLauncherArgs, "' must contain '{targetRunnerArgs}'") } process_launcher_args <- function(targetRunnerLauncherArgs, targetRunner, args) { check_launcher_args(targetRunnerLauncherArgs) targetRunnerLauncherArgs <- gsub("{targetRunner}", targetRunner, targetRunnerLauncherArgs, fixed=TRUE) targetRunnerLauncherArgs <- gsub("{targetRunnerArgs}", args, targetRunnerLauncherArgs, fixed=TRUE) targetRunnerLauncherArgs } run_target_runner <- function(experiment, scenario) { configuration.id <- experiment$id.configuration instance.id <- experiment$id.instance seed <- experiment$seed configuration <- experiment$configuration instance <- experiment$instance switches <- experiment$switches bound <- experiment$bound targetRunner <- scenario$targetRunner debugLevel <- scenario$debugLevel if (scenario$aclib) { has_value <- !is.na(configuration) # [] [] ... [--cutoff ] [--instance ] # [--seed ] --config [-param_name_1 value_1] [-param_name_2 value_2] ... args <- paste("--instance", instance, "--seed", seed, "--config", paste0("-", switches[has_value], " ", configuration[has_value], collapse = " ")) if (!is.null.or.na(bound)) args <- paste("--cutoff", bound, args) } else { args <- paste(configuration.id, instance.id, seed, instance, bound, buildCommandLine(configuration, switches)) } targetRunnerLauncher <- scenario$targetRunnerLauncher if (is.null.or.empty(scenario$targetRunnerLauncher)) { if (as.logical(file.access(targetRunner, mode = 1))) { irace.error ("targetRunner ", shQuote(targetRunner), " cannot be found or is not executable!\n") } output <- runcommand(targetRunner, args, configuration.id, debugLevel) return(list(cmd=targetRunner, output=output, args=args)) } if (as.logical(file.access(targetRunnerLauncher, mode = 1))) { irace.error ("targetRunnerLauncher ", shQuote(targetRunnerLauncher), " cannot be found or is not executable!\n") } args <- process_launcher_args(scenario$targetRunnerLauncherArgs, targetRunner, args) output <- runcommand(targetRunnerLauncher, args, configuration.id, debugLevel) return(list(cmd=targetRunnerLauncher, output=output, args=args)) } #' Default `targetRunner` function. #' #' Use it as an advanced example of how to create your own `targetRunner` function. #' #' @param experiment A list describing the experiment. It contains at least: #' \describe{ #' \item{`id.configuration`}{An alphanumeric string that uniquely identifies a configuration;} #' \item{`id.instance`}{An alphanumeric string that uniquely identifies an instance;} #' \item{`seed`}{Seed for the random number generator to be used for #' this evaluation, ignore the seed for deterministic algorithms;} #' \item{`instance`}{String giving the instance to be used for this evaluation;} #' \item{`bound`}{(only when `capping` is enabled) Time bound for the execution;} #' \item{`configuration`}{1-row data frame with a column per parameter #' name;} #' \item{`switches`}{Vector of parameter switches (labels) in the order #' of parameters used in `configuration`.} #' } #' @template arg_scenario #' #' @return If `targetEvaluator` is `NULL`, then the `targetRunner` #' function must return a list with at least one element `"cost"`, #' the numerical value corresponding to the evaluation of the given #' configuration on the given instance. #' #' If the scenario option `maxTime` is non-zero or if `capping` is enabled #' then the list must contain at least another element `"time"` that reports the #' execution time for this call to `targetRunner`. #' The return list may also contain the following optional elements that are used #' by \pkg{irace} for reporting errors in `targetRunner`: #' \describe{ #' \item{`error`}{is a string used to report an error;} #' \item{`outputRaw`}{is a string used to report the raw output of calls to #' an external program or function;} #' \item{`call`}{is a string used to report how `targetRunner` called #' an external program or function.} #' } #' #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export target.runner.default <- function(experiment, scenario) { res <- run_target_runner(experiment, scenario) cmd <- res$cmd output <- res$output args <- res$args debugLevel <- scenario$debugLevel cost <- time <- NULL err.msg <- output$error if (is.null(err.msg)) { v.output <- parse.output(output$output, verbose = (debugLevel >= 2)) if (length(v.output) > 2) { err.msg <- "The output of targetRunner should not be more than two numbers!" } else if (length(v.output) == 1) { if (!is.null(scenario$targetEvaluator)) { time <- v.output[1] } else { cost <- v.output[1] } } else if (length(v.output) == 2) { cost <- v.output[1] time <- v.output[2] } } list(cost = cost, time = time, error = err.msg, outputRaw = output$output, call = paste(cmd, args, collapse = " ")) } execute.experiments <- function(experiments, scenario) { parallel <- scenario$parallel mpi <- scenario$mpi target.runner <- .irace$target.runner execDir <- scenario$execDir if (!isTRUE (file.info(execDir)$isdir)) { irace.error ("Execution directory '", execDir, "' is not found or not a directory\n") } cwd <- setwd (execDir) on.exit(setwd(cwd), add = TRUE) target.output <- vector("list", length(experiments)) if (!is.null(scenario$targetRunnerParallel)) { # User-defined parallelization target.output <- scenario$targetRunnerParallel(experiments, exec.target.runner, scenario = scenario, target.runner = target.runner) } else if (scenario$batchmode != 0) { target.output <- cluster.lapply (experiments, scenario = scenario) } else if (parallel > 1) { if (mpi) { if (scenario$loadBalancing) { target.output <- Rmpi::mpi.applyLB(experiments, exec.target.runner, scenario = scenario, target.runner = target.runner) } else { # Without load-balancing, we need to split the experiments into chunks # of size parallel. target.output <- unlist(use.names = FALSE, tapply(experiments, ceiling(1:length(experiments) / parallel), Rmpi::mpi.apply, exec.target.runner, scenario = scenario, target.runner = target.runner)) } # FIXME: if stop() is called from mpi.applyLB, it does not # terminate the execution of the parent process, so it will # continue and give more errors later. We have to terminate # here, but is there a nicer way to detect this and terminate? if (any(sapply(target.output, inherits, "try-error"))) { # FIXME: mclapply has some bugs in case of error. In that # case, each element of the list does not keep the output of # each configuration and repetitions may occur. cat(unique(unlist(target.output[sapply( target.output, inherits, "try-error")])), file = stderr(), sep = "") irace.error("A slave process terminated with a fatal error") } } else { if (.Platform$OS.type == 'windows') { irace.assert(!is.null(.irace$cluster)) if (scenario$loadBalancing) { target.output <- parallel::parLapplyLB(.irace$cluster, experiments, exec.target.runner, scenario = scenario, target.runner = target.runner) } else { target.output <- parallel::parLapply(.irace$cluster, experiments, exec.target.runner, scenario = scenario, target.runner = target.runner) } # FIXME: if stop() is called from parLapply, then the parent # process also terminates, and we cannot give further errors. } else { target.output <- parallel::mclapply(experiments, exec.target.runner, # FALSE means load-balancing. mc.preschedule = !scenario$loadBalancing, mc.cores = parallel, scenario = scenario, target.runner = target.runner) # FIXME: if stop() is called from mclapply, it does not # terminate the execution of the parent process, so it will # continue and give more errors later. We have to terminate # here, but is there a nicer way to detect this and terminate? if (any(sapply(target.output, inherits, "try-error")) || any(sapply(target.output, is.null))) { # FIXME: mclapply has some bugs in case of error. In that # case, each element of the list does not keep the output of # each configuration and repetitions may occur. cat(unique(unlist( target.output[sapply( target.output, inherits, "try-error")])), file = stderr()) irace.error("A child process triggered a fatal error") } } } } else { # One process, all sequential target.output <- lapply(experiments, exec.target.runner, scenario = scenario, target.runner = target.runner) } target.output } execute.evaluator <- function(experiments, scenario, target.output, configurations.id) { ## FIXME: We do not need the configurations.id argument: irace.assert(isTRUE(all.equal(configurations.id, sapply(experiments, getElement, "id.configuration")))) all.conf.id <- paste(configurations.id, collapse = " ") ## Evaluate configurations sequentially for (k in seq_along(experiments)) { output <- exec.target.evaluator(experiment = experiments[[k]], num.configurations = length(configurations.id), all.conf.id, scenario = scenario, target.runner.call = target.output[[k]]$call) target.output[[k]]$cost <- output$cost if (is.null(target.output[[k]]$call)) target.output[[k]]$call <- output$call if (is.null(target.output[[k]]$time) || !is.null.or.na(output$time)) target.output[[k]]$time <- output$time } return(target.output) } irace/NEWS.md0000644000176200001440000007240214321301254012433 0ustar liggesusers# irace 3.5 ## New features and improvements * Handling of dependent parameter domains: These should be specified in the parameter domain definition and, for now, only numerical parameter can define dependent domains. A numerical domain can be dependent on one bound, e.g. `(1, "param1*2")`, where the dependent bound can include basic arithmetic operators. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * The package now provides an `ablation` executable (`ablation.exe` in Windows) that makes easier to perform ablation analysis without having any R knowledge. * The interface to functions `ablation()` and `plotAblation()` has been simplified. The `ablation()` function now allows overriding scenario settings. The `plotAblation()` function will not create the plot if the ablation log does not contain a complete ablation. (Manuel López-Ibáñez) * The argument `n.instances` of `ablation()` has been renamed to `n_instances` and it is now a factor that multiplies `scenario$firstTest`. (Manuel López-Ibáñez) * New command-line option `--quiet` to run without producing any output except errors (also available as a scenario option). (Manuel López-Ibáñez) * New command-line option `--init` to initialize a scenario. (Deyao Chen) * Added support for HTCondor cluster framework to `--batchmode`. (Filippo Bistaffa) * `--check` now also check the contents of `configurationsFile` and runs configurations provided via `initConfigurations`. (Manuel López-Ibáñez, reported by Andreea Avramescu) * New scenario options `targetRunnerLauncher` and `targetRunnerLauncherArgs` to help in cases where the target-runner must be invoked via another software with particular options (such as `python.exe` in Windows). (Manuel López-Ibáñez) * New scenario option `minMeasurableTime`. (Manuel López-Ibáñez) * An error is produced if a variable set in the scenario file is not known to irace. If your scenario file contains R code, then use variable names beginning with a dot `'.'`, which will be ignored by irace. (Manuel López-Ibáñez) * Plotting functions have been moved to the new package [iraceplot](https://auto-optimization.github.io/iraceplot/). In particular, `configurationsBoxplot()` is replaced by `iraceplot::boxplot_training()` and `iraceplot::boxplot_test()`; `parallelCoordinatesPlot()` is replaced by `iraceplot::parallel_cat()` and `iraceplot::parallel_coord()`; and `parameterFrequency()` is replaced by `iraceplot::sampling_frequency()`. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * The user-guide now contains a detailed section on "Hyper-parameter optimization of machine learning methods". (Manuel López-Ibáñez) * When `testType="F-test"` and only two configurations remain, the elimination test now uses the pseudo-median estimated by the Wilcoxon signed-rank test to decide which configuration is the best one instead of comparing the median difference. (Manuel López-Ibáñez) * New functions `testing_fromlog()` and `testing_fromfile()` for independently executing the testing phase. The function `testing.main()` was removed as it is superseded by the new ones. (Manuel López-Ibáñez) * New function `read_logfile()` to easily read the log file produced by irace. (Manuel López-Ibáñez) * New function `printParameters()` that prints a parameters R object as a valid input text. (Manuel López-Ibáñez) * `irace2pyimp` moved to its own R package. (Manuel López-Ibáñez) * Generating the file `irace.Rdata` may be disabled by setting `logFile=""`. (Manuel López-Ibáñez, reported by Johann Dreo) * `path_rel2abs()` and `checkParameters()` are now exported so that other packages may use them. (Manuel López-Ibáñez) * `path_rel2abs()` also searches in system paths. (Manuel López-Ibáñez) * `readConfigurationsFile()` will now detect duplicated configurations and error. (Manuel López-Ibáñez) * The interface to functions `getFinalElites()`, `getConfigurationById()` and `getConfigurationByIteration()` has been simplified. * The package provides a `irace.sindef` file that may be used for building a standalone container of irace using Singularity. See the `README.md` file for instructions. (Contributed by Johann Dreo) * New example `examples/target-runner-python/target-runner-python-win.bat` contributed by Levi Ribeiro. * New helper script in `bin/parallel-irace-slurm` to launch `irace` in [SLURM](https://slurm.schedmd.com/) computer clusters. (Manuel López-Ibáñez) * Rename `scenario.update.paths()` to `scenario_update_paths()`. The old name is deprecated. (Manuel López-Ibáñez) ## Fixes * Correctly handle clear out-performance cases despite strong bi-modality. (Reported by Nguyen Dang, fixed by Manuel López-Ibáñez) * Fix error when recovering from a parallel run on Windows. (Manuel López-Ibáñez, reported by Tarek Gamal) * `testNbElites` now controls how many iteration elites are tested when `testIterationElites=1`. This is the documented behavior in the user guide. (Manuel López-Ibáñez, reported by Marcelo de Souza) * Fixes to the Matlab example. (Manuel López-Ibáñez) * The default of `testType` is now set to `t-test` when capping is enabled. (Manuel López-Ibáñez, reported by Jovana Radjenovic) * Fix various issues in the user guide. (Manuel López-Ibáñez, reported by Jovana Radjenovic) * Remove duplicated elites. (Manuel López-Ibáñez, reported by Federico Naldini) * Fix (#7): warnings with partial matched parameters. (Manuel López-Ibáñez, reported by Marc Becker) * Fix (#10): wrong assert with `elitist=0`. (Manuel López-Ibáñez) * Fix (#12): irace can be run with [FastR](https://www.graalvm.org/22.1/docs/getting-started/#run-r). * Fix (#13): Maximum number configurations immediately rejected reached. (Manuel López-Ibáñez) * Fix: when setting the scenario file in the command-line, `scenarioFile` was not set correctly. The correct scenario was used, however, the debug output and the value stored in the log / recovery file was wrong. (Manuel López-Ibáñez, reported by Richard Schoonhoven) * With `sampleInstances = FALSE`, elitist irace does not change the order of instances already seen. However, if you want to make sure that the order of the instances is enforced, you also need to set `elitistNewInstances=0`. * The function `irace.usage()` was removed. It was not really useful for R users as the same result can be obtained by calling `irace.cmdline("--help")`. (Manuel López-Ibáñez) # irace 3.4.1 (31/03/2020) * `NEWS` converted to markdown. * Fix CRAN error on Solaris. # irace 3.4 (30/03/2020) * `irace2pyimp` function and executable (`irace2pyimp.exe` on Windows) to convert .Rdata files generated by irace to the input files required by the parameter importance analysis tool PyImp (https://github.com/automl/ParameterImportance). (Nguyen Dang, Manuel López-Ibáñez) * Initial configurations may also be provided directly in R using `scenario$initConfigurations` (Manuel López-Ibáñez) * Rdata files are saved in version 2 to keep compatibility with older R versions. (Manuel López-Ibáñez) * Fix invalid assert with ordered parameters: (Leslie Pérez Cáceres) ``` value >= 1L && value <= length(possibleValues) is not TRUE ``` * The `irace` executable (`irace.exe` on Windows) is a compiled binary instead of a script. On Windows, `irace.exe` replaces `irace.bat` (Manuel López-Ibáñez) * `inst/examples/Spear` contains the Spear (SAT solver) configuration scenario. (Manuel López-Ibáñez) * Fixed bug when reporting minimum `maxTime` required. (Reported by Luciana Salete Buriol, fixed by Manuel López-Ibáñez) * Fixed bug detected by assert: ```R all(apply(!is.na(elite.data$experiments), 1, any)) is not TRUE ``` (Reported by Maxim Buzdalov, fixed by Manuel López-Ibáñez) # irace 3.3 (26/04/2019) * Fix buggy test that breaks CRAN. (Manuel López-Ibáñez) * Do not print "23:59:59" when wall-clock time is actually close to zero. (Manuel López-Ibáñez) # irace 3.2 (24/04/2019) * Fix `irace --check --parallel 2` on Windows. (Manuel López-Ibáñez) * Values of real-valued parameter are now printed with sufficient precision to satisfy `digits` (up to `digits=15`). (Manuel López-Ibáñez) * It is possible to specify `boundMax` without capping. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * `irace --check` will exit with code 1 if the check is unsuccessful (Manuel López-Ibáñez) * Print where irace is installed with `--help`. (Manuel López-Ibáñez) * irace will now complain if the output of `target-runner` or `target-evaluator` contains extra lines even if the first line of output is correct. This is to avoid parsing the wrong output. Unfortunately, this may break setups that relied on this behavior. The solution is to only print the output that irace expects. (Manuel López-Ibáñez) * Completely re-implement `log` parameters to fix several bugs. Domains that contain zero or negative values are now rejected. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * New option `aclib=` (`--aclib 1`) enables compatibility with the GenericWrapper4AC (https://github.com/automl/GenericWrapper4AC/) used by AClib (http://aclib.net/). This is EXPERIMENTAL. `--aclib 1` also sets digits to 15 for compatibility with AClib defaults. (Manuel López-Ibáñez) * Fix printing of output when capping is enabled. (Manuel López-Ibáñez) * `checkTargetFiles()` (`--check`) samples an instance unless `sampleInstances` is FALSE. (Manuel López-Ibáñez) * Fix symbol printed in elimination test. (Manuel López-Ibáñez) * Use `dynGet()` to find `targetRunner` and `targetEvaluator`. As a result, we now require R >= 3.2. (Manuel López-Ibáñez) * All tests now use `testthat`. (Manuel López-Ibáñez) * New function `scenario.update.paths()` (Manuel López-Ibáñez) * Fix assert failure that may happen when `elitistNewInstances` is larger than `firstTest`. Reported by Jose Riveaux. (Manuel López-Ibáñez) * Fix bug in `checkTargetFiles()` (`--check`) with capping. (Leslie Pérez Cáceres) * Clarify a few errors/warnings when `maxTime > 0`. (Manuel López-Ibáñez, suggested by Haroldo Gambini Santos) # irace 3.1 (12/07/2018) * Use testthat for unit testing. (Manuel López-Ibáñez) * Allow instances to be a list of arbitrary R objects (`mlr` bugfix). (Manuel López-Ibáñez) # irace 3.0 (05/07/2018) * irace now supports adaptive capping for computation time minimization. The default value of the `testType` option is t-test when adaptive capping is enabled. Please see the user-guide for details. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * The package contains an `ablation()` function implementing the ablation method for parameter importance analysis by Fawcett and Hoos (2016). (Leslie Pérez Cáceres, Manuel López-Ibáñez) * New option `postselection` executes a post-selection race. (Leslie Pérez Cáceres) * At the end of each race, if the race stops before evaluating all instances seen in previous races, then the best overall may be different than the best of the race. We now print the best overall (best-so-far). Elites evaluated on more instances are considered better than those evaluated on fewer. (Manuel López-Ibáñez, Leslie Pérez Cáceres) * Last active parameter values of numerical parameters (`i` and `r`) are carried by the sampling model. When a value must be assigned and the parameter was previously not active, the sampling is performed around the last value. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * R help pages are now generated with Roxygen2. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * The user guide documents `--version`, `--help`, and `--check`. (Manuel López-Ibáñez) * A return value of `Inf` from `targetRunner`/`targetEvaluation` results in the immediate rejection of the configuration without any further evaluation. This is useful for handling unreliable or broken configurations that should not stop irace. (Manuel López-Ibáñez) * Numerical parameters may be sampled on a logarithmic scale using `i,log` or `r,log`. (Alberto Franzin) * New `target-runner.bat` for Windows contributed by André de Souza Andrade. * Fixed all shell scripts calling functions before defining them, which is not portable. (Manuel López-Ibáñez) * Fixed `--parallel` bug in Windows that resulted in `Error in checkForRemoteErrors(val)`. (Manuel López-Ibáñez) * Improve error message when no training instances are given. (Manuel López-Ibáñez) # irace 2.4 (03/08/2017) * The output of irace now specifies in which order, if any, configurations are printed. (Manuel López-Ibáñez, suggested by Markus Wagner) * Several fixes for handling paths in Windows. (Manuel López-Ibáñez) * `readConfigurationsFile()` now has a `text=` argument, which allows reading configurations from a string. (Manuel López-Ibáñez) * User-provided functions (targetRunner, targetEvaluator and repairConfiguration) and user-provided conditions for forbidden configurations are now byte-compiled when read, which should make their evaluation noticeably faster. (Manuel López-Ibáñez) * The argument `'experiment'` passed to the R function `targetRunner` does not contain anymore an element `'extra.params'`. Similarly, the `'scenario'` structure does not contain anymore the elements `'instances.extra.params'` and `'testInstances.extra.params'`. Any instance-specific parameters values now form part of the character string that defines an instance and it is up to the user-defined `targetRunner` to parse them appropriately. These changes make no difference when targetRunner is an external script, or when instances and instance-specific parameter values are read from a file. (Manuel López-Ibáñez) # irace 2.3 * Fix bug that will cause `iraceResults$experimentLog` to count calls to `targetEvaluator` as experiments, even if no call to `targetRunner` was performed. This does not affect the computation of the budget consumed and, thus, it does not affect the termination criteria of irace. The bug triggers an assertion that terminates irace, thus no run that was successful with version 2.2 is affected. (Manuel López-Ibáñez) # irace 2.2 * Command-line parameters are printed to stdout (useful for future replications). (Manuel López-Ibáñez, suggested by Markus Wagner) * Users may provide a function to repair configurations before being evaluated. See the scenario variable repairConfiguration. (Manuel López-Ibáñez) * The option `--sge-cluster` (`sgeCluster`) was removed and replaced by `--batchmode` (`batchmode`). It is now the responsibility of the target-runner to parse the output of the batch job submission command (e.g., `qsub` or `squeue`), and return just the job ID. Values supported are: "sge", "torque", "pbs" and "slurm". (Manuel López-Ibáñez) * The option `--parallel` can now be combined with `--batchmode` to limit the number of jobs submitted by irace at once. This may be useful in batch clusters that have a small queue of jobs. (Manuel López-Ibáñez) * New examples under `inst/examples/batchmode-cluster/`. (Manuel López-Ibáñez) * It is now possible to include scenario definition files from other scenario files by using: ```R eval.parent(source("scenario-common.txt", chdir = TRUE, local = TRUE)) ``` This feature is VERY experimental and the syntax is likely to change in the future. (Manuel López-Ibáñez) * Fix a bug that re-executed elite results under some circumstances. (Leslie Pérez Cáceres) * Restrict the number of maximum configurations per race to 1024. (Leslie Pérez Cáceres) * Do not warn if the last line in the instance file does not terminate with a newline. (Manuel López-Ibáñez) * Fix bug when `deterministic == 1`. (Manuel López-Ibáñez, Leslie Pérez Cáceres) * Update manual and vignette with details about the expected arguments and return value of `targetRunner` and `targetEvaluator`. (Manuel López-Ibáñez) * Many updates to the User Guide vignette. (Manuel López-Ibáñez) * Fix `\dontrun` example in `irace-package.Rd` (Manuel López-Ibáñez) * Fix bug: If testInstances contains duplicates, results of testing are not correctly saved in `iraceResults$testing$experiments` nor reported correctly at the end of a run. Now unique IDs of the form `1t, 2t, ...` are used for each testing instance. These IDs are used for the rownames of `iraceResults$testing$experiments` and the names of the `scenario$testInstances` and `iraceResults$testing$seeds` vectors. (Manuel López-Ibáñez) * Fix bug where irace keeps retrying the `target-runner` call even if it succeeds. (Manuel López-Ibáñez) * New command-line parameter ``` --only-test FILE ``` which just evaluates the configurations given in FILE on the testing instances defined by the scenario. Useful if you decide on the testing instances only after running irace. (Manuel López-Ibáñez) * Bugfix: When using `maxTime != 0`, the number of experiments performed may be miscounted in some cases. (Manuel López-Ibáñez) # irace 2.1 * Fix CRAN errors in tests. (Manuel López-Ibáñez) * Avoid generating too many configurations at once if the initial time estimation is too small. (Manuel López-Ibáñez) # irace 2.0 * Minimum R version is 2.15. * Elitist irace by default, it can be disabled with parameter `--elitist 0`. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * The parameter `--test-type` gains two additional values: (Manuel López-Ibáñez) - `t-test-bonferroni` (t-test with Bonferroni's correction for multiple comparisons), - `t-test-holm` (t-test with Holm's correction for multiple comparisons) * MPI does not create log files with `--debug-level 0`. (Manuel López-Ibáñez) * For simplicity, the `parallel-irace-*` scripts do not use an auxiliary `tune-main` script. For customizing them, make a copy and edit them directly. (Manuel López-Ibáñez) * New parameters: (Manuel López-Ibáñez) ``` --target-runner-retries : Retry target-runner this many times in case of error. ``` * We print diversity measures after evaluating on each instance: (Leslie Pérez Cáceres) - Kendall's W (also known as Kendall's coefficient of concordance) If 1, all candidates have ranked in the same order in all instances. If 0, the ranking of each candidate on each instance is essentially random. W = Friedman / (m * (k-1)) - Spearman's rho: average (Spearman) correlation coefficient computed on the ranks of all pairs of raters. If there are no repeated data values, a perfect Spearman correlation of +1 or -1 occurs when each of the variables is a perfect monotone function of the other. * Many internal and external interfaces have changed. For example, now we consistently use 'scenario' to denote the settings passed to irace and 'configuration' instead of 'candidate' to denote the parameter settings passed to the target algorithm. Other changes are: ```R parameters$boundary -> parameters$domain hookRun -> targetRunner hookEvaluate -> targetEvaluator tune-conf -> scenario.txt instanceDir -> trainInstancesDir instanceFile -> trainInstancesFile testInstanceDir -> testInstancesDir testInstanceFile -> testInstancesFile ``` * Minimal example of configuring a MATLAB program (thanks to Esteban Diaz Leiva) * Paths to files or directories given in the scenario file are relative to the scenario file (except for `--log-file`, which is an output file and it is relative to `--exec-dir`). Paths given in the command-line are relative to the current working directory. Given ```bash $ cat scenario/scenario.txt targetRunner <- "./target-runner" $ irace -s scenario/scenario.txt ``` irace will search for `"./scenario/target-runner"`, but given ```bash $ irace -s scenario/scenario.txt --target-runner ./target-runner ``` irace will search for `"./target-runner"`. (Manuel López-Ibáñez) * New command-line wrapper for Windows installed at `system.file("bin/irace.bat", package="irace")` (thanks to Anthony Antoun) * Budget can be specified as maximum time (`maxTime`, `--max-time`) consumed by the target algorithm. See the documentation for the details about how this is handled. (Leslie Pérez Cáceres, Manuel López-Ibáñez) # irace 1.07 * The best configurations found, either at the end or at each iteration of an irace run, can now be applied to a set of test instances different from the training instances. See options `testInstanceDir`, `testInstanceFile`, `testNbElites`, and `testIterationElites`. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * The R interfaces of `hookRun`, `hookEvaluate` and `hookRunParallel` have changed. See `help(hook.run.default)` and `help(hook.evaluate.default)` for examples of the new interfaces. * Printing of race progress now reports the actual configuration and instance IDs, and numbers are printed in a more human-readable format. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * Reduce memory use for very large values of `maxExperiments`. (Manuel López-Ibáñez, thanks to Federico Caselli for identifying the issue) * New option `--load-balancing` (`loadBalancing`) for disabling load-balancing when executing jobs in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster. (Manuel López-Ibáñez, thanks to Federico Caselli for identifying the issue) * The option `--parallel` in Windows now uses load-balancing by default. (Manuel López-Ibáñez) * The wall-clock time after finishing each task is printed in the output. (Manuel López-Ibáñez, thanks to Federico Caselli for providing an initial patch) # irace 1.06 * Fix bug that could introduce spurious whitespace when printing the final configurations. (Manuel López-Ibáñez) * Fix bug if there are more initial candidates than needed for the first race. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * New configuration options, mainly for R users: - `hookRunParallel`: Optional R function to provide custom parallelization of `hook.run`. - `hookRunData`: Optional data passed to `hookRun`. This is ignored by the default `hookRun` function, but it may be used by custom `hookRun` R functions to pass persistent data around. (Manuel López-Ibáñez) # irace 1.05 * New option `--version`. (Manuel López-Ibáñez) * Terminate early if there is no sufficient budget to run irace with the given settings. (Manuel López-Ibáñez) * The option `--parallel` (without `--mpi`) now works under Windows. (Manuel López-Ibáñez, thanks to Pablo Valledor Pellicer for testing it) * Improved error handling when running under Rmpi. Now irace will terminate as soon as the master node detects at least one failed slave node. This avoids irace reporting two times the same error. Also, irace will print all the unique errors returned by all slaves and not just the first one. (Manuel López-Ibáñez) * Forbidden configurations may be specified in terms of constraints on their values. Forbidden configurations will never be evaluated by irace. See `--forbidden-file` and `inst/templates/forbidden.tmpl`. (Manuel López-Ibáñez) * New option `--recovery-file` (`recoveryFile`) allows resuming a previous irace run. (Leslie Pérez Cáceres) * The confidence level for the elimination test is now configurable with parameter `--confidence`. (Leslie Pérez Cáceres) * Much more robust handling of relative/absolute paths. Improved support for Windows. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * Provide better error messages for incorrect parameter descriptions. (Manuel López-Ibáñez) Examples: ``` x "" i (0, 0) # lower and upper bounds are the same x "" r (1e-4, 5e-4) # given digits=2, ditto x "" i (-1, -2) # lower bound must be smaller than upper bound x "" c ("a", "a") # duplicated values ``` * Print elapsed time for calls to hook-run if `debugLevel >=1`. (Manuel López-Ibáñez) * `examples/hook-run-python/hook-run`: A multi-purpose `hook-run` written in Python. (Franco Mascia) * Parallel mode in an SGE cluster (`--sge-cluster`) is more robust. (Manuel López-Ibáñez) # irace 1.04 * Replace obsolete package multicore by package parallel (requires R >= 2.14.0) * Use load-balancing (`mc.preschedule = FALSE`) in `mclapply`. # irace 1.03 * Use `reg.finalizer` to finish Rmpi properly without clobbering `.Last()`. * Remove uses of deprecated `as.real()`. * Nicer error handling in `readParameters()`. * Add hypervolume (multi-objective) example. * Fix several bugs in the computation of similar candidates. # irace 1.02 * More concise output. * The parameters `expName` and `expDescription` are now useless and they were removed. * Faster computation of similar candidates (Jeremie Dubois-Lacoste and Leslie Pérez Cáceres). * Fix bug when saving instances in `tunerResults$experiments`. * `irace.cmdline ("--help")` does not try to quit R anymore. # irace 1.01 * Fix bug caused by file.exists (and possibly other functions) not handling directory names with a trailing backslash or slash on Windows. * Fix bug using per-instance parameters (Leslie Pérez Cáceres). * Fix bug when reading initial candidates from a file. irace/MD50000644000176200001440000002542614325252713011662 0ustar liggesusers69a04f8f4c0abbd39178a3d0c7f3ce10 *BUGS 5045e033d6ef0ac2cadce585596446bd *DESCRIPTION a63f58132973b19bf43020d7a5f15a8d *NAMESPACE 038a593734b0b4fcccde699be7b7ce20 *NEWS.md b1f319dfbac17e70234c634a2490b632 *R/ablation.R 9c80cd3c08b5cb818c94fd65f81907a2 *R/argparser.R 5e50e779e5017e05cd2fa56b684f6c06 *R/cluster.R a0a9019a8fff086a6b04aba345855c11 *R/generation.R 92e15c99068d5698c8b5f396abf50212 *R/irace-options.R 76968ff9efe28f18642b4d4a3163ae66 *R/irace-package.R b56a08ad22985062d39a85653a4f4e4f *R/irace.R 915e1242e234fc7b6ec17a0793781a3c *R/main.R fef3c5475713f5f09a5580f3225ab771 *R/model.R d1301df655fad850e615a3f62d536179 *R/parameterAnalysis.R 86ead6121525dc45cf04a3f54ead0c19 *R/parameterExploration.R 068f4659da9911b4e24e68c81ba0614f *R/path_rel2abs.R 213f69a714e687be269920999074c87c *R/race-wrapper.R 816656412e80a1f170fc8938d968c25d *R/race.R ee9dff2757fd0537a3f43759518bc37b *R/readConfiguration.R bc45aa88d1d4ff5d05a22778123b18a4 *R/readParameters.R 519ff55d3b1b32a21363ad4e4bbfa133 *R/testing.R bc04ea89773012f517adc374a6cc6cd9 *R/timer.R 55b4b74d608899de8109a8a136490e96 *R/tnorm.R 4fd568645b95fe328c7d2843442f1c89 *R/utils.R 4904572120e0067c272b3b2335d5a8d1 *R/version.R 2a118b7354fdedff5c74246d8423e5ad *R/zzz.R 559d78ef5ba4df1b8666192ac468df59 *README.md 92d3ace7e0265aaf74b473cadbcf470a *build/partial.rdb a852616f9564563cd4aadf7640345486 *build/vignette.rds 76b4951f93d078c5a1598e781fd9e670 *cleanup 4fb959d1be3d68345289530627355ca9 *inst/CITATION f694f4bed8633608f1914f539c2dfee4 *inst/bin/parallel-irace 36995f96e183cdd8e7e25207c4d95394 *inst/bin/parallel-irace-mpi 404f816e7b4fdf2a142c65c931f300ce *inst/bin/parallel-irace-qsub 8eda33b154a7c9cff1a90c0e168da70e *inst/bin/parallel-irace-slurm 1263d7dbd890bf549235501fbaf60c05 *inst/doc/irace-package.R c2e0a6fcffab3c32cdfc05ff2d5aa126 *inst/doc/irace-package.Rnw afa8bf9cb475e5f5f0cde602756d274b *inst/doc/irace-package.pdf 6f5adad387e00e68a7802e8d543b18ce *inst/examples/README.md 067fe2821c16cf8262dece7a4c36144e *inst/examples/Spear/README 4a9044d0341a321c352ec901cb97a8f0 *inst/examples/Spear/parameters-cat.txt cb2da2a31c96ffc3a4274e9a8ef3832a *inst/examples/Spear/parameters-mixed.txt ed518f04134d2ca0f90b7393467b4e15 *inst/examples/Spear/scenario.txt 039051117ef71079e1d2ff106d6c4e7a *inst/examples/Spear/target-runner d24870bcca1038abb7a62d97df77cbd2 *inst/examples/acotsp/README d14e5bb16ddb374e204cabfc0be4545d *inst/examples/acotsp/default.txt 0f16709b42a05455f08906576ba3ee18 *inst/examples/acotsp/forbidden.txt 45771f3ceccfe39d9568abda93d7df24 *inst/examples/acotsp/parameters-acotsp.txt cc17eef505606543c0d5536f2c34b079 *inst/examples/acotsp/scenario.txt addd8e7ae0e2ca926476aa9b9e13cb08 *inst/examples/acotsp/target-runner b7d067aba94e5b7fc1e92237a6f0a1b4 *inst/examples/batchmode-cluster/README d960f69eeadff7405e4452f8f50c710a *inst/examples/batchmode-cluster/irace-sge-cluster 0b3da55eb761846dcf247a23060ffb0a *inst/examples/batchmode-cluster/target-evaluator cb7e8b282835fd66d0745aa16dfdecbf *inst/examples/batchmode-cluster/target-runner-htcondor 682e5b6a084e3e94e0ae045b60de8f9f *inst/examples/batchmode-cluster/target-runner-pbs 3cd7828292c3927406d34868ad9266b0 *inst/examples/batchmode-cluster/target-runner-sge 40fbfe617cda99828251f4b2128a4076 *inst/examples/batchmode-cluster/target-runner-slurm 3523d53e5d616ae1343cbb787b703b5c *inst/examples/hypervolume/README 5d16974368bcb97b8dcf9594b19152f2 *inst/examples/hypervolume/target-evaluator 74406c35bc1a89a6e202f9aa27f48817 *inst/examples/hypervolume/target-runner 7a908b35eff27e6ed9fe12d5936f8018 *inst/examples/matlab/Main.m 3b0332e02daabf31651a5a0d81ba830a *inst/examples/matlab/instances.txt 570fbc605de77856756be963e4fb8c91 *inst/examples/matlab/parameters.txt a7c2b75263639028f078e5096a8c2be0 *inst/examples/matlab/scenario.txt 117ed65e989d5b1d901431e35c392b00 *inst/examples/moaco/README 6bd5d5787a551af4e396b6681960681f *inst/examples/moaco/parameters.txt 047f8de89d4747cd3526a4c7e11075c3 *inst/examples/moaco/scenario.txt 5157ef3fc2efb9230f76aef1cfe73291 *inst/examples/moaco/target-evaluator e250b38c64ed8030bc71ee66ec4f8941 *inst/examples/moaco/target-runner de151ff3fce62d302ea77b93c8a5c92b *inst/examples/target-runner-python/target-runner-acotsp.py 5ab7fbf168799bc1c7a54c9b8549a159 *inst/examples/target-runner-python/target-runner-advanced.py fc5b988b87d11f4e54a72ced66b72fb2 *inst/examples/target-runner-python/target-runner-python-win.bat 2286cd7e0e526fa1526ea38a612d476b *inst/examples/target-runner-python/trivial/instances.txt e3f5b6aee118bd70096440b0d648e812 *inst/examples/target-runner-python/trivial/parameters.txt e35dda63b5e6a98c1c15b392693ac633 *inst/examples/target-runner-python/trivial/scenario.txt 3dcae77aa87c659dff7dd85639ed2e27 *inst/examples/target-runner-python/trivial/target-runner.py fdf194d8f19dada2867eff7a0695139d *inst/exdata/irace-acotsp.Rdata 15d6164b0d369f64e77ed992d0ec3385 *inst/exdata/log-ablation.Rdata 38aec57159de03f0ff4d710331413041 *inst/exdata/sann.rda a523dd89b6ec7ba2adc26e2ed3a3a179 *inst/irace.sindef 5718f32d8359ec715f2d8bc1a0ec02b3 *inst/templates/configurations.txt.tmpl 635cce18d757a309cf38afafd9ebda9d *inst/templates/forbidden.txt.tmpl af9376428b2c2d911710fd6e51ecf784 *inst/templates/instances-list.txt.tmpl 261e74e936e2217cb612278c5d231887 *inst/templates/parameters.txt.tmpl 65bde26c9a3d3b154c68a90381fe889e *inst/templates/scenario.txt.tmpl 0185c79ae7b6f5994f89da433acd7b94 *inst/templates/target-evaluator.tmpl ce1e0647e76140eac038f0a1e0f30d2a *inst/templates/target-runner.tmpl 4850d7fbd68fbc7c06af53d3c510734f *inst/templates/windows/target-runner.bat 994adc0a852364fdcea6b274bbc4ba47 *man/CommandArgsParser.Rd 6b0dde5d934c2c16a9555cc6305bc8c1 *man/ablation.Rd 078aae4ce581e120cd576971577f5e12 *man/ablation_cmdline.Rd c1211469bba0c10e00fa80c47a384d7a *man/buildCommandLine.Rd d412ab580b7ba52bfc02370a93fe99eb *man/checkIraceScenario.Rd 7a1d06d2da2f300a7eed9fe508838824 *man/checkParameters.Rd 04e6a8b6f89522fe45d545285d6c3f4f *man/checkScenario.Rd 7e5aa8f3c75eaedc7d8e19a902864403 *man/configurations.print.Rd 14e033558771c9106b429f53bfb72320 *man/configurations.print.command.Rd c10db4cc76effd892d2d71f48d67c292 *man/defaultScenario.Rd cd7f7baaa8734863c8f7a62ae97a045b *man/getConfigurationById.Rd 9a72690609110ba97808c4284a4121ee *man/getConfigurationByIteration.Rd 147fc1fd2f1fe41f7d19efa46d981955 *man/getFinalElites.Rd ab24f5f4834c020a9e13d5d0b27456d9 *man/irace-package.Rd 9fda11a0c9063c4d668b58fbd06970ed *man/irace.Rd fc7d521a068ac95854f7aa0cd07fa8eb *man/irace.cmdline.Rd 8fb6cd730c0c3f68f950035aab00f116 *man/irace.license.Rd bfee6562847f9202097c048aea8fc38e *man/irace.main.Rd c61fcfc9965c72d1792982c4ac05487b *man/irace.version.Rd f851af63937121b12ba3818fc89d9cc4 *man/path_rel2abs.Rd 88e40f9047a1e78663eb5bb714d5862c *man/plotAblation.Rd 1e954c71985f87b3d84d20832e64ba8e *man/printParameters.Rd 1fa672e70f48ba69c3c48a9c634c199c *man/printScenario.Rd 0866f93c1c8b41b8ddf20cda0d2b37f8 *man/psRace.Rd 3933746918dd3372aef6825c963c2f6f *man/readConfigurationsFile.Rd 6416a3ab3b2c54c915f5297beba44166 *man/readParameters.Rd fd4014da95182daa5f4e036d42716aff *man/readScenario.Rd f38a57a9b8da01e0cb6e227853a7bbd2 *man/read_logfile.Rd 0013e857af07f83b4c3657a1a88525cb *man/read_pcs_file.Rd fe958fdb0536802ef73a17e55b3c2ddb *man/removeConfigurationsMetaData.Rd 8d2c92c883e214f8ac76f4a4d63a750f *man/scenario_update_paths.Rd f4388a116d42c37a4bc6d8f64cb281bd *man/target.evaluator.default.Rd 7c150396daf48a28b3dae46ecbeca84a *man/target.runner.default.Rd 8499f76c5044d2aa52df9bcf03cacbb5 *man/testConfigurations.Rd 269490247d3ceb4ea52130c545f5ee00 *man/testing_fromfile.Rd a5e75a420330b3074709671da1e3d0e7 *man/testing_fromlog.Rd a886f8cb75354603c3204d2cebc07888 *src/Makevars 7906feceff998d2b2309454c9cbb2f10 *src/Makevars-common ca5fd252fe6221d98e26e1eab7b8d9c6 *src/Makevars.win b9f27579825988f77dd3bd9cc80449c1 *src/install.libs.R 4fb82b74a5fcc75be149ef7b2ee9b2a7 *src/iracebin/ablation.c 58ea452025942c848ac07b11f57427ea *src/iracebin/irace.c f6939d43ee41ad7076812ca9404cf4e8 *src/iracebin/irace.h 4c3b685dcd16cc3d2091da956863d966 *tests/testthat.R b735d8cdd247647949863dc0db5242c5 *tests/testthat/bad_scenario.txt d2a04968b98d1dc030087ed5a3973a24 *tests/testthat/bug-13-lookup.rds 392472d97647beac3650ff564bd64455 *tests/testthat/bug_large_new_instances-confs.txt c37af8399644c4c02083ce46b2a72631 *tests/testthat/bug_large_new_instances.Rdata 0c70d9f2df1bdaae7a5427383deae61f *tests/testthat/configurations.txt 5392765ea169c97660c1a8f5e38962d5 *tests/testthat/dependencies.txt 72b9700b9daeb38f2b3d56b07e761337 *tests/testthat/dependencies2.txt a96f6d23df9bca94fd33c47b8fe54835 *tests/testthat/dummy_wrapper.py db236f7a1008af30779f40caf07060f7 *tests/testthat/forbidden.txt e53bf0109c13ffc1734dc5d66c77caac *tests/testthat/helper-common.R 5a08d4053b2e1733ac189074f84a62cc *tests/testthat/logparameters.txt 1e4c15b1cb77a4738b5431e83f59cf33 *tests/testthat/parameters.txt 34ecd3091ca49fabe295aaa76720e449 *tests/testthat/saved_maxim_bug.rds bfcde32a37798cf3fa4a0cd3a47933c9 *tests/testthat/saved_maxim_bug2.rds 802b1377a53b451f13bc190dd692eb9e *tests/testthat/scenario.txt fa5811cf38687ea0ba22857d07e43708 *tests/testthat/setup.R 390e7a8b3437a82b11f614fd1c1976f1 *tests/testthat/teardown.R f738821c6a6716067263dcb035a8f499 *tests/testthat/test-GenericWrapper4AC.R c76a2bb7c81016dd0769ea1463db40f9 *tests/testthat/test-bad_scenario.R 9001476440addf62d0bddf2d83e66391 *tests/testthat/test-bug-10.R e72a8ebe9175644123b6939ac21d9e79 *tests/testthat/test-bug-13.R 53a1fc7b10969c0a91f94585abe9a7ea *tests/testthat/test-bugs.R 968331f372e347ce39e0d3da72f58c70 *tests/testthat/test-capping.R 74f00ef9d7219ebcf467f6e63f1de3d1 *tests/testthat/test-dependencies.R 158ae05d47f940a435ee6770ec56ae86 *tests/testthat/test-forbidden.R a86211242111165f2e32a8a9105bdc54 *tests/testthat/test-maxTime.R 56754b543d36c2c63459a4befbd50c03 *tests/testthat/test-mlr.R 1dbfbc6930ca68aac00a2ecf119b0b0a *tests/testthat/test-path.R 827ba08b388c13ad94d7c47ec643b48b *tests/testthat/test-raceconfs.R 386a21c6965ec4ee93c55c6b2d4dff5d *tests/testthat/test-readconfs.R c6dbfe891bc46013458c5858ce22e7bf *tests/testthat/test-sann-irace.R 54aba5c7c5e6cacd5d809cef03a4e4ac *tests/testthat/test-similar.R 8ebd654ea8317ed80746c261c4e90b5d *tests/testthat/test-targetRunnerParallel.R 30e04b3da8a10ca41b7457a0f553d059 *tests/testthat/test-targeteval.R 038a593734b0b4fcccde699be7b7ce20 *vignettes/NEWS.txt 289ab2a15fa9dca53bb4a94dff1817e3 *vignettes/Warning-icon.png 29ce28ad39dd3182608f02197ec1ed45 *vignettes/examples.Rdata 4784f9bce400ae4624ca874739cd7c4b *vignettes/fig1u-acotsp-instances.pdf df916e74a88ae7e49d09735b04c82cd8 *vignettes/irace-acotsp-stdout.txt fdf194d8f19dada2867eff7a0695139d *vignettes/irace-acotsp.Rdata c2e0a6fcffab3c32cdfc05ff2d5aa126 *vignettes/irace-package.Rnw 003c544a7fe10a9552d74ac6bdc0a1ac *vignettes/irace-package.bib f44025a8b672c55f7c68873d309336ad *vignettes/irace-scheme.pdf 0b5a399b2478c44749ab63304194f6a7 *vignettes/light-bulb-icon.png 9d6ff066c6a773ca2b8c41999f684e0b *vignettes/section/irace-options.tex irace/inst/0000755000176200001440000000000014325242462012316 5ustar liggesusersirace/inst/examples/0000755000176200001440000000000014325242462014134 5ustar liggesusersirace/inst/examples/acotsp/0000755000176200001440000000000014325242462015425 5ustar liggesusersirace/inst/examples/acotsp/README0000644000176200001440000000073314175750646016323 0ustar liggesusersExample configuration scenario: ACOTSP ===================================== This example scenario shows how irace can be used to tune an optimization algorithm. ACOTSP is a software that implements various Ant Colony algorithms for the symmetric Traveling Salesman Problem. The ACOTSP software is available at: http://www.aco-metaheuristic.org/aco-code/ TSP instances and more information is available at: https://iridia.ulb.ac.be/supp/IridiaSupp2016-003/index.html irace/inst/examples/acotsp/scenario.txt0000644000176200001440000000276514157104332017777 0ustar liggesusers###################################################### -*- mode: r -*- ##### ## Scenario setup for Iterated Race (iRace). ############################################################################ ## To use the default value of a parameter of iRace, simply do not set ## the parameter (comment it out in this file, and do not give any ## value on the command line). ## File that contains the description of the parameters. parameterFile = "./parameters-acotsp.txt" ## Directory where the programs will be run. execDir = "./acotsp-arena" ## Directory where tuning instances are located, either absolute path or ## relative to current directory. trainInstancesDir = "./Instances" ## The maximum number of runs (invocations of targetRunner) that will performed. It ## determines the (maximum) budget of experiments for the tuning. maxExperiments = 5000 ## File that contains a set of initial configurations. If empty or NULL, ## all initial configurations are randomly generated. # configurationsFile = "" ## File that contains a list of logical expressions that cannot be TRUE ## for any evaluated configuration. If empty or NULL, do not use forbidden ## expressions. forbiddenFile = "forbidden.txt" ## Indicates the number of decimal places to be considered for the ## real parameters. digits = 2 ## A value of 0 silences all debug messages. Higher values provide ## more verbose debug messages. # debugLevel = 0 ## END of scenario file ############################################################################ irace/inst/examples/acotsp/default.txt0000644000176200001440000000150114157104332017603 0ustar liggesusers## Template for specifying initial parameter configurations, for ## example, the default configuration, in irace. ## ## This filename must be specified via the --configurations-file command-line option ## (or configurationsFile in scenario.txt). ## ## The format is one parameter configuration per line, and one parameter per ## column. The first line gives the parameter name corresponding to ## each column (names must match those given in the parameters ## file). Each configuration must satisfy the ## parameter conditions (NA should be used for those parameters that ## are not enabled for a given configuration) and, if given, the ## constraints that describe forbidden configurations. algorithm localsearch alpha beta rho ants nnls dlb q0 rasrank elitistants as 0 1.0 1.0 0.95 10 NA NA 0 NA NA irace/inst/examples/acotsp/forbidden.txt0000644000176200001440000000104514157104332020116 0ustar liggesusers## Template for specifying forbidden parameter configurations for irace. ## ## This filename must be specified via the --forbidden-file command-line option ## (or forbiddenFile in scenario.txt). ## ## The format is one constraint per line. Each constraint is a logical ## expression (in R syntax). If a parameter configuration ## is generated that makes the logical expression evaluate to TRUE, ## then the configuration is discarded. ## ## Examples of valid logical operators are: == != >= <= > < & | ! %in% (alpha == 0.0) & (beta == 0.0) irace/inst/examples/acotsp/parameters-acotsp.txt0000644000176200001440000000147614157104332021624 0ustar liggesusers### Parameter file for the ACOTSP software # name switch type values [conditions (using R syntax)] algorithm "--" c (as,mmas,eas,ras,acs) localsearch "--localsearch " c (0, 1, 2, 3) alpha "--alpha " r (0.00, 5.00) beta "--beta " r (0.00, 10.00) rho "--rho " r (0.01, 1.00) ants "--ants " i (5, 100) q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" rasrank "--rasranks " i (1, 100) | algorithm == "ras" elitistants "--elitistants " i (1, 750) | algorithm == "eas" nnls "--nnls " i (5, 50) | localsearch %in% c(1,2,3) dlb "--dlb " c (0, 1) | localsearch %in% c(1,2,3) irace/inst/examples/acotsp/target-runner0000755000176200001440000000461014226017524020150 0ustar liggesusers#!/bin/bash ############################################################################### # This script is to tune the ACOTSP software. # # PARAMETERS: # $1 is the ID of the candidate to be evaluated # $2 is the instance ID # $3 is the seed # $4 is the instance name # The rest ($* after `shift 4') are parameters for running ACOTSP # # RETURN VALUE: # This script should print a single numerical value (the value to be minimized). ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" >&2 exit 1 } # Path to the ACOTSP executable (this path is relative to the execution directory `execDir` specified in the scenario). # EXE="~/bin/acotsp" EXE="../ACOTSP-1.03/acotsp" # Fixed parameters that should be always passed to ACOTSP. # The time to be used is always 10 seconds, and we want only one run: FIXED_PARAMS=" --tries 1 --time 10 --quiet " CONFIG_ID="$1" INSTANCE_ID="$2" SEED="$3" INSTANCE="$4" # All other parameters are the candidate parameters to be passed to program shift 4 || error "Not enough parameters" CONFIG_PARAMS=$* STDOUT=c${CONFIG_ID}-${INSTANCE_ID}-${SEED}.stdout STDERR=c${CONFIG_ID}-${INSTANCE_ID}-${SEED}.stderr if [ ! -x "${EXE}" ]; then error "${EXE}: not found or not executable (pwd: $(pwd))" fi # Now we can call ACOTSP by building a command line with all parameters for it $EXE ${FIXED_PARAMS} -i $INSTANCE --seed $SEED ${CONFIG_PARAMS} 1> $STDOUT 2> $STDERR # The output of the candidate $CONFIG_ID should be written in the file # c${CONFIG_ID}.stdout (see target runner for ACOTSP). # Does this file exist? if [ ! -s "${STDOUT}" ]; then # In this case, the file does not exist. Let's exit with a value # different from 0. In this case irace will stop with an error. error "${STDOUT}: No such file or directory" fi # Ok, the file exist. It contains the whole output written by ACOTSP. # This script should return a single numerical value, the best objective # value found by this run of ACOTSP. The following line is to extract # this value from the file containing ACOTSP output. COST=$(cat ${STDOUT} | grep -o -E 'Best [-+0-9.e]+' | cut -d ' ' -f2) if ! [[ "$COST" =~ ^[-+0-9.e]+$ ]] ; then error "${STDOUT}: Output is not a number" fi # Print it! echo "$COST" # We are done with our duty. Clean files and exit with 0 (no error). rm -f "${STDOUT}" "${STDERR}" rm -f best.* stat.* cmp.* exit 0 irace/inst/examples/target-runner-python/0000755000176200001440000000000014325242462020250 5ustar liggesusersirace/inst/examples/target-runner-python/target-runner-advanced.py0000755000176200001440000004323214157104332025165 0ustar liggesusers#!/usr/bin/env python ############################################################################## # # # With this target-runner, you can: # # # # - set environment variables # # - run your jobs read the standard output / standard err, and check the # # return code # # - you can choose between 5 different ways of running your job and reading # # the standard out (you might have to try several ones until you find one # # that is robust enough for your experiments); among the 5 different ways # # there are various combinations of spawining subprocesses, using pipes or # # temporary files, and different methods to set a timeout after which the # # job you are running is killed # # - you can choose how many times a job should be run before giving up (if # # you have some heisenbugs, irace will not crash! :) but it will give the # # configuration further chances) # # - if the job fails for several times in a row, it writes in your execution # # directory a c${configuration}.stdout and stderr as the normal # # target-runner # # - set a debug level, and have a detailed output to track what happens (one # # file per target-runner, use sparely) # # # ############################################################################## import os import sys import time import socket import logging import tempfile import subprocess import threading import re # ---------------------------- DO NOT CHANGE HERE ---------------------------- # (unless you know what you are doing) # ---------------------- Search for CHANGE BELOW! ---------------------------- class Runner(object): def __init__(self, executable, candidate, instanceid, seed, parameters, parse_output, max_tests, maximize = False, log_level = logging.ERROR): self.executable = os.path.expanduser(executable) self.instanceid = instanceid self.seed = seed self.parse_output = parse_output self.candidate = candidate self.parameters = parameters self.max_tests = max_tests self.filename_prefix = 'c' + str(candidate) + '-' + str(instanceid) + '-' + str(seed) # default exec function self.execute = self.execute1 self.maximize = maximize # logging (by default only errors are logged) filename = self.filename_prefix + '.' + socket.gethostname() + '_' + str(os.getpid()) self.logger = logging.getLogger('target-runner') try : hdlr = logging.FileHandler(filename, delay=True) except OSError as e: print("Current working dir : %s" % os.getenv('PWD')) raise formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s') hdlr.setFormatter(formatter) self.logger.addHandler(hdlr) self.logger.setLevel(log_level) # changes the way the child process is executed def exec_mode(self, mode, max_time = 3600): self.execute = mode # self.max_time is used only in execute_timeout functions self.max_time = max_time # executes a process, waits until it finishes # returns the exit code, as well as the stdout and stderr # this version uses no intermediate file, it reads directly from pipes, it # is less robust though, if the child gets oddly killed it seems to hang # even if it should not def execute1(self, command): self.logger.debug('job started') self.logger.debug('PATH=' + os.environ['PATH']) self.logger.debug(command) process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=os.environ) (out, err) = process.communicate() status = process.wait() self.logger.debug(out) self.logger.debug(err) self.logger.debug('job finished') return (status, out, err) # executes a process, waits until it finishes # returns the exit code, as well as the stdout and stderr def execute2(self, command): fout = tempfile.NamedTemporaryFile(delete=False) fout.close() ferr = tempfile.NamedTemporaryFile(delete=False) ferr.close() str_cmd = ' '.join(command) + ' >' + fout.name + ' 2>' + ferr.name try: self.logger.debug('job started') self.logger.debug('PATH=' + os.environ['PATH']) self.logger.debug(str_cmd) process = subprocess.check_call(str_cmd, shell=True) status = 0 except subprocess.CalledProcessError as e: status = e.returncode self.logger.warning('Exit status: ' + str(status)) out = self.read(fout.name) err = self.read(ferr.name) os.unlink(fout.name) os.unlink(ferr.name) self.logger.debug(out) self.logger.debug(err) self.logger.debug('job finished') return (status, out, err) # executes a process, waits until it finishes # returns the exit code, as well as the stdout and stderr # if the process does not finish before the timeout it gets killed def execute_timeout1(self, command): fout = tempfile.SpooledTemporaryFile() ferr = tempfile.SpooledTemporaryFile() start_time = time.time() self.logger.debug('job started') self.logger.debug('PATH=' + os.environ['PATH']) self.logger.debug(command) process = subprocess.Popen(command, stdout=fout, stderr=ferr, env=os.environ) elapsed_time = time.time() - start_time while process.poll() is None and elapsed_time <= self.max_time: time.sleep(1) elapsed_time = time.time() - start_time self.logger.debug('elapsed time: ' + str(elapsed_time)) if elapsed_time > self.max_time: process.kill() self.logger.warning('job killed, ' + str(self.max_time) + \ ' seconds of wall-clock time elapsed') status = 1 status = process.poll() fout.seek(0) ferr.seek(0) out = str(fout.read()) err = str(ferr.read()) fout.close() ferr.close() self.logger.debug(out) self.logger.debug(err) self.logger.debug('job finished') return (status, out, err) # executes a process, waits until it finishes # returns the exit code, as well as the stdout and stderr # this version is like execute2 but requires python3.3 to set the timeouts # for the Popen.communicate() and Popen.wait() # if the process does not finish before the timeout it gets killed def execute_timeout2(self, command): self.logger.debug('job started') self.logger.debug('PATH=' + os.environ['PATH']) self.logger.debug(command) process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=os.environ) try: (out, err) = process.communicate(timeout=self.max_time) status = process.wait(timeout=self.max_time) except subprocess.TimeoutExpired: process.kill() (out, err) = process.communicate() status = 1 self.logger.warning('ob killed, ' + str(self.max_time) + \ ' seconds of CPU time elapsed') self.logger.debug(out) self.logger.debug(err) self.logger.debug('job finished') return (status, out, err) # executes a process, waits until it finishes (most robust one) # returns the exit code, as well as the stdout and stderr def execute_threaded_timeout(self, command): class ThreadedJob(threading.Thread): def __init__(self, command, logger): threading.Thread.__init__(self) self.shell_command = ' '.join(command) self.process = None self.out = None self.err = None self.logger = logger def run(self): self.logger.debug('job started') self.logger.debug('PATH=' + os.environ['PATH']) self.logger.debug(self.shell_command) self.process = subprocess.Popen(self.shell_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, env=os.environ) self.out, self.err = self.process.communicate() self.logger.debug(self.out) self.logger.debug(self.err) self.logger.debug('job finished') self.logger.debug('creating thread') thread = ThreadedJob(command, self.logger) # make daemon so that when the main program exits all (eventually # hanging) threads are killed thread.setDaemon(True) thread.start() thread.join(timeout=self.max_time) self.logger.debug('joined (timeout ' + str(self.max_time) + ')') if thread.is_alive(): # this part here is delicate we keep it in a try catch block try: self.logger.debug('thread is still alive, we timed out') # send a SIGTERM self.logger.debug('sending SIGTERM signal') thread.process.terminate() thread.join(timeout=5.0) self.logger.debug('done.') # send a SIGKILL to be very sure self.logger.debug('sending SIGKILL signal') thread.process.kill() thread.join(timeout=5.0) self.logger.debug('done') except Exception as e: self.logger.warning('exception when killing job: \n' + \ str(e)) status = thread.process.returncode out = str(thread.out) err = str(thread.err) # to be very sure del thread self.logger.debug('thread deleted, returning results') return (status, out, err) # reads and exports the environment variables def source_env(self, filename): self.logger.debug('setting environment variables') command = '"source ' + filename + ' >/dev/null 2>&1 && env"' (_, out, _) = self.execute(['bash', '-c', command]) lines = out.split('\n') for line in lines: (key, _, value) = line.partition('=') os.environ[key] = value self.logger.debug('setting ' + key + '=' + value) self.logger.debug('now PATH is: ' + os.environ['PATH']) # reads data into a string def read(self, filename): f = open(filename) content = str(f.read()) f.close() return content # writes a string to file def save(self, filename, content): f = open(filename, 'w') f.write(str(content)) f.close() # executing the program def run(self): test = 0 cost = "" while test < self.max_tests: command_list = [self.executable] + self.parameters (status, out, err) = self.execute(command_list) if status != 0: test += 1 self.logger.warning('non-zero exit status *RETRYING* ' + \ str(test) + ' of ' + str(self.max_tests)) continue # parsing the output try: cost = self.parse_output(out) except: test += 1 self.logger.warning('something failed in parse_output: *RETRYING* ' + \ str(test) + ' of ' + str(self.max_tests)) continue if cost is None: cost = "" try: check = float(cost) except: test += 1 self.logger.warning('return value of parse_output was not a number: *RETRYING* ' + \ str(test) + ' of ' + str(self.max_tests)) continue # If maximizing, simulate multiply by -1 if maximizing if self.maximize: if cost[0] == '-': cost = cost[1:] else: cost = '-' + cost break # printing the result if test < self.max_tests: self.logger.debug('returning cost: ' + cost) sys.stdout.write(cost + '\n') # force to exit all possible threads except the main one (those # launched with execute_threaded_timeout) are run as daemons so # they should be terminated automatically when exiting, but just # to be extra sure sys.exit(0) else: # in case somehting goes wrong we write stdout and stderr files self.logger.error('something went wrong after ' + \ str(self.max_tests) + ' runs') self.logger.error('saving candidate stdout to ' + \ self.filename_prefix + '.stdout') self.save(self.filename_prefix + '.stdout', out) self.save(self.filename_prefix + '.stderr', err) self.logger.error('return value of parse_output: ' + cost) self.logger.error('exit status is ' + str(status)) sys.stdout.write('something went wrong for candidate ' + \ self.candidate + '\n') if status != 0: sys.stdout.write('exit status: ' + str(status) + '\n') sys.exit(status) else: sys.stdout.write('could not cast to float the return value of parse_output: "' + \ cost + '"\n') sys.exit(1) def is_exe(fpath): fpath = os.path.expanduser(fpath) return os.path.isfile(fpath) and os.access(fpath, os.X_OK) \ and os.path.getsize(fpath) > 0 def get_execdir(): return os.path.dirname(os.path.realpath(__file__)) # ------------------------------- CHANGE BELOW! ------------------------------ # ## This example is for the ACOTSP software. Compare it with ## examples/acotsp/target-runner # Parse here directly the stdout of your job (the 'out' parameter). Or you can # ignore the out parameter and read other files produced by your job. This # function must return a string containing a floating-point number. Everything # else will generate an error. def parse_output(out): match = re.search(r'Best ([-+0-9.eE]+)', out.strip()) if match: return match.group(1); else: return "No match" if __name__=='__main__': if len(sys.argv) < 5: print("\nUsage: " + __file__ + " \n") sys.exit(1) bindir = get_execdir() # Path to the target-algorithm executable executable = '~/bin/acotsp' fixed_params = ' --tries 1 --time 1 --quiet ' # reading parameters and setting problem specific stuff ## FIXME: Convert this to a class that takes sys.argv and sets the correct ## variables. candidate_id = sys.argv[1] instance_id = sys.argv[2] seed = sys.argv[3] instance = sys.argv[4] parameters = sys.argv[5:] # maximum timeout in case the target algorithm does not terminate on its own. timeout = 180 # maximum number of trials before giving up with the configuration max_tests = 5 # Extra whitespace around options is important! parameters = [' -i ' + instance + ' --seed ' + seed + fixed_params ] + parameters runner = Runner(executable, candidate_id, instance_id, seed, parameters, parse_output, max_tests, #log_level = logging.DEBUG, maximize = False) ## FIXME: Convert this to flags with meaningful names like log_level # execute through pipes (this is the default) # runner.exec_mode(runner.execute1) ## FIXME: Convert this to flags with meaningful names # execute through temporary files (slightly more robust) # runner.exec_mode(runner.execute2) ## FIXME: Convert this to flags with meaningful names # execute through temporary files with timeout # after 5 minutes of *wallclock time* if we do not get the results we kill # the subprocess and try another time... # runner.exec_mode(runner.execute_timeout1, 300) ## FIXME: Convert this to flags with meaningful names # python3 execute through pipes with timeout (slightly less robust) # runner.exec_mode(runner.execute_timeout2, 300) ## FIXME: Convert this to flags with meaningful names # execute through temporary files with timeout # after 2 minutes of *CPU time* if we do not get the results we kill # the subprocess and try another time... runner.exec_mode(runner.execute_threaded_timeout, timeout) # Convert this to a parameter of the constructor. # environment variables that should be set for testing each configuration # runner.source_env(bindir + 'configuration') # run the target-runner runner.run() irace/inst/examples/target-runner-python/target-runner-acotsp.py0000755000176200001440000000576114157104332024716 0ustar liggesusers#!/usr/bin/python ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # This script is run in the execution directory (execDir, --exec-dir). # # PARAMETERS: # argv[1] is the candidate configuration number # argv[2] is the instance ID # argv[3] is the seed # argv[4] is the instance name # The rest (argv[5:]) are parameters to the run # # RETURN VALUE: # This script should print one numerical value: the cost that must be minimized. # Exit with 0 if no error, with 1 in case of error ############################################################################### import datetime import os.path import re import subprocess import sys # FIXME: group together everything that needs to be edited by the user and put # in functions everything that does NOT to be edited. ## This example is for the ACOTSP software. Compare it with ## examples/acotsp/target-runner # exe = "~/bin/executable" exe = "~/bin/acotsp" fixed_params = ' --tries 1 --time 10 --quiet ' # This is an example of reading a number from the output. def parse_output(out): match = re.search(r'Best ([-+0-9.eE]+)', out.strip()) if match: return match.group(1); else: return "No match" if len(sys.argv) < 5: print("\nUsage: ./target-runner.py \n") sys.exit(1) # Get the parameters as command line arguments. configuration_id = sys.argv[1] instance_id = sys.argv[2] seed = sys.argv[3] instance = sys.argv[4] conf_params = sys.argv[5:] # Build the command, run it and save the output to a file, # to parse the result from it. # # Stdout and stderr files have to be opened before the call(). # # Exit with error if something went wrong in the execution. exe = os.path.expanduser(exe) command = [exe] + fixed_params.split() + ["-i"] + [instance] + ["--seed"] + [seed] + conf_params # Define the stdout and stderr files. out_file = "c" + str(configuration_id) + "-" + str(instance_id) + str(seed) + ".stdout" err_file = "c" + str(configuration_id) + "-" + str(instance_id) + str(seed) + ".stderr" def target_runner_error(msg): now = datetime.datetime.now() print(str(now) + " error: " + msg) sys.exit(1) def check_executable(fpath): fpath = os.path.expanduser(fpath) if not os.path.isfile(fpath): target_runner_error(str(fpath) + " not found") if not os.access(fpath, os.X_OK): target_runner_error(str(fpath) + " is not executable") check_executable (exe) outf = open(out_file, "w") errf = open(err_file, "w") return_code = subprocess.call(command, stdout = outf, stderr = errf) outf.close() errf.close() if return_code != 0: target_runner_error("command returned code " + str(return_code)) if not os.path.isfile(out_file): target_runner_error("output file " + out_file + " not found.") cost = parse_output (open(out_file).read()) print(cost) os.remove(out_file) os.remove(err_file) sys.exit(0) irace/inst/examples/target-runner-python/target-runner-python-win.bat0000644000176200001440000000237014157104332025645 0ustar liggesusers::############################################################################## :: In Windows, we cannot execute directly .py files so we use this script as :: the targetRunner that calls the actual .py file. :: Contributed by Levi Ribeiro :: :: This script is run in the execution directory (execDir, --exec-dir). :: :: PARAMETERS: :: %%1 is the candidate configuration number :: %%2 is the instance ID :: %%3 is the seed :: %%4 is the instance name :: The rest are parameters to the target-algorithm :: :: RETURN VALUE: :: This script should print one numerical value: the cost that must be minimized. :: Exit with 0 if no error, with 1 in case of error ::############################################################################## @echo off :: If python is in the PATH environment variable, you can call it directly. SET "pythonexe=python.exe" :: Otherwise, you need to provide its absolute location :: SET "pythonexe=C:\bin\python.exe" :: This script is run in the execution directory (execDir, --exec-dir), which often does not coincide with the directory containing target-runner.py. Either a relative or absolute path should be provided. SET "targetrunnerpy=..\target-runner.py" set parameters=%* %pythonexe% %targetrunnerpy% %parameters% irace/inst/examples/target-runner-python/trivial/0000755000176200001440000000000014157104332021716 5ustar liggesusersirace/inst/examples/target-runner-python/trivial/scenario.txt0000644000176200001440000000016314157104332024262 0ustar liggesusersmaxExperiments = 500 trainInstancesDir = "" trainInstancesFile = "instances.txt" targetRunner = "target-runner.py" irace/inst/examples/target-runner-python/trivial/instances.txt0000644000176200001440000000002414157104332024442 0ustar liggesusersinstance1 instance2 irace/inst/examples/target-runner-python/trivial/target-runner.py0000755000176200001440000000456714157104332025104 0ustar liggesusers#!/usr/bin/python ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # This script is run in the execution directory (execDir, --exec-dir). # # PARAMETERS: # argv[1] is the candidate configuration number # argv[2] is the instance ID # argv[3] is the seed # argv[4] is the instance name # The rest (argv[5:]) are parameters to the run # # RETURN VALUE: # This script should print one numerical value: the cost that must be minimized. # Exit with 0 if no error, with 1 in case of error ############################################################################### import datetime import os.path import re import subprocess import sys ## This a dummy example that shows how to parse the parameters defined in ## parameters.txt and does not need to call any other software. if __name__=='__main__': if len(sys.argv) < 5: print("\nUsage: ./target-runner.py \n") sys.exit(1) # Get the parameters as command line arguments. configuration_id = sys.argv[1] instance_id = sys.argv[2] seed = sys.argv[3] instance = sys.argv[4] cand_params = sys.argv[5:] # Default values (if any) a = None b = None # Parse parameters while cand_params: # Get and remove first and second elements. param = cand_params.pop(0) value = cand_params.pop(0) if param == "--param1": a = float(value) elif param == "--param2": b = int(value) elif param == "--ptype": ptype = value else: target_runner_error("unknown parameter %s" % (param)) # Sanity checks if a == None and b == None: target_runner_error("either a or b must be set, something is wrong!") if ptype == "a" and a == None: target_runner_error("ptype is 'a' but no value assigned to a, something is wrong!") if ptype == "b" and b == None: target_runner_error("ptype is 'b' but no value assigned to b, something is wrong!") a = a or 1 b = b or 1 print(str(a * b) + '\n') sys.exit(0) # Useful function to print errors. def target_runner_error(msg): now = datetime.datetime.now() print(str(now) + " error: " + msg) sys.exit(1) irace/inst/examples/target-runner-python/trivial/parameters.txt0000644000176200001440000000017314157104332024623 0ustar liggesusersparam1 "--param1 " r (1,100) | ptype == "a" param2 "--param2 " i (200,300) | ptype == "b" ptype "--ptype " c ("a","b") irace/inst/examples/batchmode-cluster/0000755000176200001440000000000014325242462017541 5ustar liggesusersirace/inst/examples/batchmode-cluster/README0000644000176200001440000000020314157104332020410 0ustar liggesusersThese scripts are meant to be used with the option --batchmode. See the documentation of the option --batchmode in the user guide. irace/inst/examples/batchmode-cluster/target-runner-pbs0000644000176200001440000000267714157104332023053 0ustar liggesusers#!/bin/bash ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # This script is run in the execution directory (execDir, --exec-dir), # the same directory where target-evaluator is executed. Hence, you may # need to copy extra files needed by the executable to this directory. # # # PARAMETERS: # $1 is the candidate number # $2 is the instance id # $3 is the seed # $4 is the instance name # The rest ($* after `shift 4') are parameters to the run # # RETURN VALUE: # This script should print nothing. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" exit 1 } EXE=~/bin/program FIXED_PARAMS="" CANDIDATE=$1 INSTANCEID=$2 SEED=$3 INSTANCE=$4 shift 4 || error "Not enough parameters" CAND_PARAMS=$* STDOUT=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdout STDERR=c${CANDIDATE}-${INSTANCEID}-${SEED}.stderr # PBS just prints the jobID, so it should be good to go. exec qsub < \$TMP/$STDERR $EXE ${FIXED_PARAMS} --input $INSTANCE ${CAND_PARAMS} 1> \$TMP/$STDOUT RET=\$? echo "OK" >& 2 mv \$TMP/* ./ rmdir -p \$TMP &> /dev/null exit \$RET EOF irace/inst/examples/batchmode-cluster/target-runner-slurm0000755000176200001440000000400114157104332023413 0ustar liggesusers#!/bin/bash ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # This script is run in the execution directory (execDir, --exec-dir), # the same directory where target-evaluator is executed. Hence, you may # need to copy extra files needed by the executable to this directory. # # # PARAMETERS: # $1 is the candidate number # $2 is the instance id # $3 is the seed # $4 is the instance name # The rest ($* after `shift 4') are parameters to the run # # RETURN VALUE: # This script should print nothing. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" cat $tmpfile rm -f $tmpfile exit 1 } tmpfile=$(mktemp) EXE=~/bin/program FIXED_PARAMS="" CANDIDATE=$1 INSTANCEID=$2 SEED=$3 INSTANCE=$4 shift 4 || error "Not enough parameters" CAND_PARAMS=$* STDOUT=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdout STDERR=c${CANDIDATE}-${INSTANCEID}-${SEED}.stderr # modify with the SLURM submission template sbatch 1> $tmpfile < \$TMP/$STDOUT 2> \$TMP/$STDERR RET=\$? echo "OK" >& 2 mv \$TMP/* ./ rmdir -p \$TMP &> /dev/null exit \$RET EOF rc=$? if [[ $rc == 0 ]]; then JOBID=$(grep -o -e "Submitted batch job [^ ]\+" $tmpfile | cut -f4 -d ' ') if ! [[ "$JOBID" =~ ^[_-.@0-9A-Za-z]+$ ]] ; then error "$0: cannot parse jobID from the output of sbatch!" fi echo "$JOBID" rm -f $tmpfile exit $rc else error "$0: sbatch failed!" fi irace/inst/examples/batchmode-cluster/target-evaluator0000755000176200001440000000377014321266642022766 0ustar liggesusers#!/bin/bash ############################################################################### # This script is run for each candidate to evaluate it after all # candidate configurations have been run on a single instance. # # Check the examples in examples/ # # PARAMETERS: # $1 is the candidate number # $2 is the instance id # $3 is the seed # $4 is the instance name # $5 is the number of candidates alive in this iteration # # ONLY FOR ELITIST RACE: The rest ($* after `shift 5') are the ids of the # candidates alive in this iteration. This list can be used to calculate the # hypervolume using only active candidates. # # RETURN VALUE: # This script should print one numerical value: the cost that must be minimized. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" exit 1 } CANDIDATE="$1" INSTANCEID="$2" SEED="$3" INSTANCE="$4" TOTALCANDIDATES="$5" shift 5 || error "Not enough parameters" ALLIDS=$* STDOUT=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdout STDERR=c${CANDIDATE}-${INSTANCEID}-${SEED}.stderr ALLFILES= for FILENAME in $ALLIDS; do ALLFILES="$ALLFILES c${FILENAME}-${INSTANCEID}-${SEED}.stdout" done # # This may be used to introduce a delay if there are filesystem # # issues. # SLEEPTIME=1 # while [ ! -s "${STDOUT}" ]; do # sleep $SLEEPTIME # let "SLEEPTIME += 1" # done # This is an example of reading a number from the output of # target-runner. It assumes that the objective value is the first number in # the first column of the only line starting with a digit. if [ ! -s "${STDOUT}" ]; then error "${STDOUT}: No such file or directory" fi # You may need to update this to parse the output of your algorithm. COST=$(cat ${STDOUT} | grep -e '^[[:space:]]*[+-]\?[0-9]' | cut -f1) echo "$COST" ## Comment out if you wish to keep all output files around # TODELETE=$(comm -23 <(ls -1 c*-${INSTANCEID}-${SEED}.* | sort) <(ls -1 $ALLFILES)) # rm -f $TODELETE exit 0 irace/inst/examples/batchmode-cluster/irace-sge-cluster0000755000176200001440000000101514157104332022776 0ustar liggesusers#!/bin/bash # To use --cluster=[sge|pbs|torque], irace must be invoked in the submission # node and target-runner will launch jobs using qsub and return the jobID. CLUSTER="sge" BINDIR=$1 EXECDIR=$2 DESTMAIL=${USER}@localhost shift 2 $BINDIR/irace --exec-dir=$EXECDIR --cluster=$CLUSTER --target-runner=target-runner-${CLUSTER} $* \ 1> $EXECDIR/irace-$$.stdout \ 2> $EXECDIR/irace-$$.stderr if [ ${DESTMAIL} ]; then cat "$EXECDIR/irace-$$.stderr" \ | mail -s "[irace] $EXECDIR/irace-$$ $*" $DESTMAIL fi irace/inst/examples/batchmode-cluster/target-runner-htcondor0000755000176200001440000000336014157104332024100 0ustar liggesusers#!/bin/bash ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # This script is run in the execution directory (execDir, --exec-dir), # the same directory where target-evaluator is executed. Hence, you may # need to copy extra files needed by the executable to this directory. # # # PARAMETERS: # $1 is the candidate number # $2 is the instance id # $3 is the seed # $4 is the instance name # The rest ($* after `shift 4') are parameters to the run # # RETURN VALUE: # This script should print nothing. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" cat $tmpfile rm -f $tmpfile exit 1 } tmpfile=$(mktemp) EXE=~/bin/program FIXED_PARAMS="" CANDIDATE=$1 INSTANCEID=$2 SEED=$3 INSTANCE=$4 shift 4 || error "Not enough parameters" CAND_PARAMS=$* STDOUT=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdout STDERR=c${CANDIDATE}-${INSTANCEID}-${SEED}.stderr STDLOG=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdlog # modify with the HTCONDOR submission template condor_submit 1> $tmpfile < $tmpfile < \$TMP/$STDERR $EXE ${FIXED_PARAMS} --input $INSTANCE ${CAND_PARAMS} 1> \$TMP/$STDOUT RET=\$? echo "OK" >& 2 mv \$TMP/* ./ rmdir -p \$TMP &> /dev/null exit \$RET EOF rc=$? if [[ $rc == 0 ]]; then JOBID=$(grep -o -e "Your job [^ ]\+" $tmpfile | cut -f3 -d ' ') if ! [[ "$JOBID" =~ ^[_-.@0-9A-Za-z]+$ ]] ; then error "$0: cannot parse jobID from the output of qsub!" fi echo "$JOBID" rm -f $tmpfile exit 0 else error "$0: qsub failed!" fi irace/inst/examples/Spear/0000755000176200001440000000000014157104332015202 5ustar liggesusersirace/inst/examples/Spear/README0000644000176200001440000000202614157104332016062 0ustar liggesusersExample configuration scenario: Spear ===================================== This example scenario was inspired by ParamILS. The Spear program can be downloaded from: http://www.cs.ubc.ca/labs/beta/Projects/ParamILS/algorithms.html WARNING: Please note that no source code is available and we do not know what this executable is actually doing. We do NOT recommend running this executable in your computer. If Spear doesn't work in your computer, there is nothing we can do to fix it. Unless you absolutely must run Spear, we suggest to use a different scenario for which source code is available. * Tuning instances: http://www.cs.ubc.ca/labs/beta/Projects/ParamILS/benchmark_instances/SpearSWV/SWV-scrambled-first302.tar.gz * Testing instances: http://www.cs.ubc.ca/labs/beta/Projects/ParamILS/benchmark_instances/SpearSWV/SWV-scrambled-last302.tar.gz Two parameter files are provided: * parameters-cat.txt only contains categorical parameters. * parameters-mixed.txt contains a mix of numerical and categorical parameters. irace/inst/examples/Spear/scenario.txt0000644000176200001440000000317614157104332017555 0ustar liggesusers###################################################### -*- mode: r -*- ##### ## Scenario setup for Iterated Race (irace). ############################################################################ ## To use the default value of a parameter of iRace, simply do not set ## the parameter (comment it out in this file, and do not give any ## value on the command line). ## File that contains the description of the parameters of the target ## algorithm. parameterFile = "./parameters-cat.txt" ## Directory where the programs will be run. # execDir = "./" ## Directory where training instances are located; either absolute path or ## relative to current directory. If no trainInstancesFiles is provided, ## all the files in trainInstancesDir will be listed as instances. trainInstancesDir = "./SWV-scrambled-first302" ## Directory where testing instances are located, either absolute or ## relative to current directory. testInstancesDir = "./SWV-scrambled-last302" ## Maximum number of runs (invocations of targetRunner) that will be ## performed. It determines the maximum budget of experiments for the ## tuning. maxExperiments = 10000 ## Statistical test used for elimination. Default test is always F-test ## unless capping is enabled, in which case the default test is t-test. ## Valid values are: F-test (Friedman test), t-test (pairwise t-tests with ## no correction), t-test-bonferroni (t-test with Bonferroni's correction ## for multiple comparisons), t-test-holm (t-test with Holm's correction ## for multiple comparisons). testType = "t-test" ## END of scenario file ############################################################################ irace/inst/examples/Spear/target-runner0000755000176200001440000000405614157104332017732 0ustar liggesusers#!/bin/bash ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # This script is run in the execution directory (execDir, --exec-dir). # # PARAMETERS: # $1 is the candidate configuration number # $2 is the instance ID # $3 is the seed # $4 is the instance name # The rest ($* after `shift 4') are parameters to the run # # RETURN VALUE: # This script should print one numerical value: the cost that must be minimized. # Exit with 0 if no error, with 1 in case of error ############################################################################### EXE=~/bin/Spear-32_1.2.1 TIMEOUT=30 FIXED_PARAMS="--nosplash --time --tmout $TIMEOUT" error() { echo "`TZ=UTC date`: $0: error: $@" exit 1 } CONFIG_ID=$1 INSTANCE_ID=$2 SEED=$3 INSTANCE=$4 shift 4 || error "Not enough parameters" CONFIG_PARAMS=$* STDOUT=c${CONFIG_ID}-${INSTANCE_ID}-${SEED}.stdout STDERR=c${CONFIG_ID}-${INSTANCE_ID}-${SEED}.stderr if [ ! -x "${EXE}" ]; then error "${EXE}: not found or not executable (pwd: $(pwd))" fi if [ ! -r "$INSTANCE" ]; then error "${INSTANCE}: cannot find instance (pwd: $(pwd))" fi #echo "$EXE ${FIXED_PARAMS} --dimacs $INSTANCE ${CONFIG_PARAMS}" $EXE ${FIXED_PARAMS} --dimacs $INSTANCE ${CONFIG_PARAMS} 1> ${STDOUT} 2> ${STDERR} if [ $? -ne 0 ]; then (cat < ${STDOUT} fi ##################################################################### # Evaluate a floating point number subordinate expression. function float_cond() { local cond=0 if [[ $# -gt 0 ]]; then cond=$(echo "$*" | bc -q 2>/dev/null) if [[ -z "$cond" ]]; then cond=0; fi if [[ "$cond" != 0 && "$cond" != 1 ]]; then cond=0; fi fi local stat=$((cond == 0)) return $stat } if COST=$(cat ${STDOUT} | grep -e '^c runtime [0-9]' | cut -f3 -d ' '); then if float_cond "$COST > $TIMEOUT"; then echo "$TIMEOUT" else echo "$COST" fi rm -f ${STDOUT} ${STDERR} exit 0 fi exit 1 irace/inst/examples/Spear/parameters-mixed.txt0000644000176200001440000001022214157104332021207 0ustar liggesuserssp.var.dec.heur "--sp-var-dec-heur " c (0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19) # Originally, 3,4,9,10 not used following Domgoj's advice. 20 requires modular arithmetic input format sp.learned.clause.sort.heur "--sp-learned-clause-sort-heur " c (0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19) # All values make sense here. 20 requires modular arithmetic input format sp.orig.clause.sort.heur "--sp-orig-clause-sort-heur " c (0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19) # All values make sense here. 20 requires modular arithmetic input format sp.res.order.heur "--sp-res-order-heur " c (0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19) | sp.resolution != 0 # All values make sense here. 20 requires modular arithmetic input format sp.clause.del.heur "--sp-clause-del-heur " c (0,1,2) # All values make sense here. sp.phase.dec.heur "--sp-phase-dec-heur " c (0,1,2,3,4,5,6) # All values make sense here. sp.resolution "--sp-resolution " c (0,1,2) # 0 renders a whole bunch of subordinates irrelevant. sp.variable.decay "--sp-variable-decay " r (1.0,2.0) # Should be bigger than 1 (o/w increase not decay). sp.clause.decay "--sp-clause-decay " r (1.0,2.0) # Same thing. sp.restart.inc "--sp-restart-inc " r (1.0,2.0) # 1.3 and 1.7 were introduced later. Uniform because multiplicative. sp.learned.size.factor "--sp-learned-size-factor " r (0.1,1.6) # 0.2 and 0.8 were introduced later. Uniform on logarithmic scale (starting value). sp.learned.clauses.inc "--sp-learned-clauses-inc " r (1.0,1.5) # 1.2 and 1.4 were introduced later. Uniform because multiplicative sp.clause.activity.inc "--sp-clause-activity-inc " r (0.5,1.5) # Domagoj says these make sense. sp.var.activity.inc "--sp-var-activity-inc " r (0.5,1.5) # Same thing. sp.rand.phase.dec.freq "--sp-rand-phase-dec-freq " o (0, 0.0001, 0.001, 0.005, 0.01, 0.05) | sp.phase.dec.heur %in% c(0,1,3,4,5,6) # when heuristic is random, then additional random steps don't change anything # Never picked 0.05 in previous experiments, always zero. sp.rand.var.dec.freq "--sp-rand-var-dec-freq " o (0, 0.0001, 0.001, 0.005, 0.01, 0.05) # Never picked 0.05 in previous experiments, always zero. sp.rand.var.dec.scaling "--sp-rand-var-dec-scaling " r (0.3,1.1) | sp.rand.var.dec.freq %in% c(0.0001, 0.001, 0.005, 0.01, 0.05) # not 0 # 0.5 and 2 were introduced later. Domagoj said those are too coarse, so new values. sp.rand.phase.scaling "--sp-rand-phase-scaling " r (0.3,1.1) | sp.rand.phase.dec.freq %in% c(0.0001, 0.001, 0.005, 0.01, 0.05) # not 0 # Same thing. sp.max.res.lit.inc "--sp-max-res-lit-inc " r (0.25,4) | sp.resolution != 0 # 0.5 and 2 were introduced later. sp.first.restart "--sp-first-restart " o (25,50,100,200,400,800,1600,3200) # Uniform on logarithmic scale (starting value). sp.res.cutoff.cls "--sp-res-cutoff-cls " o (2,4,8,16,20) | sp.resolution != 0 # 4 and 20 were introduced later. Only 20 allowed, would've used 32. sp.res.cutoff.lits "--sp-res-cutoff-lits " o (100,200,400,800,1600) | sp.resolution != 0 # 200 and 800 were introduced later. sp.max.res.runs "--sp-max-res-runs " o (1,2,4,8,16,32) | sp.resolution != 0 # 2, 8, and 32 were introduced later. sp.update.dec.queue "--sp-update-dec-queue " c (0,1) # Enable by default. sp.use.pure.literal.rule "--sp-use-pure-literal-rule " c (0,1) # Enable by default. sp.clause.inversion "--sp-clause-inversion " c (0,1) | sp.learned.clause.sort.heur %in% c(19) # Enable by default. Enable reversion of learned clauses if fixed order (sp-learned-clause-sort-heur=19) irace/inst/examples/Spear/parameters-cat.txt0000644000176200001440000001033614157104332020656 0ustar liggesuserssp.var.dec.heur "--sp-var-dec-heur " c (0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19) # Originally, 3,4,9,10 not used following Domgoj's advice. 20 requires modular arithmetic input format sp.learned.clause.sort.heur "--sp-learned-clause-sort-heur " c (0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19) # All values make sense here. 20 requires modular arithmetic input format sp.orig.clause.sort.heur "--sp-orig-clause-sort-heur " c (0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19) # All values make sense here. 20 requires modular arithmetic input format sp.res.order.heur "--sp-res-order-heur " c (0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19) | sp.resolution != 0 # All values make sense here. 20 requires modular arithmetic input format sp.clause.del.heur "--sp-clause-del-heur " c (0,1,2) # All values make sense here. sp.phase.dec.heur "--sp-phase-dec-heur " c (0,1,2,3,4,5,6) # All values make sense here. sp.resolution "--sp-resolution " c (0,1,2) # 0 renders a whole bunch of subordinates irrelevant. sp.variable.decay "--sp-variable-decay " c (1.1,1.4,2.0) # Should be bigger than 1 (o/w increase not decay). sp.clause.decay "--sp-clause-decay " c (1.1,1.4,2.0) # Same thing. sp.restart.inc "--sp-restart-inc " c (1.1,1.3,1.5,1.7,1.9) # 1.3 and 1.7 were introduced later. Uniform because multiplicative. sp.learned.size.factor "--sp-learned-size-factor " c (0.1,0.2,0.4,0.8,1.6) # 0.2 and 0.8 were introduced later. Uniform on logarithmic scale (starting value). sp.learned.clauses.inc "--sp-learned-clauses-inc " c (1.1,1.2,1.3,1.4,1.5) # 1.2 and 1.4 were introduced later. Uniform because multiplicative sp.clause.activity.inc "--sp-clause-activity-inc " c (0.5,1,1.5) # Domagoj says these make sense. sp.var.activity.inc "--sp-var-activity-inc " c (0.5,1,1.5) # Same thing. sp.rand.phase.dec.freq "--sp-rand-phase-dec-freq " c (0, 0.0001, 0.001, 0.005, 0.01, 0.05) | sp.phase.dec.heur %in% c(0,1,3,4,5,6) # when heuristic is random, then additional random steps don't change anything # Never picked 0.05 in previous experiments, always zero. sp.rand.var.dec.freq "--sp-rand-var-dec-freq " c (0, 0.0001, 0.001, 0.005, 0.01, 0.05) # Never picked 0.05 in previous experiments, always zero. sp.rand.var.dec.scaling "--sp-rand-var-dec-scaling " c (0.3,0.6,0.9,1,1.1) | sp.rand.var.dec.freq %in% c(0.0001, 0.001, 0.005, 0.01, 0.05) # not 0 # 0.5 and 2 were introduced later. Domagoj said those are too coarse, so new values. sp.rand.phase.scaling "--sp-rand-phase-scaling " c (0.3,0.6,0.9,1,1.1) | sp.rand.phase.dec.freq %in% c(0.0001, 0.001, 0.005, 0.01, 0.05) # not 0 # Same thing. sp.max.res.lit.inc "--sp-max-res-lit-inc " c (0.25,0.5,1,2,4) | sp.resolution != 0 # 0.5 and 2 were introduced later. sp.first.restart "--sp-first-restart " c (25,50,100,200,400,800,1600,3200) # Uniform on logarithmic scale (starting value). sp.res.cutoff.cls "--sp-res-cutoff-cls " c (2,4,8,16,20) | sp.resolution != 0 # 4 and 20 were introduced later. Only 20 allowed, would've used 32. sp.res.cutoff.lits "--sp-res-cutoff-lits " c (100,200,400,800,1600) | sp.resolution != 0 # 200 and 800 were introduced later. sp.max.res.runs "--sp-max-res-runs " c (1,2,4,8,16,32) | sp.resolution != 0 # 2, 8, and 32 were introduced later. sp.update.dec.queue "--sp-update-dec-queue " c (0,1) # Enable by default. sp.use.pure.literal.rule "--sp-use-pure-literal-rule " c (0,1) # Enable by default. sp.clause.inversion "--sp-clause-inversion " c (0,1) | sp.learned.clause.sort.heur %in% c(19) # Enable by default. Enable reversion of learned clauses if fixed order (sp-learned-clause-sort-heur=19) irace/inst/examples/README.md0000644000176200001440000000223014157104332015404 0ustar liggesusersAdditional examples of scenarios can be found in: Capping Methods for the Automatic Configuration of Optimization Algorithms Marcelo de Souza, Marcus Ritt and Manuel López-Ibáñez https://github.com/souzamarcelo/supp-cor-capopt * ACOTSP: ant colony optimization algorithms for the symmetric traveling salesperson problem. * HEACOL: hybrid evolutionary algorithm for graph coloring. * TSBPP: tabu search for the bin packing problem. * HHBQP: hybrid heuristic for unconstrained binary quadratic programming. * LKH: a heuristic algorithm for solving the symmetric traveling salesperson problem. * SCIP: an exact solver for mixed integer programs for solving the combinatorial auction winner determination problem. http://aclib.net/ Frank Hutter, Manuel López-Ibáñez, Chris Fawcett, Marius Thomas Lindauer, Holger H. Hoos, Kevin Leyton-Brown, and Thomas Stützle. AClib: a Benchmark Library for Algorithm Configuration. In P. M. Pardalos, M. G. C. Resende, C. Vogiatzis, and J. L. Walteros, editors, Learning and Intelligent Optimization, 8th International Conference, LION 8, volume 8426 of Lecture Notes in Computer Science, pages 36–40. Springer, 2014. irace/inst/examples/hypervolume/0000755000176200001440000000000014157104332016507 5ustar liggesusersirace/inst/examples/hypervolume/README0000644000176200001440000000121614157104332017367 0ustar liggesusersThese are example hooks to tune a multi-objective algorithm (using the hypervolume indicator). You need to set targetRunner and targetEvaluator, either in the scenario file of irace (usually scenario.txt), or in the command-line (see irace --help), so irace is able to find these scripts. Update target-runner with the path, name and fixed parameters of your program. Update target-evaluator with the path and parameters for computing the hypervolume. The example uses the hypervolume code from < http://lopez-ibanez.eu/hypervolume >. Additional transformations of the input data may be applied before computing the hypervolume, such as normalization. irace/inst/examples/hypervolume/target-runner0000755000176200001440000000276614157104332021245 0ustar liggesusers#!/bin/bash ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # PARAMETERS: # $1 is the configuration number # $2 is the instance id # $3 is the seed # $4 is the instance name # The rest are parameters for running the program # # RETURN VALUE: # This script should print nothing. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: oerror: $@" exit 1 } # NOTE: You need to customize EXE and FIXED_PARAMS below. If you change the # rest, you need to also update target-evaluator ## If you want to find where target-runner is. #BINDIR=$(dirname "$(readlink -f "$(type -P $0 || echo $0)")") EXE=~/bin/program FIXED_PARAMS="--trials 1 --input $INSTANCE" CANDIDATE="$1" INSTANCEID="$2" SEED="$3" INSTANCE="$4" # All other parameters are the candidate parameters to be passed to program shift 4 || error "Not enough parameters" CAND_PARAMS=$* STDOUT=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdout STDERR=c${CANDIDATE}-${INSTANCEID}-${SEED}.stderr # TODO: Use /tmp for all temporary files, i.e., put target-data under /tmp mkdir -p target-data cd target-data || error "cannot change to directory \'target-data\'" $EXE ${FIXED_PARAMS} ${CAND_PARAMS} 1> $STDOUT 2> $STDERR # We do this to make sure this target-runner terminated correctly in target-evaluator echo "OK" >> $STDERR exit 0 irace/inst/examples/hypervolume/target-evaluator0000755000176200001440000000663214157104332021732 0ustar liggesusers#!/bin/bash ############################################################################### # This script is run for each configuration to evaluate it after all # candidate configurations have been run on a single instance. # # Check the examples in examples/ # # PARAMETERS: # $1 is the configuration number # $2 is the instance id # $3 is the seed # $4 is the instance name # $5 is the total number of configurations alive in this iteration # The rest are the IDs of all configurations run for this instance # # ONLY FOR ELITIST RACE: The rest ($* after `shift 5') are the ids of the # configurations alive in this iteration. This list can be used to calculate # the hypervolume using previous execution results. # # RETURN VALUE: # This script should print a single numerical value ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" if [ -s error.log ]; then cat error.log fi exit 1 } error_sleep() { echo "`TZ=UTC date`: $0: error: $@" >> error.log sleep 60 } NORMALIZATION_RANGE="'1 2'" REFERENCE_POINT="'2.1 2.1'" # A program that normalizes all .stdout files to normalization range # (optional, not included). NORMALIZE="~/bin/nondominated --verbose -n $NORMALIZATION_RANGE -s '_nor'" UPDATELIMITS="2>&1 | grep -E 'minimum|maximum' | sed 's/:/:\n/'" # A program for computing the hypervolume. Download hypervolume # code from http://lopez-ibanez.eu/hypervolume HV="~/bin/hv --quiet -r $REFERENCE_POINT" CONFIGURATION="$1" INSTANCEID="$2" SEED="$3" INSTANCE="$4" TOTALCONFIGURATIONS="$5" shift 5 || error "Not enough parameters to $0" ALLIDS=$* STDOUT=c${CONFIGURATION}-${INSTANCEID}-${SEED}.stdout STDERR=c${CONFIGURATION}-${INSTANCEID}-${SEED}.stderr calc_measure() { LIMITSFILE=limits-${INSTANCEID}.txt LIMITSTMP=limits-${INSTANCEID}.tmp if [ -s "$LIMITSFILE" ]; then $NORMALIZE $LIMITSFILE $ALLFILES $UPDATELIMITS > $LIMITSTMP mv $LIMITSTMP $LIMITSFILE else $NORMALIZE $ALLFILES $UPDATELIMITS > $LIMITSFILE fi } # TODO: Use /tmp for all temporary files, i.e., put target-data under /tmp cd target-data || error "cannot change to directory 'target-data'" ALLFILES= for FILENAME in ALLIDS; do ALLFILES="$ALLFILES c${FILENAME}-${INSTANCEID}-${SEED}.stdout" done # First normalize output files, unless they are ready. while ! ls ${ALLFILES}_nor &> /dev/null ; do # Check that all output files are there NUM=$(ls -1 ${ALLFILES} 2> /dev/null | wc --lines) if [ "$NUM" -ne "$TOTALCONFIGURATIONS" ]; then error_sleep "only found $NUM files out of $TOTALCONFIGURATIONS: ${ALLFILES}" continue fi # Check every STDERR for FILENAME in $ALLFILES; do FILESTDERR=${FILENAME/stdout/stderr} [ OK = "$(cat ${FILESTDERR})" ] || (error_sleep "${FILESTDERR} is not OK"; continue) [ -s "$FILENAME" ] || (error_sleep "$FILENAME is empty"; continue) done calc_measure rm -f error.log # Comment out if you wish to keep all output files around comm --output-delimiter=" " -23 <(ls -1 c*-*.* | sort) <(ls -1 ${ALLFILES//-${INSTANCEID}-${SEED}.stdout/-*.*} | sort) | xargs -r rm -f break done if [ ! -s "${STDOUT}_nor" ]; then error "${STDOUT}_nor: No such file or directory" fi COST=$(eval "$HV ${STDOUT}_nor" || error "calculating hypervolume failed") # Negative because hypervolume is maximised but irace minimises. echo "-$COST" exit 0 irace/inst/examples/matlab/0000755000176200001440000000000014325242462015374 5ustar liggesusersirace/inst/examples/matlab/scenario.txt0000644000176200001440000000614514203762252017745 0ustar liggesusers###################################################### -*- mode: r -*- ##### ## You need to install the matlabr package, for example, # install.packages("matlabr") require(matlabr, quietly=TRUE) # We assume that the matlab .m files are in the same folder as this file. # Otherwise, change the next line. (This variable has to start with '.' so that # irace does not give an error.) .matlab_script_path = getwd() # You may need to provide the path to matlab if have_matlab() cannot find it. # options(matlab.path="/opt/apps/apps/binapps/matlab/R2020a/bin/") if (have_matlab()) { cat("=== irace === Using matlab as: ", get_matlab(), "\n") } else { stop("=== irace === MATLAB not found! Maybe not installed or you need to provide the matlab.path in scenario.txt") } targetRunner <- function(experiment, scenario) { matlab_function_call <- "Main(INSTANCE,SEED,A,B)" debugLevel <- scenario$debugLevel configuration.id <- experiment$id.configuration instance.id <- experiment$id.instance seed <- experiment$seed configuration <- experiment$configuration instance <- experiment$instance args <- paste0(names(configuration), "=", configuration, collapse=";") exitcode <- matlabr::run_matlab_code( # This is the instance and seed c(paste0("INSTANCE='", instance, "';SEED=", seed), # The configuration parameters, args, matlab_function_call), paths_to_add=.matlab_script_path, verbose= (debugLevel >= 2)) ## Parse the output (just a single number!) # FIXME: Create a uniquely named temporary file instead of Result.txt to # allow parallel execution output <- scan(file="Result.txt", quiet=TRUE) cost <- as.numeric(output) if (exitcode != 0) { cat("=== irace === targetRunner: matlab returned exit code ", exitcode, "\n") } if (is.na(cost)) { cat("=== irace === targetRunner: error parsing numeric result from matlab output\n") if (!file.exists(file.path(.matlab_script_path, "Main.m"))) cat("=== irace === matlab script: '", file.path(.matlab_script_path, "Main.m"), "' not found!\n") } return(list(cost = cost, outputRaw = output)) } ## Maximum number of runs (invocations of targetRunner) that will be ## performed. It determines the maximum budget of experiments for the ## tuning. maxExperiments = 100 ## File that contains the description of the parameters of the target ## algorithm. # parameterFile = "./parameters.txt" ## Directory where the programs will be run. # execDir = "./" ## Directory where training instances are located; either absolute path or ## relative to current directory. If no trainInstancesFiles is provided, ## all the files in trainInstancesDir will be listed as instances. trainInstancesDir = "" ## File that contains a list of training instances and optionally ## additional parameters for them. If trainInstancesDir is provided, irace ## will search for the files in this folder. trainInstancesFile = "instances.txt" # Add other scenario options for irace. See the user guide. irace/inst/examples/matlab/instances.txt0000644000176200001440000000002514157104332020115 0ustar liggesusers1 2 3 4 5 6 7 8 9 10 irace/inst/examples/matlab/Main.m0000644000176200001440000000102114157104332016424 0ustar liggesusers% function Result = Main(INSTANCE,SEED,A, B) % Using INSTANCE as a seed, this function creates a pseudo-random number % drawn from a uniform distribution in the open interval (-1,1) and returns % the value of Result, where Result = A*B + r function Result = Main(INSTANCE,SEED,A,B) rng('default'); rng(SEED); minValue = -1; maxValue = 1; r = (maxValue - minValue) * rand(1) + minValue; Result = A * B + r; % fprintf('Result for irace=%g\n', Result); fid=fopen('Result.txt','wt'); fprintf(fid,'%g\n',Result); fclose(fid); end irace/inst/examples/matlab/parameters.txt0000644000176200001440000000003414157104332020271 0ustar liggesusersA "" i (1,50) B "" r (5,10) irace/inst/examples/moaco/0000755000176200001440000000000014325242462015232 5ustar liggesusersirace/inst/examples/moaco/README0000644000176200001440000000172514325210417016112 0ustar liggesusersExample configuration scenario: MOACO ===================================== This example scenario shows how irace can be used to tune a multi-objective optimization framework. It uses the multi-objective ant colony optimization (MOACO) framework proposed here for the bi-objective traveling salesperson problem (bTSP): Manuel López-Ibáñez and Thomas Stützle. The Automatic Design of Multi-Objective Ant Colony Optimization Algorithms. IEEE Transactions on Evolutionary Computation, 2012. doi:10.1109/TEVC.2011.2182651 (PDF preprint) https://iridia.ulb.ac.be/IridiaTrSeries/link/IridiaTr2011-003.pdf The MOACO framework software can be download from: http://lopez-ibanez.eu/moaco Instances of the bTSP can be generated by concatenating two single-objective TSP instances: $ cat euclidA100.tsp euclidB100.tsp > euclidAB100.tsp TSP instances and more information is available at: https://iridia.ulb.ac.be/supp/IridiaSupp2016-003/index.html irace/inst/examples/moaco/scenario.txt0000644000176200001440000000100414157104332017565 0ustar liggesusers## Directory where the programs will be run. execDir = "./execdir" ## Folder where tuning instances are located, either absolute or ## relative to working directory. trainInstancesDir = "./Instances" ## The maximum number of runs (invocations of targetRunner) that will ## performed. It determines the (maximum) budget of experiments for the tuning. maxExperiments = 10000 ## Indicates the number of decimal places to be considered for the ## real parameters. digits = 2 targetEvaluator = "./target-evaluator" irace/inst/examples/moaco/target-runner0000755000176200001440000000320614157104332017752 0ustar liggesusers#!/bin/bash ############################################################################### # # PARAMETERS: # $1 is the configuration number # $2 is the instance id # $3 is the seed # $4 is the instance name # The rest are parameters for running the program # # RETURN VALUE: # This script should print nothing. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" exit 1 } # If you want to find where target-runner is. #BINDIR=$(dirname "$(readlink -f "$(type -P $0 || echo $0)")") EXE=~/bin/moaco_btsp CONFIGURATION=$1 INSTANCEID=$2 SEED=$3 INSTANCE=$4 shift 4 || error "Not enough parameters" STDOUT=c${CONFIGURATION}-${INSTANCEID}-${SEED}.stdout STDERR=c${CONFIGURATION}-${INSTANCEID}-${SEED}.stderr # TODO: Use /tmp for all temporary files, i.e., put target-runner-data under /tmp tmp="./target-data" mkdir -p ${tmp} cd ${tmp} || error "cannot change to directory \'${tmp}\'" # Here we transform some parameters. PARAMS= while [ $# -gt 0 ]; do case "$1" in --ants) shift; ANTS="$1"; shift;; --Mants) shift; MANTS="$1"; let "ANTS = (3 * 2 * $MANTS)"; shift;; *) PARAMS="$PARAMS $1"; shift;;# terminate case esac done ISIZE=$(basename $INSTANCE | cut -f1 -d'-') let "RUNTIME = (4 * (($ISIZE / 100) ** 2))" FIXED_PARAMS="--trials 1 --wls 1 --aco mmas --time $RUNTIME --ants $ANTS --input $INSTANCE --seed $SEED" if [ ! -x "${EXE}" ]; then error "${EXE}: not found or not executable (pwd: $(pwd))" fi exec 2> $STDERR $EXE ${FIXED_PARAMS} ${PARAMS} 1> $STDOUT RET=$? echo "OK" >& 2 exit $RET irace/inst/examples/moaco/target-evaluator0000755000176200001440000000670414157104332020451 0ustar liggesusers#!/bin/bash ############################################################################### # This script is run for each configuration to evaluate it after all # candidate configurations have been run on a single instance. # # Check the examples in examples/ # # PARAMETERS: # $1 is the configuration number # $2 is the instance id # $3 is the seed # $4 is the instance name # $5 is the total number of configurations alive in this iteration # The rest are the IDs of all configurations run for this instance # # ONLY FOR ELITIST RACE: The rest ($* after `shift 5') are the ids of the # configurations alive in this iteration. This list can be used to calculate # the hypervolume using previous execution results. # # RETURN VALUE: # This script should print a single numerical value ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" if [ -s error.log ]; then cat error.log fi exit 1 } error_sleep() { echo "`TZ=UTC date`: $0: error: $@" >> error.log sleep 60 } NORMALIZATION_RANGE="'1 2'" REFERENCE_POINT="'2.1 2.1'" # A program that normalizes all _dat files to [0, 1] # (optional, not included). NORMALIZE="~/bin/nondominated --verbose -n $NORMALIZATION_RANGE -s '_nor'" UPDATELIMITS="2>&1 | grep -E '# Total minimum|# Total maximum' | sed 's/:/:\n/'" # A program for computing the hypervolume. Download hypervolume # code from http://lopez-ibanez.eu/hypervolume HV="~/bin/hv --quiet -r $REFERENCE_POINT" CONFIGURATION="$1" INSTANCEID="$2" SEED="$3" INSTANCE="$4" TOTALCONFIGURATIONS="$5" shift 5 || error "Not enough parameters to $0" ALLIDS=$* STDOUT=c${CONFIGURATION}-${INSTANCEID}-${SEED}.stdout STDERR=c${CONFIGURATION}-${INSTANCEID}-${SEED}.stderr calc_measure() { LIMITSFILE=limits-${INSTANCEID}.txt LIMITSTMP=limits-${INSTANCEID}.tmp if [ -s "$LIMITSFILE" ]; then eval "$NORMALIZE $LIMITSFILE $ALLFILES $UPDATELIMITS" > $LIMITSTMP rm -f ${LIMITSFILE} mv $LIMITSTMP $LIMITSFILE else eval "$NORMALIZE $ALLFILES $UPDATELIMITS" > $LIMITSFILE fi } # TODO: Use /tmp for all temporary files, i.e., put target-data under /tmp cd target-data || error "cannot change to directory 'target-data'" ALLFILES= for FILENAME in ALLIDS; do ALLFILES="$ALLFILES c${FILENAME}-${INSTANCEID}-${SEED}.stdout" done # First normalize output files, unless they are ready. while ! ls ${ALLFILES}_nor &> /dev/null ; do # Check that all output files are there NUM=$(ls -1 ${ALLFILES} 2> /dev/null | wc --lines) if [ "$NUM" -ne "$TOTALCONFIGURATIONS" ]; then error_sleep "only found $NUM files out of $TOTALCONFIGURATIONS: ${ALLFILES}" continue fi # Check every STDERR for FILENAME in $ALLFILES; do FILESTDERR=${FILENAME/stdout/stderr} [ OK = "$(cat ${FILESTDERR})" ] || (error_sleep "${FILESTDERR} is not OK"; continue) [ -s "$FILENAME" ] || (error_sleep "$FILENAME is empty"; continue) done calc_measure rm -f error.log # Comment out if you wish to keep all output files around comm --output-delimiter=" " -23 <(ls -1 c*-*.* | sort) <(ls -1 ${ALLFILES//-${INSTANCEID}-${SEED}.stdout/-*.*} | sort) | xargs -r rm -f break done if [ ! -s "${STDOUT}_nor" ]; then error "${STDOUT}_nor: No such file or directory" fi COST=$(eval "$HV ${STDOUT}_nor" || error "calculating hypervolume failed") # Negative because hypervolume is maximised but irace minimises. echo "-$COST" exit 0 irace/inst/examples/moaco/parameters.txt0000644000176200001440000000365214157104332020140 0ustar liggesusers## Template for parameter description file for Iterated F-Race. ## ## The format is one parameter per line. Each line contains: ## ## 1: Name of the parameter. An unquoted alphanumeric string, ## example: ants ## 2: Switch to pass the parameter. A quoted (possibly empty) string, ## if the value and the switch must be separated, add a space at ## the end of the string. Example : "--version1 --ants " ## 3: Type. An unquoted single letter, among ## i: Integer, c: component, r: real. ## 4: For c: All possible values, for i,r: minimum and maximum ## values. A variable number of numbers or unquoted strings within ## parenthesis separated by commas. ## 5: Optionally, a character '|' followed by a parameter name ## followed by a set of values enclosed in parenthesis. ## Example: | mode (acs, ras) ## # 1: 2: 3: 4: colonies "--colonies " c (1,2,3,5,10) | selection %in% c("dominance","objective") intervals "--colony-weights " c (disjoint,overlapping) | colonies %in% c(2, 3, 5, 10) cupdate "--colony-update " c (origin, region) | colonies %in% c(2, 3, 5, 10) nupdate "--num-update " c (1, 2, 5, 10) selection "--selection " c (dominance,objective,weight) weights "--weights " c (2, 3, 0.333333, 0.5, -1) directions "--directions " c (one, all) ph "--ph=" c (single,multiple) heu "--heu=" c (single,multiple) aggreg "--aggregation " c (sum, product, random) | ph == "multiple" || heu == "multiple" ants "--Mants " i (1, 25) rho "--rho " r (0.01, 0.99) onq0 " " c (" ", "--q0") q0 " " r (0.25, 0.99) | onq0 == "--q0" beta "--beta " r (0, 5) alpha "--alpha " r (0, 5) irace/inst/templates/0000755000176200001440000000000014325242462014314 5ustar liggesusersirace/inst/templates/instances-list.txt.tmpl0000644000176200001440000000066114310635542020772 0ustar liggesusers## This is an example of specifying a list of instances. # Each line is an instance relative to trainInstancesDir/testInstancesdir # unless set to "" (see scenario.txt.tmpl) and an optional sequence of # instance-specific parameters that will be passed to target-runner when # invoked on that instance. # Empty lines and comments are ignored. 100/100-1_100-2.tsp --time 1 100/100-1_100-3.tsp --time 2 100/100-1_100-4.tsp --time 3 irace/inst/templates/target-runner.tmpl0000755000176200001440000000520314321266642020014 0ustar liggesusers#!/bin/bash ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # This script is run in the execution directory (execDir, --exec-dir). # # PARAMETERS: # $1 is the candidate configuration number # $2 is the instance ID # $3 is the seed # $4 is the instance name # The rest ($* after `shift 4') are parameters to the run # # RETURN VALUE: # This script should print one numerical value: the cost that must be minimized. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" exit 1 } # This parses the arguments given by irace. Do not touch it! CONFIG_ID=$1 INSTANCE_ID=$2 SEED=$3 INSTANCE=$4 shift 4 || error "Not enough parameters" CONFIG_PARAMS=$* # End of parsing ## Find our own location. # BINDIR=$(dirname "$(readlink -f "$(type -P $0 || echo $0)")") # EDIT THIS: Path to your executable EXE=~/bin/program # EDIT THIS: Specify how parameters are given to your executable EXE_PARAMS="--time 5 -i $INSTANCE --seed ${SEED} ${CONFIG_PARAMS}" if [ ! -x "$(command -v ${EXE})" ]; then error "${EXE}: not found or not executable (pwd: $(pwd))" fi # # If the program just prints a number, we can use 'exec' to avoid creating # # another process, but there can be no other commands after exec, only exit. # exec $EXE ${EXE_PARAMS} # exit 1 # These files are saved in execDir. Saving them in /tmp may be faster. STDOUT=c${CONFIG_ID}-${INSTANCE_ID}-${SEED}.stdout STDERR=c${CONFIG_ID}-${INSTANCE_ID}-${SEED}.stderr # # Otherwise, save the output to a file, and parse the result from it. (If # # you wish to ignore segmentation faults you can use '{}' around the # # command.) # # If the command below fails, but you are not sure why, it may be useful to # # print it before executing using 'echo', as in: # echo "$EXE ${EXE_PARAMS} 1> ${STDOUT} 2> ${STDERR}" $EXE ${EXE_PARAMS} 1> ${STDOUT} 2> ${STDERR} # # This may be used to introduce a delay if there are filesystem issues. # SLEEPTIME=1 # while [ ! -s "${STDOUT}" ]; do # sleep $SLEEPTIME # let "SLEEPTIME += 1" # done if [ ! -s "${STDOUT}" ]; then error "${STDOUT}: No such file or directory" fi # This is an example of reading a number from the output. It assumes that the # objective value is the first number in the first column of the last line of # the output. COST=$(tail -n 1 ${STDOUT} | grep -e '^[[:space:]]*[+-]\?[0-9]' | cut -f1) echo "$COST" # Comment the following line if you wish to preserve temporary files rm -f "${STDOUT}" "${STDERR}" exit 0 irace/inst/templates/parameters.txt.tmpl0000644000176200001440000000316314157104332020172 0ustar liggesusers## Template for parameter description file for Iterated Race. ## ## The format is one parameter per line. Each line contains: ## ## 1: Name of the parameter. An unquoted alphanumeric string, ## example: ants ## 2: Switch to pass the parameter. A quoted (possibly empty) string, ## if the value and the switch must be separated, add a space at ## the end of the string. Example : "--version1 --ants " ## 3: Type. An unquoted single letter, among ## i: Integer, c: categorical, o: ordinal, r: real. ## 4: For c and o: All possible values, that is, a variable number of ## quoted or unquoted strings separated by commas within ## parenthesis. Empty strings and strings containing commas or ## spaces must be quoted. ## For i,r: a pair of unquoted numbers representing minimum and ## maximum values. ## 5: A conditional parameter can be defined according to the values of ## one or several other parameters. This is done by adding a ## character '|' followed by an R expression involving the names of ## other parameters. This expression must return TRUE if the ## condition is satisfied, FALSE otherwise. # 1: 2: 3: 4: 5: param1 "--param1 " i (1, 10) | mode %in% c("x1", "x2") param2 "--param2 " i (1, 10) | mode %in% c("x1", "x3") && real > 2.5 && real <= 3.5 mode "--" c ("x1" ,"x2", "x3") real "--paramreal=" r (1.5, 4.5) mutation "--mutation=" o ("none", "very low", "low", "medium", "high", "very high", "all") #unused "-u " c (1, 2, 10, 20) irace/inst/templates/forbidden.txt.tmpl0000644000176200001440000000117514157104332017764 0ustar liggesusers## Template for specifying forbidden parameter configurations in irace. ## ## This filename must be specified via the --forbidden-file command-line option ## (or forbiddenFile in scenario.txt). ## ## The format is one constraint per line. Each constraint is a logical ## expression (in R syntax). If a parameter configuration ## is generated that makes the logical expression evaluate to TRUE, ## then the configuration is discarded. ## ## Examples of valid logical operators are: == != >= <= > < & | ! %in% param1 < 5 & mode == "x1" (param2 > 6 & mode == "x1") | (param2 <= 6 & mode == "x2") real < 4 & mode %in% c("x2", "x3") irace/inst/templates/scenario.txt.tmpl0000644000176200001440000002023714321271071017631 0ustar liggesusers###################################################### -*- mode: r -*- ##### ## Scenario setup for Iterated Race (irace). ############################################################################ ## To use the default value of a parameter of iRace, simply do not set ## the parameter (comment it out in this file, and do not give any ## value on the command line). ## Directory where the programs will be run. # execDir = "./" ## File that contains the description of the parameters of the target ## algorithm. # parameterFile = "./parameters.txt" ## File that contains a list of logical expressions that cannot be TRUE for ## any evaluated configuration. If empty or NULL, do not use forbidden ## expressions. # forbiddenFile = "" ## File that contains a table of initial configurations. If empty or NULL, all ## initial configurations are randomly generated. # configurationsFile = "" ## File to save tuning results as an R dataset, either absolute path or ## relative to execDir. # logFile = "./irace.Rdata" ## Directory where training instances are located; either absolute path or ## relative to current directory. If no trainInstancesFiles is provided, all ## the files in trainInstancesDir will be listed as instances. # trainInstancesDir = "./Instances" ## File that contains a list of training instances and optionally additional ## parameters for them. If trainInstancesDir is provided, irace will search ## for the files in this folder. # trainInstancesFile = "" ## Randomly sample the training instances or use them in the order given. # sampleInstances = 1 ## Directory where testing instances are located, either absolute or relative ## to current directory. # testInstancesDir = "" ## File containing a list of test instances and optionally additional ## parameters for them. # testInstancesFile = "" ## Number of elite configurations returned by irace that will be tested if ## test instances are provided. # testNbElites = 1 ## Enable/disable testing the elite configurations found at each iteration. # testIterationElites = 0 ## Statistical test used for elimination. The default value selects t-test if ## capping is enabled or F-test, otherwise. Valid values are: F-test (Friedman ## test), t-test (pairwise t-tests with no correction), t-test-bonferroni ## (t-test with Bonferroni's correction for multiple comparisons), t-test-holm ## (t-test with Holm's correction for multiple comparisons). # testType = "" ## Number of instances evaluated before the first elimination test. It must be ## a multiple of eachTest. # firstTest = 5 ## Number of instances evaluated between elimination tests. # eachTest = 1 ## Executable called for each configuration that executes the target algorithm ## to be tuned. See the templates and examples provided. # targetRunner = "./target-runner" ## Executable that will be used to launch the target runner, when targetRunner ## cannot be executed directly (.e.g, a Python script in Windows). # targetRunnerLauncher = "" ## Command-line arguments provided to targetRunnerLauncher. The substrings ## \{targetRunner\} and \{targetRunnerArgs\} will be replaced by the value of ## the option targetRunner and by the arguments usually passed when calling ## targetRunner, respectively. Example: "-m {targetRunner --args ## {targetRunnerArgs}"}. # targetRunnerLauncherArgs = "{targetRunner} {targetRunnerArgs}" ## Number of times to retry a call to targetRunner if the call failed. # targetRunnerRetries = 0 ## Optional data passed to targetRunner. This is ignored by the default ## targetRunner function, but it may be used by custom targetRunner functions ## to pass persistent data around. # targetRunnerData = "" ## Optional R function to provide custom parallelization of targetRunner. # targetRunnerParallel = "" ## Optional script or R function that provides a numeric value for each ## configuration. See templates/target-evaluator.tmpl # targetEvaluator = "" ## If the target algorithm is deterministic, configurations will be evaluated ## only once per instance. # deterministic = 0 ## Maximum number of runs (invocations of targetRunner) that will be ## performed. It determines the maximum budget of experiments for the tuning. # maxExperiments = 0 ## Maximum total execution time in seconds for the executions of targetRunner. ## targetRunner must return two values: cost and time. # maxTime = 0 ## Fraction (smaller than 1) of the budget used to estimate the mean ## computation time of a configuration. Only used when maxTime > 0 # budgetEstimation = 0.02 ## Minimum time unit that is still (significantly) measureable. # minMeasurableTime = 0.01 ## Number of calls to targetRunner to execute in parallel. Values 0 or 1 mean ## no parallelization. # parallel = 0 ## Enable/disable load-balancing when executing experiments in parallel. ## Load-balancing makes better use of computing resources, but increases ## communication overhead. If this overhead is large, disabling load-balancing ## may be faster. # loadBalancing = 1 ## Enable/disable MPI. Use Rmpi to execute targetRunner in parallel (parameter ## parallel is the number of slaves). # mpi = 0 ## Specify how irace waits for jobs to finish when targetRunner submits jobs ## to a batch cluster: sge, pbs, torque, slurm or htcondor. targetRunner must ## submit jobs to the cluster using, for example, qsub. # batchmode = 0 ## Maximum number of decimal places that are significant for numerical (real) ## parameters. # digits = 4 ## Reduce the output generated by irace to a minimum. # quiet = 0 ## Debug level of the output of irace. Set this to 0 to silence all debug ## messages. Higher values provide more verbose debug messages. # debugLevel = 0 ## Seed of the random number generator (by default, generate a random seed). # seed = NA ## Enable/disable the soft restart strategy that avoids premature convergence ## of the probabilistic model. # softRestart = 1 ## Soft restart threshold value for numerical parameters. If NA, NULL or "", ## it is computed as 10^-digits. # softRestartThreshold = "" ## Enable/disable elitist irace. # elitist = 1 ## Number of instances added to the execution list before previous instances ## in elitist irace. # elitistNewInstances = 1 ## In elitist irace, maximum number per race of elimination tests that do not ## eliminate a configuration. Use 0 for no limit. # elitistLimit = 2 ## User-defined R function that takes a configuration generated by irace and ## repairs it. # repairConfiguration = "" ## Enable the use of adaptive capping, a technique designed for minimizing the ## computation time of configurations. This is only available when elitist is ## active. # capping = 0 ## Measure used to obtain the execution bound from the performance of the ## elite configurations: median, mean, worst, best. # cappingType = "median" ## Method to calculate the mean performance of elite configurations: candidate ## or instance. # boundType = "candidate" ## Maximum execution bound for targetRunner. It must be specified when capping ## is enabled. # boundMax = 0 ## Precision used for calculating the execution time. It must be specified ## when capping is enabled. # boundDigits = 0 ## Penalization constant for timed out executions (executions that reach ## boundMax execution time). # boundPar = 1 ## Replace the configuration cost of bounded executions with boundMax. # boundAsTimeout = 1 ## Percentage of the configuration budget used to perform a postselection race ## of the best configurations of each iteration after the execution of irace. # postselection = 0 ## Enable/disable AClib mode. This option enables compatibility with ## GenericWrapper4AC as targetRunner script. # aclib = 0 ## Maximum number of iterations. # nbIterations = 0 ## Number of runs of the target algorithm per iteration. # nbExperimentsPerIteration = 0 ## Minimum number of configurations needed to continue the execution of each ## race (iteration). # minNbSurvival = 0 ## Number of configurations to be sampled and evaluated at each iteration. # nbConfigurations = 0 ## Parameter used to define the number of configurations sampled and evaluated ## at each iteration. # mu = 5 ## Confidence level for the elimination test. # confidence = 0.95 ## END of scenario file ############################################################################ irace/inst/templates/target-evaluator.tmpl0000755000176200001440000000334414157104332020503 0ustar liggesusers#!/bin/bash ############################################################################### # This script is run for each candidate to evaluate it after all # candidate configurations have been run on a single instance. # # Check the examples in examples/ # # PARAMETERS: # $1 is the candidate configuration number # $2 is the instance id # $3 is the seed # $4 is the instance name # $5 is the number of candidates alive in this iteration # # ONLY FOR ELITIST RACE: The rest ($* after `shift 5') are the ids of the # candidates alive in this iteration. This list can be used to calculate the # hypervolume using previous execution results. # # RETURN VALUE: # This script should print one numerical value: the cost that must be minimized. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" exit 1 } CANDIDATE="$1" INSTANCEID="$2" SEED="$3" INSTANCE="$4" TOTALCANDIDATES="$5" shift 5 || error "Not enough parameters" ALLIDS=$* STDOUT=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdout STDERR=c${CANDIDATE}-${INSTANCEID}-${SEED}.stderr # # This may be used to introduce a delay if there are filesystem # # issues. # SLEEPTIME=1 # while [ ! -s "${STDOUT}" ]; do # sleep $SLEEPTIME # let "SLEEPTIME += 1" # done # This is an example of reading a number from the output of # target-runner. It assumes that the objective value is the first number in # the first column of the only line starting with a digit. if [ -s "${STDOUT}" ]; then COST=$(cat ${STDOUT} | grep -e '^[[:space:]]*[+-]\?[0-9]' | cut -f1) echo "$COST" rm -f "${STDOUT}" "${STDERR}" exit 0 else error "${STDOUT}: No such file or directory" fi irace/inst/templates/windows/0000755000176200001440000000000014157104332016002 5ustar liggesusersirace/inst/templates/windows/target-runner.bat0000755000176200001440000000431614157104332021276 0ustar liggesusers@echo off ::############################################################################## :: BAT version of target-runner for Windows. :: Contributed by Andre de Souza Andrade . :: Check other examples in examples/ :: :: This script is run in the execution directory (execDir, --exec-dir). :: :: PARAMETERS: :: %%1 is the candidate configuration number :: %%2 is the instance ID :: %%3 is the seed :: %%4 is the instance name :: The rest are parameters to the target-algorithm :: :: RETURN VALUE: :: This script should print one numerical value: the cost that must be minimized. :: Exit with 0 if no error, with 1 in case of error ::############################################################################## :: Please change the EXE and FIXED_PARAMS to the correct ones SET "exe=D:\bin\program" SET "fixed_params=--time 20 --tries 1 --quiet" FOR /f "tokens=1-4*" %%a IN ("%*") DO ( SET candidate=%%a SET instance_id=%%b SET seed=%%c SET instance=%%d SET candidate_parameters=%%e ) SET "stdout=c%candidate%-%instance_id%-%seed%.stdout" SET "stderr=c%candidate%-%instance_id%-%seed%.stderr" :: :: FIXME: How to implement this in BAT ? ::if not exist %exe% error "%exe%: not found or not executable (pwd: %(pwd))" :: Save the output to a file, and parse the result from it. %exe% %fixed_params% -i %instance% --seed %seed% %candidate_parameters% 1>%stdout% 2>%stderr% :: :: This may be used to introduce a delay if there are filesystem :: :: issues. :: setlocal EnableDelayedExpansion :: :loop :: for /f %%i in ("%stdout%") do set size=%%~zi :: if "%size%" NEQ "0" (goto endloop) :: waitfor false /t 10 >nul :: goto loop :: :endloop :: This is an example of reading a number from the output. :: It assumes that the objective value is the first number in :: the first column of the last line of the output. setlocal EnableDelayedExpansion set "cmd=findstr /R /N "^^" %stdout% | find /C ":"" for /f %%a in ('!cmd!') do set numberlines=%%a set /a lastline=%numberlines%-1 for /f "tokens=3" %%F in ('more +%lastline% %stdout%') do set COST=%%F echo %COST% :: Un-comment this if you want to delete temporary files. :: del %stdout% %stderr% exit 0 irace/inst/templates/configurations.txt.tmpl0000644000176200001440000000136114157104332021057 0ustar liggesusers## Template for specifying initial parameter configurations, for ## example, the default configuration, in irace. ## ## This filename must be specified via the --candidates-file command-line option ## (or candidatesFile in scenario.txt). ## ## The format is one parameter configuration per line, and one parameter per ## column. The first line gives the parameter name corresponding to ## each column (names must match those given in the parameters ## file). Each configuration must satisfy the ## parameter conditions (NA should be used for those parameters that ## are not enabled for a given configuration) and, if given, the ## constraints that describe forbidden configurations. param1 param2 mode real mutation 5 NA "x2" 2.0 "low" irace/inst/irace.sindef0000644000176200001440000000167314157104332014576 0ustar liggesusersBootstrap: library From: ubuntu:20.04 %post # Dependencies apt -y install software-properties-common add-apt-repository universe apt -y update apt -y dist-upgrade apt -y install git make r-base r-cran-jsonlite apt clean # Temporary directory where we are going to build everything. tmpdir=$(mktemp -d) mkdir -p ${tmpdir} # Build cd ${tmpdir} git clone --single-branch --branch master https://github.com/MLopez-Ibanez/irace.git cd irace make quick-install # Clean-up rm -rf ${tmpdir} apt -y purge software-properties-common git make apt -y --purge autoremove apt -y autoclean apt clean %help This container provides an minimal executable version of irace. You can run it with `singularity run `. e.g. `singularity run --help` %runscript /usr/local/lib/R/site-library/irace/bin/irace $* %labels Authors Johann Dreo irace/inst/doc/0000755000176200001440000000000014325242462013063 5ustar liggesusersirace/inst/doc/irace-package.Rnw0000644000176200001440000050055014325242447016237 0ustar liggesusers% !Rnw weave = knitr %%% DO NOT EDIT the .tex file directly since it is generated from the .Rnw %%% sources. %\VignetteEngine{knitr::knitr} %\VignetteIndexEntry{irace package: User Guide} %\VignetteDepends{knitr} %\VignetteCompiler{knitr} \synctex=1 \RequirePackage{xparse} \RequirePackage[dvipsnames]{xcolor} \documentclass[a4paper,english]{article} \usepackage[T1]{fontenc} \usepackage[utf8]{inputenc} \usepackage[a4paper]{geometry} % It saves some pages \usepackage[english]{babel} \usepackage{ifthen} \newboolean{Release} \setboolean{Release}{true} \usepackage{calc} \usepackage{afterpage} \usepackage{algorithm,algorithmic} \usepackage{booktabs} \usepackage{tabularx} \usepackage{xspace} \usepackage{amsmath,amssymb} \usepackage{relsize} \usepackage{fancyvrb} \usepackage{microtype} % \texttt{test -- test} keeps the "--" as "--" (and does not convert it to an en dash) \DisableLigatures{encoding = T1, family = tt* } \usepackage[hyphens]{url} \usepackage{hyperref} \usepackage[numbers]{natbib} \usepackage[nottoc]{tocbibind} %% For autoref \hypersetup{ colorlinks, linkcolor={red!50!black}, citecolor={blue!50!black}, urlcolor={blue!70!black} } \addto\extrasenglish{% \def\sectionautorefname{Section} \let\subsectionautorefname\sectionautorefname \let\subsubsectionautorefname\sectionautorefname } \usepackage[titletoc, title]{appendix} % Fix use with \autoref \newcommand*{\Appendixautorefname}{Appendix} \usepackage{tocloft} \setlength{\cftsubsecnumwidth}{3em}% Set length of number width in ToC for \subsection \usepackage[inline]{enumitem} \setlist[enumerate]{leftmargin=*,widest=00} \setlist[itemize]{leftmargin=1.5em} %% FIXME: listing is very limited, we should use 'minted' \usepackage{listings} \lstdefinestyle{BashInputStyle}{ language=bash,% basicstyle=\ttfamily,% numbers=none,% frame=tb,% rulecolor=\color{lightgray}, % framesep=1ex, framexleftmargin=1ex, columns=fullflexible,% backgroundcolor=\color{yellow!05},% linewidth=\linewidth,% % xleftmargin=1\linewidth,% identifierstyle=\color{darkgray},% keywordstyle=\color{darkgray},% keywordstyle={[2]\color{Cyan}},% keywordstyle={[3]\color{olive}},% stringstyle=\color{MidnightBlue},% commentstyle=\color{RedOrange},% morestring=[b]',% showstringspaces=false } \DefineVerbatimEnvironment{Code}{Verbatim}{} \DefineVerbatimEnvironment{CodeInput}{Verbatim}{fontshape=rm} \DefineVerbatimEnvironment{CodeOutput}{Verbatim}{} \newenvironment{CodeChunk}{}{} \newcommand{\IRACEHOME}[1]{\hyperlink{irace_home}{\path{$IRACE_HOME}}\path{#1}} \providecommand{\keywords}[1]{\textbf{\textit{Index terms---}} #1} % Simple font selection is not good enough. For example, |\texttt{--}| % gives `\texttt{--}', i.e., an endash in typewriter font. Hence, we % need to turn off ligatures, which currently only happens for commands % |\code| and |\samp| and the ones derived from them. Hyphenation is % another issue; it should really be turned off inside |\samp|. And % most importantly, \LaTeX{} special characters are a nightmare. E.g., % one needs |\~{}| to produce a tilde in a file name marked by |\file|. % Perhaps a few years ago, most users would have agreed that this may be % unfortunate but should not be changed to ensure consistency. But with % the advent of the WWW and the need for getting `|~|' and `|#|' into % URLs, commands which only treat the escape and grouping characters % specially have gained acceptance \makeatletter \DeclareRobustCommand\code{\bgroup\@makeother\_\@makeother\~\@makeother\$\@noligs\@codex} \def\@codex#1{\texorpdfstring% {{\text{\normalfont\ttfamily\hyphenchar\font=-1 #1}}}% {#1}\egroup} \makeatother \let\proglang=\textsf \newcommand{\pkg}[1]{{\fontseries{b}\selectfont #1}} \newcommand{\aR}{\proglang{R}\xspace} \newcommand{\MATLAB}{\proglang{MATLAB}\xspace} \newcommand{\eg}{e.g.,\xspace} \newcommand{\SoftwarePackage}{\pkg} \newcommand{\ACOTSP}{\SoftwarePackage{ACOTSP}\xspace} %% How to use this command: % Parameter with one short switch: \defparameter[short]{paramName}{long}{default} % Parameter without short switch: \defparameter{paramName}{long}{default} % Parameter without switch: \defparameter{paramName}{}{default} \newcommand{\defparameter}[4][]{% \item[\code{#2}]\hypertarget{opt:#2}{} ~~ % \ifthenelse{\equal{#3}{}}{}{% \emph{flag:} % \ifthenelse{\equal{#1}{}}{}{% \code{-#1}~~~\emph{or}~~~}% \code{--#3} ~~ }% \emph{default:}~\texttt{#4} \\ } \newcommand{\parameter}[1]{\hyperlink{opt:#1}{\code{#1}}} %\usepackage{showlabels} %\showlabels{hypertarget} \newcommand{\irace}{\pkg{irace}\xspace} \newcommand{\Irace}{\pkg{Irace}\xspace} \newcommand{\race}{\pkg{race}\xspace} \newcommand{\FRACE}{\text{F-Race}\xspace} \newcommand{\IFRACE}{\text{I/F-Race}\xspace} \newcommand{\PyImp}{\pkg{PyImp}\xspace} \newcommand{\iraceversion}{\Sexpr{packageVersion("irace")}} \newcommand{\Niter}{\ensuremath{N^\text{iter}}\xspace} \newcommand{\Nparam}{\ensuremath{{N^\text{param}}}\xspace} \newcommand{\iter}{\ensuremath{j}\xspace} \newcommand{\Budget}{\ensuremath{B}\xspace} \newcommand{\Budgetj}{\ensuremath{\Budget_{\iter}}\xspace} \newcommand{\Bused}{\ensuremath{\Budget_\text{used}}\xspace} \newcommand{\Ncand}[1][]{\ensuremath{N_{#1}}\xspace} \newcommand{\Mui}{\ensuremath{\mu_{\iter}}\xspace} \newcommand{\Nmin}{\ensuremath{N^\text{min}}\xspace} \newcommand{\Nsurv}{\ensuremath{N^\text{surv}}\xspace} \newcommand{\Nelite}{\ensuremath{N^\text{elite}}\xspace} \newcommand{\Nnew}{\ensuremath{N^\text{new}}\xspace} \newcommand{\bmax}{\ensuremath{b^\text{max}}\xspace} \newcommand{\bmin}{\ensuremath{b^\text{min}}\xspace} \newcommand{\Celite}{\ensuremath{\Theta^\text{elite}}\xspace} \ifthenelse {\boolean{Release}}{% \newcommand{\MANUEL}[1]{} \newcommand{\LESLIE}[1]{} \newcommand{\THOMAS}[1]{} }{% \newcommand{\MANUEL}[1]{{\footnotesize\noindent\textbf{\color{red}[~MANUEL: #1~]}}} \newcommand{\LESLIE}[1]{\footnote{\noindent\textbf{[ LESLIE: #1 ]}}} \newcommand{\THOMAS}[1]{\footnote{\noindent\textbf{[ THOMAS: #1 ]}}} } \newcommand{\hide}[1]{} \usepackage{tcolorbox} \newcommand{\infoicon}{% \parbox[c]{0.75cm}{\includegraphics[keepaspectratio=true,width=0.75cm]{light-bulb-icon}}% \hspace{1em}} \newcommand{\warningicon}{% \parbox[c]{0.75cm}{\includegraphics[keepaspectratio=true,width=0.75cm]{Warning-icon}}% \hspace{1em}} \definecolor{LightGray}{RGB}{193,193,193} \definecolor{LightYellow}{RGB}{253,247,172} \newlength\macroiconwidth \newenvironment{xwarningbox}{% \setlength{\fboxrule}{3.0\fboxrule}% \setlength{\fboxsep}{0\fboxsep}% \begin{tcolorbox}[colback=LightYellow,colframe=LightGray,boxrule=\fboxrule,boxsep=\fboxsep]% \infoicon% \settowidth{\macroiconwidth}{\infoicon}% \begin{minipage}[c]{\columnwidth - \macroiconwidth - 2.0\fboxrule - 2.0\fboxsep} \raggedright\footnotesize % }{% \end{minipage} \end{tcolorbox} % } \begin{document} <>= library(knitr) @ \author{Manuel L\'opez-Ib\'a\~nez, Leslie P\'erez C\'aceres, J\'er\'emie Dubois-Lacoste,\\ Thomas St\"utzle and Mauro Birattari \\IRIDIA, CoDE, Universit\'e Libre de Bruxelles, Brussels, Belgium} \title{The \irace Package: User Guide} \date{Version \iraceversion, \today} %\keywords{automatic % algorithm configuration, racing, parameter tuning, \aR} \maketitle \tableofcontents %Load files needed for examples <>= library("irace") load("examples.Rdata") iraceResults <- irace::read_logfile("irace-acotsp.Rdata") log_ablation_file <- file.path(system.file(package="irace"), "exdata", "log-ablation.Rdata") load(log_ablation_file) options(width = 70) @ \newpage %% %% %% %% General info %% %% %% \section{General information} \MANUEL{Some things could be taken from the intro of the irace paper and reformulated.} \MANUEL{It would be good to mention that not only opt algorithms can be configured with irace, we say this in the paper.} \subsection{Background} \MANUEL{I would add a paragraph defining what is irace (a bit longer than the abstract above) and references to the literature so people can find more info. The first reference should be the irace TR.} \LESLIE{Here i guess we should say why tune an algorithm is a good idea, and why using irace is a better one.} The \irace package implements an \emph{iterated racing} procedure, which is an extension of Iterated F-race (\IFRACE)~\cite{BirYuaBal2010:emaoa}. The main use of \irace is the automatic configuration of optimization and decision algorithms, that is, finding the most appropriate settings of an algorithm given a set of instances of a problem. However, it may also be useful for configuring other types of algorithms when performance depends on the used parameter settings. It builds upon the \pkg{race} package by Birattari and it is implemented in \aR. The \irace package is available from CRAN: % \begin{center} \url{https://cran.r-project.org/package=irace} \end{center} % More information about \irace is available at \url{https://mlopez-ibanez.github.io/irace}. \subsection{Version} The current version of the \irace package is \iraceversion. Previous versions of the package can also be found in the \href{https://cran.r-project.org/package=irace}{CRAN website}. The algorithm underlying the current version of \irace and its motivation are described by \citet{LopDubPerStuBir2016irace}. The \textbf{adaptive capping mechanism} available from version $3.0$ is described by \citet{PerLopHooStu2017:lion}. Details of the implementation before version 2.0 can be found in a previous technical report~\cite{LopDubStu2011irace}. % \begin{xwarningbox} Versions of \irace before 2.0 are not compatible with the file formats detailed in this document. \end{xwarningbox} \subsection{License} The \irace package is Copyright \copyright{} \the\year\ and distributed under the GNU General Public License version 3.0 (\url{http://www.gnu.org/licenses/gpl-3.0.en.html}). The \irace package is free software (software libre): You can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. The \irace package is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Please be aware that the fact that this program is released as Free Software does not excuse you from scientific propriety, which obligates you to give appropriate credit! If you write a scientific paper describing research that made substantive use of this program, it is your obligation as a scientist to (a) mention the fashion in which this software was used in the Methods section; (b) mention the algorithm in the References section. The appropriate citation is: \begin{itemize}[leftmargin=3em] \item[] Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, Thomas Stützle, and Mauro Birattari. The \irace package: Iterated Racing for Automatic Algorithm Configuration. \emph{Operations Research Perspectives}, 3:43--58, 2016. doi:~\href{http://dx.doi.org/10.1016/j.orp.2016.09.002}{10.1016/j.orp.2016.09.002} \end{itemize} \section{Before starting} \MANUEL{I think this could be a bit more detailed by defining what is a parameter, a configuration, an instance, etc. but ok for now.} The \irace package provides an automatic configuration tool for tuning optimization algorithms, that is, automatically finding good configurations for the parameters values of a (target) algorithm saving the effort that normally requires manual tuning. \begin{figure}[t] \centering \includegraphics[width=0.6\textwidth]{irace-scheme} \caption{Scheme of \irace flow of information.} \label{fig:irace-scheme} \end{figure} Figure~\ref{fig:irace-scheme} gives a general scheme of how \irace works. \Irace receives as input a \emph{parameter space definition} corresponding to the parameters of the target algorithm that will be tuned, a set of \emph{instances} for which the parameters must be tuned for and a set of options for \irace that define the \emph{configuration scenario}. Then, \irace searches in the parameter search space for good performing algorithm configurations by executing the target algorithm on different instances and with different parameter configurations. A \parameter{targetRunner} must be provided to execute the target algorithm with a specific parameter configuration ($\theta$) and instance ($i$). The \parameter{targetRunner} function (or program) acts as an interface between the execution of the target algorithm and \irace: It receives the instance and configuration as arguments and must return the evaluation of the execution of the target algorithm. The following user guide contains guidelines for installing \irace, defining configuration scenarios, and using \irace to automatically configure your algorithms. %% %% %% %% Installation %% %% %% \section{Installation} \subsection{System requirements} \begin{itemize} \item \aR ($\text{version} \geq 3.2.0$) is required for running irace, but you don't need to know the \aR language to use it. \aR is freely available and you can download it from the \aR project website (\url{https://www.r-project.org}). See \autoref{sec:installation} for a quick installation guide of \aR. \item For GNU/Linux and OS X, the command-line executable \code{parallel-irace} requires GNU Bash. Individual examples may require additional software. \end{itemize} \subsection{\irace installation} \label{sec:irace install} The \irace package can be installed automatically within \aR or by manual download and installation. We advise to use the automatic installation unless particular circumstances do not allow it. The instructions to install \irace with the two mentioned methods are the following: \subsubsection[Install automatically within R]{Install automatically within \aR{}} Execute the following line in the \aR console to install the package: <>= install.packages("irace") @ Select a mirror close to your location, and test the installation in the \aR console with: <>= library("irace") q() # To exit R @ Alternatively, within the \aR graphical interface, you may use the \code{Packages and data->Package installer} menu on OS X or the \code{Packages} menu on Windows. \subsubsection{Manual download and installation} From the \irace package CRAN website (\url{https://cran.r-project.org/package=irace}), download one of the three versions available depending on your operating system: \begin{itemize} \item \code{irace_\iraceversion.tar.gz} (Unix/BSD/GNU/Linux) \item \code{irace_\iraceversion.tgz} (OS X) \item \code{irace_\iraceversion.zip} (Windows) \end{itemize} To install the package on GNU/Linux and OS X, you must execute the following command at the shell (replace \code{} with the path to the downloaded file, either \code{irace_\iraceversion.tar.gz} or \code{irace_\iraceversion.zip}): % \begin{lstlisting}[style=BashInputStyle] R CMD INSTALL \end{lstlisting} To install the package on Windows, open \aR and execute the following line on the \aR console (replace \code{} with the path to the downloaded file \code{irace_\iraceversion.zip}): %\LESLIE{Check that this actually works on internet says that this: \code{Rscript -e "install.packages('foo.zip', repos = NULL)"} also works} % <>= install.packages("", repos = NULL) @ If the previous installation instructions fail because of insufficient permissions and you do not have sufficient admin rights to install \irace system-wide, then you need to force a local installation. \subsubsection{Local installation} Let's assume you wish to install \irace on a path denoted by \code{}, which is a filesystem path for which you have sufficient rights. This directory \textbf{must} exist before attempting the installation. Moreover, you must provide to \aR the path to this library when loading the package. However, the latter can be avoided by adding the path to the system variable \code{R_LIBS} or to the \aR internal variable \code{.libPaths}, as we will see below.\footnote{% On Windows, see also \url{https://cran.r-project.org/bin/windows/base/rw-FAQ.html\#I-don_0027t-have-permission-to-write-to-the-R_002d3_002e3_002e1_005clibrary-directory}.} On GNU/Linux or OS X, execute the following commands to install the package on a local directory: \begin{lstlisting}[style=BashInputStyle] export R_LIBS_USER="" # Create R_LIBS_USER if it doesn't exist mkdir $R_LIBS_USER # Replace with the path to the downloaded file. R CMD INSTALL --library=$R_LIBS_USER # Tell R where to find R_LIBS_USER export R_LIBS=${R_LIBS_USER}:${R_LIBS} \end{lstlisting} On Windows, you can install the package on a local directory by executing the following lines in the \aR console: <>= # Replace with the path to the downloaded file. # Replace with the path used for installation. install.packages("", repos = NULL, lib = "") # Tell R where to find R_LIBS_USER. # This must be executed for every new session. .libPaths(c("", .libPaths())) @ \subsubsection{Testing the installation and invoking irace} Once \irace has been installed, load the package and test that the installation was successful by opening an \aR console and executing: <>= # Load the package library("irace") # Obtain the installation path system.file(package = "irace") @ The last command must print out the filesystem path where \irace is installed. In the remainder of this guide, the variable \code{\$IRACE_HOME} is used to denote this path. When executing any provided command that includes the \code{\$IRACE_HOME} variable do not forget to replace this variable with the installation path of \irace. On GNU/Linux or OS X, you can let the operating system know where to find \irace by defining the \code{\$IRACE_HOME} variable and adding it to the system \code{PATH}. Append the following commands to \path{~/.bash_profile}, \path{~/.bashrc} or \path{~/.profile}: % %<>= \begin{lstlisting}[style=BashInputStyle] # Replace with the irace installation path export IRACE_HOME= export PATH=${IRACE_HOME}/bin/:$PATH # Tell R where to find R_LIBS_USER # Use the following line only if local installation was forced export R_LIBS=${R_LIBS_USER}:${R_LIBS} \end{lstlisting} %@ Then, open a new terminal and launch \irace as follows: %<>= \begin{lstlisting}[style=BashInputStyle] irace --help \end{lstlisting} %@ On Windows, you need to add both \aR and the installation path of \irace to the environment variable \code{PATH}. To edit the \code{PATH}, search for ``Environment variables'' in the control panel, edit \code{PATH} and add a string similar to \path{C:\R_PATH\bin;C:\IRACE_HOME\bin\x64\} where \code{R_PATH} is the installation path of \aR and \code{IRACE_HOME} is the installation path of \irace. If \irace was installed locally, you also need to edit the environment variable \code{R_LIBS} to add \code{R_LIBS_USER}. Then, open a new terminal (run program \code{cmd.exe}) and launch \irace as: % %<>= \begin{lstlisting}[style=BashInputStyle] irace.exe --help \end{lstlisting} %@ Alternatively, you may directly invoke \irace from within the \aR console by executing: <>= library("irace") irace.cmdline("--help") @ \section{Running irace}\label{sec:execution} Before performing the tuning of your algorithm, it is necessary to define a tuning scenario that will give \irace all the necessary information to optimize the parameters of the algorithm. The tuning scenario is composed of the following elements: \begin{enumerate} \item Target algorithm parameter description (see \autoref{sec:target parameters}). \item Target algorithm runner (see \autoref{sec:runner}). \item Training instances list (see \autoref{sec:training}) \item \irace options (see \autoref{sec:irace options}). \item \textit{Optional:} Initial configurations (see \autoref{sec:initial}). \item \textit{Optional:} Forbidden configurations (see \autoref{sec:forbidden}). \item \textit{Optional:} Target algorithm evaluator (see \autoref{sec:evaluator}). \end{enumerate} These scenario elements can be provided as plain text files or as \aR objects. This user guide provides examples of both types, but we advise the use of plain text files, which we consider the simpler option. For a step-by-step guide to create the scenario elements for your target algorithm continue to \autoref{sec:step}. For an example execution of \irace using the \ACOTSP scenario go to \autoref{sec:example}. \subsection{Step-by-step setup guide}\label{sec:step} This section provides a guide to setup a basic execution of \irace. The template files provided in the package (\IRACEHOME{/templates}) will be used as basis for creating your new scenario. Please follow carefully the indications provided in each step and in the template files used; if you have doubts check the the sections that describe each option in detail. \begin{enumerate}[leftmargin=*] \item Create a directory (\eg~\path{~/tuning/}) for the scenario setup. This directory will contain all the files that describe the scenario. On GNU/Linux or OS X, you can do this as follows: %<>= \begin{lstlisting}[style=BashInputStyle] mkdir ~/tuning cd ~/tuning \end{lstlisting} %@ \item Initialize the tuning directory with template config files. On GNU/Linux or OS X, you can do this as follows: %<>= \begin{lstlisting}[style=BashInputStyle] irace --init \end{lstlisting} %@ \item Define the target algorithm parameters to be tuned by following the instructions in \code{parameters.txt}. Available parameter types and other guidelines can be found in \autoref{sec:target parameters}. \item \textit{Optional}: Define the initial parameter configuration(s) of your algorithm, which allows you to provide good starting configurations (if you know some) for the tuning. Follow the instructions in \code{configurations.txt} and set \parameter{configurationsFile}\code{="configurations.txt"} in \code{scenario.txt}. More information in \autoref{sec:initial}. If you do not need to define initial configurations remove this file from the directory. \item \textit{Optional}: Define forbidden parameter value combinations, that is, configurations that \irace must not consider in the tuning. Follow the instructions in \code{forbidden.txt} and update \code{scenario.txt} with \parameter{forbiddenFile} \code{=} \code{"forbidden.txt"}. More information about forbidden configurations in \autoref{sec:forbidden}. If you do not need to define forbidden configurations remove this file from the directory. \item Place the instances you would like to use for the tuning of your algorithm in the folder \path{~/tuning/Instances/}. In addition, you can create a file (\eg~\code{instances-list.txt}) that specifies which instances from that directory should be run and which instance-specific parameters to use. To use such an instance file, set the appropriate option in \code{scenario.txt}, e.g., \parameter{trainInstancesFile} \code{= "instances-list.txt"}. See \autoref{sec:training} for guidelines. \item Uncomment and assign in \code{scenario.txt} only the options for which you need a value different from the default. %% MANUEL: I'm not sure what this means. % The names of the template files match the default names of the scenario % options. Some common parameters that you might want to adjust are: \begin{description} \item[\parameter{execDir}] (\code{--exec-dir}): the directory in which \irace will execute the target algorithm; the default value is the current directory. %\item[\parameter{logFile}] (\code{--log-file}): a file the name of the results \aR data file that produces \irace. \item[\parameter{maxExperiments}] (\code{--max-experiments}): the maximum number of executions of the target algorithm that \irace will perform. \item[\parameter{maxTime}] (\code{--max-time}): maximum total execution time in seconds for the executions of \code{targetRunner}. In this case, \code{targetRunner} must return two values: cost and time. Note that you must provide either \parameter{maxTime} or \parameter{maxExperiments}. \end{description} For setting the tuning budget see \autoref{sec:budget}. For more information on \irace options and their default values, see \autoref{sec:irace options}. \item Modify the \code{target-runner} script to run your algorithm. This script must execute your algorithm with the parameters and instance specified by \irace and return the evaluation of the execution and \textit{optionally} the execution time (\code{cost [time]}). When the \parameter{maxTime} option is used, returning \code{time} is mandatory. The \code{target-runner} template is written in \proglang{GNU Bash} scripting language, which can be executed easily in GNU/Linux and OS X systems. However, you may use any other programming language. We provide examples written in \proglang{Python}, \proglang{MATLAB} and other languages in \IRACEHOME{/examples/}. % Follow these instructions to adjust the given \code{target-runner} template to your algorithm: \begin{enumerate} \item Set the \code{EXE} variable with the path to the executable of the target algorithm. \item Set the \code{FIXED_PARAMS} if you need extra arguments in the execution line of your algorithm. An example could be the time that your algorithm is required to run (\code{FIXED_PARAMS}\hspace{0pt}\code{=}\hspace{0pt}\code{"--time 60"}) or the number of evaluations required (\code{FIXED_PARAMS}\hspace{0pt}\code{=}\hspace{0pt}\code{"--evaluations 10000"}). \item The line provided in the template executes the executable described in the \code{EXE} variable. % \begin{center} \code{\$EXE \$\{FIXED_PARAMS\} -i \$\{INSTANCE\} --seed \$\{SEED\} \$\{CONFIG_PARAMS\}} \end{center} % You must change this line according to the way your algorithm is executed. In this example, the algorithm receives the instance to solve with the flag \code{-i} and the seed of the random number generator with the flag \code{--seed}. The variable \code{CONFIG_PARAMS} adds to the command line the parameters that \irace has given for the execution. You must set the command line execution as needed. For example, the instance might not need a flag and might need to be the first argument: \begin{center} \code{\$EXE \$\{INSTANCE\} \$\{FIXED_PARAMS\} --seed \$\{SEED\} \$\{CONFIG_PARAMS\}} \end{center} The output of your algorithm is saved to the file defined in the \code{\$STDOUT} variable, and error output is saved in the file given by \code{\$STDERR}. The line: \begin{center} \code{if [ -s "\${STDOUT}" ]; then} \end{center} checks if the file containing the output of your algorithm is not empty. The example provided in the template assumes that your algorithm prints in the last output line the best result found (only a number). The line: \begin{center} \code{COST=\$(cat \$\{STDOUT\} | grep -e '\^{}[[:space:]]*[+-]\textbackslash{}?[0-9]' | cut -f1)} \end{center} parses the output of your algorithm to obtain the result from the last line. The \code{target-runner} script must print \textbf{only} one number. In the template example, the result is printed with \code{echo "\$COST"} (assuming \parameter{maxExperiments} is used) and the generated files are deleted (you may remove that line if you wish to keep them). \begin{xwarningbox} The \code{target-runner} script must be an executable file, unless you specify \parameter{targetRunnerLauncher} and \parameter{targetRunnerLauncherArgs}. \end{xwarningbox} You can test the target runner from the \aR console by checking the scenario as explained earlier in \autoref{sec:execution}. If you have problems related to the \code{target-runner} script when executing \irace, see \autoref{sec:check list} for a check list to help diagnose common problems. For more information about the \parameter{targetRunner}, please see \autoref{sec:runner}, \end{enumerate} \item \textit{Optional}: Modify the \code{target-evaluator} file. This is rarely needed and the \code{target-runner} template does not use it. \autoref{sec:evaluator} explains when a \parameter{targetEvaluator} is needed and how to define it. \item The \irace executable provides an option (\parameter{--check}) to check that the scenario is correctly defined. We recommend to perform a check every time you create a new scenario. When performing the check, \irace will verify that the scenario and parameter definitions are correct and will test the execution of the target algorithm. To check your scenario execute the following commands: \begin{itemize} \item From the command-line (on Windows, execute \code{irace.bat}): %<>= \begin{lstlisting}[style=BashInputStyle] # $IRACE_HOME is the installation directory of irace. $IRACE_HOME/bin/irace --scenario scenario.txt --check \end{lstlisting} %@ \item Or from the \aR console: <>= library("irace") scenario <- readScenario(filename = "scenario.txt", scenario = defaultScenario()) checkIraceScenario(scenario = scenario) @ \end{itemize} \item Once all the scenario elements are prepared you can execute \irace, either using the command-line wrappers provided by the package or directly from the \aR console: \begin{itemize} \item \textbf{From the command-line console}, call the command (on Windows, you should execute \code{irace.exe}): \begin{lstlisting}[style=BashInputStyle] cd ~/tuning/ # $IRACE_HOME is the installation directory of irace # By default, irace reads scenario.txt, you can specify a different file # with --scenario. $IRACE_HOME/bin/irace \end{lstlisting} For this example we assume that the needed scenario files have been set properly in the \code{scenario.txt} file using the options described in \autoref{sec:irace options}. Most \irace options can be specified in the command line or directly in the \code{scenario.txt} file. \item \textbf{From the \aR console}, evaluate: <>= library("irace") # Go to the directory containing the scenario files setwd("~/tuning") scenario <- readScenario(filename = "scenario.txt", scenario = defaultScenario()) irace.main(scenario = scenario) @ \end{itemize} This will perform one run of \irace. See the output of \code{irace --help} in the command-line or \code{irace.usage()} in \aR for quick information on additional \irace parameters. For more information about \irace options, see \autoref{sec:irace options}. \end{enumerate} \begin{xwarningbox} Command-line options override the same options specified in the \code{scenario.txt} file. \end{xwarningbox} \subsection{Setup example for ACOTSP}\label{sec:example} The \ACOTSP tuning example can be found in the package installation in the folder \IRACEHOME{/examples/acotsp}. % Other example scenarios can be found in the same folder. More examples of tuning scenarios can be found in the Algorithm Configuration Library (AClib, \url{http://www.aclib.net/}). In this section, we describe how to execute the \ACOTSP scenario. If you wish to start setting up your own scenario, continue to the next section. For this example, we assume a GNU/Linux system such as Ubuntu with a working \proglang{C} compiler such as \code{gcc}. To execute this scenario follow these steps: \begin{enumerate} \item Create a directory for the tuning (\eg~\path{~/tuning/}) and copy the example scenario files located in the \code{examples} folder to the created directory: %<>= \begin{lstlisting}[style=BashInputStyle] mkdir ~/tuning cd ~/tuning # $IRACE_HOME is the installation directory of irace. cp $IRACE_HOME/examples/acotsp/* ~/tuning/ \end{lstlisting} %@ \item Download the training instances from \url{https://iridia.ulb.ac.be/supp/IridiaSupp2016-003/index.html} to the \path{~/tuning/} directory. \item Create the instance directory (\eg~\path{~/tuning/Instances}) and decompress the instance files on it. %<>= \begin{lstlisting}[style=BashInputStyle] mkdir ~/tuning/Instances/ cd ~/tuning/ tar -xvf tsp-instances-training.tar.bz2 Instances/ \end{lstlisting} %@ \item Download the \ACOTSP software from \url{http://www.aco-metaheuristic.org/aco-code/} to the \path{~/tuning/} directory and compile it. %<>= \begin{lstlisting}[style=BashInputStyle] cd ~/tuning/ tar -xvf ACOTSP-1.03.tgz cd ~/tuning/ACOTSP-1.03 make \end{lstlisting} %@ \item Create a directory for executing the experiments and execute \irace: %<>= \begin{lstlisting}[style=BashInputStyle] mkdir ~/tuning/acotsp-arena/ cd ~/tuning/ # $IRACE_HOME is the installation directory of irace. $IRACE_HOME/bin/irace \end{lstlisting} %@ \item Or you can also execute \irace from the \aR console using: <>= library("irace") setwd("~/tuning/") irace.cmdline() @ \end{enumerate} %% %% %% %% Scenario settings %% %% %% \section{Defining a configuration scenario}\label{sec:scenario} \subsection{Target algorithm parameters} \label{sec:target parameters} The parameters of the target algorithm are defined by a parameter file as described in \autoref{sec:parameters file}. Optionally, when executing \irace from the \aR console, the parameters can be specified directly as an \aR object (see \autoref{sec:parameters object}). For defining your parameters follow the guidelines provided in the following sections. \subsubsection{Parameter types} Each target parameter has an associated type that defines its domain and the way \irace handles them internally. Understanding the nature of the domains of the target parameters is important to select appropriate types. The four basic types supported by \irace are the following: \begin{itemize} \item \textit{Real} parameters are numerical parameters that can take floating-point values within a given range. The range is specified as an interval `\code{(,)}'. This interval is closed, that is, the parameter value may eventually be one of the bounds. The possible values are rounded to a number of \emph{decimal places} specified by option \parameter{digits}. For example, given the default number of digits of $4$, the values $0.12345$ and $0.12341$ are both rounded to $0.1234$. Selected real-valued parameters can be optionally sampled on a logarithmic scale (base $e$). \item \textit{Integer} parameters are numerical parameters that can take only integer values within the given range. Their range is specified as the range of real parameters and they can also be optionally sampled on a logarithmic scale (base $e$). \item \textit{Categorical} parameters are defined by a set of possible values specified as `\code{(,} \code{...,} \code{)}'. The values are quoted or unquoted character strings. Empty strings and strings containing commas or spaces must be quoted. \item \emph{Ordinal} parameters are defined by an \emph{ordered} set of possible values in the same format as for categorical parameters. They are handled internally as integer parameters, where the integers correspond to the indexes of the values. \end{itemize} \subsubsection{Parameter domains} For each target parameter, an interval or a set of values must be defined according to its type, as described above. There is no limit for the size of the set or the length of the interval, but keep in mind that larger domains could increase the difficulty of the tuning task. Choose always values that you consider relevant for the tuning. In case of doubt, we recommend to choose larger intervals, as occasionally best parameter settings may be not intuitive a priori. All intervals are considered as closed intervals. It is possible to define parameters that will have always the same value. Such ``\emph{fixed}'' parameters will not be tuned but their values are used when executing the target algorithm and they are affected by constraints defined on them. All fixed parameters must be defined as categorical parameters and have a domain of one element. \subsubsection{Parameter dependent domains}\label{sec:paramdependant} Domains that are dependent on the values of other parameters can be specified only for numerical parameters (both integer and real). To do so, the dependent domain must be expressed in function of another parameter, which must be a numerical parameter. The expression that defines a dependency must be written between quotes: \code{(value, "expression")} or \code{("expression", value)} or \code{("expression", "expression")}. The expressions can only use the following operators and \aR functions: \code{+}, \code{-}, \code{*}, \code{/}, \code{\%\%}, \code{min}, \code{max}, \code{round}, \code{floor}, \code{ceiling}, \code{trunc}. If you need to use an operator or function not listed here, please contact us. \begin{xwarningbox} The user must ensure that the defined domain is valid at all times since \irace currently is not able to detect possible invalid domains based on the expressions provided. \end{xwarningbox} If you have a parameter \code{p2} that is just a transformation of another \code{p1}, then instead of using a dependent domain (left-hand side of the following example), it will be better to create a dummy parameter that controls the transformation (right-hand side) and do the transformation within \code{target-runner}. For example: % \begin{center} \begin{minipage}{0.4\linewidth} \small\centering% \begin{CodeInput}[frame=single] # With dependent domains p1 "" r (0, 100) p2 "" r ("p1", "p1 + 10") \end{CodeInput} \end{minipage} \hspace{\stretch{1}} should be \hspace{\stretch{1}} \begin{minipage}{0.4\linewidth} \small\centering% \begin{CodeInput}[frame=single] # With a dummy parameter p1 "" r (0, 100) p2dum "" r (0, 10) \end{CodeInput} \end{minipage} \end{center} % and \code{target-runner} will compute $\code{p2} = \code{p2dum} \cdot \code{p1}$. \subsubsection{Conditional parameters}\label{sec:conditional} Conditional parameters are active only when others have certain values. These dependencies define a hierarchical relation between parameters. For example, the target algorithm may have a parameter \code{localsearch} that takes values \code{(sa, ts)} and another parameter \code{ts-length} that only needs to be set if the first parameter takes precisely the value \code{ts}. Thus, parameter \code{ts-length} is conditional on \code{localsearch == "ts"}. \subsubsection{Parameter file format}\label{sec:parameters file} For simplicity, the description of the parameters space is given as a table. Each line of the table defines a configurable parameter % \begin{center} \code{\

C׿\q?^_kN|s.c;%Se1*xK<}{ Yθم{<]|^UzvMu[wC?Wβ;oKo6|WSFe{/kCzg=r_ѫ9~{7z=c؝;?z%9.p@_~_~ϟ1O[q<{`s\F}w|sܩ/Uuxj֟j]cΟ#ޖv%vɻo_vqŨî:7oy:_V k2/仮N}mE?<6s&ݱwC=g[_,og>sK.˩~OsGM}f^3=gE[>v={;ݸ+^w;cO>(n~o*T7?~|+Z\X|~0ͩ?C;wq3ovms(7mn:|Czg ;/{캯% 6^MGy矽9wmL]e>5m/z~÷|g/zȆݳܕE#szTuG~ӄI׾znjזϫmQ/*⒉KK{w9げ׭x[߽j-YkpҾxoɻxsGɋ.]~ W7 z |jiwfI|}}^q):Uw<}߆uns/Wtٰ~UG\p;N_̿O=.LǸA9gv Zu9sM {ۊ¯Rv=}Yww>u3 -|B r ~3CNcE`{mNxH)Dw! 3j9zYڎҮ |q+hB"ǷXK" n*ව&h`\ &G_q ( L蠛Lo.@~6),k ;Nʌa2p%Q@PtbE)+N.|G{|iI?}z)%7iy.O/XZ Z{rԓ-ڴg}|蜘{4wl>t%VShƌv[;`7u { =pwՃ+7Ξx9pgWޣ˺N毖ZΟ>^!y_u{n;#!3gerV>w;.ew,Gm#>4z}[+~k:yRGKzϾo>BeO=Wח=eAvm7߽9cѫdS'ϙ&aQy_>0}Bޖ'ߛ/F.]eq￢ac~;b׆dwƔfޱd@/ʯ?c`N+y+GmxQ_ϙ?ME|zhk/s˓]W{YلӇrޫa?Ov =kk/ {QM]9;ξcewƐ.>]g}>{i'MF._WRtۯzmyu3n˖ٽ/uV>.K3i7xgLOq̜d{'(7M~jK/ yCj>y-,{y^9ᗇOn\m+7.\w'qQ^yh)kwz?;E5Uko:7qOU}e*N:5˧ޜ1W,m`z /?lּu%˯}=I?yʸ|]LfIӦ|cy߾n*zUGμh=N'A.dhz$&(eTrIb`R?gC]2ىI)@o^fa $TҶY{ЪhtUa!t;3O i g̉;;޳IDvN@7 솿=@+ n2OF= a d0ly;pCDg(? \˱SGr7ny#EJ^ _FC ߠ Jɷ+F9y@o2,é,wo`<i%@c07\Bo0ʍN k>kI.ƮmevvQ:g5𺃮'0}A L' C (iv@bBp`lMRkfUL{p"QClX%ذJXL 0&`$lH$o1khi9q?* tw诿UmǗ*%%m&Iu NR%"J5sΦf2mhg6d|},X<pVrk#9gi[^߹cnt ڒMfmU,`lߢ*4,I4V W*js"mHaד!f rUQWVۘч2S4%e5vy>:mmj(aݬYɲ#k\9$~(|}4XM)h/ԥ9~h;EO mmup}]mMKN3qLoVt Ż _?]PM7 5Xd m=Jjά@> z$hY]2b́ZOd)Joa$|or}ˎ1kcu{#H9V3<l,n6z4/o u ?(~#1U~pSlty` uEʊ B!` KYo9~fnP7s"f.d6psiN p.y:ӥLm?z!Ctzx\pc9ЖB9[ ޿ʦa:k>DWhPMKA|ăkA=C џ.mUAȖ0iX"6:uԂvQ;Xং 59n05o^=xmLp3M 0-i1ϚVH5O7S.$h Kx3ap),HO+e_>'9*R߬AOCG2/5?#5n,npۃY5fwC9Ò<| s6cs #MDNyp$WIBt5WA0´~λfDA ӸTm+9Gq9V1'{e?aIhw0}/'!}4&0#VͥXtid߰`r'4+x\m!VC<_Ȓ |k~i[dT6LHvfC[Ap0fM1-dׂE(;v6H^IL[yҲn>0^-r"ө-[ ;B-tX_T-` Jƍ pJ3:V bJڀô Z.ӫ֗K$(UzIpͫ*kM42+3zK*+Hvo'yrԞ ǪG0ڿ¸i# HzȲiAxq}R~~VwHE3w%K l6g LC6"!yD, ~9eZO37C i:'o)0}8yo.k;Q;;X-"xb~ĝDp'{LStpg;YӀcz0w||e.K-숸+=pWXq׀pW;|k1D-p7Y6PYG<ȥ"Y*BP. $Jjpko >iL/ S2ɶCZ<7 0\J!a)¬.OI޶,ûDlӠO(n,ugwluڔpX"=tCbAXNbAX)`=(so*{loV.Xr=na'mvy/rxFn1Jafm nXn#T3\[;]4}'3Y~4Aw!Gt%_I?Np8K^ga\M|@0.R{$ X~V'ljoU<`% ,NpߜB' ~xo5`8,rm,[lȋDg>>~rP3PxO>5>~thFɳvr[<\wrTS,Jz¸Gu:ዌFOG(w[Ka_NaY7([K&K qXKc+XNl^hZ A~߉{o;@GPIG]#iM q2( |gЍ@Gx;Ա7@? ЗK "x T :@>P@R(zx $ &L!0-JK/gG`b ,u8]jX0<@*6@*TQ mf#wrY:T*ԗbSZeµpe,9ߜHng ؋XGPc7:å @خ1 ɵ+d?VFg2hd3X h)LF%VB:^ ҷH x\lxgn L?)ld^L&Mߎ$qi.w| 㶆^@Gb;˅.Lo&O*z >%+Tۃ]Cr"W8wAXoLTWgv.?i=y:D:-[؁9}k\A!MS?5vP:S;-AbGy^WiBWBW"2:L+K\7#6h 7I5UL ?tql6.^k7/;7:[gG#D5ƏF.eO*^<5Uʪu,}{Ӷ(* kM/$P2,~4YiN+vD;‰G:,8,afF4MDV4J!h0'Zx!^kyk!La>#c6"rJ#Êqx$ / ֦ A,01@rb2-A󺛱xo'̧7詐@\=>ZF#mLCWh[)5# a\R*eULF e8lw'߾*.vEh݌&rfYEZ^ ( ja0&]Y3_/dz?UaCюm|Fw e@w3t6]ЯRmVС NW |ꈭgnywi oO /uPصiʶH]=q?8之 g #s-Gy!8_!+e{ пH Ž+2a73pܫ :q-zH("0>,-n 6],\MzaBn>p_{?p/ /両5B0O S3!THp~ ih[(u Dc3cbw  =xO& 0{{>|<|~ ybǓDx;\anZEx + u9˳_Qm&$kCd4aҫ%u5vm˟I71-TǯgQ6K6H.]+{&HZ+u̯lB1qܔYxY28]Vo#(Z6diЕN]H[Bafixms&;xIHz^`+ %`MWxگ֦y-~m3gkm2yLϜ;Iý7i]8?IKc6Y'K3 D|V"trVj:7rj+8~?F_˃|jz6v6K163m/𢡄T$JZ?ᾤW[MxՓg#(K"P*"d%`<7Y熮WАaӥΎ89swS%h<嗦[k%2~9>b<\@mmN;5G|#3Ͽ1QuCȿ4an8oO0/"D3~"d_f]}S(45Fc6i&`݁e&MQc8ntv7+MGouʂiGD͝`j{DW{3Kطv-f5j[ PqVւ ^3ϣ^+BAcBn#SFx tKʧH!Z#Vx/!hח9ȽܛQ qZw+!N4uzvƲJMJtV e{z%`ZRE⦸=^i"զ]xٔISeek7e7kd͖w@5Fq_8αPCe~/&2sINEYeS|mW/RB fBM,ќQ28KOSkw V\_t~4t䭫s%Nl [lD:CWaGAGƟ)U,,5["vU'BYgcRs.SMsݎ[\*l**sC En禚) ?4MMx1jēy3\LGG|)Opi"(͒p}7D$7Ta'maJJ_ÅoZ[Q'{!ڱNgքiBH ~dx&Qoԗ%Gpӏ#t| b5-oILJ>CF>d؂aX7:$@r?FwKDvOL{K7=*ݾr-|S$>daL.8s3E>L7ѡ|Bǡ:G8 u-ȋ<<|=Cl)`_,)"?U |O^2Mį\LFGD })LT.fr1 :2q;)1|'ljoU<"];c?NVŃ7܅zSi{ "=a)'Jzҿح0+vma9>mlbIb\(lhuYQg>5>=j;yRǒtWJ<%_Io5kk+u?ų8K:=]q@ߍb:׻w7ŏ=aP"eDyK$Ri\wl0}x:%=)_@M˷ѻɷA>on|F=|:5 /wm`o -:ۑ_;vSѮƒW nUPulPpZzo#^j"V@5kD3|%u:pkwvm$=ł=/qQmD=m3ʸxSpʒ:U=iQ-PcVKn5lN6rމrBnpO/e NZQ0=Gn* o☶wwƃtLpW~:TA,Ʃr[9yb*rޢ. "ܸϖܥM &BExh'kEARՠ qjA'lmdK"+E--wm56re6ЁAnz8@o}`?C>`!0p<4@ !-X&`Z)fpfBo`>򤶘QTcI"@[,#P`% [`#<  P`oA P PClw;U{~'85p&<p.<?O \$p\p<nXյ;ܥx  L!G@{?Fqx6pQ w4% JM?*=["} PO]'B_8KmKx02'Gu c#;S 2w3UBN[>$I2^g8rv%Gr]@ΎAx:QAx&x|>>W]т =\4 ۈQY]o42;SDaljke{|7s|RcS- nؠH. ~.ux9taowENhGYa#f*>j/YݍsD~`Ur2&hW $=<| ?RH1ž>Y_/}!0y d'>FIw>-&{*,qφ@Q^,w_4/,*OkX4VU6U5 [3w_ <?Tl(V5Q4N~1B<.*`|^Oxo/1GMr|AaԚuJoβGl!7H[ ;cTv Te!1VW[,u^omkmBmeˑ`a$}_m2o_鶵_Z|+c6*aV@piګ,Oí6c6'Ѷkl6@:k(0\WeօcR61y! -/_F[‰vmoWbh;W-S+;31V;dI;;-u? _TXdc~ZtX?VhWYƩrIZ㕯 Զt ?KS<_xTh .+L!-?:p1r;Djt&l%{P?vm#բe|ֱ&ᶨ亴wEs5<㱳veӱAYY(Q2H]umS,ZQ]O&ԕP7j=ⳅL{5&Eإ9wnz$p+tz L 0e%l3+=3 '89m'ܷ*h[72 G!"*??ΩdƏxtkوzT&' uڸ`r׍uٓ, 6N/0 bRi߳3ˍ5>?J}ʙ-RURX.=[3`,t0{QU,o`/9,Ӷsc(傊B^䎡(/EchheheR]`,䎃]Ztݖ7;E[55r;_􎵼CewG{8QO.k,`͂6|v|=_xԨj#T\f;'D mk('uQo*مX~s53Gj,[.h lsWP/l}^>T'&e-_8{?J7y(yQj,jQ+3^ƏS+xU :*X䪢$2AXr\]Rt]!zHtTRvRBaߐ"sO6S dWM1rf|z֜i*,Z!HJ52M؛_\t e|0BݜKWvg{?۶yj?L<5v~fe;{X=_x o85e4Ba]}}XAWU W$-v_"_,Ry5Ӹɥ:,8QRYrytEs`|hl0?- Kh51vBuo[I@u0rB UHi,YS=-7jB2rKqb6ޱ{%^Avq ݠcM*`?M̱ZP25"ύK[+v%{Sd.ocMl"Y܎$Ut\L'B\{؁[.Eu3 yٓR*aNew4U\Õcɬy Κr6rwhK-[2`}@*?|#^#|m|mxh/^{co~`_o|'5>]\L 7igV2m}h7d;fneɖoZ63u_uk栃J_o1f*閩}ͭV#n4|`,|76]KNՌr3eSؠ628K6QLkkrnԚ;̣iW'˯p9ttd6ۅ ?1;zSz O@Hb7MjzŻZEf;}U.w:s*;pfZvo w9ww0+?BhT.ng(+|Gu9z8'I?>L.z&p (Y۷wi@ttY@HyrH`ʻ}e{(oM!71%/yr_xͅĚLSv4<.'m#X5K-Ot饾ZN|H"mex6't4~ ('H~$bFGvS~DD\R.nT{DDewX+n8n>nnT w ;O^axl}`/E9G.Sd(t57S/r\Y1Wn͍GP "_0 }2}/ p{At3UމAq$HTa~SݚA` 53 XE v:tP~\$x}8?ON *e%ô7L Gc;)cc 8>A~ ?z i>hq/.?=vYӸ|r(86!Q?-/0>/ҧZ_Yp&R9#qL;pG nAvy;pڨo\Cp|68{qFƝqb,| b,1ṉFp±#c n4v6N׺_'a0v]aDd-Oq냲?/15t2:H1]"4d бݠuJ?jLy=tF> bK|p0[Z8]#IZlpBNI,#yzkHxʉd~ۓ)ƙ4v9X :/=::3@nE7j;7"" ܶѤ7˥Fಇ?Wy夳0f^w7vqN nOpWߎR%{ sCs[v`t>x݉ p`pl@gp =Fw1' cv%[OH\Q˸dR=o*G*JWa<Z I82.Ct5dzG-,fPO{UlsLItBo!^1}Ąj5܆ۃ$a6pCg٨"\/8ܠ#2 dT^!f9+V6IQp1T905NRǙ>#/?Ƽ> )MAR3h߅641O'N|m}ҾGӟH&s |EzŊm!87~Ÿ"[ ٪-l|q $[];6ʷ$>W #N}ԇ nq/MAWaoW4Jf:qyFɈgmIMl䶼Do;.oHY"6A9~{nG؃‘u+S#􈃝~oH8<aGvDdf~3GڃǼ];Gt1OؼR!ޔ^/vw{َ_TO[<ryzGghM;kh|kEüp' ݷI }GoГQ/H%̸ج!,?^B9a=p m#<҅)~OyXz޽LPӱvtpυ}vό}p7Ճ]dFg%uapymc].J:t߸ގ؎W;ҸoUfܝǍ)AƖJ\4}7vpyzz>ҋނޜ }#}]ȇn vs9:oҫzK"rmKR4\ v_ȴ,t${B Ei./@} p0%b(%XH}4NO $ ^(c`XM-`"c `N&`e)V||0dtQ0\ QpW s;K[6 -nO_{a>G->I>t_5M[Y\G0E) C:Rg.r <77]cӾ~tľ;ɫu^}nzb\u=/*(wOOMꟂe}CsK%2?Ek'Vhmy zuvhlQI+ΥyWz'qS;ֆp.e7=D5+/@* o?!Ӎڷ ho= FZfObu_xo@Ĵn1q_1N_N_?dp-!ep b4b;`yp9Y(>a1IDgQoMw6sZؗyH|+.v{ CQkGcyw_Bp烻 ܥ.2utSLD(q)\Xg^:v$x\gFEAS@GY8źы[^cujH{GoywZ_>'S <$L'r~#I77 ҠY^^B% )n$TAAnu0r`BSh+Q.7 2x1(p Hi0&-9o&Bp쓀+\'>Kzr,ߛLJ>|i2w¼ֈS&&6 d_.uf!eY s/p<R/91VҝdeeŀVkׇF A{,V)td ae"rRh7xV*pꅹLՁEDPE6j'}LD Nz']@&kˏSA2&rabmtnѨw8>[~'Dy9kkl;b헭E|s֖#b6S[-ְ󑭩=ֺxVTJ+;9Oʁk6vz˝fNmEі{jyqy3}y'&ey|aE9]2Kg%6(C 䚺@A`Kފ?Uȟ}ğOJzSHt'FzOo}2ٲ~z2Y>OG O3ؖv\1t{݅|{z zqKe|&Uz*{1ǟğ}B×>J'jGU<~|^H9ا:߁¾A¾j=lD"#?Ԍg !<0܌%~P (=7Bط_hGG9(᩾{@(nh?şihXXxpƙaS?og6>^0>R"Lj8䓛~&|%|a|8Eqm%LxQ1x̷1qox`x-Nnİ3.~Հ-ՙ~A$aBOH:r5@6]rB4r("W2>i6Cà/\5d)3mHs|Jc85tNilߑUTO|E|;3\(1_pUꝸf,\|`зJSLC9;oC0¸,2u[q>23V+K~l0pyo5zrC0Ot?k7ôX-7`Eb͇$aD=H;yvS9;rNoȣ-3<;[Ì_қ.oֽ1Bl?Diws ;8d{^g4H`p0 ` .K 0 &Nh),`Y0h~<%0XXDXC`gXO% l('PJUPC(.#av;.8@q9*'ɧƎ'hM{RbV ]{6|CJ:ҿu[-ź*b:271 AodX0UaՂc)5t #}䮢DwD?FmQaD HQkqvsR3S=Y'@~>e{(\'nMG*ŏk}>d/!"nTfުy']B zG6Nw(T ݏ[P+bW\3|Cw#:pz\A,ѓ7. v׺cC|.+'ICùK(M͡KXZ{/Te5~-1GՂw.k:*o28IĶڥX*oyN"٩HbtGFGIpj.jE=$jY{x>]-f|U+:8Ǜ@\gp#[?tn| ܦpE!||%eE FIte_̰[k 13Պb!dH+_Hw*)TN'+V^0]ɤ+H\ڸB,kYLtt9 BەRJWGqŴʈ~J V7i7(#ZEV+ Tt(^-\JppeWWeI4 `F?p%=tg2<=][?!p6  ;. p! -#OKK.'p` O\C5p->HuBxz n[`u;m3>p/ <3-6[3@=Eig\%< +O1xb  2u_BK)21?g0ǐJH\oJ]EYeIJ ͌/?*u%gzr(gofta+] f:,p<ߗ,Zs:3 9 ru`9K] =!lZfp%E}A~~9xZr y@c):[5p0fR-({a-{7C$T3!OB>,% YBך(4[-0P#߲sg{zLW., nؑϖ]w4{2MM$&eb{'<>ϟgw!ӱaSV(~33唪eS%z,wͅ^mSdSqvy]Mm@v]CBt9C?+"w* B]ZuY`[o,X^~$ӱ+#lmU&d?4W]xtN`aovMmub]a_`h Z,,;3fzPSqRU]Y, :AɅ2j\#0ՕuU,q;uڛ3yJZWTYZl 2 QsNpVR*ti8kUt,wVX맃 .̟E!g7{w9%}v@R!7Y@_Z]QY1tC]EreEvme.Jla9ZimI LA9hD}8C5%K\ ^B{mc珴|eU"VrUaϿᦪQzÄ}km`Aч x>$"gZd`&sKp[ \&)͂jkn:43}l%UY,(Eo9FĸX̲H@&#lbH)( Kbc*]n7pÂ) I1lJBg/K#\4_Qc×+A!?(~ltZK;ZZHE$?_} ȧp;B81_ǃ7"1HXgp Ruͨ%d?!z hnY3 NrcGGHp̓Ϥ,嵐X$Z5~n.Hk$ވxa,t|vlRjGG)@'TE,,kI>0>L!pim A9$$ᲅiq t 2 }t!"/d? '~Gs_* YXr 0g#lQ1ƊO&8@!l L֟AYiv0"0_/eZ."?+BG% z VY~P啰m `Z#[w)~o!nx0 ,N֬-NqK|(F{v50/rjJl '\QdI zv͊_}I H#ՖW [oɓ[3kEs/3rdnA0PjGڇd2=l_UMuɆ`Y_jO3H^ T6?& M0OC #{9dryX\T0eph|uI n7rCT :xʱY˶Me%f5Adw hm//2l RIRõZWӅ`42k42|W/)/11`)r8K+: a|ЉGTFCShtOҭ`Iv! _h /9!25DF5 -l`P<6s[.dJlkr* }sx5'oNʰf\R%'LinmL^Vou" or~iQRt|O!C~>mv|$~[rI,a8( 6I!]AH܊^{s:!cvee%u jrq6A't1.ѫS[d-|Z/CqBI6kaA w'r>YE79{CD^ڋE?t2wC9нub'UZCqE p@.&[+ANeOĕ21 o7&}j]>gW"?aQF;vxŶ\%~l{u^s=uĭ=GgS6X-ߓ V(Jc`Bo6 KTVln [zMٷ%o8c~tXjB06Yl*]A+Vjo.g=o]<"@m C+CaxpF~ה}H$H ̜RY^(- QY֗IHIsn|cYnyoy%<$:D< 'I+aiXyh=tߒG[fp>+ہ_HOrt9f9l3 a2mw =n_pqM|c ,gfA7xJ9@7#y2^yk0#"*V8FC"zViI](W<y<^hMH7Cn: 2e:rY䟧H7/nd@[doԓM0ؗ)bY aE,^vIe?d_ wȶւgqh~=%8/ӗ1\d& gt[&dCzIw08U8TV)6m+P jBwZS6WGl8;v ] g'Ȼ=?)0CaSv+"vƐtr hܳQ9Y#v\^w}0 BW2m/0A{9We(og^ W s:$uZ : ܟ{ýQD֟aB[Y[,x/m> Y¹›,:s':#˅+-3A{} A{oh| <<}Jd iBDanKhlf~,V }}Y@IC~7fRbgς=}=OxQ?W;䵖zx1^/ :Hx],_xMV߀xR?/[ ۛB{owmdzxl}ǧh%O!a#B!(Daw~ LxP3) Kȑ?~vw? 6HJ@~ ? F'.áM 6:Yo@?!g)qq(` ͱ|rLa r|vTght3NwGL2+**J\k+B*C$+6F//ǁ!;tI{qk姐1+2J 7fIlEivM]UZ,XYIӬֿ, /b7kɘiBYîH7 YCviQ[tzhVBr+[&ajϯ0)N23|\; {; rx.Jjg9(NzRu.Ha{a_,ܺ:-uh7sNfM΂ GJIR|NF.$N帛Pr\!g^?یW\_~j$.ʩ?ܣVs|n䵒{Bn?1;e'PVۡ&Z;qXsǏ3ǯgrl7:Lys<0NL4'IxVov*f0`[9e.uo]R]biiGD`j{{wX}/˨q=G#mJ0lDrJWAx*_5ϣ^+BAcBncSj tNfSڋE yE،ě|vV'hח9Ƚܛv|'hW8ȽқvMljfmk;Xs[:=u=_Sa'Z5UtKSETl?f|Wӝ&KؽLz._*bM}rmɮoxQM'bɌ?jWPuo J-[=+zfottQe zT7tUOJopUUtI vrW+~Scƿá)!Xיſڪכ)9Ȼ6a_w{J_~h9^y&cYY4"mmMu偊ښ!jk7l8d돓X2If?Y07p***N`LcT)^x&$|6˩nP0FtQg+8* g@FJ_Å0p:vl%jGߚ0@)@{a~e0iDVy* Ғ4 DĹ_ ;\My:zDDBXL2G٥ɏoŪDLJ>z[2l0\ 'e$ݯٓ9 H]m$3S2-tX_/"0P/Q酗R`~T1O~A ڀ_@MȓҞ2 Lfaxрx~!W#]e]ӄ.e Kץ6".)dxFĮLfǪ*, ѓ SV09wzӉ.̷\*RX 7Ń/g0?5Nx#|>y xۅa:bW+ӭGtvD3MDnZI.ƴ.<\In0x.%7Q}pUO$Ϥ})N&66Au>L7ѡ|Bǡ:G8 u-ȋKZ X'mbKD da4dOTL~(Ri"~b0:"Z.fLa*5?r1CYs$H88N|\?qc̷*|%y.ӟ|OS #O8Uғn̈́_nۗAn3fK OR"pEafTG'blஈ>+9 OW>'V#ɳ8|}=ʝWR,Jz]Np}]sl_Io;-IX_F?ɻNў0(F[2l嚆I6W07[ȯܝ|)hWcWҫ*v:v(_Io^V/ﵩXMߋ)Tx)G\07$v_:pkwvmo$=ł6B `;z7Q/_Hq=^ ,X%?f<{ Z BFeHk . Qk>U߅_*إfwҊ>to_t>B.54c>HD=3= \mSK:7KoY1{@{8'8'rKM &BExih'kEARՠpjA'lmdK"+E--[׭ @km,"`ǡ=@m.p8~h| }z +|'c` 6hq&xV3+i{Y\ЛC`<-xa3`!X`)E,XIÖ1X b~3 6;@9@ATs]Na`՞DN 4j 8O.$S9.s+ \:ps3[qLWvǃ/GzL<@U27T{^ r.,˜< d|*V u"WJpeYmY gB:\IDvS?/C'#y`"L/>E]N eGY6#v)>j{YbsDu~(sZvŵIЮ4Izx~N"]O}-[xxR m0eJV`\GZV"r}4a^[Lmൽ;_>7=am ˰dkarw.Ʋn,!dXЃ0Y{60>Yj퀎 z>fy'Gz8#x+]la~-]wm @z)S)KaARJ[amll.]i.7". WW8 d ˃0'*ʼ!ׯVf1+J/p0jmh]tꁎ-$asҎW$irTҧ#{eS^Dd¥y s0חi`!K L5yF(8SD.l/2lN[;7Dg)hWE |Tض6H:lҖ;).F N a+!l0'ۺuĦk KB2^d,K~7٤Hd>!\Bt.'E* Yd* \Ӕq-d?V+@.$g`zv6[ov O 6M2l'"-L;q_v R]S=$ ˙`,O mfGcy#9!c ie1iTm⤝. њtQNJX.v@ֽK .'q])u_VWMBPe>7vK{.=׀ͯ]qu`#Kh0ʒ׃eFpl7dپ4n_Ӏspn||FsŽiz[{p>LaW1}¨SCN;Sr ? 녾08쏃h\< N'ݤ|OE6:n@ِ3h>wu\:ߋ KE%/Y^?¡lm; {[E/f\hXHvOdJ|[x']qJ4sC)WOsrE{No3і+i 6u?Ш㯾xɏݖ~}˲o/!qS(qN 3~˙E\F3!x9yՏjl s|m)|v ֖fQ[\ ?nvc7.6* WZ{YCW}Sa'Sz'\ rqsOX%9W-ǻyZ'V+Y߹ϲ 6on,hWi9W#Q?Œ4_;%Wm; ^;.>ᆯg~.zp?hT{db}O0e_@QNze8.WExq'(tksղetdWRBa"sO6s^dW}L1+σKf7,Xha!R//O.%BY^3<~P ޱFPIst=MK%f:yj>n*{>v{Vuv9lm UEU7◈Ɨ+LAe"x1;#\uإ:,6M f|9󿭚D;7c1r.G.^F͡XPa3<΅>;Fm׾{8VkɡCˣPn/8T(۠oVU(5uUj[ 'Dl!_TGgw|Uq#|~\ ]ȇn+z§]K;~hN}+._?XRE*_9rj,߿mMA>_m{3;~ښv#c<~[GP1"͝:\OlO|0[ܚe S&[G>=;=y~l% ]p^Ns~4wY8޼]=k ap.wSLA 9{'~DmUu[* lMݜ_iߣj$}/W eo+iXOѾvj=@NӑX}5zUqr} |_g3t\`t^D[o;ČjJR3b'`ߨK o_[ɴA%2~93&s^&stFU :ȽG36tVr}+__WI|wnlP%aj)l|[e.SԱZaN6Ɯ.1ECy<麙[Q~u¾ clI^AGoz](?zCx;1^o!ToԷa=nGXU3+ʎ,2雮v)SفǛDr߿}x rzx /JVU?aW^v|д~M5]zMK9Aǻ\<"v!ŏe Data['GџA?>JwE.+? |AKGjBRV .C:H_n*ÃZ9(n0$)6*̏s[3HLAE&+ߎ@N6Ё@ʏ :LKYXw0D1AD +Ouy;Dqhq1]84X[#Ʃy<w!E,ƸzXb#>Ƕ6xM8_Iu=S1fNuǷd\Z,8>q[Pw|ă/ҙBr#t !6pBvv~ԥK+nH 1jV9)w2Kz2O--ClitT'iiF 8%4Z[!">0mg:`.HSqQ_hxM)wmucApC0p{udNLۚ+Qv¼VYsUL/û np8`nv'}oG^}9)1=vGhw^ N%owvKq= iY`?A-MrU}Kd?-۞LHVK{&e<0֌?ٟzo>Q&B0A!>z3z 4Zh }L-dYwˉM0oYC<ͧI8gǘg4Y8 @8HW t& І&PډTOڗHۛ6{7&Hi {TQO7Ӊ'OFl}t5SGTHmb# ~[qyE) u;xԟ[9wdž􈃝~H8|c.t\K;t۹7Uh%}9Pҁ6ץAKp ) ^oC d@ :=`zw!d"a|_o?AC`m1@ hM$pyk''F/g10,&X0@0O'+>P>f }Jc:( M`.ُcic_mN%qEǖc7'0#@Ϥd& ޭ,Sx#S"!)kc }ig"3!,߲t!vh>3VOp9~x1yib?:bߝ?qW}ȫ>BF=j_գtKA/>u,\I\DR\&[ʗYy ;ꧧ`OADz>ѡm~"h{r+k6ӼMG:;irAljvΤzR+= a㸩 km8gu̗|pG[ vrx_̟FF~7A#-_'/7S bF 8pbp'axp'cD&s2Eo`TjBÐ2n ӄy_j]̋ `¼f Jݜ ,a |$"dz;ܹ]_-< ]$e>\ؕy|_;N|__I=!ޡ5#Dxӱݎ/w!]Rp:}iJ_"v}ӸRkE3/;}<3ݢ|y,_bE-[{YݱB}:e~$=z#Y<;A\i-t)dav& 9{@\?a|ޑ^niP,H/ vZ K! t7 trow H7:Tr9@s0)m(恛Gors@{$4Zpcha|yc7G^!ELI@WF%=9M^BWL ~4l;a^kD)hy`Pk/Y:XD,K9{]M)œG] Ndвb]5gvCqu@ =@n:o2Y^dRV) <ˀVpWWm[ 8\@"c|dk j&b"u'Zd= ܓ.{Jw ǩ ku0 9{h;mrl-ָ#~w/[抭-Gl7. [a;#[SS>ZoUVX~R\{\431}^OW{˛!; x)6'tmY:Ӝ/&PA߬:X (Pk N.U+Kx+FS&jU{>}rt%=)QTK:#G=ا7>nlY\ F=n,dG#rlK;.xBY~>=}= d|82W~*{ e=JOw>K]ZWO *?Z>tS/S@a_e a_e`d6| j3iLG \vnW?O(ב F!m/\#xT= X mfPc?ƌϟ 4NlsPsZOxUO3BA6xuOUM]}qƾe'w|7j{aMS5/1 ͨ4CU}' P矌/1/==B/N6c{VLS2M(ˋe?]!ePYS~؝T\^fWݲW/+/+}0%`ExpMg?YX~yƧ W'Cޡ>."_K' V[\]䍧P/I8lԊ٨%`ΨDF+ẏV}}-7$?ȗj_<_̵1{6:Al2IlrmZ]F,I0 K|,84S:XnQޣp[+Ӆ/?@?^ 3x_Ï<%xj%n¸$ă6TNf sSv4ԁK}!bZ>D݅KoݿtN?2C}(8#/qχL ӡ3M+vtOc񐩈R x">1P1`e[劷v/GqtSXsÄ~s}}:GziaaI/W)ycDM?K^0>8^L<ۨoc<ۘ8Ko1 B@'\o4h**'e"}./*lN\3.H>n0[]%E{ ZZߜ_J!M`a\ :D8V`+KSf SON=qPE5MEaZvA1uopv?ٝ<)~ ۜS t 7і\[-aF /M7^t|ہ]I!EDCV"4a {ǻi ڹzO = @/|ˁ޳w$08 0p%FcS\O'a4i0,fo| ]K ,V,ep!3'PP@%*!P `; `BTSTc4\&=])K1+]W =kp>t!S%_ :bg}1ܛ_ 72,Yߪ0jr챔 >rWzqT6騰I(5֎8C;fZǹw)), zr?2?C[oƓQXk7#MyRnq'Gm-1#n;9w-X p+Es Ypgpd֞Da#-1ce+nr:prbv: C]ĵ>tsX7sd3DWowj=ңf^lv';k}f҅Z].!nnps p_IUXSk]1!>QD_$vܥM &%d=XtזޘVjAvC5exQ$bcZ{Yx<'Thh:#q#ˣ$8 ZMDkf]"[pu t= uoMU} . j7HDFPnP q"HWf>" $/fح5ׄљHjE`2ĕC/$`;jW*]KedPK_.i m\!EZ Lg,&UNJ`\!]MJ)]UbZBeD?]%@Mڍ P" j*:A.]fx2$pNq`ƞ{h3`E8\Ho'ץU jx'!Vي `~:\m n<[lV Ϻ6 m p  Q-yݞ"43.ixTh^xSJܺG1*FO sb9s791YWAx0v ܮg ~:CJC8x^;`69j7b?wԈM~ς. v[+̻5`+ 0M0N?59B@^ xHБ2ev=qJxg@tx!ex>@Wg0>O3sN+YZ\iOO0n?i|hf'W +w s Вs\dCO?ԁ0 \^F2(a] ex荇|$;/̷| S K y˜7J沶Hc!0n asגPτ74 ` ~ U\zg ӱKSV(~33䔪eS%z՞*wͅ^mSdSqvy]Mm@v]CBt9C?+"w* B]ZuY`f,X^Y$ӱ#nlmJ4X\/gV+6:0LYv6 qs.YҩXN-6%ػ+I:V1$$$ %B 5H!Z{ogrw޽ޛ7}쀵}:7[ DcxS٩5 05;_0a'k%pg9rGś{M*1qB~.ۦ=,J/nbP78t\b+};ԯ&zo,qq~}~ok9Ko[@ u4o?tv_u~M7ŗŗ)~Υ zp]ehzX):BP]ZMW0D(?|Q9r|bNwAr=C>IXT)|[P,RzCYH}Z#xǮgsp9"ъ~t/p&. h;oAg>(&[#< /QcxH .CG|hـ @;RZH'L8GPӅ_复,>H,Rz#&ĸ G;((U D=T*,)RF|d?R/,,`=<BL ƎuW.\ XXh 'Ky-4HECCUEF',~һHWLQODXxѡ|jàゅ)MGМ\goaq9q';T<H"ۊ` 2m Cl:i8gy{>{W< 0o- @7?5!O|/ Wx_B7Q X2rxVp4*пx4h/>> KM]!+! `!n ě7HgE+VU,$Rh6}`Zt#kw N ih|ύ'q>;aaM-~MgAK`Ffc &%:8֓T~Wpo0V%x?G? .:ňun&eˤwImPEPPrPt2H;9nW(iK$=H:hFp}1}H_nr d8(4nO(0'ѱ# xVA jihOߧ92aCتZ0$J#Oa!X swM`Jļ=yC`-Miom46wn`h2O 8'tq"hu 9$ejmbAfï3uK^N %OLIlQJhUo*fCFauG1p vW3AsB-9ce,G[a͝_[۳ޯ:Iu); CIPY#z3K֓8aQ>&aEjZ:T3 \u؍gyD?w2\$ќ PI[=`zO^ʇ =vxB!De2ĥӈ~.GmqoɄ# HGeG=d/<'zY/'39 ;̟yR)HyߕJ6L(]xdŅ,-xTJ֑)\>(yPVA9r`]`7'E@XqM%'屜yxzļ*c8 "y6m&0g!@ Yg3*!R+ S-J ,jP'PUz:S&@=J`ؽATJ O@b/<+g!Unׇ7tcr[G4K)?1K.c^Fs;s[H*f'c%nάg.8",t't;9 v;s"tJ7DHu$P:薏 н(&;cB􃤰;c!c_8׷uZci_ox/ϊ6060Mb j]>Z7?L-]SW Ġ.c!#h5uY0/Z3 ]B2ID:`Wa^bq߂uj!8±X"vm.mM{aT29/܄gR-J8*-UଘdJaO[`%M-x^K?do52}ӆy2w R><1A}ŖA>2k^}@@g9sȉ@6Xq}[IY bPk],|T ZџBc1!nKXcWmYk5üe$:,': b`OH'b*jv%sX_#ՃGqrd~`93yҜup*Y* AIDEV&VJ!+P?^SLxH9~.@DsHǭCĖ~EB`ŞsjThfF|zhif!pYOTLbOEWR9~*3N֤OŶNŎ//È/(p| +pE#Vt%-My1t&U}Wm?Skm|nHU8E3*@b/EWR9~i`L;*IU_TlT4FX]0t}\Փh`C3`z zr(\8ECQ5j7[nvb{i[|f=HDVK8)ADo{C]>^-#h~bѽ+,3D(+> k[.bBQ}EP8k@\ x -~ RL"QGB@#3Aq௏ 8  ޕl N-@L>>%D{Sf4B` `| __ȁ]`)7[>as9X Pk"h}VJϑ|Q΢FG!F"B4=„@L#$̈=IX#<+FZPW=~>> uŶ{cس~mK^fblc<#YtO{E:JQʩLބ.鬯!.ҙ^ў @ ~/Agv "^!F{:#s ơ::c3g&}C@x/ Nz $&qstˈk}KJ> sY6RqmJwdShlɄwPl%J1O'Տt ]ԗT9wÎ[MHgL^m5O?Ht?z(ȥ)/&w K0ںΉuQ< B3s3,DxVA z_'s (7RD  ,BJ.ԎmB=.j$UIQ.0($\azQ(ҧkEzpӥpӥcxS(CJ;Hթ 1*yeV(rqÍt]#w,_x0clg,@N8_ՙwGh?jLb#OM:E*DnAqrB(NRԸ1q Zp_!odZT9 "E:g 9n Bg';!CgdOǺseFw ?SßDsI@:9!OPb@Ñ>hCJ:K1/ :Da]s.2X;;ʹP9 7)}қb ZW$6@v 0M//t~] bR Dhl-vÄ._&0]Y76vZ<,6_ $GLMbI,-Ӽ]=qsp%DtK䐴Լd%7HS{;Fz4H=, U%l a\ TKf=nN7 >^ sÂ?RwG頶76Wv]s磻wX@s.7)1qrcVs4/tuT[,(M /WNُ¬hDcHk,׀a=MfĚQbN?VC*=yu55Uc66QJ1^婩uUqiWK(a}" Rd 1vi-JA=۝x]~p-RLtjP~ĭJ>%~#Oqp.F!BL?qYg gyIN*#p˅nQu F6t?MԽvAdo!V`I2|ݴBB8$$W4Km-VEA's;}-Ӧ7duUuD%F(YnV1焮Dں\xq;! ]"7(.pvyi8=]NѦn4r95A.p+ x:A$@Y ZOÎy9r)k@ BcPǷur4<3ђ5IJO}zql7 c{d:gbM= 9]L|x&(͝>k IU꠿K)տ:q3 jb00oOrp"C'A)I+ D=;AX5'!CZAWB?.qe.hk&+p G{ ^\?HsM0Ryѷ;^B|8K^0?T,x ѯqz o<]xz~8YǺ1|!aul\ x Y Vg/man [ ^c#x1@+7YI JM "P}b1)f!7D_)3ATgU{JDHJP`(R}+h`4n Bۀb80(3!baz8=J.U1՛#p9i@_8fu{|]F1pyɭksGyo̓BhI.uwlRDfu0i4Jat in)r r>l7o R%r _+tL" {4wv%jZ¬0O[–vUHvrNe.4 _uNX=(&yyLĽ{Dmk*Ş)2v<>,B| .&o.Ѳg#}S:M6 {A~噴I 1ђKEuOH (\21Bm}s3ng ?GldknAc#n>uQq{S bKSӟ85OI<iz5o:/05%oovtv=}oϹ@oG+Tphsh 7\ qns(# 4;pBT8%VڨVAAJu1J '7 B™9dwf_{/pIrS!hэ1ZtWDdx{S baՎ/xZ! cdޛ-Ǿ0>YT=T#kEJuP bPءvXΥhYG4JVD$l9GXXj#5{ X(EKĠ'*셌m=O_s3D JHme+t7],z^\M+g6#8,4K 4]}gZ3pt rZ:;W9HFht:X҉Ë+zZqFxU-p!j/pèf-;$\%. :¸wI(ǵ1ѝ*pJxX]1c#pꁚv'i,{$ۺA-@;|]j.EМj^#r6'TlU ,]#AXdmClƖ4գ [Bv7 EuXF1SW}X)V,|b bnei_99Du[֡D21j<.+ iy[Ƭ.ar&t5_=x8@Wp?J13M4NIt29,r8@<s@8~p{9A<9( J9(s Rf:j_/90ᤙ239t.~=Hsx>̋`[&36$`o3y@ 0 0G5DJaB0% 4€ˬ0A|!h'kU_Q}2gr |a=*Czc㪳&N|Sl>~+8A|U/iݾ/VqKn~Otm1o[tL0&p_ eV1V8ܙÞäK3{ˬ<קouM0/ˬ-?`olGנw̲"yͼ[gS+>2ÔS3K+|z#YYcCg:Ǯ z^1]Vܺ_-33(f@*tsL/sܕF7[3=DN:{[{z[Z=ޞN/?]m^ EJ7'u͗vgQ|Ϙ 8TLo9{s}͊7XU6n/6-~6gw0WzsoW?3E͒6_d>9T,\ŝ3|(rFר`)*Z۵m'@%>&ev;|$<4ͦ@n׃yOoOCr:ZZmDϔΎZݷgZÓi H0,sbՕ'. g.\7`6Zi1x x"IqDF ) n\OtpO pLp,pw={{s=s=1{{!ʀQA"_   |1/pm—@_K |)/—A2_ ,? ς |9/W@ _+ |%WA*_ |5>j_ k | 5TgC~67Bx#7B8n^oZ 1V}}}} }oM F }3o͐7!}3Kw:+V\xbsX|ZwpZsYšXP܂J2rYǵν;< )Cxl#w2yu8<)-k[PRrv .s:T:pϵh#Dte /ױӲãqXs*SNJSP)ٜρ9>9!Cc֪Y?pwA:9ˡ<8h,O(t:FI>pC=e)'uMS Y|%فkPvL>)K/;rjқSaoK2G<4vYr՟.9'>Zq?0_NQ WJGՃDO 'FNOqC^O;ziY,rv~Kiw)!{.W(Z-kɿ9ߜC9 5X F>-9{Qiދ嬞Wnt ]aީ/~:~ki~{lg$mr{釜9_MOz(muCQmA5rbҟOLq{UKGՈ%3⥃7]U}Kpxʭ9G/RNQ͑>@:QeK+|CXEIۤ }}R:WIA9Iv͔WS'n>e'rڨҊ&,ԌIǰugnkߗw' ݬX 5crR쌗NQR__tKoPp5(g~0Z-v)}҅$%ug;_Y/{۞^h_xtOvÊr"ՓI/~+/<+S?H;qB.v)z{K>W#E{.rtٿ@>j)r;m*''4 i|- {tϾϧi)ltڋ֧JW*-% i^'|"|[vӳ]+WNzM'}=֏nSWO># 4 _~^N飨r;3T~-kftշz{Nyt!w8@v}ez󛲔t!h;uZ5C 2C?5wJ/Se痓oV7v2eOj ]0G6?%g;'Oir&߁X[o3%u%߹(Y@;'gZՆ {Ng_7WNV偗%Kέe q9{Eͧ=tl)H=rߧ<|h/|Ju: LzUn:u#}66Q)jk߆ycQ7-EmHҋj6k]j\"mql*g/HW&}Ul]欔v(= 05Wʣ0 ?d=M ^?=Kz%Ko?56nHOIBu|i[ix:HE0 W]f)rr[[evĬQ,i+kw~ߎ9Bz^ie"z͗!>kV0.W%mcH۔Nl쏤]TBz;KTw^)ΦҾ ʸW󺽗.r}yxu z3hUxnx>~֫/zI ZVv.%ȻvH>nkᏱ8逺5:(^,,[~5x_\|+]M]^һ`}8cv>Xu^/ưV~ OVy_3/^ͼ#mң5?s߅}'Wڤ۷(҃TAtVǡ|(T #Oo5I捯Kl{q/u=[sj_w8ci2P~, :bbO,)}Z,Mb%lP" (p ,P|B^ ^U%\XU8tRyGG7,QRc_8 V 5̩nC$n 卹>r [Uv=tuD:<ֆNڝ1tO|Y6p}mT?F_lWL8 ߱A`n'X|]9[?vK鈧nӋؤ"|9_nG/1/z1E>"]LDz0[%W`1_b|]҉tx|1dkµ%1xwuE*|%]5./Ӟʵ R{ҞV{Ӟ4@ GƣBQxThWטBFwn%WL LCFF ~q!ԡe!dѯezb=wE,~WIkV/yxF?n #?-ѝ{똜n$&Q_0:2~-XjƯZ&gLO>njoIމ>/Zҭazv'W!{5KW\2NVq/eF{U&O70zGľN/qS'Ve=+a2~Kb~&YY͌;ZF-?km`baz^L?`~/ftWS3\}Kw[LIVi{|2=~V|Mu32y,^=L/332o/dYSd<&ke?[̬ge>VMޔ1zÿ—^2@;[ g dr!.ncrlg)WMwFA/sL#~|YI0/敁qi{?`y,7&O쪚{+uSX7ߛ"Ooi¦)+V7qW" ^p"]t[G5uNzOIO7vN׽vjf)T,/`M\Wva3FV$屧-J2}G(irace/inst/exdata/log-ablation.Rdata0000644000176200001440000000516414324747572017132 0ustar liggesusers lu}SH:wO 6q])4MB煽c;|4mTFj %%ZTZѦ QԆ RhU@w}liڞdy̼εek8pb/B+JmXGe5ngوs`_RR^d$01ER./" #T(ޘ"FEMj#[W zNɮMx&mW l8>> 0MǰE.6 `Xa 0 b؈a0 p2<}xA Nvtʌ:LϷ 1f.~YbNd&˟$0n ۫wVş,43!|ş!*mS$ߖy(y@wہyKwMtwۀB39?A=yゴKVOks@:ß^VqNK;Ά\q7lȴl;6ðDuW6,n0&| ]}C |J5ui{xt4fa2AU#ҞFj*DGa: Qa*G5BtP.#< 2 ,|ðfƮ&GDǸyz:_:ru[ρcWۏWO!;Ol  oA:_<Վ#1}?^/Dǎ Dr֡ѽ~o?gY[@niDY<l-F7Ƕg@Sys߽okWngq9o-9v<K~W~zP^VyT?w_8]DXD`;P|V{AB-=8 pa/m7t}z+[-4.ϼ z=wx`/_;`/ x-Y pBq ?ꑞye q 5@|s7(\!ȵ{׀-乲[43x9|[߄CE_9W9~KoA|m_:6U@ywF_0 v9 u@Kw{qϽ(l`fD7"ԫ ?˞{݇/+8:\͞ط8E[ KW@^,-JBBq \Fހ|*%5 WM|8w!hyw7k|Oԧ`hx._yr/+ ?q6+C\4݄ڌ1y'_?s9%EH^e24A S=O҅aTsj?RF04Q QG)jt[B|6\:c/pp'h:w>"HQ~z^XXxWw|oC?~9IU.J')N(E1D6kHxyREOР;1(LG:jf:jV2li7Ib^hlx'.YE^3E&XRvhq} FnW\Yn؟fjr8!$4,Tdb:\eNqC r,d8TeXoH3Y3 C%NI#JsTfјFg*F_KBLcMK6%EMJq(UCqT9jQ)ֻ& %jZi%Mk\9$56,i6&F]Ť`i#wS6ihNauJ2*!)nH)]!D7jl.["f̫Zua"s@E8U>bIJlD  a$ƈd$$Ĩ-jza1`W׈l44*7~Gs[YRB"%L*(DUS1rDd!R?O&y;tI|U&UEY (Y"J.bf%,~QĬ=}AS$'R+v rt&L2 hĖd }O~@~@WM~& S_*NC+DtnbB&|jۨC8}Er$]L<-kkM@oic޴zx=bƁq9JBOBTͩ6yAuN y*b&irace/inst/bin/0000755000176200001440000000000014325242462013066 5ustar liggesusersirace/inst/bin/parallel-irace-mpi0000755000176200001440000000667014240651520016460 0ustar liggesusers#!/bin/bash ############################################################################### # This script launches several runs of irace in parallel in a SGE Cluster, and # each run is parallelized using MPI. Execute without parameters to see usage. ############################################################################### set -e set -o pipefail # Find our own location. BINDIR=$(dirname "$(readlink -f "$(type -P $0 || echo $0)")") IRACE="$BINDIR/irace" # You may need to customize this part according to the setup of your own # cluster. QUEUE=long #MACHINE=opteron2216 #MACHINE=xeon5410 #MACHINE=opteron6128 MACHINE=opteron6272 MPIRUN=/opt/openmpi/bin/mpirun #PARALLEL_ENV=mpich_fu PARALLEL_ENV=mpich_rr # This function launches one run of irace. irace_main() { # We would like to use $BASHPID here, but OS X version of bash does not # support it. JOBNAME=irace-$$-$1 exec qsub -v PATH <&2 exit 1 } usage() { cat <&2 exit 1 } usage() { cat < $EXECDIR/irace-$THISPID.stdout exec 2> $EXECDIR/irace-$THISPID.stderr exec $IRACE --exec-dir=$EXECDIR --seed $RUNSEED $PARAMS } ## End of customization error () { echo "$0: error: $@" >&2 exit 1 } usage() { cat <&2 exit 1 fi echo \$TMP/irace.stdout >&2 exec 1> \$TMP/irace.stdout echo \$TMP/irace.stdout >&2 $IRACE --exec-dir=\$TMP --seed $RUNSEED $PARAMS RET=\$? # Use command to avoid aliases command cp --force -R \$TMP/./ $EXECDIR/ rm -rf \$TMP exit \$RET EOF } ## END OF CUSTOMIZATION error () { echo "$0: error: $@" >&2 exit 1 } usage() { cat <