irace/0000755000176200001440000000000013460705512011340 5ustar liggesusersirace/BUGS0000644000176200001440000002521013320453242012016 0ustar liggesusers------------------------------------------------------------*- mode: text -*- #12 ------------------------------------------------------------ irace --check or checkIraceScenario do not work with targetRunnerParallel if the user does not use target.runner at all. #11 ------------------------------------------------------------ The semantics of recovery are not clear. Should it recover the exact parameters given to the original run? If so, then moving to a new execdir will mess up things. Should it allow changing parameters after recovery (that is, what to do if it finds a new scenario file with different values or it is passed additional command-line parameters?). One possible behavior could be: Recover everything as given originally; if new settings are given, either by a configuration file or command-line, override them; for settings that are still set to default values in the original, recompute the default (because overriden settings may influence the default of other settings). Report everything that is restored and overriden. This means recovery should happen much earlier, before reading command-line/scenario file. #9 ------------------------------------------------------------ By default # trainInstancesDir = "./Instances/" # testInstancesDir = "" So not specifying any directory, like: ## File with a list of instances and (optionally) parameters. ## If empty or NULL, do not use a file. trainInstancesFile = "instances-train.txt" ## File containing a list of test instances and optionally additional ## parameters for them. If empty or NULL, do not use a file. testInstancesFile = "instances-test.txt" defaults to for train instances to Instances/train-instance-1 ... but to ./test-instance-1 ... for test instances, which is unexpected. # FIXME: Use NA as default value for testInstancesFile/testInstancesDir. If testInstancesFile != NA and testInstancesDir == NA, then make testInstancesDir = trainInstancesDir. Maybe give a warning when doing this. #8 ------------------------------------------------------------ We do not re-evaluate the elitist (using which.exe), but we do call target.evaluator on them. See this code in race.R: # Execute commands if (length(which.exe) > 0) { which.exps <- which(which.alive %in% which.exe) target.output[which.exps] <- execute.experiments (experiments[which.exps], scenario) } irace.assert(!any(sapply(target.output, is.null))) # target.evaluator may be NULL. If so, target.output must # contain the right output already. if (!is.null(.irace$target.evaluator)) target.output <- execute.evaluator (experiments, scenario, target.output, configurations[which.alive, ".ID."]) The reason for that is, in some scenarios, we want to normalize the output of the algorithm, so when we evaluate new configurations, there is a need to re-normalize the output of the elites as well. So we need to call target-evaluator again. However, when recovering, the files needed by target.evaluator are likely lost. We would need to call again target-runner. This is perhaps a design mistake: We could simply do the normalization internally in irace or create a separate target script (target-re-evaluator?) that only people needing something like re-normalization will use. As a short-term fix, we could implement something like: If recovering and target-evaluator is in use, keep elites as elites, but re-run target-runner on them. (Perhaps we need a new .irace$ flag saying: first race from recovery). #6 ------------------------------------------------------------ Sometimes system2 does not say what exactly failed or what command was run when something failed, it just prints: "error in running command". Can we provide more details? #2 ------------------------------------------------------------ A parameter like: a "" c (0, 5, 10, 20) and a condition like a > 10 will return TRUE when a is 5, because a is actually "5" so the comparison is lexicographic. How to solve this? - Fix 1. Check if a can be converted to numeric, then force it before evaluating conditions. However, we cannot know for sure if the user actually wants to convert a to numeric or not. Imagine a = "+3", then a == "+3" would fail! - Fix 2. Force users to use quotes when defining categorical and ordinal parameters. Hopefully this will make obvious that these values are strings and cannot be compared with other numbers. This is already documented in the user-guide. ############################################################### # FIXED bugs ############################################################### #1 --- fixed at revision 1590 $ irace --debug-level Error in if (scenario$debugLevel >= 1) { : missing value where TRUE/FALSE needed Calls: irace.cmdline -> irace.main -> checkScenario Execution halted #5 --- fixed at revision 1343 Paths in the command-line should be relative to the current working directory. Paths in scenario.txt should be relative to scenario.txt. However, currently the latter are also relative to the working directory of irace. Example: $ ls test test/hook-run test/scenario.txt $ cat test/scenario.txt ######################## hookRun <- "./hook-run" ####################### $ irace -c test/scenario.txt Error in file.check(scenario$hookRun, executable = TRUE, text = "run program hook") : run program hook '/home/manu/./hook-run' does not exist Calls: irace.cmdline -> irace.main -> checkScenario -> file.check #3 -- fixed at revision 1101 $ irace --version (or any other unknown parameter) It should say "unknown parameter" or something like that. #4 -- fixed at revision 827 $ irace --hook-run my-hook-run doesn't work. One needs to use ./my-hook-run #7 -- fixed at revision r882 ------------------------------ When using MPI, if hook-run exits with status 1, the error-handling does not work correctly. It should print the hookRun call, but it doesn't. It prints: Error: The output of `hookRun' is not numeric! The output was: Error : running command '/home/mascia/tuning_paradiseo/experiments/PFSPWT/tuning_2/./hook-run /home/mascia/tuning_paradiseo/experiments/PFSPWT/tuning_2/./Instances//90x20_5 402 --sa-code_definition%ps-initialisation=1 --sa-code_definition%sa-algo_choice@0=7 --sa-code_definition%sa-algo_choice@0%7%sa-perturbation=0 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance=3 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule=0 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule%0%sa-sa_r_initial_temp=1.0607 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule%0%sa-sa_r_factor=0.6901 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule%0%sa-sa_r_max_step=562 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule%0%sa-sa_r_final_temp=825 --sa-code_definition%sa-algo_choice@0%7%sa-time_ratio=7 --sa-code_definition%s Execution halted Warning message: running command 'ls *.26602+1.*.log 2>/dev/null' had status 2 which looks like the master code continued execution even after the slave died and the error output from the master was lost. The truncation is also a problem. In theory, the following program should reproduce the issue, but it doesn't: #!/usr/bin/Rscript # Load the R MPI package if it is not already loaded. if (!is.loaded("mpi_initialize")) { library("Rmpi") } # Spawn as many slaves as possible mpi.spawn.Rslaves(nslaves=5) # In case R exits unexpectedly, have it automatically clean up # resources taken up by Rmpi (slaves, memory, etc...) . Last <- function(){ if (is.loaded("mpi_initialize")){ if (mpi.comm.size(1) > 0){ print("Please use mpi.close.Rslaves() to close slaves.") mpi.close.Rslaves() } print("Please use mpi.quit() to quit R") .Call("mpi_finalize") } } runcommand <- function(command, args) { cat (format(Sys.time(), usetz=TRUE), ":", command, args, "\n") elapsed <- proc.time()["elapsed"] err <- NULL output <- withCallingHandlers( tryCatch(system2(command, args, stdout = TRUE, stderr = TRUE), error=function(e) { err <<- paste(err, conditionMessage(e), sep ="\n") NULL }), warning=function(w) { err <<- paste(err, conditionMessage(w), sep ="\n") invokeRestart("muffleWarning") }) # If e is a warning, the command failed. if (!is.null(err)) { stop (call. = FALSE, format(Sys.time(), usetz=TRUE), ": ERROR (", paste(output, sep="\n"), "):", err,"\n") } cat (format(Sys.time(), usetz=TRUE), ": DONE (", ") Elapsed: ", proc.time()["elapsed"] - elapsed, "\n") return(list(output = output, error = NULL)) } candidates <- list("x", "--xx /home/mascia/tuning_paradiseo/experiments/PFSPWT/tuning_2/./hook-run /home/mascia/tuning_paradiseo/experiments/PFSPWT/tuning_2/./Instances//90x20_5 402 --sa-code_definition%ps-initialisation=1 --sa-code_definition%sa-algo_choice\ @0=7 --sa-code_definition%sa-algo_choice@0%7%sa-perturbation=0 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance=3 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule=0 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cool\ ing_schedule%0%sa-sa_r_initial_temp=1.0607 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule%0%sa-sa_r_factor=0.6901 --sa-code_definition%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule%0%sa-sa_r_max_step=562 --sa-code_definiti\ on%sa-algo_choice@0%7%sa-acceptance%3%sa-cooling_schedule%0%sa-sa_r_final_temp=825 --sa-code_definition%sa-algo_choice@0%7%sa-time_ratio=7 --sa-code_definition%something-else and") # Tell all slaves to return a message identifying themselves output <- Rmpi::mpi.applyLB(candidates, runcommand, command = "ls") cat(paste(output, sep="\n")) #mpi.remote.exec(stop("error", mpi.comm.rank(),"of",mpi.comm.size())) # Tell all slaves to close down, and exit the program mpi.close.Rslaves() #mpi.quit() #10 --- fixed at revision 1919 ------------------------------------ Related to #8. With --maxTime, irace assumes that target-runner will report the time, however, this is not possible with --batchmode. If you are confident in modifying the package sources, you need to add a check on scenario$batchmode before checking the scenario$maxTime condition that you quote, you need to handle the case where that function returns no time and you need to handle the case that target-evaluator returns time and how to use that to increment the time budget. However, targetEvaluator is called again for elitist configurations even if no target-runner call is made. This means that we may double count the time? In the meantime, I would suggest that you don't use --maxTime for now. irace/inst/0000755000176200001440000000000013435236163012320 5ustar liggesusersirace/inst/examples/0000755000176200001440000000000013263477454014147 5ustar liggesusersirace/inst/examples/hypervolume/0000755000176200001440000000000013355676012016517 5ustar liggesusersirace/inst/examples/hypervolume/target-evaluator0000755000176200001440000000663213355676012021742 0ustar liggesusers#!/bin/bash ############################################################################### # This script is run for each configuration to evaluate it after all # candidate configurations have been run on a single instance. # # Check the examples in examples/ # # PARAMETERS: # $1 is the configuration number # $2 is the instance id # $3 is the seed # $4 is the instance name # $5 is the total number of configurations alive in this iteration # The rest are the IDs of all configurations run for this instance # # ONLY FOR ELITIST RACE: The rest ($* after `shift 5') are the ids of the # configurations alive in this iteration. This list can be used to calculate # the hypervolume using previous execution results. # # RETURN VALUE: # This script should print a single numerical value ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" if [ -s error.log ]; then cat error.log fi exit 1 } error_sleep() { echo "`TZ=UTC date`: $0: error: $@" >> error.log sleep 60 } NORMALIZATION_RANGE="'1 2'" REFERENCE_POINT="'2.1 2.1'" # A program that normalizes all .stdout files to normalization range # (optional, not included). NORMALIZE="~/bin/nondominated --verbose -n $NORMALIZATION_RANGE -s '_nor'" UPDATELIMITS="2>&1 | grep -E 'minimum|maximum' | sed 's/:/:\n/'" # A program for computing the hypervolume. Download hypervolume # code from http://lopez-ibanez.eu/hypervolume HV="~/bin/hv --quiet -r $REFERENCE_POINT" CONFIGURATION="$1" INSTANCEID="$2" SEED="$3" INSTANCE="$4" TOTALCONFIGURATIONS="$5" shift 5 || error "Not enough parameters to $0" ALLIDS=$* STDOUT=c${CONFIGURATION}-${INSTANCEID}-${SEED}.stdout STDERR=c${CONFIGURATION}-${INSTANCEID}-${SEED}.stderr calc_measure() { LIMITSFILE=limits-${INSTANCEID}.txt LIMITSTMP=limits-${INSTANCEID}.tmp if [ -s "$LIMITSFILE" ]; then $NORMALIZE $LIMITSFILE $ALLFILES $UPDATELIMITS > $LIMITSTMP mv $LIMITSTMP $LIMITSFILE else $NORMALIZE $ALLFILES $UPDATELIMITS > $LIMITSFILE fi } # TODO: Use /tmp for all temporary files, i.e., put target-data under /tmp cd target-data || error "cannot change to directory 'target-data'" ALLFILES= for FILENAME in ALLIDS; do ALLFILES="$ALLFILES c${FILENAME}-${INSTANCEID}-${SEED}.stdout" done # First normalize output files, unless they are ready. while ! ls ${ALLFILES}_nor &> /dev/null ; do # Check that all output files are there NUM=$(ls -1 ${ALLFILES} 2> /dev/null | wc --lines) if [ "$NUM" -ne "$TOTALCONFIGURATIONS" ]; then error_sleep "only found $NUM files out of $TOTALCONFIGURATIONS: ${ALLFILES}" continue fi # Check every STDERR for FILENAME in $ALLFILES; do FILESTDERR=${FILENAME/stdout/stderr} [ OK = "$(cat ${FILESTDERR})" ] || (error_sleep "${FILESTDERR} is not OK"; continue) [ -s "$FILENAME" ] || (error_sleep "$FILENAME is empty"; continue) done calc_measure rm -f error.log # Comment out if you wish to keep all output files around comm --output-delimiter=" " -23 <(ls -1 c*-*.* | sort) <(ls -1 ${ALLFILES//-${INSTANCEID}-${SEED}.stdout/-*.*} | sort) | xargs -r rm -f break done if [ ! -s "${STDOUT}_nor" ]; then error "${STDOUT}_nor: No such file or directory" fi COST=$(eval "$HV ${STDOUT}_nor" || error "calculating hypervolume failed") # Negative because hypervolume is maximised but irace minimises. echo "-$COST" exit 0 irace/inst/examples/hypervolume/README0000644000176200001440000000121612763515445017403 0ustar liggesusersThese are example hooks to tune a multi-objective algorithm (using the hypervolume indicator). You need to set targetRunner and targetEvaluator, either in the scenario file of irace (usually scenario.txt), or in the command-line (see irace --help), so irace is able to find these scripts. Update target-runner with the path, name and fixed parameters of your program. Update target-evaluator with the path and parameters for computing the hypervolume. The example uses the hypervolume code from < http://lopez-ibanez.eu/hypervolume >. Additional transformations of the input data may be applied before computing the hypervolume, such as normalization. irace/inst/examples/hypervolume/target-runner0000755000176200001440000000276613301316723021244 0ustar liggesusers#!/bin/bash ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # PARAMETERS: # $1 is the configuration number # $2 is the instance id # $3 is the seed # $4 is the instance name # The rest are parameters for running the program # # RETURN VALUE: # This script should print nothing. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: oerror: $@" exit 1 } # NOTE: You need to customize EXE and FIXED_PARAMS below. If you change the # rest, you need to also update target-evaluator ## If you want to find where target-runner is. #BINDIR=$(dirname "$(readlink -f "$(type -P $0 || echo $0)")") EXE=~/bin/program FIXED_PARAMS="--trials 1 --input $INSTANCE" CANDIDATE="$1" INSTANCEID="$2" SEED="$3" INSTANCE="$4" # All other parameters are the candidate parameters to be passed to program shift 4 || error "Not enough parameters" CAND_PARAMS=$* STDOUT=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdout STDERR=c${CANDIDATE}-${INSTANCEID}-${SEED}.stderr # TODO: Use /tmp for all temporary files, i.e., put target-data under /tmp mkdir -p target-data cd target-data || error "cannot change to directory \'target-data\'" $EXE ${FIXED_PARAMS} ${CAND_PARAMS} 1> $STDOUT 2> $STDERR # We do this to make sure this target-runner terminated correctly in target-evaluator echo "OK" >> $STDERR exit 0 irace/inst/examples/target-runner-python/0000755000176200001440000000000013337033652020251 5ustar liggesusersirace/inst/examples/target-runner-python/target-runner-acotsp.py0000755000176200001440000000576113337031143024714 0ustar liggesusers#!/usr/bin/python ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # This script is run in the execution directory (execDir, --exec-dir). # # PARAMETERS: # argv[1] is the candidate configuration number # argv[2] is the instance ID # argv[3] is the seed # argv[4] is the instance name # The rest (argv[5:]) are parameters to the run # # RETURN VALUE: # This script should print one numerical value: the cost that must be minimized. # Exit with 0 if no error, with 1 in case of error ############################################################################### import datetime import os.path import re import subprocess import sys # FIXME: group together everything that needs to be edited by the user and put # in functions everything that does NOT to be edited. ## This example is for the ACOTSP software. Compare it with ## examples/acotsp/target-runner # exe = "~/bin/executable" exe = "~/bin/acotsp" fixed_params = ' --tries 1 --time 10 --quiet ' # This is an example of reading a number from the output. def parse_output(out): match = re.search(r'Best ([-+0-9.eE]+)', out.strip()) if match: return match.group(1); else: return "No match" if len(sys.argv) < 5: print("\nUsage: ./target-runner.py \n") sys.exit(1) # Get the parameters as command line arguments. configuration_id = sys.argv[1] instance_id = sys.argv[2] seed = sys.argv[3] instance = sys.argv[4] conf_params = sys.argv[5:] # Build the command, run it and save the output to a file, # to parse the result from it. # # Stdout and stderr files have to be opened before the call(). # # Exit with error if something went wrong in the execution. exe = os.path.expanduser(exe) command = [exe] + fixed_params.split() + ["-i"] + [instance] + ["--seed"] + [seed] + conf_params # Define the stdout and stderr files. out_file = "c" + str(configuration_id) + "-" + str(instance_id) + str(seed) + ".stdout" err_file = "c" + str(configuration_id) + "-" + str(instance_id) + str(seed) + ".stderr" def target_runner_error(msg): now = datetime.datetime.now() print(str(now) + " error: " + msg) sys.exit(1) def check_executable(fpath): fpath = os.path.expanduser(fpath) if not os.path.isfile(fpath): target_runner_error(str(fpath) + " not found") if not os.access(fpath, os.X_OK): target_runner_error(str(fpath) + " is not executable") check_executable (exe) outf = open(out_file, "w") errf = open(err_file, "w") return_code = subprocess.call(command, stdout = outf, stderr = errf) outf.close() errf.close() if return_code != 0: target_runner_error("command returned code " + str(return_code)) if not os.path.isfile(out_file): target_runner_error("output file " + out_file + " not found.") cost = parse_output (open(out_file).read()) print(cost) os.remove(out_file) os.remove(err_file) sys.exit(0) irace/inst/examples/target-runner-python/trivial/0000755000176200001440000000000013337033704021721 5ustar liggesusersirace/inst/examples/target-runner-python/trivial/parameters.txt0000644000176200001440000000017313337032476024633 0ustar liggesusersparam1 "--param1 " r (1,100) | ptype == "a" param2 "--param2 " i (200,300) | ptype == "b" ptype "--ptype " c ("a","b") irace/inst/examples/target-runner-python/trivial/scenario.txt0000644000176200001440000000016313337030646024267 0ustar liggesusersmaxExperiments = 500 trainInstancesDir = "" trainInstancesFile = "instances.txt" targetRunner = "target-runner.py" irace/inst/examples/target-runner-python/trivial/target-runner.py0000755000176200001440000000456713337033446025112 0ustar liggesusers#!/usr/bin/python ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # This script is run in the execution directory (execDir, --exec-dir). # # PARAMETERS: # argv[1] is the candidate configuration number # argv[2] is the instance ID # argv[3] is the seed # argv[4] is the instance name # The rest (argv[5:]) are parameters to the run # # RETURN VALUE: # This script should print one numerical value: the cost that must be minimized. # Exit with 0 if no error, with 1 in case of error ############################################################################### import datetime import os.path import re import subprocess import sys ## This a dummy example that shows how to parse the parameters defined in ## parameters.txt and does not need to call any other software. if __name__=='__main__': if len(sys.argv) < 5: print("\nUsage: ./target-runner.py \n") sys.exit(1) # Get the parameters as command line arguments. configuration_id = sys.argv[1] instance_id = sys.argv[2] seed = sys.argv[3] instance = sys.argv[4] cand_params = sys.argv[5:] # Default values (if any) a = None b = None # Parse parameters while cand_params: # Get and remove first and second elements. param = cand_params.pop(0) value = cand_params.pop(0) if param == "--param1": a = float(value) elif param == "--param2": b = int(value) elif param == "--ptype": ptype = value else: target_runner_error("unknown parameter %s" % (param)) # Sanity checks if a == None and b == None: target_runner_error("either a or b must be set, something is wrong!") if ptype == "a" and a == None: target_runner_error("ptype is 'a' but no value assigned to a, something is wrong!") if ptype == "b" and b == None: target_runner_error("ptype is 'b' but no value assigned to b, something is wrong!") a = a or 1 b = b or 1 print(str(a * b) + '\n') sys.exit(0) # Useful function to print errors. def target_runner_error(msg): now = datetime.datetime.now() print(str(now) + " error: " + msg) sys.exit(1) irace/inst/examples/target-runner-python/trivial/instances.txt0000644000176200001440000000002413337030553024444 0ustar liggesusersinstance1 instance2 irace/inst/examples/target-runner-python/target-runner-advanced.py0000755000176200001440000004323213337033652025172 0ustar liggesusers#!/usr/bin/env python ############################################################################## # # # With this target-runner, you can: # # # # - set environment variables # # - run your jobs read the standard output / standard err, and check the # # return code # # - you can choose between 5 different ways of running your job and reading # # the standard out (you might have to try several ones until you find one # # that is robust enough for your experiments); among the 5 different ways # # there are various combinations of spawining subprocesses, using pipes or # # temporary files, and different methods to set a timeout after which the # # job you are running is killed # # - you can choose how many times a job should be run before giving up (if # # you have some heisenbugs, irace will not crash! :) but it will give the # # configuration further chances) # # - if the job fails for several times in a row, it writes in your execution # # directory a c${configuration}.stdout and stderr as the normal # # target-runner # # - set a debug level, and have a detailed output to track what happens (one # # file per target-runner, use sparely) # # # ############################################################################## import os import sys import time import socket import logging import tempfile import subprocess import threading import re # ---------------------------- DO NOT CHANGE HERE ---------------------------- # (unless you know what you are doing) # ---------------------- Search for CHANGE BELOW! ---------------------------- class Runner(object): def __init__(self, executable, candidate, instanceid, seed, parameters, parse_output, max_tests, maximize = False, log_level = logging.ERROR): self.executable = os.path.expanduser(executable) self.instanceid = instanceid self.seed = seed self.parse_output = parse_output self.candidate = candidate self.parameters = parameters self.max_tests = max_tests self.filename_prefix = 'c' + str(candidate) + '-' + str(instanceid) + '-' + str(seed) # default exec function self.execute = self.execute1 self.maximize = maximize # logging (by default only errors are logged) filename = self.filename_prefix + '.' + socket.gethostname() + '_' + str(os.getpid()) self.logger = logging.getLogger('target-runner') try : hdlr = logging.FileHandler(filename, delay=True) except OSError as e: print("Current working dir : %s" % os.getenv('PWD')) raise formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s') hdlr.setFormatter(formatter) self.logger.addHandler(hdlr) self.logger.setLevel(log_level) # changes the way the child process is executed def exec_mode(self, mode, max_time = 3600): self.execute = mode # self.max_time is used only in execute_timeout functions self.max_time = max_time # executes a process, waits until it finishes # returns the exit code, as well as the stdout and stderr # this version uses no intermediate file, it reads directly from pipes, it # is less robust though, if the child gets oddly killed it seems to hang # even if it should not def execute1(self, command): self.logger.debug('job started') self.logger.debug('PATH=' + os.environ['PATH']) self.logger.debug(command) process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=os.environ) (out, err) = process.communicate() status = process.wait() self.logger.debug(out) self.logger.debug(err) self.logger.debug('job finished') return (status, out, err) # executes a process, waits until it finishes # returns the exit code, as well as the stdout and stderr def execute2(self, command): fout = tempfile.NamedTemporaryFile(delete=False) fout.close() ferr = tempfile.NamedTemporaryFile(delete=False) ferr.close() str_cmd = ' '.join(command) + ' >' + fout.name + ' 2>' + ferr.name try: self.logger.debug('job started') self.logger.debug('PATH=' + os.environ['PATH']) self.logger.debug(str_cmd) process = subprocess.check_call(str_cmd, shell=True) status = 0 except subprocess.CalledProcessError as e: status = e.returncode self.logger.warning('Exit status: ' + str(status)) out = self.read(fout.name) err = self.read(ferr.name) os.unlink(fout.name) os.unlink(ferr.name) self.logger.debug(out) self.logger.debug(err) self.logger.debug('job finished') return (status, out, err) # executes a process, waits until it finishes # returns the exit code, as well as the stdout and stderr # if the process does not finish before the timeout it gets killed def execute_timeout1(self, command): fout = tempfile.SpooledTemporaryFile() ferr = tempfile.SpooledTemporaryFile() start_time = time.time() self.logger.debug('job started') self.logger.debug('PATH=' + os.environ['PATH']) self.logger.debug(command) process = subprocess.Popen(command, stdout=fout, stderr=ferr, env=os.environ) elapsed_time = time.time() - start_time while process.poll() is None and elapsed_time <= self.max_time: time.sleep(1) elapsed_time = time.time() - start_time self.logger.debug('elapsed time: ' + str(elapsed_time)) if elapsed_time > self.max_time: process.kill() self.logger.warning('job killed, ' + str(self.max_time) + \ ' seconds of wall-clock time elapsed') status = 1 status = process.poll() fout.seek(0) ferr.seek(0) out = str(fout.read()) err = str(ferr.read()) fout.close() ferr.close() self.logger.debug(out) self.logger.debug(err) self.logger.debug('job finished') return (status, out, err) # executes a process, waits until it finishes # returns the exit code, as well as the stdout and stderr # this version is like execute2 but requires python3.3 to set the timeouts # for the Popen.communicate() and Popen.wait() # if the process does not finish before the timeout it gets killed def execute_timeout2(self, command): self.logger.debug('job started') self.logger.debug('PATH=' + os.environ['PATH']) self.logger.debug(command) process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=os.environ) try: (out, err) = process.communicate(timeout=self.max_time) status = process.wait(timeout=self.max_time) except subprocess.TimeoutExpired: process.kill() (out, err) = process.communicate() status = 1 self.logger.warning('ob killed, ' + str(self.max_time) + \ ' seconds of CPU time elapsed') self.logger.debug(out) self.logger.debug(err) self.logger.debug('job finished') return (status, out, err) # executes a process, waits until it finishes (most robust one) # returns the exit code, as well as the stdout and stderr def execute_threaded_timeout(self, command): class ThreadedJob(threading.Thread): def __init__(self, command, logger): threading.Thread.__init__(self) self.shell_command = ' '.join(command) self.process = None self.out = None self.err = None self.logger = logger def run(self): self.logger.debug('job started') self.logger.debug('PATH=' + os.environ['PATH']) self.logger.debug(self.shell_command) self.process = subprocess.Popen(self.shell_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, env=os.environ) self.out, self.err = self.process.communicate() self.logger.debug(self.out) self.logger.debug(self.err) self.logger.debug('job finished') self.logger.debug('creating thread') thread = ThreadedJob(command, self.logger) # make daemon so that when the main program exits all (eventually # hanging) threads are killed thread.setDaemon(True) thread.start() thread.join(timeout=self.max_time) self.logger.debug('joined (timeout ' + str(self.max_time) + ')') if thread.is_alive(): # this part here is delicate we keep it in a try catch block try: self.logger.debug('thread is still alive, we timed out') # send a SIGTERM self.logger.debug('sending SIGTERM signal') thread.process.terminate() thread.join(timeout=5.0) self.logger.debug('done.') # send a SIGKILL to be very sure self.logger.debug('sending SIGKILL signal') thread.process.kill() thread.join(timeout=5.0) self.logger.debug('done') except Exception as e: self.logger.warning('exception when killing job: \n' + \ str(e)) status = thread.process.returncode out = str(thread.out) err = str(thread.err) # to be very sure del thread self.logger.debug('thread deleted, returning results') return (status, out, err) # reads and exports the environment variables def source_env(self, filename): self.logger.debug('setting environment variables') command = '"source ' + filename + ' >/dev/null 2>&1 && env"' (_, out, _) = self.execute(['bash', '-c', command]) lines = out.split('\n') for line in lines: (key, _, value) = line.partition('=') os.environ[key] = value self.logger.debug('setting ' + key + '=' + value) self.logger.debug('now PATH is: ' + os.environ['PATH']) # reads data into a string def read(self, filename): f = open(filename) content = str(f.read()) f.close() return content # writes a string to file def save(self, filename, content): f = open(filename, 'w') f.write(str(content)) f.close() # executing the program def run(self): test = 0 cost = "" while test < self.max_tests: command_list = [self.executable] + self.parameters (status, out, err) = self.execute(command_list) if status != 0: test += 1 self.logger.warning('non-zero exit status *RETRYING* ' + \ str(test) + ' of ' + str(self.max_tests)) continue # parsing the output try: cost = self.parse_output(out) except: test += 1 self.logger.warning('something failed in parse_output: *RETRYING* ' + \ str(test) + ' of ' + str(self.max_tests)) continue if cost is None: cost = "" try: check = float(cost) except: test += 1 self.logger.warning('return value of parse_output was not a number: *RETRYING* ' + \ str(test) + ' of ' + str(self.max_tests)) continue # If maximizing, simulate multiply by -1 if maximizing if self.maximize: if cost[0] == '-': cost = cost[1:] else: cost = '-' + cost break # printing the result if test < self.max_tests: self.logger.debug('returning cost: ' + cost) sys.stdout.write(cost + '\n') # force to exit all possible threads except the main one (those # launched with execute_threaded_timeout) are run as daemons so # they should be terminated automatically when exiting, but just # to be extra sure sys.exit(0) else: # in case somehting goes wrong we write stdout and stderr files self.logger.error('something went wrong after ' + \ str(self.max_tests) + ' runs') self.logger.error('saving candidate stdout to ' + \ self.filename_prefix + '.stdout') self.save(self.filename_prefix + '.stdout', out) self.save(self.filename_prefix + '.stderr', err) self.logger.error('return value of parse_output: ' + cost) self.logger.error('exit status is ' + str(status)) sys.stdout.write('something went wrong for candidate ' + \ self.candidate + '\n') if status != 0: sys.stdout.write('exit status: ' + str(status) + '\n') sys.exit(status) else: sys.stdout.write('could not cast to float the return value of parse_output: "' + \ cost + '"\n') sys.exit(1) def is_exe(fpath): fpath = os.path.expanduser(fpath) return os.path.isfile(fpath) and os.access(fpath, os.X_OK) \ and os.path.getsize(fpath) > 0 def get_execdir(): return os.path.dirname(os.path.realpath(__file__)) # ------------------------------- CHANGE BELOW! ------------------------------ # ## This example is for the ACOTSP software. Compare it with ## examples/acotsp/target-runner # Parse here directly the stdout of your job (the 'out' parameter). Or you can # ignore the out parameter and read other files produced by your job. This # function must return a string containing a floating-point number. Everything # else will generate an error. def parse_output(out): match = re.search(r'Best ([-+0-9.eE]+)', out.strip()) if match: return match.group(1); else: return "No match" if __name__=='__main__': if len(sys.argv) < 5: print("\nUsage: " + __file__ + " \n") sys.exit(1) bindir = get_execdir() # Path to the target-algorithm executable executable = '~/bin/acotsp' fixed_params = ' --tries 1 --time 1 --quiet ' # reading parameters and setting problem specific stuff ## FIXME: Convert this to a class that takes sys.argv and sets the correct ## variables. candidate_id = sys.argv[1] instance_id = sys.argv[2] seed = sys.argv[3] instance = sys.argv[4] parameters = sys.argv[5:] # maximum timeout in case the target algorithm does not terminate on its own. timeout = 180 # maximum number of trials before giving up with the configuration max_tests = 5 # Extra whitespace around options is important! parameters = [' -i ' + instance + ' --seed ' + seed + fixed_params ] + parameters runner = Runner(executable, candidate_id, instance_id, seed, parameters, parse_output, max_tests, #log_level = logging.DEBUG, maximize = False) ## FIXME: Convert this to flags with meaningful names like log_level # execute through pipes (this is the default) # runner.exec_mode(runner.execute1) ## FIXME: Convert this to flags with meaningful names # execute through temporary files (slightly more robust) # runner.exec_mode(runner.execute2) ## FIXME: Convert this to flags with meaningful names # execute through temporary files with timeout # after 5 minutes of *wallclock time* if we do not get the results we kill # the subprocess and try another time... # runner.exec_mode(runner.execute_timeout1, 300) ## FIXME: Convert this to flags with meaningful names # python3 execute through pipes with timeout (slightly less robust) # runner.exec_mode(runner.execute_timeout2, 300) ## FIXME: Convert this to flags with meaningful names # execute through temporary files with timeout # after 2 minutes of *CPU time* if we do not get the results we kill # the subprocess and try another time... runner.exec_mode(runner.execute_threaded_timeout, timeout) # Convert this to a parameter of the constructor. # environment variables that should be set for testing each configuration # runner.source_env(bindir + 'configuration') # run the target-runner runner.run() irace/inst/examples/acotsp/0000755000176200001440000000000013320545016015420 5ustar liggesusersirace/inst/examples/acotsp/parameters-acotsp.txt0000644000176200001440000000147013320545016021615 0ustar liggesusers### Parameter file for the ACOTSP software # name switch type values [conditions (using R syntax)] algorithm "--" c (as,mmas,eas,ras,acs) localsearch "--localsearch " c (0, 1, 2, 3) alpha "--alpha " r (0.00, 5.00) beta "--beta " r (0.00, 10.00) rho "--rho " r (0.01, 1.00) ants "--ants " i (5, 100) q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" rasrank "--rasranks " i (1, 100) | algorithm == "ras" elitistants "--elitistants " i (1, 750) | algorithm == "eas" nnls "--nnls " i (5, 50) | localsearch %in% c(1, 2, 3) dlb "--dlb " c (0, 1) | localsearch %in% c(1,2,3) irace/inst/examples/acotsp/scenario.txt0000644000176200001440000000246113313524427017774 0ustar liggesusers###################################################### -*- mode: r -*- ##### ## Scenario setup for Iterated Race (iRace). ############################################################################ ## To use the default value of a parameter of iRace, simply do not set ## the parameter (comment it out in this file, and do not give any ## value on the command line). ## File that contains the description of the parameters. parameterFile = "./parameters-acotsp.txt" ## Directory where the programs will be run. execDir = "./acotsp-arena" ## Directory where tuning instances are located, either absolute path or ## relative to current directory. trainInstancesDir = "./Instances" ## The maximum number of runs (invocations of targetRunner) that will performed. It ## determines the (maximum) budget of experiments for the tuning. maxExperiments = 5000 ## File that contains a set of initial configurations. If empty or NULL, ## all initial configurations are randomly generated. # configurationsFile = "" ## Indicates the number of decimal places to be considered for the ## real parameters. digits = 2 ## A value of 0 silences all debug messages. Higher values provide ## more verbose debug messages. # debugLevel = 0 ## END of scenario file ############################################################################ irace/inst/examples/acotsp/README0000644000176200001440000000070512763515445016317 0ustar liggesusersExample configuration scenario: ACOTSP ===================================== This example scenario shows how irace can be used to tune an optimization algorithm. ACOTSP is a software that implements various Ant Colony algorithms for the symmetric Traveling Salesman Problem. The ACOTSP software is available at: http://www.aco-metaheuristic.org/aco-code/ TSP instances and more information is available at: http://iridia.ulb.ac.be/irace/#ACOTSP irace/inst/examples/acotsp/forbidden.txt0000644000176200001440000000104512763515445020132 0ustar liggesusers## Template for specifying forbidden parameter configurations for irace. ## ## This filename must be specified via the --forbidden-file command-line option ## (or forbiddenFile in scenario.txt). ## ## The format is one constraint per line. Each constraint is a logical ## expression (in R syntax). If a parameter configuration ## is generated that makes the logical expression evaluate to TRUE, ## then the configuration is discarded. ## ## Examples of valid logical operators are: == != >= <= > < & | ! %in% (alpha == 0.0) & (beta == 0.0) irace/inst/examples/acotsp/default.txt0000644000176200001440000000150113313524315017603 0ustar liggesusers## Template for specifying initial parameter configurations, for ## example, the default configuration, in irace. ## ## This filename must be specified via the --configurations-file command-line option ## (or configurationsFile in scenario.txt). ## ## The format is one parameter configuration per line, and one parameter per ## column. The first line gives the parameter name corresponding to ## each column (names must match those given in the parameters ## file). Each configuration must satisfy the ## parameter conditions (NA should be used for those parameters that ## are not enabled for a given configuration) and, if given, the ## constraints that describe forbidden configurations. algorithm localsearch alpha beta rho ants nnls dlb q0 rasrank elitistants as 0 1.0 1.0 0.95 10 NA NA 0 NA NA irace/inst/examples/acotsp/target-runner0000755000176200001440000000441713304313537020154 0ustar liggesusers#!/bin/bash ############################################################################### # This script is to tune the ACOTSP software. # # PARAMETERS: # $1 is the ID of the candidate to be evaluated # $2 is the instance ID # $3 is the seed # $4 is the instance name # The rest ($* after `shift 4') are parameters for running ACOTSP # # RETURN VALUE: # This script should print a single numerical value (the value to be minimized). ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" >&2 exit 1 } # Path to the ACOTSP software: EXE=~/bin/acotsp # Fixed parameters that should be always passed to ACOTSP. # The time to be used is always 10 seconds, and we want only one run: FIXED_PARAMS=" --tries 1 --time 10 --quiet " CONFIG_ID="$1" INSTANCE_ID="$2" SEED="$3" INSTANCE="$4" # All other parameters are the candidate parameters to be passed to program shift 4 || error "Not enough parameters" CONFIG_PARAMS=$* STDOUT=c${CONFIG_ID}-${INSTANCE_ID}-${SEED}.stdout STDERR=c${CONFIG_ID}-${INSTANCE_ID}-${SEED}.stderr if [ ! -x "${EXE}" ]; then error "${EXE}: not found or not executable (pwd: $(pwd))" fi # Now we can call ACOTSP by building a command line with all parameters for it $EXE ${FIXED_PARAMS} -i $INSTANCE --seed $SEED ${CONFIG_PARAMS} 1> $STDOUT 2> $STDERR # The output of the candidate $CONFIG_ID should be written in the file # c${CONFIG_ID}.stdout (see target runner for ACOTSP). # Does this file exist? if [ ! -s "${STDOUT}" ]; then # In this case, the file does not exist. Let's exit with a value # different from 0. In this case irace will stop with an error. error "${STDOUT}: No such file or directory" fi # Ok, the file exist. It contains the whole output written by ACOTSP. # This script should return a single numerical value, the best objective # value found by this run of ACOTSP. The following line is to extract # this value from the file containing ACOTSP output. COST=$(cat ${STDOUT} | grep -o -E 'Best [-+0-9.e]+' | cut -d ' ' -f2) if ! [[ "$COST" =~ ^[-+0-9.e]+$ ]] ; then error "${STDOUT}: Output is not a number" fi # Print it! echo "$COST" # We are done with our duty. Clean files and exit with 0 (no error). rm -f "${STDOUT}" "${STDERR}" rm -f best.* stat.* cmp.* exit 0 irace/inst/examples/moaco/0000755000176200001440000000000013301316503015221 5ustar liggesusersirace/inst/examples/moaco/parameters.txt0000644000176200001440000000365212763515445020154 0ustar liggesusers## Template for parameter description file for Iterated F-Race. ## ## The format is one parameter per line. Each line contains: ## ## 1: Name of the parameter. An unquoted alphanumeric string, ## example: ants ## 2: Switch to pass the parameter. A quoted (possibly empty) string, ## if the value and the switch must be separated, add a space at ## the end of the string. Example : "--version1 --ants " ## 3: Type. An unquoted single letter, among ## i: Integer, c: component, r: real. ## 4: For c: All possible values, for i,r: minimum and maximum ## values. A variable number of numbers or unquoted strings within ## parenthesis separated by commas. ## 5: Optionally, a character '|' followed by a parameter name ## followed by a set of values enclosed in parenthesis. ## Example: | mode (acs, ras) ## # 1: 2: 3: 4: colonies "--colonies " c (1,2,3,5,10) | selection %in% c("dominance","objective") intervals "--colony-weights " c (disjoint,overlapping) | colonies %in% c(2, 3, 5, 10) cupdate "--colony-update " c (origin, region) | colonies %in% c(2, 3, 5, 10) nupdate "--num-update " c (1, 2, 5, 10) selection "--selection " c (dominance,objective,weight) weights "--weights " c (2, 3, 0.333333, 0.5, -1) directions "--directions " c (one, all) ph "--ph=" c (single,multiple) heu "--heu=" c (single,multiple) aggreg "--aggregation " c (sum, product, random) | ph == "multiple" || heu == "multiple" ants "--Mants " i (1, 25) rho "--rho " r (0.01, 0.99) onq0 " " c (" ", "--q0") q0 " " r (0.25, 0.99) | onq0 == "--q0" beta "--beta " r (0, 5) alpha "--alpha " r (0, 5) irace/inst/examples/moaco/scenario.txt0000644000176200001440000000100412763515445017601 0ustar liggesusers## Directory where the programs will be run. execDir = "./execdir" ## Folder where tuning instances are located, either absolute or ## relative to working directory. trainInstancesDir = "./Instances" ## The maximum number of runs (invocations of targetRunner) that will ## performed. It determines the (maximum) budget of experiments for the tuning. maxExperiments = 10000 ## Indicates the number of decimal places to be considered for the ## real parameters. digits = 2 targetEvaluator = "./target-evaluator" irace/inst/examples/moaco/target-evaluator0000755000176200001440000000670413301316403020443 0ustar liggesusers#!/bin/bash ############################################################################### # This script is run for each configuration to evaluate it after all # candidate configurations have been run on a single instance. # # Check the examples in examples/ # # PARAMETERS: # $1 is the configuration number # $2 is the instance id # $3 is the seed # $4 is the instance name # $5 is the total number of configurations alive in this iteration # The rest are the IDs of all configurations run for this instance # # ONLY FOR ELITIST RACE: The rest ($* after `shift 5') are the ids of the # configurations alive in this iteration. This list can be used to calculate # the hypervolume using previous execution results. # # RETURN VALUE: # This script should print a single numerical value ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" if [ -s error.log ]; then cat error.log fi exit 1 } error_sleep() { echo "`TZ=UTC date`: $0: error: $@" >> error.log sleep 60 } NORMALIZATION_RANGE="'1 2'" REFERENCE_POINT="'2.1 2.1'" # A program that normalizes all _dat files to [0, 1] # (optional, not included). NORMALIZE="~/bin/nondominated --verbose -n $NORMALIZATION_RANGE -s '_nor'" UPDATELIMITS="2>&1 | grep -E '# Total minimum|# Total maximum' | sed 's/:/:\n/'" # A program for computing the hypervolume. Download hypervolume # code from http://lopez-ibanez.eu/hypervolume HV="~/bin/hv --quiet -r $REFERENCE_POINT" CONFIGURATION="$1" INSTANCEID="$2" SEED="$3" INSTANCE="$4" TOTALCONFIGURATIONS="$5" shift 5 || error "Not enough parameters to $0" ALLIDS=$* STDOUT=c${CONFIGURATION}-${INSTANCEID}-${SEED}.stdout STDERR=c${CONFIGURATION}-${INSTANCEID}-${SEED}.stderr calc_measure() { LIMITSFILE=limits-${INSTANCEID}.txt LIMITSTMP=limits-${INSTANCEID}.tmp if [ -s "$LIMITSFILE" ]; then eval "$NORMALIZE $LIMITSFILE $ALLFILES $UPDATELIMITS" > $LIMITSTMP rm -f ${LIMITSFILE} mv $LIMITSTMP $LIMITSFILE else eval "$NORMALIZE $ALLFILES $UPDATELIMITS" > $LIMITSFILE fi } # TODO: Use /tmp for all temporary files, i.e., put target-data under /tmp cd target-data || error "cannot change to directory 'target-data'" ALLFILES= for FILENAME in ALLIDS; do ALLFILES="$ALLFILES c${FILENAME}-${INSTANCEID}-${SEED}.stdout" done # First normalize output files, unless they are ready. while ! ls ${ALLFILES}_nor &> /dev/null ; do # Check that all output files are there NUM=$(ls -1 ${ALLFILES} 2> /dev/null | wc --lines) if [ "$NUM" -ne "$TOTALCONFIGURATIONS" ]; then error_sleep "only found $NUM files out of $TOTALCONFIGURATIONS: ${ALLFILES}" continue fi # Check every STDERR for FILENAME in $ALLFILES; do FILESTDERR=${FILENAME/stdout/stderr} [ OK = "$(cat ${FILESTDERR})" ] || (error_sleep "${FILESTDERR} is not OK"; continue) [ -s "$FILENAME" ] || (error_sleep "$FILENAME is empty"; continue) done calc_measure rm -f error.log # Comment out if you wish to keep all output files around comm --output-delimiter=" " -23 <(ls -1 c*-*.* | sort) <(ls -1 ${ALLFILES//-${INSTANCEID}-${SEED}.stdout/-*.*} | sort) | xargs -r rm -f break done if [ ! -s "${STDOUT}_nor" ]; then error "${STDOUT}_nor: No such file or directory" fi COST=$(eval "$HV ${STDOUT}_nor" || error "calculating hypervolume failed") # Negative because hypervolume is maximised but irace minimises. echo "-$COST" exit 0 irace/inst/examples/moaco/README0000644000176200001440000000167212064616354016123 0ustar liggesusersExample configuration scenario: MOACO ===================================== This example scenario shows how irace can be used to tune a multi-objective optimization framework. It uses the multi-objective ant colony optimization (MOACO) framework proposed here for the bi-objective traveling salesperson problem (bTSP): Manuel López-Ibáñez and Thomas Stützle. The Automatic Design of Multi-Objective Ant Colony Optimization Algorithms. IEEE Transactions on Evolutionary Computation, 2012. doi:10.1109/TEVC.2011.2182651 (PDF preprint) http://iridia.ulb.ac.be/IridiaTrSeries/IridiaTr2011-003.pdf The MOACO framework software can be download from: http://iridia.ulb.ac.be/~manuel/moaco Instances of the bTSP can be generated by concatenating two single-objective TSP instances: $ cat euclidA100.tsp euclidB100.tsp > euclidAB100.tsp TSP instances and more information is available at: http://iridia.ulb.ac.be/irace/ irace/inst/examples/moaco/target-runner0000755000176200001440000000320613301316503017745 0ustar liggesusers#!/bin/bash ############################################################################### # # PARAMETERS: # $1 is the configuration number # $2 is the instance id # $3 is the seed # $4 is the instance name # The rest are parameters for running the program # # RETURN VALUE: # This script should print nothing. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" exit 1 } # If you want to find where target-runner is. #BINDIR=$(dirname "$(readlink -f "$(type -P $0 || echo $0)")") EXE=~/bin/moaco_btsp CONFIGURATION=$1 INSTANCEID=$2 SEED=$3 INSTANCE=$4 shift 4 || error "Not enough parameters" STDOUT=c${CONFIGURATION}-${INSTANCEID}-${SEED}.stdout STDERR=c${CONFIGURATION}-${INSTANCEID}-${SEED}.stderr # TODO: Use /tmp for all temporary files, i.e., put target-runner-data under /tmp tmp="./target-data" mkdir -p ${tmp} cd ${tmp} || error "cannot change to directory \'${tmp}\'" # Here we transform some parameters. PARAMS= while [ $# -gt 0 ]; do case "$1" in --ants) shift; ANTS="$1"; shift;; --Mants) shift; MANTS="$1"; let "ANTS = (3 * 2 * $MANTS)"; shift;; *) PARAMS="$PARAMS $1"; shift;;# terminate case esac done ISIZE=$(basename $INSTANCE | cut -f1 -d'-') let "RUNTIME = (4 * (($ISIZE / 100) ** 2))" FIXED_PARAMS="--trials 1 --wls 1 --aco mmas --time $RUNTIME --ants $ANTS --input $INSTANCE --seed $SEED" if [ ! -x "${EXE}" ]; then error "${EXE}: not found or not executable (pwd: $(pwd))" fi exec 2> $STDERR $EXE ${FIXED_PARAMS} ${PARAMS} 1> $STDOUT RET=$? echo "OK" >& 2 exit $RET irace/inst/examples/batchmode-cluster/0000755000176200001440000000000013460635612017543 5ustar liggesusersirace/inst/examples/batchmode-cluster/target-evaluator0000755000176200001440000000402413442476657022773 0ustar liggesusers#!/bin/bash ############################################################################### # This script is run for each candidate to evaluate it after all # candidate configurations have been run on a single instance. # # Check the examples in examples/ # # PARAMETERS: # $1 is the candidate number # $2 is the instance id # $3 is the seed # $4 is the instance name # $5 is the number of candidates alive in this iteration # # ONLY FOR ELITIST RACE: The rest ($* after `shift 5') are the ids of the # candidates alive in this iteration. This list can be used to calculate the # hypervolume using only active candidates. # # RETURN VALUE: # This script should print one numerical value: the cost that must be minimized. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" exit 1 } CANDIDATE="$1" INSTANCEID="$2" SEED="$3" INSTANCE="$4" TOTALCANDIDATES="$5" shift 5 || error "Not enough parameters" ALLIDS=$* STDOUT=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdout STDERR=c${CANDIDATE}-${INSTANCEID}-${SEED}.stderr ALLFILES= for FILENAME in $ALLIDS; do ALLFILES="$ALLFILES c${FILENAME}-${INSTANCEID}-${SEED}.stdout" done # # This may be used to introduce a delay if there are filesystem # # issues. # SLEEPTIME=1 # while [ ! -s "${STDOUT}" ]; do # sleep $SLEEPTIME # let "SLEEPTIME += 1" # done # This is an example of reading a number from the output of # target-runner. It assumes that the objective value is the first number in # the first column of the only line starting with a digit. if [ -s "${STDOUT}" ]; then # You may need to update this to parse the output of your algorithm. COST=$(cat ${STDOUT} | grep -e '^[[:space:]]*[+-]\?[0-9]' | cut -f1) echo "$COST" TODELETE=$(comm -23 <(ls -1 c*-${INSTANCEID}-${SEED}.* | sort) <(ls -1 $ALLFILES)) ## Comment out if you wish to keep all output files around # rm -f $TODELETE exit 0 else error "${STDOUT}: No such file or directory" fi irace/inst/examples/batchmode-cluster/README0000644000176200001440000000020313053052355020411 0ustar liggesusersThese scripts are meant to be used with the option --batchmode. See the documentation of the option --batchmode in the user guide. irace/inst/examples/batchmode-cluster/target-runner-slurm0000755000176200001440000000400113301314461023407 0ustar liggesusers#!/bin/bash ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # This script is run in the execution directory (execDir, --exec-dir), # the same directory where target-evaluator is executed. Hence, you may # need to copy extra files needed by the executable to this directory. # # # PARAMETERS: # $1 is the candidate number # $2 is the instance id # $3 is the seed # $4 is the instance name # The rest ($* after `shift 4') are parameters to the run # # RETURN VALUE: # This script should print nothing. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" cat $tmpfile rm -f $tmpfile exit 1 } tmpfile=$(mktemp) EXE=~/bin/program FIXED_PARAMS="" CANDIDATE=$1 INSTANCEID=$2 SEED=$3 INSTANCE=$4 shift 4 || error "Not enough parameters" CAND_PARAMS=$* STDOUT=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdout STDERR=c${CANDIDATE}-${INSTANCEID}-${SEED}.stderr # modify with the SLURM submission template sbatch 1> $tmpfile < \$TMP/$STDOUT 2> \$TMP/$STDERR RET=\$? echo "OK" >& 2 mv \$TMP/* ./ rmdir -p \$TMP &> /dev/null exit \$RET EOF rc=$? if [[ $rc == 0 ]]; then JOBID=$(grep -o -e "Submitted batch job [^ ]\+" $tmpfile | cut -f4 -d ' ') if ! [[ "$JOBID" =~ ^[_-.@0-9A-Za-z]+$ ]] ; then error "$0: cannot parse jobID from the output of sbatch!" fi echo "$JOBID" rm -f $tmpfile exit $rc else error "$0: sbatch failed!" fi irace/inst/examples/batchmode-cluster/target-runner-pbs0000644000176200001440000000267713301314523023046 0ustar liggesusers#!/bin/bash ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # This script is run in the execution directory (execDir, --exec-dir), # the same directory where target-evaluator is executed. Hence, you may # need to copy extra files needed by the executable to this directory. # # # PARAMETERS: # $1 is the candidate number # $2 is the instance id # $3 is the seed # $4 is the instance name # The rest ($* after `shift 4') are parameters to the run # # RETURN VALUE: # This script should print nothing. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" exit 1 } EXE=~/bin/program FIXED_PARAMS="" CANDIDATE=$1 INSTANCEID=$2 SEED=$3 INSTANCE=$4 shift 4 || error "Not enough parameters" CAND_PARAMS=$* STDOUT=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdout STDERR=c${CANDIDATE}-${INSTANCEID}-${SEED}.stderr # PBS just prints the jobID, so it should be good to go. exec qsub < \$TMP/$STDERR $EXE ${FIXED_PARAMS} --input $INSTANCE ${CAND_PARAMS} 1> \$TMP/$STDOUT RET=\$? echo "OK" >& 2 mv \$TMP/* ./ rmdir -p \$TMP &> /dev/null exit \$RET EOF irace/inst/examples/batchmode-cluster/target-runner-sge0000755000176200001440000000336013301314413023027 0ustar liggesusers#!/bin/bash ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # This script is run in the execution directory (execDir, --exec-dir), # the same directory where target-evaluator is executed. Hence, you may # need to copy extra files needed by the executable to this directory. # # # PARAMETERS: # $1 is the candidate number # $2 is the instance id # $3 is the seed # $4 is the instance name # The rest ($* after `shift 4') are parameters to the run # # RETURN VALUE: # This script should print nothing. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" cat $tmpfile rm -f $tmpfile exit 1 } tmpfile=$(mktemp) EXE=~/bin/program FIXED_PARAMS="" CANDIDATE=$1 INSTANCEID=$2 SEED=$3 INSTANCE=$4 shift 4 || error "Not enough parameters" CAND_PARAMS=$* STDOUT=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdout STDERR=c${CANDIDATE}-${INSTANCEID}-${SEED}.stderr qsub &> $tmpfile < \$TMP/$STDERR $EXE ${FIXED_PARAMS} --input $INSTANCE ${CAND_PARAMS} 1> \$TMP/$STDOUT RET=\$? echo "OK" >& 2 mv \$TMP/* ./ rmdir -p \$TMP &> /dev/null exit \$RET EOF rc=$? if [[ $rc == 0 ]]; then JOBID=$(grep -o -e "Your job [^ ]\+" $tmpfile | cut -f3 -d ' ') if ! [[ "$JOBID" =~ ^[_-.@0-9A-Za-z]+$ ]] ; then error "$0: cannot parse jobID from the output of qsub!" fi echo "$JOBID" rm -f $tmpfile exit 0 else error "$0: qsub failed!" fi irace/inst/examples/batchmode-cluster/irace-sge-cluster0000755000176200001440000000101513040676306023003 0ustar liggesusers#!/bin/bash # To use --cluster=[sge|pbs|torque], irace must be invoked in the submission # node and target-runner will launch jobs using qsub and return the jobID. CLUSTER="sge" BINDIR=$1 EXECDIR=$2 DESTMAIL=${USER}@localhost shift 2 $BINDIR/irace --exec-dir=$EXECDIR --cluster=$CLUSTER --target-runner=target-runner-${CLUSTER} $* \ 1> $EXECDIR/irace-$$.stdout \ 2> $EXECDIR/irace-$$.stderr if [ ${DESTMAIL} ]; then cat "$EXECDIR/irace-$$.stderr" \ | mail -s "[irace] $EXECDIR/irace-$$ $*" $DESTMAIL fi irace/inst/examples/matlab/0000755000176200001440000000000013301317000015354 5ustar liggesusersirace/inst/examples/matlab/parameters.txt0000644000176200001440000000004012763515445020302 0ustar liggesusersA "A=" i (1,50) B "B=" r (5,10) irace/inst/examples/matlab/RUN.m0000644000176200001440000000070112763515445016224 0ustar liggesusers% function Result = RUN(INSTANCE, A, B) % Using INSTANCE as a seed, this function creates a pseudo-random number % drawn from a uniform distribution in the open interval (-1,1) and returns % the value of Result, where Result = A*B + r function Result = RUN(INSTANCE,A,B) rng('default'); rng(INSTANCE); minValue = -1; maxValue = 1; r = (maxValue - minValue) * rand(1) + minValue; Result = A * B + r; fprintf('Result for irace=%g\n', Result); end irace/inst/examples/matlab/scenario2.txt0000644000176200001440000000136513214240657020027 0ustar liggesusers###################################################### -*- mode: r -*- ##### maxExperiments = 1000 targetRunner <- function(function(experiment, scenario) { main.script <- "Main.m" require(matlabr) debugLevel <- scenario$debugLevel configuration.id <- experiment$id.configuration instance.id <- experiment$id.instance seed <- experiment$seed configuration <- experiment$configuration instance <- experiment$instance switches <- experiment$switches args <- irace::buildCommandLine(configuration, switches) args <- strsplit(args, split=" ")[[1]] matlablr::run_matlab_code(c(args, paset0("run('", main.script, "')"))) cost <- as.numeric(readLines("hv")) return(list(cost = cost)) } irace/inst/examples/matlab/target-runner0000755000176200001440000000366513301317000020111 0ustar liggesusers#!/bin/bash ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # This target-runner is run in the execution directory (execDir, --exec-dir). # # # PARAMETERS: # $1 is the candidate number # $2 is the instance ID # $3 is the seed # $4 is the instance name # The rest ($* after `shift 4') are parameters to the run # # RETURN VALUE: # This script should print one numerical value: the cost that must be minimized. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" exit 1 } EXE=matlab FIXED_PARAMS="-nosplash -nodesktop -nodisplay" CANDIDATE=$1 INSTANCEID=$2 SEED=$3 INSTANCE=$4 shift 4 || error "Not enough parameters" CAND_PARAMS=$* STDOUT=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdout STDERR=c${CANDIDATE}-${INSTANCEID}-${SEED}.stderr ## If the program just prints a number, we can use 'exec' to avoid ## creating another process, but there can be no other commands after exec. #exec $EXE ${FIXED_PARAMS} -i $INSTANCE ${CAND_PARAMS} #exit 1 ## Otherwise, save the output to a file, and parse the result from it. ## (If you wish to ignore segmentation faults you can use '{}' around ## the command.) ## This example assumes that CAND_PARAMS contains something like "A=10 B=5" $EXE ${FIXED_PARAMS} -r "INSTANCE=${INSTANCEID};SEED=${SEED};${CAND_PARAMS// /;};RUN(INSTANCE,A,B);exit" < RUN.m 1> ${STDOUT} 2> ${STDERR} ## This may be used to introduce a delay if there are filesystem ## issues. # SLEEPTIME=1 # while [ ! -s "${STDOUT}" ]; do # sleep $SLEEPTIME # let "SLEEPTIME += 1" # done if [ -s "${STDOUT}" ]; then COST=$(cat ${STDOUT} | grep 'Result for irace=' | cut -f2 -d '=') echo "$COST" rm -f "${STDOUT}" "${STDERR}" exit 0 else error "${STDOUT}: No such file or directory" fi irace/inst/exdata/0000755000176200001440000000000013256533654013574 5ustar liggesusersirace/inst/exdata/sann.rda0000644000176200001440000005256513451373556015240 0ustar liggesusersXS>j&DGDiR ^aAAT"]+VaRQ6wB{w}Y׻fӂ;F$M]rs qi(I&Ʀ$ӪIb7`GHMƧr,RHj|xRL]F%^&9ǜ/3 MMJHG$%DNɩ)1 tY?8Ҙэ}k>2xc)S>wjfCI& A I+ Z KÉ Q,d1s&E~5Z $H ‘<@;SB:Jy aR41]KkZZ ڗZi&ҳ`b$jJjz$X-JLiOpxrJRLtL<U9%<>:@#8腜& [‘ cS)VQ2!ׯĕ4@kxRc&},REλ/m3gT|6,RN=u6c_{z!Ǯ^ EGfy\Eg~lY6h3//@6hӜg>h;F{.hw+] ލ|TZh'Ikp8o MQzCckaKM$kAۻ#@N]~ hKu=h{.4O.5'=<.?${SKh<|QRoVi !*hcsnȈg/H6"F~h/Mߨ)k@[UJ>h/tIYuIdLDV *M}þ;]A[-4eBTAr ȠIY9к0ZV( ^0)hJ“yMA>eEwf)h0},mwק]n3%h'6h#+k&IKuvڷ'W/ E!ziٕRH jU<]TЦ_5h;)acvȔk}| oO6hmS'^mA~Abq Ǝ[% ڭ}Xڒ_:E@{–md?zbfenA h;Pi**$~Pbc˲['G*~rx|^RT8SODVRRXSq3fiԴXjxjJ u*ѧᕚ®4.oO<I85 MmH[ѡuQ23cq%-eK3?&>؄v)q Q8`rOILESh+) QDh~L=@F8ƤY}?J@YWL֠!KL\ ^h 2."!5>@g=sgYMg& Didj,5GDKP!XHДD?Dr~k~fNf9@$bg C_!Abl8񱐱+Ƒhv0Gk/7\Wt/8I?L  ,r!sY.l_TU oI__1dZ%Y9VXԌp* ,|@4Yolqo胦!~v~.#ޕf }mˁKJol+}Eڕ'AGr .hSaOۦ0F IK?>(LWkpwGhG髕χ^}4RL}]>}rr]UG)7M.jJ\J!E 6PaFs YO/ו2O<|vw})ɓ-# E,@ppdO1P0rBo}MAyPPX|J\,?r^^11苝g nSd,v딧PΩCnoUS׃S)-5)S#oҕA WU#}O5Atj{+-++H/3x ~r*_pS} =6/`7YpTB(e5nHI_$L\8d}"[)gڊAak'zGQg[ 7(  7kc$n7^|)V> 9?tlV>>ɇ%+6Ho]wZ?)zkPinXECABe}^o&áb^p5 ]#st{~6dεi?R <.8( j8S= VO ޖn1'z*l$7r\#F/6^ z>n 3Fz^Zs/ OZOBQ,iAWux4a,1@ıރ֌:[`g_BR (~ [ ttzF5c~9_ (_r ^wT,r<>*|+?wx&(![u4"z7oT!'} PCaOԧn#-FlųیZ?t->E_h뗜_ނ{Wӛc6݂ wAK{͟m0%!]\t^w$7*vz@J2oEۛg}/HrT_s[ ?t_0TV~-.S6+E ZkYǶ  ޵Y PN_)tI8*=ZR:+=dy‘]xAEae/Aȫ)R-L@Ð&<kKC6U_^ٟ7뇯?9ʷ=_'fhLwB_߬fIuj?ܙxIrW J rF߁Rg/8]:x{9:~Nɏ>a\5X E[b[KCަ9hn]wN|<w 왗eTh#XSѣ$o`8(8#yyX+ }{A)-07}C砐oOKA;[ )ɡFC/-h}`BAW*y$(>nh e5̀ǔPvX˼ۓcsŌ@Ί#y&_(PN-rwt8y5mZs_ʊ*/飮.T[FNޟ' uMB6蕪%߿ -L\&F˂[JB]0Rg(Ž/|w2?%XE^%kѪ/RxL{)ۗ-/{QXx_-^..HZҾw]+/e |e<ʻMKۯ|q8>2T(9-=Tɶh1M'e*'@mU`}[ھPQMh#vz(>&[lۻ%a}PD=7oyx66X(' .@N!.3/XC'fqGG4A~! ~=7M%cy\pvĒ?Oy9Ơ[8g{ie(cކwg8~_ Oy."4"9îD˧f>x_3S.ளc 7O9DJ@TNg=-Z߁j=1.7>j_4M\j|9mn4әX~JƼ_ȵJ~fOӷ vH}ݟ},GuM)~7?N:kفΪGٗ|qL 9wtBkϋNf~au(ܝ{j-r&6m\q?f' W^~rDힿ"e{ωzmWӛMrLaZ / K -{|WXy/-7xL wNSѠ;3¸j]ቴz;`pkOe<<Ŗ !6bp+7oW~gp' u X$=o)$_::R{[6C NM9Ո|~ttO y{yo&sn}[uae Ń!Kh(;۴cߎ?qFGԿ^;Y+52zͿ7~if?fH +H(ˁ#V)r['n,gۚ˹hW~.J0Or{ {ܪtU98ƨG7(\B:(oi k_"63G lg1PxUOܸ/q ǶL-\,5XZR|&)k@–ʛCv?\ʹUS@FE'+p3;dmpDy toĻPmuPŨ lkո* @ibӉ`mSAb8^Z>s $tdAreNϰhbWQ;UaFG\_wր3˼eO>E N[3v>B[#IW)ͻwl 㾴,|&AY6( Q̹{[;7k=}6HA=t8y7G`/_inHtoz+0*pHFg^P$dɯU6f,nxXEZ&mJG43%Wަk>铃 0lu  uyþxAêƋ>Y-ZPZ\Tw~MWss{ H\/&r/:Vv+m 9:]4-`әz>ک&'=nl߶Í֐;zNp4$̺j) Fۃ5Blw/ݴ/]o 䧃…$CއTm4 @_'7A2tY8l=* K:qj=o4ʸy ?9p qVCXFxkP6ur ePrQJc84>9[?VQ H~2y8,~NIp(ysMM|/$w7 1AegwcsW PeWC`͵*=PNWr^!wN ERC :9!xtZC*[yXlaE( ^Xvo866i֙`- >^-;>y[χ$}^Ny9 GGoME?EQ^b]'׷n ߘ-"2pd&5m&96Zpl{æ(\~P(Siuͻs'IODD;\ iLw'Mm{~ݛf/s@>mϊ=/5shK{rށN!yu=)pl8?d%;+rzH'M=Awo-`SaґnOG}Oϐbvw 3l13 ml?~΅`sgo<{f?pM}`;gowAqFQkppwXGz@TuE箛yؼlt8qcu {e{{gOnm{SD?k p!ykP(L+]|/ImmPUv]p-9d^[t RmT}[.J덱 c [V|ϾWܙZ@ݜ\"6$CR!KA{%Amw 4akqXsOg]i)h̳k@"\7Pi[8X&H}ܔ<_zȐ $8pPNpχm`Ȉփyi ʐ6 |px1YU`C-`q!;>ޙ`3#>U:r̤\yRt)';.qT˘̘3VR9{*$>4ybR>u8P$*JqYN)&/BJ`nP y))p󄃷hP*$R6-tO)k7USf]ʑy1nq- >y+AIz EkREO[suzi߼鳆LPp _0nw, >:~~3]F:X5NN>+r_)jE$H~uQq;H $^ChPlrŪ`|cH%RF8Oo6U@[X5N z Vm it }Zt*vǭ2P鵇-| eIsm]#Typր;<;o|K喁<&_DπʊPhV8HQ! rWkq~NOf?5^. Zk0)׆88 0/ CD?I_WFĕ1qeB\Wfĕ9qeA\FaDpFaDpFaLp1aLp1aLp& aBp& aBp& aJp)aJp)aJpfaFpfaFpfaNp9aNp9aNpaApXaApXaIpX%aIpX%aIpX\KCƥҘqi¸4e\1.K!͐f`3d2 l 6C!͐f`3b1،lF 6#͈f`3b3،l 6cm( D ld#F6!ld#F6F6ld#F6_!_ܳ/"(I?GQQ߂u?rTc  d? }5/$>z-9'A&Bm 4Dh BSfZ Dhp41"FB3h6(-ۣۡ=;ʏCq((?ǣ;9eґ7:q#;n< !;nt/2~7@8Y՟Yi0 ]1t՟Q]Q>1(5t@r$@r$DrO$DrO$Br/$Br/$Fro$Fro$Ar$Ar$Er_$Er_$Cr?$Cr?$Gr"9ۏGH#GTN Hҏo'YK/m@v ./ύ#? @$@$D@$D@$B $B $F`$F`$䓐|OB$A$AP$EP$E0$SAꅅ!Dv"Dc !Bl!Dr"9_/HB$g !⯌߿{ld#F6ld#F6ld#F6ld#F6ld#F6ld#F6ld#F6ld# I$nNɤ%5Mr z9(?e$~ǤPSbQ_L|rJx|$#'DĝWHH@ifp&Hhi"; uAY[>6GLkQ\*Bc_$w~`[秷uq#O~Tr]j5ؽ\WzQJ&ƕ6\.}d.Uܰ,#|GmcH]Yg_pՖHp"`EksW+ L4{TΏm+}fXk _;1by{OxOݣ㲥FM琅G(-ٗ囗+}%WJܺ23z6(qfQϘ[F]}EROWtyԼ7?]]2֩ӌ5OxT(9͇=r?r'D]xB4b؍VvZ7;q]qsjmO~9RiR15WrwӪ8AZcL޻gM֦ En@RS%RIi]TIQv>A'WP0,snMϾ5~{Z-3@޺k+l ?s\|N>cÚG-[ wݍ87/ö(8n>u/ɲAeA=mv*.&"Y]%S0.x$9Iӌ;sڵZtBzW G^.?x|¸:ҩ,!#|6E;o1ڝs<^gfSƆyFO!\[{8iFHTgK{*7.T8G<'jϳ?lh}izf7ƽ'\zȻ2HͽZdmڻ\{/ZZq,#)aǷ]n+{p.h|=S>t30AùG_=s(;3!wQ1;;KV<\p|hu$o˥3ֻ_ͩ<{i)57q]"撵QKu;{m֭/~7USqenV)'7=Z)yJGz|;gh^?%^R,a&<2mjW[lW8+)3j/-7qW|gizTnxPEڍ~'rtۜsʟ=ߦ]ܘ}~bzUQ1n^=|h%UY+_,'WMUoUMLқȹKbOT)4l/7@}_w.>nZU£AzFI[Eɱ7sm7m]v,mzG)Te66|pKU/pzkO0v ]1_ym;zZPM@pR.Lz)ᥫ&:p?>+"/ܳscw߼_cكԭ:'^_gQ_BM.3;zU඙ô+Fԅ-,$tXW 7E,lcpI4ۤ꺶.54ێ\q\/r;yTf.uȗӊOo:W(p麈+\=ZYGk"rvy *D.0BeӳuO.INybxܦ9 WmP^aSܧpiٲ犸yih׳nX_!"h|#(2<{T=Md[{LcߓrH 9|{ő˦߷٨f}e-#=ܱ%< vVo7CpDg|ugۨ啚5늛  B]vZoH-дrUh+뱈>1|BZHkYdؙyjuß}/r/![aqeΦdIPT:q]J5 #S97cЖs_͕_%Rx#c>e)JAxbF,h 5eŋѐhIBFJm#:ͿĜڲi%MND ׺\TȒj"'4Xhw)ǞhK~#ƣz> yݹZ9"r~Oz׫*ཀྵy*E#7e.~Ӷ{lG:[)k-9I1G,)I$L77RR“)zᱩ) Iຓ]bSh _|K[r!*z5Xe+P%Wb镦% @^/GK1S+Ӓ<-a0^; {%-a?J_-Z^y@Kh {1*ހSG~B{<*Y2-%u+hi}AO^*@KثlP$`a~~[c?׀Jb%U*`a {;- }F^) QZR%mZ^KKEK"8 c^? ђ5JH{"?ۢx"_ظ#_k6s!^2(ƈWE?a+9-}VȏBǂ'n(>(z(ȟ_\oPoG# bqBRE1DF>O/_/b} OQc}Xܱ>[`wnIrȏ=0/|cxF#X8#$>(%^7?(o0c1F>)2 ~`G>7+~,B(>b 'ևQ̆!c} X,֧Ç_?|@>jhyHK1r ;S j j, P۰C~ C DqbI b1 Fq&1 7X'!aװ(/?wO,(X~1`>(X3w PL~#I뮧M2[:&ӬRmqᯍuxxF&XV-D~:rd:e/j]I)xg *-R%F;:Ow=-=wV%uoNڰI~xkRB%ŧ>՚tD;tx$x8m\Vh_p+mɭ¯fd˥'B)V}-2].]tˎSSSrG@…|szN)σS7W6[zؤ}_bsM-:=]am%"Uʞs/,}rewF\[c#j _6GXIq܉ .?=՗wmZ*/F Wv:mK%9 ,tV>amLPi$E_ͣ)LKšÐr89r{1}:t^stΈ-qoJ#W!O̹1V>j﷑c}y3 |~]JrIȞ{BGl9ֻ7MJ~ˊ1CTShݣ[[mLtX  gO T[F:3Z`Rƻ&:$M? j3yO濙ag3a7vı Jy:兠ӫYT8O)m'ȼ,Hړr5fN~YV)Uzk2  Hv{vbcg=CS|%8׽ [nh?Vrg'Pha7ޚM֛kVqF:. jxw6e񷭮=̲{93Fw!Wa5}~,v{M}@lwTUJϖ.bLZ!BiFW\JOM1)y/mudצ&G*$RUz928jt )5/$U@ ݠZIMޜQ-],۱F3Sxgrw."XLigǓC3^ُ|RGgpji]pIa#yͤ*y|>U(B֎K=/4~<t?[xn!('2.}(  Nv J bPS6b3=҃RLCb [H>_cF" `jߠ3:{`@}w ]/ʲ\W`ﳎ?;- (eڎw=Cܤ!>ӿ"K}΁?X_O|W8W{X$п 6ߎsk:Pg=#X#L3js\6H=YzC]s6 u^t]:xb`WocڗDcyJ5Xɒd6Hc)>zI}^(?]-Eu-vA"۱22VCb<3=< ?fbҫlc^s~.eaY2~ XUq5Ԑ XOɱȲ~b#gp̲&񾌏ٌq 9PXtڙ$.IKu4|e/#qb?'!t Ce-WC:`lee8Ze6 X'>>O ߜ|6#iy S".-9%<%y ABbJLsBĈU!O:f}seJb OB0 :CAzM<;&MEeJ2~Dֻt>!OI>aT -:wڌo[871N}ˁ/DߞpokGㄻ}<Φ!HS˼jğ`֝77> Wx`Տ6!q_2$F,=ohGӧat#>b^3K)ra4Љ8Sa(I)'̧HjBeJH:(3ߑv>c>?&sIzп)tܫO~n\s9[*蕡[f?$l*aΧ ёo+\\Ғ(40cq2>F9ߘ.gz6x=qu#*989UC/B'6? É_qB}]Oý1ʋ'RsNN3c€(3[~E0ni=ouFrn :kt}5@C`kl9_Q~xP&Qcc{f ̟o2rHmLNFPqu“"biti1Ţߞ@ ` y"y "y!y!y0#y0#$$'q|1 C<C<C<C<Ðn#KKx{qS"x`61߁+§t,_!:fgu5@`]/鸯tE_o蘆(c:o5Az?ZBǹ[6xX{"%tD~*F! &#h~\T( գv",@W\zD =Yw鸆>| :.Fj .E߷emPCxTx !+,To/Z~TW:/GK^'*7C-ը2*G>*Dh߾by_| !gH_IHE)ғ]Q(B;aB$/F~O$ R1g±Tr3q\v7q/A[n'&,3aYpRddh^HooN gʥW= 'v(BZ :T<'uvZ)Yנzl!VM/S%4U0kՓQJ}Αo7 ZkXg8~=Y(',}K 8k΄Upt~ӃOY-'MuYK6ߏhZ!{ҟgC{Mnfn'斦~6QunAߍMc Q ?|mҙ}t?+vOz4].ˮCn@9YK@Ucھr7h>ח7%@byOSWjwBקtߵoا 'n1 z466*A5pDBi~|: ț=oN(BS8-~{a\oX˕I֝_&CZjG! _6Co&w#@, KZUwt0W-a@rZ{$8֝bqȩPkoGSwIpclMtO MyڏjuȱPrseEP[%hԟL׷puZr)M+?3(o7!G7w7HZ\X ǟfSoӎ.d+ h`̳rZO Uƿ SW(*{xzhqo76]=_U%āħ/GW=O z%(l]#|EX ,,`0f!hyKȽi]ʉLSua6y]O} ȳ Yv>zCzߟ/oAyHzv{oJ?]~q~a_২!*!qfhH\W1 qeJ\WqPkHpFaDpFaDpF1aLp1aLp1aLp& aBp& aBp&)aJp)aJp)aJp1\ 3q4kFpfaFp9aNp9aNp98: 8, 3aApXaApX%aIpX%aIpXEҐqiĸ4f\0.MfKsƥf`3d2 l 6C!͐f`3d1،lF 6#͈f`3b1،l 6c1͘F mEE>Gs:+cfHj|xRL Ao5)q/Q Q xl^ MQ Y ~H/F2 SτBsL(drb6TBuYem=@i7qbfNv{d8G)cFg$J9"+='/#HLx3#G&!gI=Ҕ IN x$IŔH%J_Mer$٪l6)v}KwDIX&T#K,Kr$2u{ScDz˝xRFH[GWO_4%W>@Fn{S-G,KQ](C"Sqs MzjaLʹ5+?2$jz?L,YuˮǶ]ǓS6zϱOcWؔ4"bobb_b?bG4 %IRйع&$)5K*E,rfʲhU-ԘZ4JL(MuL,·HeHPGUR'13#Y"sle8/4/K"K2~] J>K9-&NP (L TEd~ fgeI$J~UJΔ%H$*ƀ1,aV^$<βD!qS‚%ȓb9t)I)9$I䇩meY(&rnАJ3U1 frLSg}@XMX` YQ֠S`:SK3UH*Kٔ;s5K芞E|BZWUwu|A.~`[H,D֑_mŋW~RVڌ$j ς"39O5_;hu92Y4dJ6LL(E/ǒ쥚Z}1/j埑MqLRԂ R>ʠVU$#4#!Z$O2et둨HLrdp壐[v Ώ<e&{|ł7DOM~ ?ş0^kDD_;Xw}GqP$x `\ YLۂZ፾ͽ1絵›x A̪ ,l Wt' \O= KwnڐS/eRbNG男Zᝲ Ha5,~vC=k%Ή@nfY`?(B?7`,xjZV+;f@olH%OK7 %Q[8FZ= \@0}~ .Ϣyߝ# @̶*I0>0?{_+vtՅ~%t|a:ЫvEO~?_AQ+=ˀ~Og;0f>à~e›M!M}v CVx߆}\ kk[qhvVmux6 (m>t,k<u'agNߺ`n[@=̷ C~w I~ k<1Z?0y}ŵWqۮByy#uI/jNQTׇq^xTfK-UQޙ-XA ե0j-E5ɛ-yݶB'.}&yn8_Ng18@$݆O@~9k#qE_ 4@VFUxqVn/[#`CA!@t<寚ix };NUo_ Ί H \F jߑ7݊#*@]x,(z`x0/}]~7Tyg'%ee5ɛ77 ͛o yDxOɠ̟^3 Fy'?O*{edwQ*h jc'л͛[;1o7dy@"Tx~'OF\FoQ o@oǴ8u_l^) D/lq#tghG:`:L+4Kk?X![h?vAPF7:_ZA4"YR~* 8*͛kRJ6[䯌=k7ۣ硟1|5oHwx.?rq-ad -Tyv)c嗧}?Î`Enl?~6lJה{lsu蹫}Oa\hoA#q?xsO<9Z ҧ*[z;~JcTosJ˯%B{Wu{5T%ݹ\~nt!ϋs*o|ި)*V*?.YeśY>3֪CY~ x.bTym TҎ*wv.W}"Aq^vVѫ:K5α+!oZv~yǑP{쉧JzsU;uwI'UϠL]v{ S7@%4!';Nǀޡ}6 (#.}|Ƒ7&=@9%|MvVλ wah߸/.,^2'6^#'|5Š|Q)^2~8 M[>7 ʙt_ 0&;wMq5<1)7W?_&~2N(앟5v4\PxEaQ Q W>P].@_'g `~lIASr _ rӳc/Ů&)a]'oEaށ8+"v@܃E ćnwGV^'z@=;L4#Kvx+lA;6vvy/EU=⻚lsESYXg`DiP%?/?kB%U& ӽa< a `_B^öGmO]RuDjs':|p|W-v{t[a-[ɀ bӡP؁xUZ8m }ޒ>_h+n uF4[/?Iqο{T?8\|OSX.uxӞ6+ccV/kA=ZF~OB)f(?6xC.P )}CaU;X|k |N&f A˯n|zMm!hv m.sh_dh׋Go z%d5q0{U/. )/5ԣ??d=(ll { |al;uvS|oQa4%:@o7 )! vkBZ^aȮn"htY{Wg¸+_"l!)x_R&-ת%.Gߛ5YՓYhSC=^}ordM6i7}bA^P.?g9,Rz&@ 1E"{}[G]gA{ @K22^-/0bAbMڃ|iɃwڷ<W7bVC;oKg @Q++ZU93OFޞKɃ3~o)ͼa<w5Y^cyv@ a\A{{] <6?ߏ˯ +߭B0u<OB}j+z{.ЯX~h)v+a(wWr6`J  5>l> zO!wm)? ˙vwÞ qhvRyR%0j 7X;_wQÛwb(>c4 h~$øyO|c:[^ҞhC_, yF:ޅz^ _WtLg?B{V0m|_Ti`ŀw,9EP+:U|ha1~ חfu?{AvȀ~6c\/,/2<`<^‹-ʗoC49zIR#T(tuL1m5tO|/>'$˓9b΄̙9f|QS]F3_̏9g!!d8 2BCp!!d8D 1"Cp!b8D f8oÛf8oÛf8Ça8|Ça8|Çe8|_×e8|_×e8|?Ïc8?Ïc8?Ïg8ßg8ßg8#`8#`8#syyz*OBHy|#P:D!DH{"y$G"y$G"yG!yG!y4G#y4G#y/$佐`~iUk~a?}'rOteUE8r׏A47*{#yo$EX$EX$}AH"y_$CHۯC֏ `psx~V/ҕŏ~̏a?<<| DH><^G!J!Q}1/FZhq*=١~W^rb$t GV?ԢE-jQZԢE-jQZԢE-jQZԢE-jQZԢE-jQZԢE-jQZԢE-jQZԢuuyPy&Hs$2q43eK3sHHU HqLyFTkSfx<糖VuBԂyG5kҐ1>,:j:b F&_Qθag>I5/Mè :pN%$y͝S6|bɁޓ2&Ȑ}oNg{+_6M.36&=`wzofukv]}FѼr3teC C*xeaV9iQ}S[mh8(,z}m~>Ǚ8Woz~'pϢwq+}6²G)Eܸ'osܟ[]2aӋ)+W3| qe<5tY=c/X;3c<]cfU[YxᅥAO&;joZMKh]q>䭧)}  ?S\չ~Mdoߙωݺ{FRVC q˝x0/Fω(јiZ$o6x_){GGYZiU{z ~ֽY^Jިv-r5dpp띇9\V~cݟ$\%9}fN'NOt."Ej营Ϳpy-ʲyV40~!oJW|)#dGoH3l.!eFZHݴgt5Of7; ;paxLgy_/L4;Vm2(nl[s-6my..-m:8pӣk7pղez}1?|j\v'3}x;Z㤬O.^ih3״':oqgwG#.w&{LϹ1y#:$>e!lU"gVGv2h뾻zc }0[r}qü9Iz^,o{Q֭^yC[u_kcs3;ѽ:첎r˘ 3Z~C{[MuםS7n\d+fx39[_ԒyMv~?s4¢ K}iX{C[ta`y͞}e E?֊U#^<[="?=w6[9W#.3챹iyqz'Xw(dxnH5]|?ho^zt߾9! M{. kݤ#=kV[ٮΛsM\O[<շcn 'l=zsUdΥuv;vj yLL:%!t>\`aaisM3UWΥ-h,k;'/se-޵9znIz?i_Wv:r^]hu?u7nˎw4X7xI܅_~hߟD"=l8Lc\ fw?߳=ۇ7/=xh7" K_ɻ}Nzi=l#׈ݗ}g{&M,<ZKפe~7yߚim7?s;+g_}M|voS7欉_:lCg^[MкgFb7o\ݞ=,-#7RY&O\k؝σ z3|Sm|`h~317ϩ͋8x9ƿ/o?bMm'1hBwԻz [(6GiÚk~ҷ۬_ģmIBS,0mnN 8d밒GG~69vpghӁO=k0nL ymᤌ>_ޫnp˔Ͽ5\yFRI=7fߴ~7M)s俒I#w\wxbpW{񬆻x[*֛eCp4nACmݔt,RyQ6?ItxDI԰:{.9vj]W CZ5C=?pom+gݾzV~Uw.kšMߎ{|Rq*pԊ-/^4{o91;ļ桉OF=mgX`wҭ.s]~L{WmÞ W x\'/Ks>KN>}~FK.i7] dXaxnZ"WpILryVfHNs6KxSaܜ怬d^[_t7q?N2>e|(-nwHoto~+,~Żm >G:ܝ$22|/ 7]f-=$hk ='h IsAO9I=fHrܙ)%gI ?"?G,KK48'SV֝UCM|DC0U <9S!u>uQ=T^U7EP]N= at]tn,,b[}&>Cŋ}E,='# 2/) [ƱvaHnmR~ۄ0`:bX1eaǶň1f6vb#nw>+'@I?6vM(k:)Z%iqv[ӌѱI*4N&⚕+i]=x3bo,xnɂ@AKuٔ@Q?~zѬu_zIb|*kq,ϮEftոz;h֎_fIAdl3tx)W,m3N"4˯β x:S_=?~%}[\IvsbթSGٙv$nW|Fӷ6 ~#~Q_y][9}c6Q!F$g_.H{ ~zfݟ*?hst^(^_-pR=fܘ\vp* }nHf>}#\FGKNluHVЯ4GqAMQn,{O-7s_u# =GU͇ ¦Cm#J-^o:Xw{72å7ȺwH¤ajc؆glr\}|[5k46nzwhA#_%Ϯ4Wu}(qu6ZmˉCdMm79,sĐwo[*djРo#߭;) vvO I/O5'G;[6iwS&K?HkGl$s2eg%b-?Ab8;1L$q=YFUrE7L<#CB-5a 21c]Qb*˒Ȥ V{ˑg'J22i&3RG;NG4آ!'5١!I#Ck(di}fHnFNG-`Z# J!z,Hހ:A$5BFHN }ΨLtޔ:SG3tГ/'hP-PVHOkvE[6HeOsC=z"B(D(B#_|XPm"lA.na{VPA(;yNCdK8슰gwdC-z"=X>uD!?t#Y .*:U4飿gC1_cM)eի,(Ǖnj#y~By}/cz Fƃ}kl:OycIu)TaVD MrWX&ST$H}YOU=GEQVʬD7u3c%V n3eTʌ2U(cVMxղQehOv4'1LͪgO1=ETfe1M$SJ4+rNYįEp*:gse6U0J9%m*(i*vOL&񳭠Y4M.NLd{$6=OLxPFbQe6nE&eD&dG㳊Y[FmWnԞ ▩ Q#HkF)Ht$O('ILD<%w;,q6׷z ԍ܁{mW@OKӒ:d3J38v4.P (LyN>J":U"I,P#ffӒ莕)rRLA<=A"v$ʮD٭ DU3J@+ڣu8=U xպ"+NCO㯐ypˋ*|TQ>hoFT'H]+͇T`k:;br%ǎfuٶ*e,SMӤxZGjPϙ-8~GsY OR噀> LC:۪lniʝVN )ۯb=C"D{8_!o F8oMZ|7i[i'Dh q> L7iz]NP#dcS9ݘy4.ӌ9]֡Sh>R:}P[0ӭ4a ^)G |F-z1{]|t/,ضrG~jU}-zRK.\.ݥ壻u,?6-֢';;W߬&-OH;ZoUA~ +SmTќ\>{˿[__4q5ɐ#~wxt!Oj?zWP;k͓>6Og8Z΂Z˯9OT4[쉥CsҪ*.Ը⌤tw&<)f"Q.I2r˓R$9LX cC ٽ%IQ2$8ߗ3KfDe B?{ ~FZ&FγI iFڴ5Ml*[}7IIs$Ia!XEr,A$B8XtA+L})Hc|h9[/HBƋgn/Rp/`^E"/B?A- I- F(;tAEPq< ;tga(x]P JsPJxwhmάу)2C='DH$DH$D($B($Bh$Fh$F^H {!y/$A$A$佑7FX$EX$E>H y$}/E~Hə.H#y$|@HqHqH>"@$H ţrQyA$W QQ}1/FL1\bXԿSG]RV\mQ=bzu*TyJ|lzMꪫ Ke*8J jMPQVj+I}nW.ǵOGWUP\z5pZ>\[]BN4mʝxV܉2UYN^PjiU6/ PC./`OULՠZ6k@>?t]5Bݮ}߇d[|=1-&[KZYνo:|@!ؕ Muiyw}vǓ&OͮO?iۀ4ID7E#[4/i `K9vҽDZJ-޹`FkiӀ睾L񎶲#.u04?2i+b߻veFWHQ_Vu ZVf?4=Ō$3|aukoHi1{7As'Audy;aI{O7T_51;7{EuvpC]jBG~K<9zwfWHz3`{4g[g/*Ҏ]z-4zӍNP&o+3D֩ݤY*}9%}EN]6 '8Cי^/?5MEHvMאF[O;xѽeWMrn'r5锝CSHSd9u/`捛T/YK|@q|w.fyʣN tQW<] ǼsjQ%M!=ۛ1lg߻Y']:u Q{㩤yiΩSK̉װ:@G#;Hm`8NuVsHv`&t;x%iE4{,4+G0~$IdUC7$z0^C.;.J4^Lߐ2_?U!h_8xr._v@I}PGJLud_I#Ʒ&ޯӴot~v(Fa >.%>&]0ޖ8MZt7K[.4҉?,A#)믬H O:7zUTN:\O48i]+ M\Oz4.lzD:[@=@!`$& z䷤p$[D `׼%HE6O54 sJBo(}=iR@DZ2/ݚMdQuٳWzUCHunW~.&qF^4p1]$fxL:ht_&@NZ> ٘nU]sRoU۟vWE?XYJ ez^6pt./_^/*#='HdG67 :Ga-t!i A~A9q>Xq{!.WoA5j}+q<.0 w Mez`"8@30.8 g%M2y@ܧ/u$6t!WqQ4'dB[`ZMo} }Uh~v x&΁iB7vt9E[Փr\n ]:QJu%3<F|7~d`%=;qG0%v;,_G-8$닸Ik8tw1q I\yއ1qt^gmԞW[y]U'gY*|}4[q9ip|8 z9\7bl7~'? `a?r _h`sK6v1z0G_hq {VP_A>Iq^9'z/`8N]py _;N> 7ܗt9u x\ D}>\~`'?QfkU1{~~Zq8~ӄLX~^g<`qON9/q֝ƈqG~rZ|\-.q YLk%no4yj;ו7g1iXilIzi5\~)|J<'N,v(C⯿6d%L /Iz y/ѫd}V \+L*ybJɿ aúBޛSWLx\cA+_5~ G܇]LZ ?q;z5q3Lq-qo~9ʉtϠ7Z|I?pBC~܁՘~](r;q:}3X> .b?q2Zȉ`$"nwsxC[jE؏ -{ LN+c>~`L\2+Vp_Z]OuF ێ}=wƿFG}O0 xӘ8q<'~Zߢ_Vssu*#/TT; PK e5U\IG]TR=X]%OTUګEvROi/O;)״UqӼJCVz &`M,++W`,WU^.j>P+מSsxٟr5=TdƟ?@ xbeOa,A,J=r \Y_:>-j7z\|3U?1]Y ǥ~V _Ը^R5nV\+ZgƟ,>+˯}8XS(Y.~I %\׏<"8qUǿد@}~Js4yq[0S2,RE$.-SRr?E'+:] 3!s&bμ3̗9c3Cp!!d8 2BCp0_l1"Cp0s1"Cp!b8z3 7px3 7px3> p0> p0> p2 /p2 /p2~ p1~ p1~ p3 ?p3 ?p3ʯ0 Gp0_ `8#@Or|hإ0 `= P_W}hg..c0~|to3AG7QєWO 3s͜0g̙sϜ0g۲P$T{Y.WQ|xYCD~Yft2,o)_cLcקs܏#"E(ˀNR:{Rى L'QVHdf$D97-Rvʬ)A9YSZ~,;39'RBa߭KI*Fi4#w@irace/inst/CITATION0000644000176200001440000000163513137612506013460 0ustar liggesuserscitHeader("To cite package 'irace' in publications use:") citEntry( entry = "article", title = "The irace package: Iterated Racing for Automatic Algorithm Configuration", author = personList(as.person("Manuel López-Ibáñez"), as.person("Jérémie Dubois-Lacoste"), person(given = "Leslie", family = "Pérez Cáceres"), as.person("Thomas Stützle"), as.person("Mauro Birattari")), year = 2016, journal = "Operations Research Perspectives", doi = "10.1016/j.orp.2016.09.002", volume = "3", pages = "43--58", textVersion = paste0( "Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, Thomas Stützle, and Mauro Birattari. ", "The irace package: Iterated Racing for Automatic Algorithm Configuration. ", "Operations Research Perspectives, 3:43-58, 2016. doi:10.1016/j.orp.2016.09.002")) irace/inst/bin/0000755000176200001440000000000013314232645013065 5ustar liggesusersirace/inst/bin/parallel-irace-qsub0000755000176200001440000000555312763515444016660 0ustar liggesusers#!/bin/bash ############################################################################### # This script launches several runs of irace in parallel on a SGE cluster using # qsub. Execute without parameters to see usage. ############################################################################### set -e set -o pipefail # Find our own location. BINDIR=$(dirname "$(readlink -f "$(type -P $0 || echo $0)")") IRACE="$BINDIR/irace" # You may need to customize this part according to the setup of your own # cluster. QUEUE="long" #QUEUE="short" #MACHINE=opteron2216 #MACHINE=xeon5410 #MACHINE=opteron6128 MACHINE=opteron6272 # This function launches one run of irace. irace_main() { # We would like to use $BASHPID here, but OS X version of bash does not # support it. JOBNAME="irace-$$-$1" exec qsub -v PATH <&2 exit 1 fi echo \$TMP/irace.stdout >&2 exec 1> \$TMP/irace.stdout echo \$TMP/irace.stdout >&2 $IRACE --exec-dir=\$TMP --seed $RUNSEED $PARAMS RET=\$? # Use command to avoid aliases command cp --force -R \$TMP/./ $EXECDIR/ rm -rf \$TMP exit \$RET EOF } ## End of customization error () { echo "$0: error: $@" >&2 exit 1 } usage() { cat <&2 exit 1 } usage() { cat < [--machine MACHINE] [IRACE PARAMS] Parameters: N an integer giving the number of repetitions of irace or a sequence N-M giving which repetitions to redo. EXECDIR job M will use EXECDIR-M directory (default: execdir-) as the execDir (--exec-dir) parameter of irace. --parallel number of parallel jobs --machine MACHINE qsub queue type, e.g., opteron6272 IRACE PARAMS additional parameters for irace. EOF exit 1 } # Issue usage if no parameters are given. test $# -ge 1 || usage # Number of repetitions of iRace REPETITIONS=$1 shift START=1 if [[ "$REPETITIONS" =~ ^([0-9]+)-([0-9]+)$ ]] ; then START=${BASH_REMATCH[1]} REPETITIONS=${BASH_REMATCH[2]} elif ! [[ "$REPETITIONS" =~ ^[0-9]+$ ]] ; then error "number of repetitions must be an integer" fi # execDir (--exec-dir) directory EXECDIR_PREFIX=${1:-execdir} shift SEED=1234567 PARAMS= while [ $# -gt 0 ]; do case "$1" in --parallel) shift; NB_SLAVES="$1"; shift;; --seed) shift; SEED="$1"; shift;; --machine) shift; MACHINE="$1"; shift;; *) PARAMS="$PARAMS $1"; shift;;# terminate case esac done if [ $NB_SLAVES -lt 2 ]; then error "--parallel must be larger than 1" exit 1 fi let NB_PARALLEL_PROCESS=NB_SLAVES+1 for i in $(seq $START $REPETITIONS); do EXECDIR=$(printf '%s-%002d' ${EXECDIR_PREFIX} $i) echo "execution directory: $EXECDIR" rm -rf $EXECDIR mkdir -p $EXECDIR let RUNSEED=SEED+i-1 irace_main $(printf '%002d' $i) & sleep 1 done irace/inst/bin/parallel-irace0000755000176200001440000000411612763515445015703 0ustar liggesusers#!/bin/bash ############################################################################### # This script launches several runs of irace in parallel. # Execute without parameters to see usage. ############################################################################### set -e set -o pipefail # Find our own location. BINDIR=$(dirname "$(readlink -f "$(type -P $0 || echo $0)")") IRACE="$BINDIR/irace" # You may want to customize this function. irace_main() { # We would like to use $BASHPID here, but OS X version of bash does not # support it. THISPID=$(exec sh -c 'echo $PPID') # It may be faster to use /tmp. exec 1> $EXECDIR/irace-$THISPID.stdout exec 2> $EXECDIR/irace-$THISPID.stderr exec $IRACE --exec-dir=$EXECDIR --seed $RUNSEED $PARAMS } ## End of customization error () { echo "$0: error: $@" >&2 exit 1 } usage() { cat <= <= > < & | ! %in% param1 < 5 & mode == "x1" (param2 > 6 & mode == "x1") | (param2 <= 6 & mode == "x2") real < 4 & mode %in% c("x2", "x3") irace/inst/templates/target-evaluator.tmpl0000755000176200001440000000334413301314315020475 0ustar liggesusers#!/bin/bash ############################################################################### # This script is run for each candidate to evaluate it after all # candidate configurations have been run on a single instance. # # Check the examples in examples/ # # PARAMETERS: # $1 is the candidate configuration number # $2 is the instance id # $3 is the seed # $4 is the instance name # $5 is the number of candidates alive in this iteration # # ONLY FOR ELITIST RACE: The rest ($* after `shift 5') are the ids of the # candidates alive in this iteration. This list can be used to calculate the # hypervolume using previous execution results. # # RETURN VALUE: # This script should print one numerical value: the cost that must be minimized. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" exit 1 } CANDIDATE="$1" INSTANCEID="$2" SEED="$3" INSTANCE="$4" TOTALCANDIDATES="$5" shift 5 || error "Not enough parameters" ALLIDS=$* STDOUT=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdout STDERR=c${CANDIDATE}-${INSTANCEID}-${SEED}.stderr # # This may be used to introduce a delay if there are filesystem # # issues. # SLEEPTIME=1 # while [ ! -s "${STDOUT}" ]; do # sleep $SLEEPTIME # let "SLEEPTIME += 1" # done # This is an example of reading a number from the output of # target-runner. It assumes that the objective value is the first number in # the first column of the only line starting with a digit. if [ -s "${STDOUT}" ]; then COST=$(cat ${STDOUT} | grep -e '^[[:space:]]*[+-]\?[0-9]' | cut -f1) echo "$COST" rm -f "${STDOUT}" "${STDERR}" exit 0 else error "${STDOUT}: No such file or directory" fi irace/inst/templates/scenario.txt.tmpl0000644000176200001440000001721513420364214017634 0ustar liggesusers###################################################### -*- mode: r -*- ##### ## Scenario setup for Iterated Race (irace). ############################################################################ ## To use the default value of a parameter of iRace, simply do not set ## the parameter (comment it out in this file, and do not give any ## value on the command line). ## File that contains the description of the parameters of the target ## algorithm. # parameterFile = "./parameters.txt" ## Directory where the programs will be run. # execDir = "./" ## File to save tuning results as an R dataset, either absolute path or ## relative to execDir. # logFile = "./irace.Rdata" ## Previously saved log file to recover the execution of irace, either ## absolute path or relative to the current directory. If empty or NULL, ## recovery is not performed. # recoveryFile = "" ## Directory where training instances are located; either absolute path or ## relative to current directory. If no trainInstancesFiles is provided, ## all the files in trainInstancesDir will be listed as instances. # trainInstancesDir = "./Instances" ## File that contains a list of training instances and optionally ## additional parameters for them. If trainInstancesDir is provided, irace ## will search for the files in this folder. # trainInstancesFile = "" ## File that contains a set of initial configurations. If empty or NULL, ## all initial configurations are randomly generated. # configurationsFile = "" ## File that contains a list of logical expressions that cannot be TRUE ## for any evaluated configuration. If empty or NULL, do not use forbidden ## expressions. # forbiddenFile = "" ## Script called for each configuration that executes the target algorithm ## to be tuned. See templates. # targetRunner = "./target-runner" ## Number of times to retry a call to targetRunner if the call failed. # targetRunnerRetries = 0 ## Optional data passed to targetRunner. This is ignored by the default ## targetRunner function, but it may be used by custom targetRunner ## functions to pass persistent data around. # targetRunnerData = "" ## Optional R function to provide custom parallelization of targetRunner. # targetRunnerParallel = "" ## Optional script or R function that provides a numeric value for each ## configuration. See templates/target-evaluator.tmpl # targetEvaluator = "" ## Maximum number of runs (invocations of targetRunner) that will be ## performed. It determines the maximum budget of experiments for the ## tuning. # maxExperiments = 0 ## Maximum total execution time in seconds for the executions of ## targetRunner. targetRunner must return two values: cost and time. # maxTime = 0 ## Fraction (smaller than 1) of the budget used to estimate the mean ## computation time of a configuration. Only used when maxTime > 0 # budgetEstimation = 0.02 ## Maximum number of decimal places that are significant for numerical ## (real) parameters. # digits = 4 ## Debug level of the output of irace. Set this to 0 to silence all debug ## messages. Higher values provide more verbose debug messages. # debugLevel = 0 ## Number of iterations. # nbIterations = 0 ## Number of runs of the target algorithm per iteration. # nbExperimentsPerIteration = 0 ## Randomly sample the training instances or use them in the order given. # sampleInstances = 1 ## Statistical test used for elimination. Default test is always F-test ## unless capping is enabled, in which case the default test is t-test. ## Valid values are: F-test (Friedman test), t-test (pairwise t-tests with ## no correction), t-test-bonferroni (t-test with Bonferroni's correction ## for multiple comparisons), t-test-holm (t-test with Holm's correction ## for multiple comparisons). # testType = "F-test" ## Number of instances evaluated before the first elimination test. It ## must be a multiple of eachTest. # firstTest = 5 ## Number of instances evaluated between elimination tests. # eachTest = 1 ## Minimum number of configurations needed to continue the execution of ## each race (iteration). # minNbSurvival = 0 ## Number of configurations to be sampled and evaluated at each iteration. # nbConfigurations = 0 ## Parameter used to define the number of configurations sampled and ## evaluated at each iteration. # mu = 5 ## Confidence level for the elimination test. # confidence = 0.95 ## If the target algorithm is deterministic, configurations will be ## evaluated only once per instance. # deterministic = 0 ## Seed of the random number generator (by default, generate a random ## seed). # seed = NA ## Number of calls to targetRunner to execute in parallel. Values 0 or 1 ## mean no parallelization. # parallel = 0 ## Enable/disable load-balancing when executing experiments in parallel. ## Load-balancing makes better use of computing resources, but increases ## communication overhead. If this overhead is large, disabling ## load-balancing may be faster. # loadBalancing = 1 ## Enable/disable MPI. Use Rmpi to execute targetRunner in parallel ## (parameter parallel is the number of slaves). # mpi = 0 ## Specify how irace waits for jobs to finish when targetRunner submits ## jobs to a batch cluster: sge, pbs, torque or slurm. targetRunner must ## submit jobs to the cluster using, for example, qsub. # batchmode = 0 ## Enable/disable the soft restart strategy that avoids premature ## convergence of the probabilistic model. # softRestart = 1 ## Soft restart threshold value for numerical parameters. If NA, NULL or ## "", it is computed as 10^-digits. # softRestartThreshold = "" ## Directory where testing instances are located, either absolute or ## relative to current directory. # testInstancesDir = "" ## File containing a list of test instances and optionally additional ## parameters for them. # testInstancesFile = "" ## Number of elite configurations returned by irace that will be tested if ## test instances are provided. # testNbElites = 1 ## Enable/disable testing the elite configurations found at each ## iteration. # testIterationElites = 0 ## Enable/disable elitist irace. # elitist = 1 ## Number of instances added to the execution list before previous ## instances in elitist irace. # elitistNewInstances = 1 ## In elitist irace, maximum number per race of elimination tests that do ## not eliminate a configuration. Use 0 for no limit. # elitistLimit = 2 ## User-defined R function that takes a configuration generated by irace ## and repairs it. # repairConfiguration = "" ## Enable the use of adaptive capping, a technique designed for minimizing ## the computation time of configurations. This is only available when ## elitist is active. # capping = 0 ## Measure used to obtain the execution bound from the performance of the ## elite configurations: median, mean, worst, best. # cappingType = "median" ## Method to calculate the mean performance of elite configurations: ## candidate or instance. # boundType = "candidate" ## Maximum execution bound for targetRunner. It must be specified when ## capping is enabled. # boundMax = 0 ## Precision used for calculating the execution time. It must be specified ## when capping is enabled. # boundDigits = 0 ## Penalization constant for timed out executions (executions that reach ## boundMax execution time). # boundPar = 1 ## Replace the configuration cost of bounded executions with boundMax. # boundAsTimeout = 1 ## Percentage of the configuration budget used to perform a postselection ## race of the best configurations of each iteration after the execution ## of irace. # postselection = 0 ## Enable/disable AClib mode. This option enables compatibility with ## GenericWrapper4AC as targetRunner script. # aclib = 0 ## END of scenario file ############################################################################ irace/inst/templates/instances-list.txt.tmpl0000644000176200001440000000062212763515445021000 0ustar liggesusers## This is an example of specifying instances with a file. # Each line is an instance relative to trainInstancesDir # (see scenario.txt.tmpl) and an optional sequence of instance-specific # parameters that will be passed to target-runnerx when invoked on that # instance. # Empty lines and comments are ignored. 100/100-1_100-2.tsp --time 1 100/100-1_100-3.tsp --time 2 100/100-1_100-4.tsp --time 3 irace/inst/templates/windows/0000755000176200001440000000000013366056446016017 5ustar liggesusersirace/inst/templates/windows/target-runner.bat0000755000176200001440000000431613366056446021313 0ustar liggesusers@echo off ::############################################################################## :: BAT version of target-runner for Windows. :: Contributed by Andre de Souza Andrade . :: Check other examples in examples/ :: :: This script is run in the execution directory (execDir, --exec-dir). :: :: PARAMETERS: :: %%1 is the candidate configuration number :: %%2 is the instance ID :: %%3 is the seed :: %%4 is the instance name :: The rest are parameters to the target-algorithm :: :: RETURN VALUE: :: This script should print one numerical value: the cost that must be minimized. :: Exit with 0 if no error, with 1 in case of error ::############################################################################## :: Please change the EXE and FIXED_PARAMS to the correct ones SET "exe=D:\bin\program" SET "fixed_params=--time 20 --tries 1 --quiet" FOR /f "tokens=1-4*" %%a IN ("%*") DO ( SET candidate=%%a SET instance_id=%%b SET seed=%%c SET instance=%%d SET candidate_parameters=%%e ) SET "stdout=c%candidate%-%instance_id%-%seed%.stdout" SET "stderr=c%candidate%-%instance_id%-%seed%.stderr" :: :: FIXME: How to implement this in BAT ? ::if not exist %exe% error "%exe%: not found or not executable (pwd: %(pwd))" :: Save the output to a file, and parse the result from it. %exe% %fixed_params% -i %instance% --seed %seed% %candidate_parameters% 1>%stdout% 2>%stderr% :: :: This may be used to introduce a delay if there are filesystem :: :: issues. :: setlocal EnableDelayedExpansion :: :loop :: for /f %%i in ("%stdout%") do set size=%%~zi :: if "%size%" NEQ "0" (goto endloop) :: waitfor false /t 10 >nul :: goto loop :: :endloop :: This is an example of reading a number from the output. :: It assumes that the objective value is the first number in :: the first column of the last line of the output. setlocal EnableDelayedExpansion set "cmd=findstr /R /N "^^" %stdout% | find /C ":"" for /f %%a in ('!cmd!') do set numberlines=%%a set /a lastline=%numberlines%-1 for /f "tokens=3" %%F in ('more +%lastline% %stdout%') do set COST=%%F echo %COST% :: Un-comment this if you want to delete temporary files. :: del %stdout% %stderr% exit 0 irace/inst/templates/target-runner.tmpl0000755000176200001440000000407513301314302020002 0ustar liggesusers#!/bin/bash ############################################################################### # This script is the command that is executed every run. # Check the examples in examples/ # # This script is run in the execution directory (execDir, --exec-dir). # # PARAMETERS: # $1 is the candidate configuration number # $2 is the instance ID # $3 is the seed # $4 is the instance name # The rest ($* after `shift 4') are parameters to the run # # RETURN VALUE: # This script should print one numerical value: the cost that must be minimized. # Exit with 0 if no error, with 1 in case of error ############################################################################### error() { echo "`TZ=UTC date`: $0: error: $@" exit 1 } EXE=~/bin/program FIXED_PARAMS="" CONFIG_ID=$1 INSTANCE_ID=$2 SEED=$3 INSTANCE=$4 shift 4 || error "Not enough parameters" CONFIG_PARAMS=$* STDOUT=c${CONFIG_ID}-${INSTANCE_ID}-${SEED}.stdout STDERR=c${CONFIG_ID}-${INSTANCE_ID}-${SEED}.stderr if [ ! -x "${EXE}" ]; then error "${EXE}: not found or not executable (pwd: $(pwd))" fi # If the program just prints a number, we can use 'exec' to avoid # creating another process, but there can be no other commands after exec. #exec $EXE ${FIXED_PARAMS} -i $INSTANCE ${CONFIG_PARAMS} # exit 1 # # Otherwise, save the output to a file, and parse the result from it. # (If you wish to ignore segmentation faults you can use '{}' around # the command.) $EXE ${FIXED_PARAMS} -i $INSTANCE --seed ${SEED} ${CONFIG_PARAMS} 1> ${STDOUT} 2> ${STDERR} # # This may be used to introduce a delay if there are filesystem # # issues. # SLEEPTIME=1 # while [ ! -s "${STDOUT}" ]; do # sleep $SLEEPTIME # let "SLEEPTIME += 1" # done # This is an example of reading a number from the output. # It assumes that the objective value is the first number in # the first column of the last line of the output. if [ -s "${STDOUT}" ]; then COST=$(tail -n 1 ${STDOUT} | grep -e '^[[:space:]]*[+-]\?[0-9]' | cut -f1) echo "$COST" rm -f "${STDOUT}" "${STDERR}" exit 0 else error "${STDOUT}: No such file or directory" fi irace/inst/templates/configurations.txt.tmpl0000644000176200001440000000136112763515445021073 0ustar liggesusers## Template for specifying initial parameter configurations, for ## example, the default configuration, in irace. ## ## This filename must be specified via the --candidates-file command-line option ## (or candidatesFile in scenario.txt). ## ## The format is one parameter configuration per line, and one parameter per ## column. The first line gives the parameter name corresponding to ## each column (names must match those given in the parameters ## file). Each configuration must satisfy the ## parameter conditions (NA should be used for those parameters that ## are not enabled for a given configuration) and, if given, the ## constraints that describe forbidden configurations. param1 param2 mode real mutation 5 NA "x2" 2.0 "low" irace/inst/templates/parameters.txt.tmpl0000644000176200001440000000316312763515445020206 0ustar liggesusers## Template for parameter description file for Iterated Race. ## ## The format is one parameter per line. Each line contains: ## ## 1: Name of the parameter. An unquoted alphanumeric string, ## example: ants ## 2: Switch to pass the parameter. A quoted (possibly empty) string, ## if the value and the switch must be separated, add a space at ## the end of the string. Example : "--version1 --ants " ## 3: Type. An unquoted single letter, among ## i: Integer, c: categorical, o: ordinal, r: real. ## 4: For c and o: All possible values, that is, a variable number of ## quoted or unquoted strings separated by commas within ## parenthesis. Empty strings and strings containing commas or ## spaces must be quoted. ## For i,r: a pair of unquoted numbers representing minimum and ## maximum values. ## 5: A conditional parameter can be defined according to the values of ## one or several other parameters. This is done by adding a ## character '|' followed by an R expression involving the names of ## other parameters. This expression must return TRUE if the ## condition is satisfied, FALSE otherwise. # 1: 2: 3: 4: 5: param1 "--param1 " i (1, 10) | mode %in% c("x1", "x2") param2 "--param2 " i (1, 10) | mode %in% c("x1", "x3") && real > 2.5 && real <= 3.5 mode "--" c ("x1" ,"x2", "x3") real "--paramreal=" r (1.5, 4.5) mutation "--mutation=" o ("none", "very low", "low", "medium", "high", "very high", "all") #unused "-u " c (1, 2, 10, 20) irace/inst/doc/0000755000176200001440000000000013460635613013066 5ustar liggesusersirace/inst/doc/irace-package.Rnw0000644000176200001440000044314713460635577016260 0ustar liggesusers%%% DO NOT EDIT the .tex file directly since it is generated from the .Rnw %%% sources. %\VignetteEngine{knitr::knitr} %\VignetteIndexEntry{irace package: User Guide} %\VignetteDepends{knitr} %\VignetteCompiler{knitr} \RequirePackage[dvipsnames]{xcolor} \documentclass[a4paper,english]{article} \usepackage[a4paper]{geometry} % It saves some pages \usepackage[utf8]{inputenc} \usepackage[T1]{fontenc} \usepackage[english]{babel} \usepackage{ifthen} \newboolean{Release} \setboolean{Release}{false} \setboolean{Release}{true} \usepackage{calc} \usepackage{afterpage} \usepackage{algorithm,algorithmic} \usepackage{booktabs} \usepackage{tabularx} \usepackage{xspace} \usepackage{amsmath,amssymb} \usepackage{relsize} \usepackage{fancyvrb} \usepackage[hyphens]{url} \usepackage{hyperref} \usepackage[numbers]{natbib} \usepackage[nottoc]{tocbibind} %% For autoref \hypersetup{ colorlinks, linkcolor={red!50!black}, citecolor={blue!50!black}, urlcolor={blue!70!black} } \addto\extrasenglish{% \def\sectionautorefname{Section} \let\subsectionautorefname\sectionautorefname \let\subsubsectionautorefname\sectionautorefname } \usepackage[titletoc, title]{appendix} % Fix use with \autoref \newcommand*{\Appendixautorefname}{Appendix} \usepackage{tocloft} \setlength{\cftsubsecnumwidth}{3em}% Set length of number width in ToC for \subsection \usepackage[inline]{enumitem} \setlist[enumerate]{leftmargin=*,widest=00} \setlist[itemize]{leftmargin=1.5em} %% FIXME: listing is very limited, we should use 'minted' \usepackage{listings} \lstdefinestyle{BashInputStyle}{ language=bash,% basicstyle=\ttfamily,% numbers=none,% frame=tb,% rulecolor=\color{lightgray}, % framesep=1ex, framexleftmargin=1ex, columns=fullflexible,% backgroundcolor=\color{yellow!05},% linewidth=\linewidth,% % xleftmargin=1\linewidth,% identifierstyle=\color{darkgray},% keywordstyle=\color{darkgray},% keywordstyle={[2]\color{Cyan}},% keywordstyle={[3]\color{olive}},% stringstyle=\color{MidnightBlue},% commentstyle=\color{RedOrange},% morestring=[b]',% showstringspaces=false } \DefineVerbatimEnvironment{Code}{Verbatim}{} \DefineVerbatimEnvironment{CodeInput}{Verbatim}{fontshape=rm} \DefineVerbatimEnvironment{CodeOutput}{Verbatim}{} \newenvironment{CodeChunk}{}{} \newcommand{\IRACEHOME}[1]{\hyperlink{irace_home}{\path{$IRACE_HOME}}\path{#1}} \providecommand{\keywords}[1]{\textbf{\textit{Index terms---}} #1} % Simple font selection is not good enough. For example, |\texttt{--}| % gives `\texttt{--}', i.e., an endash in typewriter font. Hence, we % need to turn off ligatures, which currently only happens for commands % |\code| and |\samp| and the ones derived from them. Hyphenation is % another issue; it should really be turned off inside |\samp|. And % most importantly, \LaTeX{} special characters are a nightmare. E.g., % one needs |\~{}| to produce a tilde in a file name marked by |\file|. % Perhaps a few years ago, most users would have agreed that this may be % unfortunate but should not be changed to ensure consistency. But with % the advent of the WWW and the need for getting `|~|' and `|#|' into % URLs, commands which only treat the escape and grouping characters % specially have gained acceptance \makeatletter \DeclareRobustCommand\code{\bgroup\@makeother\_\@makeother\~\@makeother\$\@noligs\@codex} \def\@codex#1{\texorpdfstring% {{\normalfont\ttfamily\hyphenchar\font=-1 #1}}% {#1}\egroup} \makeatother \let\proglang=\textsf \newcommand{\pkg}[1]{{\fontseries{b}\selectfont #1}} \newcommand{\aR}{\proglang{R}\xspace} \newcommand{\MATLAB}{\proglang{MATLAB}\xspace} \newcommand{\eg}{e.g.,\xspace} \newcommand{\SoftwarePackage}{\pkg} \newcommand{\ACOTSP}{\SoftwarePackage{ACOTSP}\xspace} %% How to use this command: % Parameter with one short switch: \defparameter[short]{paramName}{long}{default} % Parameter without short switch: \defparameter{paramName}{long}{default} % Parameter without switch: \defparameter{paramName}{}{default} \newcommand{\defparameter}[4][]{% \item[\code{#2}]\hypertarget{opt:#2}{} ~~ % \ifthenelse{\equal{#3}{}}{}{% \emph{flag:} % \ifthenelse{\equal{#1}{}}{}{% \code{-#1}~~~\emph{or}~~~}% \code{-{}-#3} ~~ }% \emph{default:}~\texttt{#4} \\ } \newcommand{\parameter}[1]{\hyperlink{opt:#1}{\code{#1}}} %\usepackage{showlabels} %\showlabels{hypertarget} \newcommand{\irace}{\pkg{irace}\xspace} \newcommand{\Irace}{\pkg{Irace}\xspace} \newcommand{\race}{\pkg{race}\xspace} \newcommand{\FRACE}{\text{F-Race}\xspace} \newcommand{\IFRACE}{\text{I/F-Race}\xspace} \newcommand{\iraceversion}{3.3} \newcommand{\Niter}{\ensuremath{N^\text{iter}}\xspace} \newcommand{\Nparam}{\ensuremath{{N^\text{param}}}\xspace} \newcommand{\iter}{\ensuremath{j}\xspace} \newcommand{\Budget}{\ensuremath{B}\xspace} \newcommand{\Budgetj}{\ensuremath{\Budget_{\iter}}\xspace} \newcommand{\Bused}{\ensuremath{\Budget_\text{used}}\xspace} \newcommand{\Ncand}[1][]{\ensuremath{N_{#1}}\xspace} \newcommand{\Mui}{\ensuremath{\mu_{\iter}}\xspace} \newcommand{\Nmin}{\ensuremath{N^\text{min}}\xspace} \newcommand{\Nsurv}{\ensuremath{N^\text{surv}}\xspace} \newcommand{\Nelite}{\ensuremath{N^\text{elite}}\xspace} \newcommand{\Nnew}{\ensuremath{N^\text{new}}\xspace} \newcommand{\bmax}{\ensuremath{b^\text{max}}\xspace} \newcommand{\bmin}{\ensuremath{b^\text{min}}\xspace} \newcommand{\Celite}{\ensuremath{\Theta^\text{elite}}\xspace} \ifthenelse {\boolean{Release}}{% \newcommand{\MANUEL}[1]{} \newcommand{\LESLIE}[1]{} \newcommand{\THOMAS}[1]{} }{% \newcommand{\MANUEL}[1]{{\footnotesize\noindent\textbf{\color{red}[~MANUEL: #1~]}}} \newcommand{\LESLIE}[1]{\footnote{\noindent\textbf{[ LESLIE: #1 ]}}} \newcommand{\THOMAS}[1]{\footnote{\noindent\textbf{[ THOMAS: #1 ]}}} } \newcommand{\hide}[1]{} \usepackage{tcolorbox} \newcommand{\infoicon}{% \parbox[c]{0.75cm}{\includegraphics[keepaspectratio=true,width=0.75cm]{light-bulb-icon}}% \hspace{1em}} \newcommand{\warningicon}{% \parbox[c]{0.75cm}{\includegraphics[keepaspectratio=true,width=0.75cm]{Warning-icon}}% \hspace{1em}} \definecolor{LightGray}{RGB}{193,193,193} \definecolor{LightYellow}{RGB}{253,247,172} \newlength\macroiconwidth \newenvironment{xwarningbox}{% \setlength{\fboxrule}{3.0\fboxrule}% \setlength{\fboxsep}{0\fboxsep}% \begin{tcolorbox}[colback=LightYellow,colframe=LightGray,boxrule=\fboxrule,boxsep=\fboxsep]% \infoicon% \settowidth{\macroiconwidth}{\infoicon}% \begin{minipage}[c]{\columnwidth - \macroiconwidth - 2.0\fboxrule - 2.0\fboxsep} \raggedright\footnotesize % }{% \end{minipage} \end{tcolorbox} % } \begin{document} <>= library(knitr) @ \author{Manuel L\'opez-Ib\'a\~nez, Leslie P\'erez C\'aceres, J\'er\'emie Dubois-Lacoste,\\ Thomas St\"utzle and Mauro Birattari \\IRIDIA, CoDE, Universit\'e Libre de Bruxelles, Brussels, Belgium} \title{The \irace Package: User Guide} \date{Version \iraceversion, \today} %\keywords{automatic % algorithm configuration, racing, parameter tuning, \aR} \maketitle \tableofcontents %Load files needed for examples <>= library("irace") load("examples.Rdata") load("irace-acotsp.Rdata") load("log-ablation.Rdata") options(width = 70) @ \newpage %% %% %% %% General info %% %% %% \section{General information} \MANUEL{Some things could be taken from the intro of the irace paper and reformulated.} \MANUEL{It would be good to mention that not only opt algorithms can be configured with irace, we say this in the paper.} \subsection{Background} \MANUEL{I would add a paragraph defining what is irace (a bit longer than the abstract above) and references to the literature so people can find more info. The first reference should be the irace TR.} \LESLIE{Here i guess we should say why tune an algorithm is a good idea, and why using irace is a better one.} The \irace package implements an \emph{iterated racing} procedure, which is an extension of Iterated F-race (\IFRACE)~\cite{BirYuaBal2010:emaoa}. The main use of \irace is the automatic configuration of optimization and decision algorithms, that is, finding the most appropriate settings of an algorithm given a set of instances of a problem. However, it may also be useful for configuring other types of algorithms when performance depends on the used parameter settings. It builds upon the \pkg{race} package by Birattari and it is implemented in \aR. The \irace package is available from CRAN: % \begin{center} \url{https://cran.r-project.org/package=irace} \end{center} % More information about \irace is available at \url{http://iridia.ulb.ac.be/irace}. \subsection{Version} The current version of the \irace package is \iraceversion. Previous versions of the package can also be found in the \href{https://cran.r-project.org/package=irace}{CRAN website}. The algorithm underlying the current version of \irace and its motivation are described by \citet{LopDubPerStuBir2016irace}. The \textbf{adaptive capping mechanism} available from version $3.0$ is described by \citet{PerLopHooStu2017:lion}. Details of the implementation before version 2.0 can be found in a previous technical report~\cite{LopDubStu2011irace}. % \begin{xwarningbox} Versions of \irace before 2.0 are not compatible with the file formats detailed in this document. \end{xwarningbox} \subsection{License} The \irace package is Copyright \copyright{} \the\year\ and distributed under the GNU General Public License version 3.0 (\url{http://www.gnu.org/licenses/gpl-3.0.en.html}). The \irace package is free software (software libre): You can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. The \irace package is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Please be aware that the fact that this program is released as Free Software does not excuse you from scientific propriety, which obligates you to give appropriate credit! If you write a scientific paper describing research that made substantive use of this program, it is your obligation as a scientist to (a) mention the fashion in which this software was used in the Methods section; (b) mention the algorithm in the References section. The appropriate citation is: \begin{itemize}[leftmargin=3em] \item[] Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, Thomas Stützle, and Mauro Birattari. The \irace package: Iterated Racing for Automatic Algorithm Configuration. \emph{Operations Research Perspectives}, 3:43--58, 2016. doi:~\href{http://dx.doi.org/10.1016/j.orp.2016.09.002}{10.1016/j.orp.2016.09.002} \end{itemize} \section{Before starting} \MANUEL{I think this could be a bit more detailed by defining what is a parameter, a configuration, an instance, etc. but ok for now.} The \irace package provides an automatic configuration tool for tuning optimization algorithms, that is, automatically finding good configurations for the parameters values of a (target) algorithm saving the effort that normally requires manual tuning. \begin{figure}[t] \centering \includegraphics[width=0.6\textwidth]{irace-scheme} \caption{Scheme of \irace flow of information.} \label{fig:irace-scheme} \end{figure} Figure~\ref{fig:irace-scheme} gives a general scheme of how \irace works. \Irace receives as input a \emph{parameter space definition} corresponding to the parameters of the target algorithm that will be tuned, a set of \emph{instances} for which the parameters must be tuned for and a set of options for \irace that define the \emph{configuration scenario}. Then, \irace searches in the parameter search space for good performing algorithm configurations by executing the target algorithm on different instances and with different parameter configurations. A \parameter{targetRunner} must be provided to execute the target algorithm with a specific parameter configuration ($\theta$) and instance ($i$). The \parameter{targetRunner} function (or program) acts as an interface between the execution of the target algorithm and \irace: It receives the instance and configuration as arguments and must return the evaluation of the execution of the target algorithm. The following user guide contains guidelines for installing \irace, defining configuration scenarios, and using \irace to automatically configure your algorithms. %% %% %% %% Installation %% %% %% \section{Installation} \subsection{System requirements} \begin{itemize} \item \aR ($\text{version} \geq 3.2.0$) is required for running irace, but you don't need to know the \aR language to use it. \aR is freely available and you can download it from the \aR project website (\url{https://www.r-project.org}). See \autoref{sec:installation} for a quick installation guide of \aR. \item For GNU/Linux and OS X, the command-line executables \code{irace} and \code{parallel-irace} require GNU Bash. There is also an \code{irace.bat} for Windows. Individual examples may require additional software. \end{itemize} \subsection{\irace installation} \label{sec:irace install} The \irace package can be installed automatically within \aR or by manual download and installation. We advise to use the automatic installation unless particular circumstances do not allow it. The instructions to install \irace with the two mentioned methods are the following: \subsubsection[Install automatically within R]{Install automatically within \aR{}} Execute the following line in the \aR console to install the package: <>= install.packages("irace") @ Select a mirror close to your location, and test the installation in the \aR console with: <>= library("irace") q() # To exit R @ Alternatively, within the \aR graphical interface, you may use the \code{Packages and data->Package installer} menu on OS X or the \code{Packages} menu on Windows. \subsubsection{Manual download and installation} From the \irace package CRAN website (\url{https://cran.r-project.org/package=irace}), download one of the three versions available depending on your operating system: \begin{itemize} \item \code{irace_\iraceversion.tar.gz} (Unix/BSD/GNU/Linux) \item \code{irace_\iraceversion.tgz} (OS X) \item \code{irace_\iraceversion.zip} (Windows) \end{itemize} To install the package on GNU/Linux and OS X, you must execute the following command at the shell (replace \code{} with the path to the downloaded file, either \code{irace_\iraceversion.tar.gz} or \code{irace_\iraceversion.zip}): % \begin{lstlisting}[style=BashInputStyle] R CMD INSTALL \end{lstlisting} To install the package on Windows, open \aR and execute the following line on the \aR console (replace \code{} with the path to the downloaded file \code{irace_\iraceversion.zip}): %\LESLIE{Check that this actually works on internet says that this: \code{Rscript -e "install.packages('foo.zip', repos = NULL)"} also works} % <>= install.packages("", repos = NULL) @ If the previous installation instructions fail because of insufficient permissions and you do not have sufficient admin rights to install \irace system-wide, then you need to force a local installation. \subsubsection{Local installation} Let's assume you wish to install \irace on a path denoted by \code{}, which is a filesystem path for which you have sufficient rights. This directory \textbf{must} exist before attempting the installation. Moreover, you must provide to \aR the path to this library when loading the package. However, the latter can be avoided by adding the path to the system variable \code{R_LIBS} or to the \aR internal variable \code{.libPaths}, as we will see below.\footnote{% On Windows, see also \url{https://cran.r-project.org/bin/windows/base/rw-FAQ.html\#I-don_0027t-have-permission-to-write-to-the-R_002d3_002e3_002e1_005clibrary-directory}.} On GNU/Linux or OS X, execute the following commands to install the package on a local directory: \begin{lstlisting}[style=BashInputStyle] export R_LIBS_USER="" # Create R_LIBS_USER if it doesn't exist mkdir $R_LIBS_USER # Replace with the path to the downloaded file. R CMD INSTALL --library=$R_LIBS_USER # Tell R where to find R_LIBS_USER export R_LIBS=${R_LIBS_USER}:${R_LIBS} \end{lstlisting} On Windows, you can install the package on a local directory by executing the following lines in the \aR console: <>= # Replace with the path to the downloaded file. # Replace with the path used for installation. install.packages("", repos = NULL, lib = "") # Tell R where to find R_LIBS_USER. # This must be executed for every new session. .libPaths(c("", .libPaths())) @ \subsubsection{Testing the installation and invoking irace} Once \irace has been installed, load the package and test that the installation was successful by opening an \aR console and executing: <>= # Load the package library("irace") # Obtain the installation path system.file(package = "irace") @ The last command must print out the filesystem path where \irace is installed. In the remainder of this guide, the variable \code{\$IRACE_HOME} is used to denote this path. When executing any provided command that includes the \code{\$IRACE_HOME} variable do not forget to replace this variable with the installation path of \irace. On GNU/Linux or OS X, you can let the operating system know where to find \irace by defining the \code{\$IRACE_HOME} variable and adding it to the system \code{PATH}. Append the following commands to \path{~/.bash_profile}, \path{~/.bashrc} or \path{~/.profile}: % %<>= \begin{lstlisting}[style=BashInputStyle] # Replace with the irace installation path export IRACE_HOME= export PATH=${IRACE_HOME}/bin/:$PATH # Tell R where to find R_LIBS_USER # Use the following line only if local installation was forced export R_LIBS=${R_LIBS_USER}:${R_LIBS} \end{lstlisting} %@ Then, open a new terminal and launch \irace as follows: %<>= \begin{lstlisting}[style=BashInputStyle] irace --help \end{lstlisting} %@ On Windows, you need to add both \aR and the installation path of \irace to the environment variable \code{PATH}. To edit the \code{PATH}, search for ``Environment variables'' in the control panel, edit \code{PATH} and add a string similar to \path{C:\R_PATH\bin;C:\IRACE_HOME\bin} where \code{R_PATH} is the installation path of \aR and \code{IRACE_HOME} is the installation path of \irace. If \irace was installed locally, you also need to edit the environment variable \code{R_LIBS} to add \code{R_LIBS_USER}. Then, open a new terminal (run program \code{cmd.exe}) and launch \irace as: % %<>= \begin{lstlisting}[style=BashInputStyle] irace.bat --help \end{lstlisting} %@ Alternatively, you may directly invoke \irace from within the \aR console by executing: <>= library("irace") irace.cmdline("--help") @ \section{Running irace}\label{sec:execution} Before performing the tuning of your algorithm, it is necessary to define a tuning scenario that will give \irace all the necessary information to optimize the parameters of the algorithm. The tuning scenario is composed of the following elements: \begin{enumerate} \item Target algorithm parameter description (see \autoref{sec:target parameters}). \item Target algorithm runner (see \autoref{sec:runner}). \item Training instances list (see \autoref{sec:training}) \item \irace options (see \autoref{sec:irace options}). \item \textit{Optional:} Initial configurations (see \autoref{sec:initial}). \item \textit{Optional:} Forbidden configurations (see \autoref{sec:forbidden}). \item \textit{Optional:} Target algorithm evaluator (see \autoref{sec:evaluator}). \end{enumerate} These scenario elements can be provided as plain text files or as \aR objects. This user guide provides examples of both types, but we advise the use of plain text files, which we consider the simpler option. For a step-by-step guide to create the scenario elements for your target algorithm continue to \autoref{sec:step}. For an example execution of \irace using the \ACOTSP scenario go to \autoref{sec:example}. \subsection{Step-by-step setup guide}\label{sec:step} This section provides a guide to setup a basic execution of \irace. The template files provided in the package (\IRACEHOME{/templates}) will be used as basis for creating your new scenario. Please follow carefully the indications provided in each step and in the template files used; if you have doubts check the the sections that describe each option in detail. \begin{enumerate}[leftmargin=*] \item Create a directory (\eg~\path{~/tuning/}) for the scenario setup. This directory will contain all the files that describe the scenario. On GNU/Linux or OS X, you can do this as follows: %<>= \begin{lstlisting}[style=BashInputStyle] mkdir ~/tuning cd ~/tuning \end{lstlisting} %@ \item Copy all the template files from the \IRACEHOME{/templates/} directory to the scenario directory. %<>= \begin{lstlisting}[style=BashInputStyle] # $IRACE_HOME is the installation directory of irace. cp $IRACE_HOME/templates/*.tmpl ~/tuning/ \end{lstlisting} %@ % Remember that \IRACEHOME{} is the path to the installation % directory of \irace. It can be obtained in the \aR console with: % % <>= % library("irace") % system.file(package = "irace") % @ \item For each template in your tuning directory, remove the \code{.tmpl} suffix, and modify them following the next steps. \item Define the target algorithm parameters to be tuned by following the instructions in \code{parameters.txt}. Available parameter types and other guidelines can be found in \autoref{sec:target parameters}. \item \textit{Optional}: Define the initial parameter configuration(s) of your algorithm, which allows you to provide good starting configurations (if you know some) for the tuning. Follow the instructions in \code{configurations.txt} and set \parameter{configurationsFile}\code{="configurations.txt"} in \code{scenario.txt}. More information in \autoref{sec:initial}. If you do not need to define initial configurations remove this file from the directory. \item \textit{Optional}: Define forbidden parameter value combinations, that is, configurations that \irace must not consider in the tuning. Follow the instructions in \code{forbidden.txt} and update \code{scenario.txt} with \parameter{forbiddenFile} \code{=} \code{"forbidden.txt"}. More information about forbidden configurations in \autoref{sec:forbidden}. If you do not need to define forbidden configurations remove this file from the directory. \item Place the instances you would like to use for the tuning of your algorithm in the folder \path{~/tuning/Instances/}. In addition, you can create a file (\eg~\code{instances-list.txt}) that specifies which instances from that directory should be run and which instance-specific parameters to use. To use such an instance file, set the appropriate option in \code{scenario.txt}, e.g., \parameter{trainInstancesFile} \code{= "instances-list.txt"}. See \autoref{sec:training} for guidelines. \item Uncomment and assign in \code{scenario.txt} only the options for which you need a value different from the default. %% MANUEL: I'm not sure what this means. % The names of the template files match the default names of the scenario % options. Some common parameters that you might want to adjust are: \begin{description} \item[\parameter{execDir}] (\code{--exec-dir}): the directory in which \irace will execute the target algorithm; the default value is the current directory. %\item[\parameter{logFile}] (\code{--log-file}): a file the name of the results \aR data file that produces \irace. \item[\parameter{maxExperiments}] (\code{--max-experiments}): the maximum number of executions of the target algorithm that \irace will perform. \item[\parameter{maxTime}] (\code{--max-time}): maximum total execution time in seconds for the executions of \code{targetRunner}. In this case, \code{targetRunner} must return two values: cost and time. Note that you must provide either \parameter{maxTime} or \parameter{maxExperiments}. \end{description} For setting the tuning budget see \autoref{sec:budget}. For more information on \irace options and their default values, see \autoref{sec:irace options}. \item Modify the \code{target-runner} script to run your algorithm. This script must execute your algorithm with the parameters and instance specified by \irace and return the evaluation of the execution and \textit{optionally} the execution time (\code{cost [time]}). When the \parameter{maxTime} option is used, returning \code{time} is mandatory. The \code{target-runner} template is written in \proglang{GNU Bash} scripting language, which can be executed easily in GNU/Linux and OS X systems. However, you may use any other programming language. As an example, we provide a \proglang{Python} example in the directory \IRACEHOME{/examples/python}. % Follow these instructions to adjust the given \code{target-runner} template to your algorithm: \begin{enumerate} \item Set the \code{EXE} variable with the path to the executable of the target algorithm. \item Set the \code{FIXED_PARAMS} if you need extra arguments in the execution line of your algorithm. An example could be the time that your algorithm is required to run (\code{FIXED_PARAMS}\hspace{0pt}\code{=}\hspace{0pt}\code{"--time 60"}) or the number of evaluations required (\code{FIXED_PARAMS}\hspace{0pt}\code{=}\hspace{0pt}\code{"--evaluations 10000"}). \item The line provided in the template executes the executable described in the \code{EXE} variable. % \begin{center} \code{\$EXE \$\{FIXED_PARAMS\} -i \$\{INSTANCE\} --seed \$\{SEED\} \$\{CONFIG_PARAMS\}} \end{center} % You must change this line according to the way your algorithm is executed. In this example, the algorithm receives the instance to solve with the flag \code{-i} and the seed of the random number generator with the flag \code{--seed}. The variable \code{CONFIG_PARAMS} adds to the command line the parameters that \irace has given for the execution. You must set the command line execution as needed. For example, the instance might not need a flag and might need to be the first argument: \begin{center} \code{\$EXE \$\{INSTANCE\} \$\{FIXED_PARAMS\} --seed \$\{SEED\} \$\{CONFIG_PARAMS\}} \end{center} The output of your algorithm is saved to the file defined in the \code{\$STDOUT} variable, and error output is saved in the file given by \code{\$STDERR}. The line: \begin{center} \code{if [ -s "\${STDOUT}" ]; then} \end{center} checks if the file containing the output of your algorithm is not empty. The example provided in the template assumes that your algorithm prints in the last output line the best result found (only a number). The line: \begin{center} \code{COST=\$(cat \$\{STDOUT\} | grep -e '\^{}[[:space:]]*[+-]\textbackslash{}?[0-9]' | cut -f1)} \end{center} parses the output of your algorithm to obtain the result from the last line. The \code{target-runner} script must return \textbf{only} one number. In the template example, the result is returned with \code{echo "\$COST"} (assuming \parameter{maxExperiments} is used) and the used files are deleted. \begin{xwarningbox} The \code{target-runner} script must be executable. \end{xwarningbox} You can test the target runner from the \aR console by checking the scenario as explained earlier in \autoref{sec:execution}. If you have problems related to the \code{target-runner} script when executing \irace, see \autoref{sec:check list} for a check list to help diagnose common problems. For more information about the \parameter{targetRunner}, please see \autoref{sec:runner}, \end{enumerate} \item \textit{Optional}: Modify the \code{target-evaluator} file. This is rarely needed and the \code{target-runner} template does not use it. \autoref{sec:evaluator} explains when a \parameter{targetEvaluator} is needed and how to define it. \item The \irace executable provides an option (\parameter{-{}-check}) to check that the scenario is correctly defined. We recommend to perform a check every time you create a new scenario. When performing the check, \irace will verify that the scenario and parameter definitions are correct and will test the execution of the target algorithm. To check your scenario execute the following commands: \begin{itemize} \item From the command-line (on Windows, execute \code{irace.bat}): %<>= \begin{lstlisting}[style=BashInputStyle] # $IRACE_HOME is the installation directory of irace. $IRACE_HOME/bin/irace --scenario scenario.txt --check \end{lstlisting} %@ \item Or from the \aR console: <>= library("irace") scenario <- readScenario(filename = "scenario.txt", scenario = defaultScenario()) checkIraceScenario(scenario = scenario) @ \end{itemize} \item Once all the scenario elements are prepared you can execute \irace, either using the command-line wrappers provided by the package or directly from the \aR console: \begin{itemize} \item{% \textbf{From the command-line console}, call the command (on Windows, you should execute \code{irace.bat}): \begin{lstlisting}[style=BashInputStyle] cd ~/tuning/ # $IRACE_HOME is the installation directory of irace # By default, irace reads scenario.txt, you can specify a different file # with --scenario. $IRACE_HOME/bin/irace \end{lstlisting} For this example we assume that the needed scenario files have been set properly in the \code{scenario.txt} file using the options described in \autoref{sec:irace options}. Most \irace options can be specified in the command line or directly in the \code{scenario.txt} file. } \item{ \textbf{From the \aR console}, evaluate: <>= library("irace") # Go to the directory containing the scenario files setwd("~/tuning") scenario <- readScenario(filename = "scenario.txt", scenario = defaultScenario()) irace.main(scenario = scenario) @ } \end{itemize} This will perform one run of \irace. See the output of \code{irace --help} in the command-line or \code{irace.usage()} in \aR for quick information on additional \irace parameters. For more information about \irace options, see \autoref{sec:irace options}. \end{enumerate} \begin{xwarningbox} Command-line options override the same options specified in the \code{scenario.txt} file. \end{xwarningbox} \subsection{Setup example for ACOTSP}\label{sec:example} The \ACOTSP tuning example can be found in the package installation at \IRACEHOME{/examples/acotsp}. % Additionally, a number of example scenarios can be found in the \code{examples} folder. More examples of tuning scenarios can be found in the Algorithm Configuration Library (AClib, \url{http://www.aclib.net/}). In this section, we describe how to execute the \ACOTSP scenario. If you wish to start setting up your own scenario, continue to the next section. For this example, we assume a GNU/Linux system but making the necessary changes in the commands and \parameter{targetRunner}, it can be executed in any system that has a \proglang{C} compiler. %\MANUEL{I don't think this is true, since the target-runner script needs Bash} To execute this scenario follow these steps: \begin{enumerate} \item Create a directory for the tuning (\eg~\path{~/tuning/}) and copy the example scenario files located in the \code{examples} folder to the created directory: %<>= \begin{lstlisting}[style=BashInputStyle] mkdir ~/tuning cd ~/tuning # $IRACE_HOME is the installation directory of irace. cp $IRACE_HOME/examples/acotsp/* ~/tuning/ \end{lstlisting} %@ \item Download the training instances from \url{http://iridia.ulb.ac.be/irace/} to the \path{~/tuning/} directory. \item Create the instance directory (\eg~\path{~/tuning/Instances}) and decompress the instance files on it. %<>= \begin{lstlisting}[style=BashInputStyle] mkdir ~/tuning/Instances/ cd ~/tuning/ tar -xvf tsp-instances-training.tar.bz2 Instances/ \end{lstlisting} %@ \item Download the \ACOTSP software from \url{http://www.aco-metaheuristic.org/aco-code/} to the \path{~/tuning/} directory and compile it. %<>= \begin{lstlisting}[style=BashInputStyle] cd ~/tuning/ tar -xvf ACOTSP-1.03.tgz cd ~/tuning/ACOTSP-1.03 make \end{lstlisting} %@ \item Create a directory for the executable and copy it: %<>= \begin{lstlisting}[style=BashInputStyle] mkdir ~/bin/ cp ~/tuning/ACOTSP-1.03/acotsp ~/bin/ \end{lstlisting} %@ \item Create a directory for executing the experiments and execute \irace: %<>= \begin{lstlisting}[style=BashInputStyle] mkdir ~/tuning/acotsp-arena/ cd ~/tuning/ # $IRACE_HOME is the installation directory of irace. $IRACE_HOME/bin/irace \end{lstlisting} %@ \item Or you can also execute \irace from the \aR console using: <>= library("irace") setwd("~/tuning/") irace.cmdline() @ \end{enumerate} %% %% %% %% Scenario settings %% %% %% \section{Defining a configuration scenario}\label{sec:scenario} \subsection{Target algorithm parameters} \label{sec:target parameters} The parameters of the target algorithm are defined by a parameter file as described in \autoref{sec:parameters file}. Optionally, when executing \irace from the \aR console, the parameters can be specified directly as an \aR object (see \autoref{sec:parameters object}). For defining your parameters follow the guidelines provided in the following sections. \subsubsection{Parameter types} Each target parameter has an associated type that defines its domain and the way \irace handles them internally. Understanding the nature of the domains of the target parameters is important to select appropriate types. The four basic types supported by \irace are the following: \begin{itemize} \item \textit{Real} parameters are numerical parameters that can take floating-point values within a given range. The range is specified as an interval `\code{(,)}'. This interval is closed, that is, the parameter value may eventually be one of the bounds. The possible values are rounded to a number of \emph{decimal places} specified by option \parameter{digits}. For example, given the default number of digits of $4$, the values $0.12345$ and $0.12341$ are both rounded to $0.1234$. % However, the values $0.00001$ and $0.00005$ remain the same. Selected real-valued parameters can be optionally sampled on a logarithmic scale (base $e$). \item \textit{Integer} parameters are numerical parameters that can take only integer values within the given range. Their range is specified as the range of real parameters and they can also be optionally sampled on a logarithmic scale (base $e$). \item \textit{Categorical} parameters are defined by a set of possible values specified as `\code{(,} \code{...,} \code{)}'. The values are quoted or unquoted character strings. Empty strings and strings containing commas or spaces must be quoted. \item \emph{Ordinal} parameters are defined by an \emph{ordered} set of possible values in the same format as for categorical parameters. They are handled internally as integer parameters, where the integers correspond to the indexes of the values. \end{itemize} \subsubsection{Parameter domains} For each target parameter, an interval or a set of values must be defined according to its type, as described above. There is no limit for the size of the set or the length of the interval, but keep in mind that larger domains could increase the difficulty of the tuning task. Choose always values that you consider relevant for the tuning. In case of doubt, we recommend to choose larger intervals, as occasionally best parameter settings may be not intuitive a priori. All intervals are considered as closed intervals. It is possible to define parameters that will have always the same value. Such ``\emph{fixed}'' parameters will not be tuned but their values are used when executing the target algorithm and they are affected by constraints defined on them. All fixed parameters must be defined as categorical parameters and have a domain of one element. \subsubsection{Conditional parameters} Conditional parameters are active only when others have certain values. These dependencies define a hierarchical relation between parameters. For example, the target algorithm may have a parameter \code{localsearch} that takes values \code{(sa, ts)} and another parameter \code{ts-length} that only needs to be set if the first parameter takes precisely the value \code{ts}. Thus, parameter \code{ts-length} is conditional on \code{localsearch == "ts"}. \subsubsection{Parameter file format}\label{sec:parameters file} For simplicity, the description of the parameters space is given as a table. Each line of the table defines a configurable parameter % \begin{center} \code{\