parallelMap/0000755000176200001440000000000014066267304012514 5ustar liggesusersparallelMap/NAMESPACE0000644000176200001440000000131714066072250013727 0ustar liggesusers# Generated by roxygen2: do not edit by hand S3method(print,ParallelMapOptions) S3method(print,RegisteredLevels) export(parallelExport) export(parallelGetOptions) export(parallelGetRegisteredLevels) export(parallelLapply) export(parallelLibrary) export(parallelMap) export(parallelRegisterLevels) export(parallelSapply) export(parallelSource) export(parallelStart) export(parallelStartBatchJobs) export(parallelStartBatchtools) export(parallelStartLocal) export(parallelStartMPI) export(parallelStartMulticore) export(parallelStartSocket) export(parallelStop) import(BBmisc) import(checkmate) import(parallel) importFrom(stats,setNames) importFrom(utils,getFromNamespace) importFrom(utils,head) importFrom(utils,tail) parallelMap/LICENSE0000644000176200001440000000007414066072250013514 0ustar liggesusersYEAR: 2013-2016 COPYRIGHT HOLDER: Bernd Bischl, Michel Lang parallelMap/man/0000755000176200001440000000000014066072250013261 5ustar liggesusersparallelMap/man/parallelMap.Rd0000644000176200001440000000607314066072250016010 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/parallelMap.R \name{parallelMap} \alias{parallelMap} \title{Maps a function over lists or vectors in parallel.} \usage{ parallelMap( fun, ..., more.args = list(), simplify = FALSE, use.names = FALSE, impute.error = NULL, level = NA_character_, show.info = NA ) } \arguments{ \item{fun}{\link{function}\cr Function to map over \code{...}.} \item{...}{(any)\cr Arguments to vectorize over (list or vector).} \item{more.args}{\link{list}\cr A list of other arguments passed to \code{fun}. Default is empty list.} \item{simplify}{(\code{logical(1)})\cr Should the result be simplified? See \link{simplify2array}. If \code{TRUE}, \code{simplify2array(higher = TRUE)} will be called on the result object. Default is \code{FALSE}.} \item{use.names}{(\code{logical(1)})\cr Should result be named? Use names if the first \code{...} argument has names, or if it is a character vector, use that character vector as the names.} \item{impute.error}{(\code{NULL} | \verb{function(x)})\cr This argument can be used for improved error handling. \code{NULL} means that, if an exception is generated on one of the slaves, it is also thrown on the master. Usually all slave jobs will have to terminate until this exception on the master can be thrown. If you pass a constant value or a function, all jobs are guaranteed to return a result object, without generating an exception on the master for slave errors. In case of an error, this is a \code{\link[=simpleError]{simpleError()}} object containing the error message. If you passed a constant object, the error-objects will be substituted with this object. If you passed a function, it will be used to operate on these error-objects (it will ONLY be applied to the error results). For example, using \code{identity} would keep and return the \code{simpleError}-object, or \code{function(x) 99} would impute a constant value (which could be achieved more easily by simply passing \code{99}). Default is \code{NULL}.} \item{level}{(\code{character(1)})\cr If a (non-missing) level is specified in \code{\link[=parallelStart]{parallelStart()}}, this call is only parallelized if the level specified here matches. Useful if this function is used in a package. Default is \code{NA}.} \item{show.info}{(\code{logical(1)})\cr Verbose output on console? Can be used to override setting from options / \code{\link[=parallelStart]{parallelStart()}}. Default is NA which means no overriding.} } \value{ Result. } \description{ Uses the parallelization mode and the other options specified in \code{\link[=parallelStart]{parallelStart()}}. Libraries and source file can be initialized on slaves with \code{\link[=parallelLibrary]{parallelLibrary()}} and \code{\link[=parallelSource]{parallelSource()}}. Large objects can be separately exported via \code{\link[=parallelExport]{parallelExport()}}, they can be simply used under their exported name in slave body code. Regarding error handling, see the argument \code{impute.error}. } \examples{ parallelStart() parallelMap(identity, 1:2) parallelStop() } parallelMap/man/parallelGetRegisteredLevels.Rd0000644000176200001440000000175614066072250021206 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/parallelRegisterLevels.R \name{parallelGetRegisteredLevels} \alias{parallelGetRegisteredLevels} \title{Get registered parallelization levels for all currently loaded packages.} \usage{ parallelGetRegisteredLevels(flatten = FALSE) } \arguments{ \item{flatten}{(\code{logical(1)})\cr Flatten to character vector or not? See description. Default is \code{FALSE}.} } \value{ \code{RegisteredLevels} | \code{character}. See above. } \description{ With \code{flatten = FALSE}, a structured S3 object is returned. The S3 object only has one slot, which is called \code{levels}. This contains a named list. Each name refers to \code{package} from the call to \code{\link[=parallelRegisterLevels]{parallelRegisterLevels()}}, while the entries are character vectors of the form \dQuote{package.level}. With \code{flatten = TRUE}, a simple character vector is returned that contains all concatenated entries of \code{levels} from above. } parallelMap/man/parallelLapply.Rd0000644000176200001440000000273514066072250016535 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/parallelApply.R \name{parallelLapply} \alias{parallelLapply} \alias{parallelSapply} \title{Parallel versions of apply-family functions.} \usage{ parallelLapply(xs, fun, ..., impute.error = NULL, level = NA_character_) parallelSapply( xs, fun, ..., simplify = TRUE, use.names = TRUE, impute.error = NULL, level = NA_character_ ) } \arguments{ \item{xs}{(\code{vector} | \code{list})\cr \code{fun} is applied to the elements of this argument.} \item{fun}{\code{\link{function}}\cr Function to map over \code{xs}.} \item{...}{(any)\cr Further arguments passed to \code{fun}.} \item{impute.error}{(\code{NULL} | \verb{function(x)})\cr See \code{\link[=parallelMap]{parallelMap()}}.} \item{level}{(\code{character(1)})\cr See \code{\link[=parallelMap]{parallelMap()}}.} \item{simplify}{(\code{logical(1)})\cr See \code{\link[=sapply]{sapply()}}. Default is \code{TRUE}.} \item{use.names}{(\code{logical(1)})\cr See \code{\link[=sapply]{sapply()}}. Default is \code{TRUE}.} } \value{ For \code{parallelLapply} a named list, for \code{parallelSapply} it depends on the return value of \code{fun} and the settings of \code{simplify} and \code{use.names}. } \description{ \code{parallelLapply}: A parallel \code{\link[=lapply]{lapply()}} version.\cr \code{parallelSapply}: A parallel \code{\link[=sapply]{sapply()}} version.\cr All functions are simple wrappers for \code{\link[=parallelMap]{parallelMap()}}. } parallelMap/man/parallelRegisterLevels.Rd0000644000176200001440000000307014066072250020224 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/parallelRegisterLevels.R \name{parallelRegisterLevels} \alias{parallelRegisterLevels} \title{Register a parallelization level} \usage{ parallelRegisterLevels(package = "custom", levels) } \arguments{ \item{package}{(\code{character(1)})\cr Name of your package. Default is \dQuote{custom} (we are not in a package).} \item{levels}{(\code{character(1)})\cr Available levels that are used in the \code{\link[=parallelMap]{parallelMap()}} operations of your package or code. If \code{package} is not missing, all levels will be prefixed with \dQuote{package.}.} } \value{ Nothing. } \description{ Package developers should call this function in their packages' \code{base::.onLoad()}. This enables the user to query available levels and bind parallelization to specific levels. This is especially helpful for nested calls to \code{\link[=parallelMap]{parallelMap()}}, e.g. where the inner call should be parallelized instead of the outer one. To avoid name clashes, we encourage developers to always specify the argument \code{package}. This will prefix the specified levels with the string containing the package name, e.g. \code{parallelRegisterLevels(package="foo", levels="dummy")} will register the level \dQuote{foo.dummy} and users can start parallelization for this level with \verb{parallelStart(, level = "parallelMap.dummy")}. If you do not provide \code{package}, the level names will be associated with category \dQuote{custom} and can there be later referred to with \dQuote{custom.dummy}. } parallelMap/man/parallelStop.Rd0000644000176200001440000000127314066072250016215 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/parallelStop.R \name{parallelStop} \alias{parallelStop} \title{Stops parallelization.} \usage{ parallelStop() } \value{ Nothing. } \description{ Sets mode to \dQuote{local}, i.e., parallelization is turned off and all necessary stuff is cleaned up. For socket and mpi mode \code{parallel::stopCluster()} is called. For BatchJobs mode the subdirectory of the \code{storagedir} containing the exported objects is removed. After a subsequent call of \code{\link[=parallelStart]{parallelStart()}}, no exported objects are present on the slaves and no libraries are loaded, i.e., you have clean R sessions on the slaves. } parallelMap/man/parallelExport.Rd0000644000176200001440000000266414066072250016556 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/parallelExport.R \name{parallelExport} \alias{parallelExport} \title{Export R objects for parallelization.} \usage{ parallelExport( ..., objnames, master = TRUE, level = NA_character_, show.info = NA ) } \arguments{ \item{...}{\code{\link{character}}\cr Names of objects to export.} \item{objnames}{(\code{character(1)})\cr Names of objects to export. Alternative way to pass arguments.} \item{master}{(\code{logical(1)})\cr Really export to package environment on master for local and multicore mode? If you do not do this your objects might not get exported for the mapping function call. Only disable when you are really sure. Default is \code{TRUE}.} \item{level}{(\code{character(1)})\cr If a (non-missing) level is specified in \code{\link[=parallelStart]{parallelStart()}}, the function only exports if the level specified here matches. See \code{\link[=parallelMap]{parallelMap()}}. Useful if this function is used in a package. Default is \code{NA}.} \item{show.info}{(\code{logical(1)})\cr Verbose output on console? Can be used to override setting from options / \code{\link[=parallelStart]{parallelStart()}}. Default is NA which means no overriding.} } \value{ Nothing. } \description{ Makes sure that the objects are exported to slave process so that they can be used in a job function which is later run with \code{\link[=parallelMap]{parallelMap()}}. } parallelMap/man/parallelGetOptions.Rd0000644000176200001440000000126114066072250017360 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/parallelShowOptions.R \name{parallelGetOptions} \alias{parallelGetOptions} \title{Retrieve the configured package options.} \usage{ parallelGetOptions() } \value{ \code{ParallelMapOptions}. See above. } \description{ Returned are current and default settings, both as lists. The return value has slots elements \code{settings} and \code{defaults}, which are both lists of the same structure, named by option names. A printer exists to display this object. For details on the configuration procedure please read \code{\link[=parallelStart]{parallelStart()}} and \url{https://github.com/mlr-org/parallelMap}. } parallelMap/man/parallelStart.Rd0000644000176200001440000001546114066072250016371 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/parallelStart.R \name{parallelStart} \alias{parallelStart} \alias{parallelStartLocal} \alias{parallelStartMulticore} \alias{parallelStartSocket} \alias{parallelStartMPI} \alias{parallelStartBatchJobs} \alias{parallelStartBatchtools} \title{Parallelization setup for parallelMap.} \usage{ parallelStart( mode, cpus, socket.hosts, bj.resources = list(), bt.resources = list(), logging, storagedir, level, load.balancing = FALSE, show.info, suppress.local.errors = FALSE, reproducible, ... ) parallelStartLocal(show.info, suppress.local.errors = FALSE, ...) parallelStartMulticore( cpus, logging, storagedir, level, load.balancing = FALSE, show.info, reproducible, ... ) parallelStartSocket( cpus, socket.hosts, logging, storagedir, level, load.balancing = FALSE, show.info, reproducible, ... ) parallelStartMPI( cpus, logging, storagedir, level, load.balancing = FALSE, show.info, reproducible, ... ) parallelStartBatchJobs( bj.resources = list(), logging, storagedir, level, show.info, ... ) parallelStartBatchtools( bt.resources = list(), logging, storagedir, level, show.info, ... ) } \arguments{ \item{mode}{(\code{character(1)})\cr Which parallel mode should be used: \dQuote{local}, \dQuote{multicore}, \dQuote{socket}, \dQuote{mpi}, \dQuote{BatchJobs}. Default is the option \code{parallelMap.default.mode} or, if not set, \dQuote{local} without parallel execution.} \item{cpus}{(\code{integer(1)})\cr Number of used cpus. For local and BatchJobs mode this argument is ignored. For socket mode, this is the number of processes spawned on localhost, if you want processes on multiple machines use \code{socket.hosts}. Default is the option \code{parallelMap.default.cpus} or, if not set, \code{\link[parallel:detectCores]{parallel::detectCores()}} for multicore mode, \verb{max(1, [mpi.universe.size][Rmpi::mpi.universe.size] - 1)} for mpi mode and 1 for socket mode.} \item{socket.hosts}{\link{character}\cr Only used in socket mode, otherwise ignored. Names of hosts where parallel processes are spawned. Default is the option \code{parallelMap.default.socket.hosts}, if this option exists.} \item{bj.resources}{\link{list}\cr Resources like walltime for submitting jobs on HPC clusters via BatchJobs. See \code{\link[BatchJobs:submitJobs]{BatchJobs::submitJobs()}}. Defaults are taken from your BatchJobs config file.} \item{bt.resources}{\link{list}\cr Analog to \code{bj.resources}. See \code{\link[batchtools:submitJobs]{batchtools::submitJobs()}}.} \item{logging}{(\code{logical(1)})\cr Should slave output be logged to files via \code{\link[=sink]{sink()}} under the \code{storagedir}? Files are named \verb{.log} and put into unique subdirectories named \verb{parallelMap_log_} for each subsequent \code{\link[=parallelMap]{parallelMap()}} operation. Previous logging directories are removed on \code{parallelStart} if \code{logging} is enabled. Logging is not supported for local mode, because you will see all output on the master and can also run stuff like \code{\link[=traceback]{traceback()}} in case of errors. Default is the option \code{parallelMap.default.logging} or, if not set, \code{FALSE}.} \item{storagedir}{(\code{character(1)})\cr Existing directory where log files and intermediate objects for BatchJobs mode are stored. Note that all nodes must have write access to exactly this path. Default is the current working directory.} \item{level}{(\code{character(1)})\cr You can set this so only calls to \code{\link[=parallelMap]{parallelMap()}} that have exactly the same level are parallelized. Default is the option \code{parallelMap.default.level} or, if not set, \code{NA} which means all calls to \code{\link[=parallelMap]{parallelMap()}} are are potentially parallelized.} \item{load.balancing}{(\code{logical(1)})\cr Enables load balancing for multicore, socket and mpi. Set this to \code{TRUE} if you have heterogeneous runtimes. Default is \code{FALSE}} \item{show.info}{(\code{logical(1)})\cr Verbose output on console for all further package calls? Default is the option \code{parallelMap.default.show.info} or, if not set, \code{TRUE}.} \item{suppress.local.errors}{(\code{logical(1)})\cr Should reporting of error messages during function evaluations in local mode be suppressed? Default ist FALSE, i.e. every error message is shown.} \item{reproducible}{(\code{logical(1)})\cr Should parallel jobs produce reproducible results when setting a seed? With this option, \code{parallelMap()} calls will be reproducible when using \code{set.seed()} with the default RNG kind. This is not the case by default when parallelizing in R, since the default RNG kind "Mersenne-Twister" is not honored by parallel processes. Instead RNG kind \code{"L'Ecuyer-CMRG"} needs to be used to ensure paralllel reproducibility. Default is the option \code{parallelMap.default.reproducible} or, if not set, \code{TRUE}.} \item{...}{(any)\cr Optional parameters, for socket mode passed to \code{parallel::makePSOCKcluster()}, for mpi mode passed to \code{\link[parallel:makeCluster]{parallel::makeCluster()}} and for multicore passed to \code{parallel::mcmapply()} (\code{mc.preschedule} (overwriting \code{load.balancing}), \code{mc.set.seed}, \code{mc.silent} and \code{mc.cleanup} are supported for multicore).} } \value{ Nothing. } \description{ Defines the underlying parallelization mode for \code{\link[=parallelMap]{parallelMap()}}. Also allows to set a \dQuote{level} of parallelization. Only calls to \code{\link[=parallelMap]{parallelMap()}} with a matching level are parallelized. The defaults of all settings are taken from your options, which you can also define in your R profile. For an introductory tutorial and information on the options configuration, please go to the project's github page at https://github.com/mlr-org/parallelMap. } \details{ Currently the following modes are supported, which internally dispatch the mapping operation to functions from different parallelization packages: \itemize{ \item \strong{local}: No parallelization with \code{\link[=mapply]{mapply()}} \item \strong{multicore}: Multicore execution on a single machine with \code{parallel::mclapply()}. \item \strong{socket}: Socket cluster on one or multiple machines with \code{parallel::makePSOCKcluster()} and \code{parallel::clusterMap()}. \item \strong{mpi}: Snow MPI cluster on one or multiple machines with \code{\link[parallel:makeCluster]{parallel::makeCluster()}} and \code{parallel::clusterMap()}. \item \strong{BatchJobs}: Parallelization on batch queuing HPC clusters, e.g., Torque, SLURM, etc., with \code{\link[BatchJobs:batchMap]{BatchJobs::batchMap()}}. } For BatchJobs mode you need to define a storage directory through the argument \code{storagedir} or the option \code{parallelMap.default.storagedir}. } parallelMap/man/parallelLibrary.Rd0000644000176200001440000000253614066072250016677 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/parallelLibrary.R \name{parallelLibrary} \alias{parallelLibrary} \title{Load packages for parallelization.} \usage{ parallelLibrary( ..., packages, master = TRUE, level = NA_character_, show.info = NA ) } \arguments{ \item{...}{\link{character}\cr Names of packages to load.} \item{packages}{(\code{character(1)})\cr Names of packages to load. Alternative way to pass arguments.} \item{master}{(\code{logical(1)})\cr Load packages also on master for any mode? Default is \code{TRUE}.} \item{level}{(\code{character(1)})\cr If a (non-missing) level is specified in \code{\link[=parallelStart]{parallelStart()}}, the function only loads the packages if the level specified here matches. See \code{\link[=parallelMap]{parallelMap()}}. Useful if this function is used in a package. Default is \code{NA}.} \item{show.info}{(\code{logical(1)})\cr Verbose output on console? Can be used to override setting from options / \code{\link[=parallelStart]{parallelStart()}}. Default is NA which means no overriding.} } \value{ Nothing. } \description{ Makes sure that the packages are loaded in slave process so that they can be used in a job function which is later run with \code{\link[=parallelMap]{parallelMap()}}. For all modes, the packages are also (potentially) loaded on the master. } parallelMap/man/parallelSource.Rd0000644000176200001440000000250114066072250016523 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/parallelSource.R \name{parallelSource} \alias{parallelSource} \title{Source R files for parallelization.} \usage{ parallelSource( ..., files, master = TRUE, level = NA_character_, show.info = NA ) } \arguments{ \item{...}{\link{character}\cr File paths to sources.} \item{files}{\link{character}\cr File paths to sources. Alternative way to pass arguments.} \item{master}{(\code{logical(1)})\cr Source files also on master for any mode? Default is \code{TRUE}.} \item{level}{(\code{character(1)})\cr If a (non-missing) level is specified in \code{\link[=parallelStart]{parallelStart()}}, the function only sources the files if the level specified here matches. See \code{\link[=parallelMap]{parallelMap()}}. Useful if this function is used in a package. Default is \code{NA}.} \item{show.info}{(\code{logical(1)})\cr Verbose output on console? Can be used to override setting from options / \code{\link[=parallelStart]{parallelStart()}}. Default is NA which means no overriding.} } \value{ Nothing. } \description{ Makes sure that the files are sourced in slave process so that they can be used in a job function which is later run with \code{\link[=parallelMap]{parallelMap()}}. For all modes, the files are also (potentially) loaded on the master. } parallelMap/DESCRIPTION0000644000176200001440000000341714066267304014227 0ustar liggesusersPackage: parallelMap Title: Unified Interface to Parallelization Back-Ends Version: 1.5.1 Authors@R: c(person(given = "Bernd", family = "Bischl", role = c("cre", "aut"), email = "bernd_bischl@gmx.net"), person(given = "Michel", family = "Lang", role = "aut", email = "michellang@gmail.com", comment = c(ORCID = "0000-0001-9754-0393")), person(given = "Patrick", family = "Schratz", role = "aut", email = "patrick.schratz@gmail.com", comment = c(ORCID = "0000-0003-0748-6624"))) Description: Unified parallelization framework for multiple back-end, designed for internal package and interactive usage. The main operation is parallel mapping over lists. Supports 'local', 'multicore', 'mpi' and 'BatchJobs' mode. Allows tagging of the parallel operation with a level name that can be later selected by the user to switch on parallel execution for exactly this operation. License: BSD_2_clause + file LICENSE URL: https://parallelmap.mlr-org.com, https://github.com/mlr-org/parallelMap BugReports: https://github.com/mlr-org/parallelMap/issues Depends: R (>= 3.0.0) Imports: BBmisc (>= 1.8), checkmate (>= 1.8.0), parallel, stats, utils Suggests: BatchJobs (>= 1.8), batchtools (>= 0.9.6), data.table, Rmpi, rpart, snow, testthat ByteCompile: yes Encoding: UTF-8 RoxygenNote: 7.1.0 NeedsCompilation: no Packaged: 2021-06-27 13:21:26 UTC; pjs Author: Bernd Bischl [cre, aut], Michel Lang [aut] (), Patrick Schratz [aut] () Maintainer: Bernd Bischl Repository: CRAN Date/Publication: 2021-06-28 06:40:04 UTC parallelMap/tests/0000755000176200001440000000000014066072250013650 5ustar liggesusersparallelMap/tests/testthat/0000755000176200001440000000000014066072250015510 5ustar liggesusersparallelMap/tests/testthat/helper_sockettest.R0000644000176200001440000000142614066072250021365 0ustar liggesuserssockettest = function() { expect_error(parallelStartSocket(cpus = 2, socket.hosts = "localhost"), "You cannot set both") parallelStop() parallelStartSocket(2) partest1() parallelStop() parallelStartSocket(2, load.balancing = TRUE) partest1() parallelStop() # check with host args as strings too parallelStartSocket(socket.hosts = c("localhost", "localhost")) partest1() parallelStop() parallelStartSocket(2, logging = TRUE, storagedir = tempdir()) partest2(tempdir()) parallelStop() parallelStartSocket(2) partest3() parallelStop() parallelStartSocket(2) partest4(slave.error.test = TRUE) parallelStop() parallelStartSocket(2) partest5() parallelStop() parallelStartSocket(2) partest6(slave.error.test = TRUE) parallelStop() } parallelMap/tests/testthat/test_parallelLibrary.R0000644000176200001440000000065314066072250022017 0ustar liggesuserscontext("parallelLibrary") # issue https://github.com/mlr-org/parallelMap/issues/67 test_that("exported packages honor the given parallelization level", { parallelRegisterLevels(levels = "test") parallelStartSocket(1, level = "custom.test") parallelLibrary(packages = "rpart") out = parallelMap(function(.) search(), 1, level = "custom.test")[[1]] parallelStop() expect_true(any(grepl("package:rpart", out))) }) parallelMap/tests/testthat/test_01_registerLevels.R0000644000176200001440000000206714066072250022176 0ustar liggesuserscontext("register levels") test_that("register levels", { expect_equal(parallelGetRegisteredLevels()$levels, list()) parallelRegisterLevels("p1", "lev1") parallelRegisterLevels("p2", c("a", "b")) parallelRegisterLevels("p2", c("a", "b")) expect_equal( parallelGetRegisteredLevels()$levels, list(p1 = "p1.lev1", p2 = c("p2.a", "p2.b")) ) expect_equal(parallelGetRegisteredLevels(flatten = TRUE), c("p1.lev1", "p2.a", "p2.b")) parallelRegisterLevels(levels = c("x", "y")) expect_equal( parallelGetRegisteredLevels()$levels, list(p1 = "p1.lev1", p2 = c("p2.a", "p2.b"), custom = c("custom.x", "custom.y")) ) expect_equal(parallelGetRegisteredLevels(flatten = TRUE), c("p1.lev1", "p2.a", "p2.b", "custom.x", "custom.y")) }) test_that("warn on unregisterred level", { # check that we warn expect_warning(parallelStartBatchJobs(level = "foo"), "not registered") parallelStop() # check that we DONT warn for no level opt = getOption("warn") options(warn = 2L) parallelStartBatchJobs() parallelStop() options(warn = opt) }) parallelMap/tests/testthat/test_batchtools.R0000644000176200001440000000347614066072250021046 0ustar liggesuserscontext("batchtools mode") test_that("batchtools mode", { requireNamespace("batchtools") reg = batchtools::makeRegistry(NA) storagedir = reg$temp.dir if (is.null(storagedir) || is.na(storagedir)) { storagedir = tempfile() } parallelStartBatchtools(storagedir = storagedir) partest1() parallelStop() # FIXME: Wait until https://github.com/hadley/testthat/issues/460 is fixed # parallelStartBatchtools(logging = TRUE, storagedir = storagedir) # partest2(storagedir) # parallelStop() parallelStartBatchtools(storagedir = storagedir) partest3() parallelStop() parallelStartBatchtools(storagedir = storagedir) # we cannot really check that wrong libraries are not loaded on slave here. # because we only load them during the job. but the error will show up then partest4(slave.error.test = FALSE) parallelStop() parallelStartBatchtools(storagedir = storagedir) partest5() parallelStop() parallelStartBatchtools(storagedir = storagedir) partest6(slave.error.test = FALSE) parallelStop() # test that expire generate exceptions # we can of course only do that on a true batch system if (batchtools::makeRegistry(file.dir = NA)$cluster.functions$name %in% c("SLURM", "Torque")) { parallelStartBatchtools(storagedir = storagedir, bj.resources = list(walltime = 5)) f = function(i) Sys.sleep(30 * 60) expect_error(suppressWarnings(parallelMap(f, 1:2)), "expired") parallelStop() } # test that working dir on master is working dir on slave oldwd = getwd() bn = "parallelMap_test_temp_dir_123" newwd = file.path(storagedir, bn) dir.create(newwd) setwd(newwd) parallelStartBatchtools(storagedir = storagedir) f = function(i) getwd() y = parallelMap(f, 1) parallelStop() expect_equal(basename(y[[1]]), bn) setwd(oldwd) unlink(newwd, recursive = TRUE) }) parallelMap/tests/testthat/helpers.R0000644000176200001440000001143614066072250017302 0ustar liggesusers options(BBmisc.ProgressBar.style = "off") options(parallelMap.default.show.info = FALSE) # a test for normal functionality of mapping an its options partest1 = function() { # normal lapply expect_equal(parallelMap(identity, 1), list(1)) expect_equal(parallelMap(identity, 1:2), list(1, 2)) y = c("a", "b") names(y) = y # simplify and names expect_equal(parallelMap(identity, y, simplify = TRUE, use.names = TRUE), y) # more.args and mapping over 2 vectors f = function(x, y) x + y expect_equal(parallelMap(f, 1:2, more.args = list(y = 1)), list(2, 3)) expect_equal(parallelMap(f, 1:2, 2:3), list(3, 5)) # with level expect_equal(parallelMap(identity, 1:2), list(1, 2), level = "foo") } # test that log files are correctly generated partest2 = function(log.dir, test.warning = TRUE) { # do log files exist under correct path / name? check.exists = function(iter, n) { fp = file.path(log.dir, sprintf("parallelMap_logs_%03i", iter)) sapply(seq_len(n), function(i) { expect_true(file.exists(file.path(fp, sprintf("%05i.log", i)))) }) } # do log files contain the printed output fromn the slave? check.contains = function(iter, xs) { fp = file.path(log.dir, sprintf("parallelMap_logs_%03i", iter)) Map(function(i, x) { s = readLines(file.path(fp, sprintf("%05i.log", i))) s = collapse(s, sep = "\n") expect_true(grep(x, s) == 1) }, seq_along(xs), xs) } parallelMap(cat, c("xxx", "yyy")) check.exists(iter = 1, n = 2) check.contains(iter = 1, c("xxx", "yyy")) parallelMap(print, c("xxx", "yyy")) check.exists(iter = 2, n = 2) check.contains(iter = 2, c("xxx", "yyy")) if (test.warning) { parallelMap(warning, c("xxx", "yyy")) check.exists(iter = 3, n = 2) check.contains(iter = 3, c("xxx", "yyy")) } } # test that exported variables exist on slave partest3 = function() { # export nothing, no change parallelExport() foo = 100 parallelExport("foo") f = function(i) { i + foo } expect_equal(parallelMap(f, 1:2), list(101, 102)) # now test with foo2 defined one level fruther above f = function(i) { i + foo2 } foo2 = 100 g = function() { parallelExport("foo2") } g() expect_equal(parallelMap(f, 1:2), list(101, 102)) # now test with foo3 defined in global env assign("foo3", 100, env = .GlobalEnv) f = function(i) { i + foo3 } parallelExport("foo3") expect_equal(parallelMap(f, 1:2), list(101, 102)) } # test that exported libraries are loaded partest4 = function(slave.error.test) { # testhat is basically the only lib we have in suggests... parallelLibrary("testthat") f = function(i) { expect_true } res = parallelMap(f, 1:2) expect_true(is.list(res) && length(res) == 2 && is.function(res[[1]])) if (slave.error.test) { expect_error(parallelLibrary("foo", master = FALSE), "Packages could not be loaded on all slaves: foo.") expect_error(parallelLibrary("foo1", "foo2", master = FALSE), "Packages could not be loaded on all slaves: foo1,foo2.") expect_error(parallelLibrary("testthat", "foo", master = FALSE), "Packages could not be loaded on all slaves: foo.") } } # test error handling partest5 = function() { # exception is thrown on master f = function(i) { if (i == 1) { stop("foo") } else { i } } y = parallelMap(f, 1:2, impute.error = identity) expect_true(BBmisc::is.error(y[[1L]])) expect_equal(y[[2L]], 2L) y = parallelMap(f, 1:2, impute.error = function(x) 123) expect_equal(y[[1L]], 123L) expect_equal(y[[2L]], 2L) y = parallelMap(f, 1:2, impute.error = 99) expect_equal(y[[1L]], 99L) expect_equal(y[[2L]], 2L) } # test that exported files are sourced partest6 = function(slave.error.test) { # this is all so fucking crappy and broken # apparently the name/path changes depending on how I run the test... fn = system.file("test_source_file.R", package = "parallelMap") if (fn == "") { fn = system.file("inst/test_source_file.R", package = "parallelMap") } parallelSource(fn, master = FALSE) f = function(i) i + xxx res = parallelMap(f, 1:2, simplify = TRUE) expect_equal(res, 124:125) if (slave.error.test) { expect_error(parallelSource("foo", master = FALSE), "Files could not be sourced on all slaves: foo.") expect_error(parallelSource("foo1", "foo2", master = FALSE), "Files could not be sourced on all slaves: foo1,foo2.") expect_error(parallelSource(fn, "foo", master = FALSE), "Files could not be sourced on all slaves: foo.") } } # test that load balancing is active partest7 = function() { f = function(i) { if (i == 1) { Sys.sleep(9) } else { Sys.sleep(1) } return(i) } st = system.time({ ys = parallelMap(f, 1:10, simplify = TRUE) }) expect_equal(ys, 1:10) expect_true(st[3L] < 10) } parallelMap/tests/testthat/test_multicore.R0000644000176200001440000000206414066072250020677 0ustar liggesuserscontext("multicore mode") # cran allows no multicore mode testing # FIXME: I also get strange messages in "make test" and interactive test, but # apparently not when I really use the pkg...? # .Warning in selectChildren(ac, 1) : # error 'Interrupted system call' in select if (isExpensiveExampleOk()) { test_that("multicore mode", { parallelStartMulticore(2) partest1() parallelStop() parallelStartMulticore(2, load.balancing = TRUE) partest1() parallelStop() expect_error(parallelStartMulticore(storagedir = "xxx")) parallelStartMulticore(2, logging = TRUE, storagedir = tempdir()) # FIXME: see issue #33 partest2(tempdir(), test.warning = FALSE) parallelStop() parallelStartMulticore(2, mc.preschedule = FALSE) partest4(slave.error.test = FALSE) parallelStop() parallelStartMulticore(2) partest5() parallelStop() parallelStartMulticore(2) partest6(slave.error.test = FALSE) parallelStop() parallelStartMulticore(2, load.balancing = TRUE) partest7() parallelStop() }) } parallelMap/tests/testthat/test_stopWithJobErrorMessages.R0000644000176200001440000000036514066072250023654 0ustar liggesusers context("stopWithJobErrorMessages") test_that("stopWithJobErrorMessages", { inds = seq_along(letters) msgs = letters expect_error(stopWithJobErrorMessages(inds, msgs), sprintf("Errors occurred in %i slave jobs", length(letters))) }) parallelMap/tests/testthat/test_socket.R0000644000176200001440000000032214066072250020157 0ustar liggesusers context("socket mode") # FIXME: I do NOT know why this blocks in R CMD check if (interactive()) { # cran allows socket mode with 2 localhost processes test_that("socket mode", { sockettest() }) } parallelMap/tests/testthat/test_autodetectCpus.R0000644000176200001440000000022114066072250021661 0ustar liggesuserscontext("BatchJobs mode") test_that("BatchJobs mode", { k = autodetectCpus(MODE_MULTICORE) expect_true(is.integer(k) && length(k) == 1L) }) parallelMap/tests/testthat/test_batchjobs.R0000644000176200001440000000346114066072250020635 0ustar liggesuserscontext("BatchJobs mode") test_that("BatchJobs mode", { requireNamespace("BatchJobs") storagedir = tempdir() # if on lido or SLURM for test, tempdir is not shared and test wih torque wont run if (BatchJobs::getConfig()$cluster.functions$name %in% c("SLURM", "Torque")) { storagedir = getwd() } parallelStartBatchJobs(storagedir = storagedir) partest1() parallelStop() parallelStartBatchJobs(logging = TRUE, storagedir = storagedir) expect_warning(partest2(storagedir), "xxx") parallelStop() parallelStartBatchJobs(storagedir = storagedir) partest3() parallelStop() parallelStartBatchJobs(storagedir = storagedir) # we cannot really check that wrong libraries are not loaded on slave here. # because we only load them during the job. but the error will show up then partest4(slave.error.test = FALSE) parallelStop() parallelStartBatchJobs(storagedir = storagedir) partest5() parallelStop() parallelStartBatchJobs(storagedir = storagedir) partest6(slave.error.test = FALSE) parallelStop() # test that expire generate exceptions # we can of course only do that on a true batch system if (BatchJobs::getConfig()$cluster.functions$name %in% c("SLURM", "Torque")) { parallelStartBatchJobs(storagedir = storagedir, bj.resources = list(walltime = 1)) f = function(i) Sys.sleep(30 * 60) expect_error(suppressWarnings(parallelMap(f, 1:2)), "expired") parallelStop() } # test that working dir on master is working dir on slave oldwd = getwd() bn = "parallelMap_test_temp_dir_123" newwd = file.path(storagedir, bn) dir.create(newwd) setwd(newwd) parallelStartBatchJobs(storagedir = storagedir) f = function(i) getwd() y = parallelMap(f, 1) parallelStop() expect_equal(basename(y[[1]]), bn) setwd(oldwd) unlink(newwd, recursive = TRUE) }) parallelMap/tests/testthat/test_parallelStart.R0000644000176200001440000000071414066072250021506 0ustar liggesuserscontext("parallelStart") # FIXME: I do NOT know why this blocks in R CMD check if (interactive()) { test_that("parallelStart finds regged level", { oldopt = getOption("warn") options(warn = 2L) parallelRegisterLevels(levels = "bla") parallelStartSocket(level = "custom.bla", cpus = 2L) parallelStop() # remove regged level again for clean state options(parallelMap.registered.levels = list()) options(warn = oldopt) }) } parallelMap/tests/testthat/test_parallelApply.R0000644000176200001440000000116014066072250021472 0ustar liggesuserscontext("parallel apply") test_that("parallelLapply", { parallelStart(mode = "local") ys = parallelLapply(1:2, identity) expect_equal(ys, as.list(1:2)) xs = list(a = 1, b = 2) ys = parallelLapply(xs, identity) expect_equal(ys, xs) parallelStop() }) test_that("parallelSapply", { parallelStart(mode = "local") xs = c("a", "b") ys = parallelSapply(xs, identity, use.names = FALSE) expect_equal(ys, xs) ys = parallelSapply(xs, identity, use.names = TRUE) expect_equal(ys, setNames(xs, xs)) expect_equal( parallelSapply(1:2, identity), sapply(1:2, identity) ) parallelStop() }) parallelMap/tests/testthat/test_parallelGetOptions.R0000644000176200001440000000113214066072250022477 0ustar liggesuserscontext("get / print options") test_that("get / print options", { opts = parallelGetOptions() expect_true(is.list(opts)) expect_equal(names(opts), c("settings", "defaults")) expect_equal(names(opts$settings), names(opts$defaults)) capture.output( print(parallelGetOptions()) ) }) # issue https://github.com/mlr-org/parallelMap/issues/41 test_that("parallelGetOptions() prints the state of the object", { parallelStartSocket(2) y = parallelGetOptions() parallelStop() expect_match( capture.output(print(y))[3], "mode : socket \\(not set\\)" ) }) parallelMap/tests/testthat/test_mpi.R0000644000176200001440000000145314066072250017462 0ustar liggesuserscontext("mpi mode") # cran allows no mpi mode testing # FIXME: same problem as for socket mode test, but we cannot test on cran anyway # FIXME: Spurious errors on CI as well -> Rmpi bad :/ test_that("mpi mode", { skip_on_cran() skip_on_ci() parallelStartMPI(2) partest1() parallelStop() parallelStartMPI(2, load.balancing = TRUE) partest1() parallelStop() parallelStartMPI(2, logging = TRUE, storage = tempdir()) partest2(tempdir()) parallelStop() parallelStartMPI(2) partest3() parallelStop() parallelStartMPI(2) partest4(slave.error.test = TRUE) parallelStop() parallelStartMPI(2) partest5() parallelStop() parallelStartMPI(2) partest6(slave.error.test = TRUE) parallelStop() parallelStartMPI(2, load.balancing = TRUE) partest7() parallelStop() }) parallelMap/tests/testthat/test_reproducibility.R0000644000176200001440000000516314066072250022110 0ustar liggesuserscontext("reproducibility") test_that("reproducibility with standard RNG kind works for 'socket' mode", { set.seed(42) parallelStartSocket(cpus = 2) foo1 <- parallelMap(runif, rep(3, 2)) parallelStop() set.seed(42) parallelStartSocket(cpus = 2) foo2 <- parallelMap(runif, rep(3, 2)) parallelStop() expect_equal(foo1, foo2) }) test_that("socket: reproducibility can be turned off", { set.seed(42) parallelStartSocket(cpus = 2, reproducible = FALSE) foo1 <- parallelMap(runif, rep(3, 2)) parallelStop() set.seed(42) parallelStartSocket(cpus = 2, reproducible = FALSE) foo2 <- parallelMap(runif, rep(3, 2)) parallelStop() # would be class "logical" if TRUE expect_is(all.equal(foo1, foo2), "character") }) test_that("reproducibility with standard RNG kind works for 'multicore' mode", { skip_on_os("windows") set.seed(42) parallelStartMulticore(2) foo3 = parallelMap(rnorm, 1:2) parallelStop() set.seed(42) parallelStartMulticore(2) foo4 = parallelMap(rnorm, 1:2) parallelStop() expect_equal(foo3, foo4) }) # This is how it was before v1.5 -> set.seed() with standard RNG kind had no # effect test_that("multicore: reproducibility with standard RNG kind can be turned off", { skip_on_os("windows") set.seed(42) parallelStartMulticore(2, reproducible = FALSE) foo3 = parallelMap(rnorm, 1:2) parallelStop() set.seed(42) parallelStartMulticore(2, reproducible = FALSE) foo4 = parallelMap(rnorm, 1:2) parallelStop() # would be class "logical" if TRUE expect_is(all.equal(foo3, foo4), "character") }) test_that("reproducibility with L'Ecuyer-CMRG RNG kind works for 'multicore' mode", { skip_on_os("windows") # NB: this RNG kind should always work, even without reproducible = TRUE set.seed(42, "L'Ecuyer-CMRG") parallelStartMulticore(2) foo5 = parallelMap(rnorm, 1:2) parallelStop() set.seed(42, "L'Ecuyer-CMRG") parallelStartMulticore(2) foo6 = parallelMap(rnorm, 1:2) parallelStop() expect_equal(foo5, foo6) }) # I do not think we can do anything here: # 1. This RNG kind always forces reproducibility in parallel processes, # even when mc.set.seed = FALSE is FALSE test_that("reproducibility with L'Ecuyer-CMRG RNG kind works even when 'reproducible = FALSE'", { skip_on_os("windows") # NB: this RNG kind should always work, even without reproducible = TRUE set.seed(42, "L'Ecuyer-CMRG") parallelStartMulticore(2, reproducible = FALSE) foo5 = parallelMap(rnorm, 1:2) parallelStop() set.seed(42, "L'Ecuyer-CMRG") parallelStartMulticore(2, reproducible = FALSE) foo6 = parallelMap(rnorm, 1:2) parallelStop() expect_equal(foo5, foo6) }) parallelMap/tests/testthat/test_local.R0000644000176200001440000000035114066072250017763 0ustar liggesuserscontext("local mode") test_that("local mode", { parallelStartLocal(suppress.local.errors = TRUE) partest1() partest3() partest4(slave.error.test = FALSE) partest5() partest6(slave.error.test = FALSE) parallelStop() }) parallelMap/tests/run-all.R0000644000176200001440000000101514066072250015342 0ustar liggesusers# FIXME: R CMD check does not run on true batch systems, but maybe it does not have to # we can check there with 'make test' and run 'make check' locally library(testthat) test_check("parallelMap") # #FIXME: bad hack # # for some reason using test_package and opening a socked node # # blocks R CMD check # # but we really want to test at least one real parallel mode on cran # if (!interactive()) { library(BBmisc) library(parallelMap) source("testthat/helpers.R") source("testthat/helper_sockettest.R") sockettest() # } parallelMap/R/0000755000176200001440000000000014066072250012707 5ustar liggesusersparallelMap/R/displayErrorMessages.R0000644000176200001440000000126114066072250017201 0ustar liggesuserscheckResultsAndStopWithErrorsMessages = function(result.list) { if (length(result.list) > 0L) { inds = which(vlapply(result.list, inherits, what = "parallelMapErrorWrapper")) if (length(inds) > 0L) { stopWithJobErrorMessages(inds, vcapply(result.list[inds], as.character)) } } } stopWithJobErrorMessages = function(inds, msgs, extra.msg = NULL) { n = length(inds) msgs = head(msgs, 10L) inds = head(inds, 10L) msgs = sprintf("%05i: %s", inds, msgs) extra.msg = ifelse(is.null(extra.msg), "", sprintf("\n%s\n", extra.msg)) stopf("Errors occurred in %i slave jobs, displaying at most 10 of them:\n\n%s\n%s", n, collapse(msgs, sep = "\n"), extra.msg) } parallelMap/R/parallelExport.R0000644000176200001440000000620214066072250016030 0ustar liggesusers#' @title Export R objects for parallelization. #' #' @description Makes sure that the objects are exported to slave process so #' that they can be used in a job function which is later run with #' [parallelMap()]. #' #' @param ... [`character`]\cr #' Names of objects to export. #' @param objnames (`character(1)`)\cr #' Names of objects to export. #' Alternative way to pass arguments. #' @param level (`character(1)`)\cr #' If a (non-missing) level is specified in [parallelStart()], #' the function only exports if the level specified here matches. #' See [parallelMap()]. #' Useful if this function is used in a package. #' Default is `NA`. #' @param master (`logical(1)`)\cr #' Really export to package environment on master for local and multicore #' mode? If you do not do this your objects might not get exported for the #' mapping function call. Only disable when you are really sure. Default is #' `TRUE`. #' @param show.info (`logical(1)`)\cr #' Verbose output on console? #' Can be used to override setting from options / [parallelStart()]. #' Default is NA which means no overriding. #' @return Nothing. #' @export parallelExport = function(..., objnames, master = TRUE, level = NA_character_, show.info = NA) { args = list(...) assertList(args, types = "character") if (!missing(objnames)) { assertCharacter(objnames, any.missing = FALSE) objnames = c(as.character(args), objnames) } else { objnames = as.character(args) } assertFlag(master) assertString(level, na.ok = TRUE) assertFlag(show.info, na.ok = TRUE) mode = getPMOptMode() # remove duplicates objnames = unique(objnames) if (length(objnames) > 0L) { if (isParallelizationLevel(level)) { if (master && (isModeLocal() || isModeMulticore())) { showInfoMessage("Exporting objects to package env on master for mode: %s", mode, collapse(objnames)) for (n in objnames) { # FIXME 2x envir! assign(n, get(n, envir = sys.parent()), envir = PKG_LOCAL_ENV) } } if (isModeSocket() || isModeMPI()) { showInfoMessage("Exporting objects to slaves for mode %s: %s", mode, collapse(objnames)) # export via our helper function for (n in objnames) { exportToSlavePkgParallel(n, get(n, envir = sys.parent())) } } else if (isModeBatchJobs()) { showInfoMessage("Storing objects in files for BatchJobs slave jobs: %s", collapse(objnames)) objs = setNames(lapply(objnames, get, envir = sys.parent()), objnames) suppressMessages({ BatchJobs::batchExport(getBatchJobsReg(), li = objs, overwrite = TRUE) }) } else if (isModeBatchtools()) { old = getOption("batchtools.verbose") options(batchtools.verbose = FALSE) on.exit(options(batchtools.verbose = old)) showInfoMessage("Storing objects in files for batchtools slave jobs: %s", collapse(objnames)) reg = getBatchtoolsReg() objs = setNames(lapply(objnames, get, envir = sys.parent()), objnames) batchtools::batchExport(export = objs, reg = reg) } } } invisible(NULL) } parallelMap/R/autodetectCpus.R0000644000176200001440000000070014066072250016023 0ustar liggesusersautodetectCpus = function(mode) { if (mode == MODE_MULTICORE) { cpus = detectCores() } else if (mode == MODE_MPI) { cpus = max(1L, Rmpi::mpi.universe.size() - 1L) } else { cpus = NA_integer_ } if (!testCount(cpus, positive = TRUE)) { stopf("The was some problem in autodetecting the number of cpus. Autodetection returned:\n%s", printStrToChar(cpus)) } showInfoMessage("Autodetecting cpus: %i", cpus) return(cpus) } parallelMap/R/parallelStop.R0000644000176200001440000000375014066072250015501 0ustar liggesusers#' @title Stops parallelization. #' #' @description #' Sets mode to \dQuote{local}, i.e., parallelization is turned #' off and all necessary stuff is cleaned up. #' #' For socket and mpi mode `parallel::stopCluster()` is called. #' #' For BatchJobs mode the subdirectory of the `storagedir` #' containing the exported objects is removed. #' #' After a subsequent call of [parallelStart()], no exported objects #' are present on the slaves and no libraries are loaded, #' i.e., you have clean R sessions on the slaves. #' #' @return Nothing. #' @export parallelStop = function() { # only do something if we are in "started" state if (isStatusStarted()) { if (isModeSocket() || isModeMPI()) { # only stop if we registred one (exception in parallelStart can happen) # the whole following code is full of horrible stuff but I cannot change that # parallel is really buggy and the design is horrible # a) stopCluster will not work when called via stopCluster(NULL) on the default cluster # Through some envir assign "magic" cl gets set to NULL before it is stopped # via S3 inheritance # b) stopCluster will also throw amn exception when none is registered. great, and apparently # we have no way of asking whether one is alrealdy registered. cl = get("default", envir = getFromNamespace(".reg", ns = "parallel")) if (!is.null(cl)) { stopCluster(cl = cl) setDefaultCluster(NULL) } } else if (isModeBatchJobs()) { # delete registry file dir unlink(getBatchJobsRegFileDir(), recursive = TRUE) } if (!isModeLocal()) { showInfoMessage("Stopped parallelization. All cleaned up.") } } # remove our local export collection (local + multicore mode) rm(list = ls(PKG_LOCAL_ENV), envir = PKG_LOCAL_ENV) # in any case be in local / stopped mode now options(parallelMap.mode = MODE_LOCAL) options(parallelMap.status = STATUS_STOPPED) # FIXME do we clean up log files? invisible(NULL) } parallelMap/R/parallelSource.R0000644000176200001440000000743214066072250016015 0ustar liggesusers#' @title Source R files for parallelization. #' #' @description Makes sure that the files are sourced in slave process so that #' they can be used in a job function which is later run with [parallelMap()]. #' #' For all modes, the files are also (potentially) loaded on the master. #' #' @param ... [character]\cr #' File paths to sources. #' @param files [character]\cr #' File paths to sources. #' Alternative way to pass arguments. #' @param master (`logical(1)`)\cr #' Source files also on master for any mode? #' Default is `TRUE`. #' @param level (`character(1)`)\cr #' If a (non-missing) level is specified in [parallelStart()], #' the function only sources the files if the level specified here matches. #' See [parallelMap()]. #' Useful if this function is used in a package. #' Default is `NA`. #' @param show.info (`logical(1)`)\cr #' Verbose output on console? #' Can be used to override setting from options / [parallelStart()]. #' Default is NA which means no overriding. #' @return Nothing. #' @export parallelSource = function(..., files, master = TRUE, level = NA_character_, show.info = NA) { args = list(...) assertList(args, types = "character") if (!missing(files)) { assertCharacter(files, any.missing = FALSE) files = c(as.character(args), files) } else { files = as.character(args) } assertFlag(master) assertString(level, na.ok = TRUE) assertFlag(show.info, na.ok = TRUE) mode = getPMOptMode() # remove duplicates files = unique(files) if (length(files) > 0L) { if (master) { showInfoMessage("Sourcing files on master: %s", collapse(files)) lapply(files, source) } # if level matches, load on slaves if (isParallelizationLevel(level)) { # only source when we have not already done on master if (mode %in% c(MODE_LOCAL, MODE_MULTICORE)) { if (master) { showInfoMessage("Source files are already available on the slave") } else { showInfoMessage("Sourcing files on master (to be available on slaves for this mode): %s", collapse(files), show.info = show.info) lapply(files, source) } } else if (mode %in% c(MODE_SOCKET, MODE_MPI)) { showInfoMessage("Sourcing files on slaves: %s", collapse(files), show.info = show.info) .parallelMap.srcs = files exportToSlavePkgParallel(".parallelMap.srcs", .parallelMap.srcs) errs = clusterEvalQ(cl = NULL, { sapply(.parallelMap.srcs, function(f) { r = try(source(f)) if (inherits(r, "try-error")) { as.character(r) } else { NA_character_ } }, USE.NAMES = TRUE) }) # to vector, remove NA = ok, we also dont wnat to read error multiple times for multiple slaves errs = unlist(errs) errs = errs[!is.na(errs), drop = FALSE] errs = errs[!duplicated(names(errs))] if (length(errs) > 0L) { stopf("Files could not be sourced on all slaves: %s\n%s", collapse(names(errs)), collapse(paste(names(errs), errs, sep = "\n"), sep = "\n")) } } else if (isModeBatchJobs()) { showInfoMessage("Storing source file info for BatchJobs slave jobs: %s", collapse(files), show.info = show.info) suppressMessages({ reg = getBatchJobsReg() BatchJobs::addRegistrySourceFiles(reg, files, src.now = FALSE) }) } else if (isModeBatchtools()) { showInfoMessage("Storing source file info for batchtools slave jobs: %s", collapse(files), show.info = show.info) suppressMessages({ reg = getBatchtoolsReg() reg$source = unique(c(reg$source, files)) batchtools::saveRegistry(reg = reg) }) } } } invisible(NULL) } parallelMap/R/utils.R0000644000176200001440000000265414066072250014201 0ustar liggesusers# show message if OPTION show.info is TRUE # show.info ARGUMENT provides an immediate OVERRIDE to that option showInfoMessage = function(msg, ..., show.info = NA) { if (ifelse(is.na(show.info), getPMOptShowInfo(), show.info)) { messagef(msg, ...) } } showStartupMsg = function(mode, cpus, socket.hosts) { if (mode != MODE_LOCAL) { if (mode %in% c(MODE_MULTICORE, MODE_MPI) || (mode == MODE_SOCKET && !is.na(cpus))) { showInfoMessage("Starting parallelization in mode=%s with cpus=%i.", mode, cpus) } else if (mode == MODE_SOCKET) { showInfoMessage("Starting parallelization in mode=%s on %i hosts.", mode, length(socket.hosts)) } else if (mode == MODE_BATCHJOBS) { showInfoMessage("Starting parallelization in mode=%s-%s.", mode, BatchJobs::getConfig()$cluster.functions$name) } } } # either the option level is not set or it is and the level of parmap matches isParallelizationLevel = function(level) { optlevel = getPMOptLevel() is.na(optlevel) || (!is.na(level) && level == optlevel) } exportToSlavePkgParallel = function(objname, objval) { # clusterExport is trash because of envir argument, we cannot easily export # stuff defined in the scope of an R function # cl = NULL is default cluster, pos=1 is always globalenv # I really hope the nextline does what I think in all cases... clusterCall(cl = NULL, assign, x = objname, value = objval, pos = 1) } parallelMap/R/zzz.R0000644000176200001440000000252614066072250013674 0ustar liggesusers#' @import BBmisc #' @import checkmate #' @import parallel #' @importFrom utils getFromNamespace head tail #' @importFrom stats setNames # define constants MODE_LOCAL = "local" MODE_MULTICORE = "multicore" MODE_SOCKET = "socket" MODE_MPI = "mpi" MODE_BATCHJOBS = "BatchJobs" MODE_BATCHTOOLS = "batchtools" MODES = c(MODE_LOCAL, MODE_MULTICORE, MODE_SOCKET, MODE_MPI, MODE_BATCHJOBS, MODE_BATCHTOOLS) STATUS_STARTED = "started" STATUS_STOPPED = "stopped" PKG_LOCAL_ENV = new.env() .MulticoreCluster = new.env() .onLoad = function(libname, pkgname) { # init all settings from defaults # we cant call any function here in onload that dispatch to BBmisc... options( parallelMap.mode = getPMDefOption("mode", MODE_LOCAL), parallelMap.cpus = getPMDefOption("cpus", NA_integer_), parallelMap.socket.hosts = getPMDefOption("socket.hosts", NULL), parallelMap.level = getPMDefOption("level", NA_character_), parallelMap.logging = getPMDefOption("logging", FALSE), parallelMap.show.info = getPMDefOption("show.info", TRUE), parallelMap.storagedir = getPMDefOption("storagedir", getwd()), parallelMap.status = STATUS_STOPPED, parallelMap.on.slave = FALSE, parallelMap.registered.levels = list(), parallelMap.suppress.local.errors = FALSE, parallelMap.reproducible = TRUE ) # set defaults makeMulticoreCluster() } parallelMap/R/mclapply_fixed.R0000644000176200001440000000301114066072250016025 0ustar liggesusersif (getRversion() >= "3.1.1") { mcmapply_fixed = mcmapply } else { mcmapply_fixed = function(FUN, ..., MoreArgs = NULL, SIMPLIFY = TRUE, USE.NAMES = TRUE, mc.preschedule = TRUE, mc.set.seed = TRUE, mc.silent = FALSE, mc.cores = getOption("mc.cores", 2L), mc.cleanup = TRUE) { FUN <- match.fun(FUN) dots <- list(...) if (!length(dots)) { return(list()) } lens <- sapply(dots, length) n <- max(lens) if (n && min(lens) == 0L) { stop("Zero-length inputs cannot be mixed with those of non-zero length") } answer <- if (mc.cores == 1L) { # <- only touched this line! .mapply(FUN, dots, MoreArgs) } else { X <- if (!all(lens == n)) { lapply(dots, function(x) rep(x, length.out = n)) } else { dots } do_one <- function(indices, ...) { dots <- lapply(X, function(x) x[indices]) .mapply(FUN, dots, MoreArgs) } answer <- mclapply(seq_len(n), do_one, mc.preschedule = mc.preschedule, mc.set.seed = mc.set.seed, mc.silent = mc.silent, mc.cores = mc.cores, mc.cleanup = mc.cleanup) do.call(c, answer) } if (USE.NAMES && length(dots)) { if (is.null(names1 <- names(dots[[1L]])) && is.character(dots[[1L]])) { names(answer) <- dots[[1L]] } else if (!is.null(names1)) { names(answer) <- names1 } } if (!identical(SIMPLIFY, FALSE) && length(answer)) { simplify2array(answer, higher = (SIMPLIFY == "array")) } else { answer } } } parallelMap/R/batchjobs.R0000644000176200001440000000054714066072250014777 0ustar liggesusersgetBatchJobsNewRegFileDir = function() { fd = tempfile(pattern = "parallelMap_BatchJobs_reg_", tmpdir = getPMOptStorageDir()) options(parallelMap.bj.reg.filedir = fd) return(fd) } getBatchJobsRegFileDir = function() { getOption("parallelMap.bj.reg.filedir") } getBatchJobsReg = function() { BatchJobs::loadRegistry(getBatchJobsRegFileDir()) } parallelMap/R/batchtools.R0000644000176200001440000000061214066072250015173 0ustar liggesusersgetBatchtoolsNewRegFileDir = function() { fd = tempfile(pattern = "parallelMap_batchtools_reg_", tmpdir = getPMOptStorageDir()) options(parallelMap.bt.reg.filedir = fd) return(fd) } getBatchtoolsRegFileDir = function() { getOption("parallelMap.bt.reg.filedir") } getBatchtoolsReg = function() { batchtools::loadRegistry(file.dir = getBatchtoolsRegFileDir(), writeable = TRUE) } parallelMap/R/getExtraPackages.R0000644000176200001440000000025014066072250016251 0ustar liggesusersgetExtraPackages = function(mode) { switch(mode, MODE_MPI = "Rmpi", MODE_BATCHJOBS = "BatchJobs", MODE_BATCHTOOLS = "batchtools", character(0L) ) } parallelMap/R/parallelMap.R0000644000176200001440000003100014066072250015256 0ustar liggesusers#' @title Maps a function over lists or vectors in parallel. #' #' @description #' Uses the parallelization mode and the other options specified in #' [parallelStart()]. #' #' Libraries and source file can be initialized on slaves with #' [parallelLibrary()] and [parallelSource()]. #' #' Large objects can be separately exported via [parallelExport()], #' they can be simply used under their exported name in slave body code. #' #' Regarding error handling, see the argument `impute.error`. #' #' @param fun [function]\cr #' Function to map over `...`. #' @param ... (any)\cr #' Arguments to vectorize over (list or vector). #' @param more.args [list]\cr #' A list of other arguments passed to `fun`. #' Default is empty list. #' @param simplify (`logical(1)`)\cr #' Should the result be simplified? See [simplify2array]. If `TRUE`, #' `simplify2array(higher = TRUE)` will be called on the result object. #' Default is `FALSE`. #' @param use.names (`logical(1)`)\cr #' Should result be named? #' Use names if the first `...` argument has names, or if it is a #' character vector, use that character vector as the names. #' @param impute.error (`NULL` | `function(x)`)\cr #' This argument can be used for improved error handling. `NULL` means that, #' if an exception is generated on one of the slaves, it is also thrown on the #' master. Usually all slave jobs will have to terminate until this exception #' on the master can be thrown. If you pass a constant value or a function, #' all jobs are guaranteed to return a result object, without generating an #' exception on the master for slave errors. In case of an error, this is a #' [simpleError()] object containing the error message. If you passed a #' constant object, the error-objects will be substituted with this object. If #' you passed a function, it will be used to operate on these error-objects #' (it will ONLY be applied to the error results). For example, using #' `identity` would keep and return the `simpleError`-object, or `function(x) #' 99` would impute a constant value (which could be achieved more easily by #' simply passing `99`). Default is `NULL`. #' @param level (`character(1)`)\cr #' If a (non-missing) level is specified in [parallelStart()], #' this call is only parallelized if the level specified here matches. #' Useful if this function is used in a package. #' Default is `NA`. #' @param show.info (`logical(1)`)\cr #' Verbose output on console? #' Can be used to override setting from options / [parallelStart()]. #' Default is NA which means no overriding. #' @return Result. #' @export #' @examples #' parallelStart() #' parallelMap(identity, 1:2) #' parallelStop() parallelMap = function(fun, ..., more.args = list(), simplify = FALSE, use.names = FALSE, impute.error = NULL, level = NA_character_, show.info = NA) { assertFunction(fun) assertList(more.args) assertFlag(simplify) assertFlag(use.names) # if it is a constant value construct function to impute if (!is.null(impute.error)) { if (is.function(impute.error)) { impute.error.fun = impute.error } else { impute.error.fun = function(x) impute.error } } assertString(level, na.ok = TRUE) assertFlag(show.info, na.ok = TRUE) if (!is.na(level) && level %nin% unlist(getPMOption("registered.levels", list()))) { stopf("Level '%s' not registered", level) } cpus = getPMOptCpus() load.balancing = getPMOptLoadBalancing() logging = getPMOptLogging() reproducible = getPMOptReproducible() # use NA to encode "no logging" in logdir logdir = ifelse(logging, getNextLogDir(), NA_character_) if (isModeLocal() || !isParallelizationLevel(level) || getPMOptOnSlave()) { if (!is.null(impute.error)) { # so we behave in local mode as in parallelSlaveWrapper fun2 = function(...) { res = try(fun(...), silent = getOption("parallelMap.suppress.local.errors")) if (BBmisc::is.error(res)) { res = list(try.object = res) class(res) = "parallelMapErrorWrapper" } return(res) } } else { fun2 = fun } assignInFunctionNamespace(fun, env = PKG_LOCAL_ENV) res = mapply(fun2, ..., MoreArgs = more.args, SIMPLIFY = FALSE, USE.NAMES = FALSE) } else { iters = seq_along(..1) showInfoMessage("Mapping in parallel%s: mode = %s; level = %s; cpus = %i; elements = %i.", ifelse(load.balancing, " (load balanced)", ""), getPMOptMode(), level, getPMOptCpus(), length(iters), show.info = show.info) if (isModeMulticore()) { more.args = c(list(.fun = fun, .logdir = logdir), more.args) if (reproducible) { old.seed = .Random.seed old.rng.kind = RNGkind() seed = sample(1:100000, 1) # we need to reset the seed first in case the user supplied a seed, # otherwise "L'Ecuyer-CMRG" won't be used rm(.Random.seed, envir = globalenv()) set.seed(seed, "L'Ecuyer-CMRG") } res = MulticoreClusterMap(slaveWrapper, ..., .i = iters, MoreArgs = more.args, mc.cores = cpus, SIMPLIFY = FALSE, USE.NAMES = FALSE) if (reproducible) { # restore initial RNGkind .Random.seed = old.seed RNGkind(old.rng.kind[1], old.rng.kind[2], old.rng.kind[3]) } } else if (isModeSocket() || isModeMPI()) { more.args = c(list(.fun = fun, .logdir = logdir), more.args) if (load.balancing) { res = clusterMapLB(cl = NULL, slaveWrapper, ..., .i = iters, MoreArgs = more.args) } else { res = clusterMap(cl = NULL, slaveWrapper, ..., .i = iters, MoreArgs = more.args, SIMPLIFY = FALSE, USE.NAMES = FALSE) } } else if (isModeBatchJobs()) { # dont log extra in BatchJobs more.args = c(list(.fun = fun, .logdir = NA_character_), more.args) suppressMessages({ reg = getBatchJobsReg() # FIXME: this should be exported by BatchJobs ... asNamespace("BatchJobs")$dbRemoveJobs(reg, BatchJobs::getJobIds(reg)) BatchJobs::batchMap(reg, slaveWrapper, ..., more.args = more.args) # increase max.retries a bit, we dont want to abort here prematurely # if no resources set we submit with the default ones from the bj conf BatchJobs::submitJobs(reg, resources = getPMOptBatchJobsResources(), max.retries = 15) ok = BatchJobs::waitForJobs(reg, stop.on.error = is.null(impute.error)) }) # copy log files of terminated jobs to designated dir if (!is.na(logdir)) { term = BatchJobs::findTerminated(reg) fns = BatchJobs::getLogFiles(reg, term) dests = file.path(logdir, sprintf("%05i.log", term)) file.copy(from = fns, to = dests) } ids = BatchJobs::getJobIds(reg) ids.err = BatchJobs::findErrors(reg) ids.exp = BatchJobs::findExpired(reg) ids.done = BatchJobs::findDone(reg) ids.notdone = c(ids.err, ids.exp) # construct notdone error messages msgs = rep("Job expired!", length(ids.notdone)) msgs[ids.err] = BatchJobs::getErrorMessages(reg, ids.err) # handle errors (no impute): kill other jobs + stop on master if (is.null(impute.error) && length(c(ids.notdone)) > 0) { extra.msg = sprintf("Please note that remaining jobs were killed when 1st error occurred to save cluster time.\nIf you want to further debug errors, your BatchJobs registry is here:\n%s", reg$file.dir) onsys = BatchJobs::findOnSystem(reg) suppressMessages( BatchJobs::killJobs(reg, onsys) ) onsys = BatchJobs::findOnSystem(reg) if (length(onsys) > 0L) { warningf("Still %i jobs from operation on system! kill them manually!", length(onsys)) } if (length(ids.notdone) > 0L) { stopWithJobErrorMessages(ids.notdone, msgs, extra.msg) } } # if we reached this line and error occurred, we have impute.error != NULL (NULL --> stop before) res = vector("list", length(ids)) res[ids.done] = BatchJobs::loadResults(reg, simplify = FALSE, use.names = FALSE) res[ids.notdone] = lapply(msgs, function(s) impute.error.fun(simpleError(s))) } else if (isModeBatchtools()) { # don't log extra in batchtools more.args = insert(more.args, list(.fun = fun, .logdir = NA_character_)) old = getOption("batchtools.verbose") options(batchtools.verbose = FALSE) on.exit(options(batchtools.verbose = old)) reg = getBatchtoolsReg() if (nrow(reg$status) > 0L) { batchtools::clearRegistry(reg = reg) } ids = batchtools::batchMap(fun = slaveWrapper, ..., more.args = more.args, reg = reg) batchtools::submitJobs(ids = ids, resources = getPMOptBatchtoolsResources(), reg = reg) ok = batchtools::waitForJobs(ids = ids, stop.on.error = is.null(impute.error), reg = reg) # copy log files of terminated jobs to designated directory if (!is.na(logdir)) { x = batchtools::findStarted(reg = reg) x$log.file = file.path(reg$file.dir, "logs", sprintf("%s.log", x$job.hash)) .mapply(function(id, fn) writeLines(batchtools::getLog(id, reg = reg), con = fn), x, NULL) } if (ok) { res = batchtools::reduceResultsList(ids, reg = reg) } else { if (is.null(impute.error)) { extra.msg = sprintf("Please note that remaining jobs were killed when 1st error occurred to save cluster time.\nIf you want to further debug errors, your batchtools registry is here:\n%s", reg$file.dir) batchtools::killJobs(reg = reg) ids.notdone = batchtools::findNotDone(reg = reg) stopWithJobErrorMessages( inds = ids.notdone$job.id, batchtools::getErrorMessages(ids.notdone, missing.as.error = TRUE, reg = reg)$message, extra.msg) } else { # if we reached this line and error occurred, we have impute.error != NULL (NULL --> stop before) res = batchtools::findJobs(reg = reg) res$result = list() ids.complete = batchtools::findDone(reg = reg) ids.incomplete = batchtools::findNotDone(reg = reg) res[ids.complete, data.table::`:=`("result", batchtools::reduceResultsList(ids.complete, reg = reg)), with = FALSE] ids[ids.complete, data.table::`:=`("result", lapply(batchtools::getErrorMessages(ids.incomplete, reg = reg)$message, simpleError)), with = FALSE] } } } } # handle potential errors in res, depending on user setting if (is.null(impute.error)) { checkResultsAndStopWithErrorsMessages(res) } else { res = lapply(res, function(x) { if (inherits(x, "parallelMapErrorWrapper")) { impute.error.fun(attr(x$try.object, "condition")) } else { x } }) } if (use.names && !is.null(names(..1))) { names(res) = names(..1) } else if (use.names && is.character(..1)) { names(res) = ..1 } else if (!use.names) { names(res) = NULL } if (isTRUE(simplify) && length(res) > 0L) { res = simplify2array(res, higher = simplify) } # count number of mapping operations for log dir options(parallelMap.nextmap = (getPMOptNextMap() + 1L)) return(res) } slaveWrapper = function(..., .i, .fun, .logdir = NA_character_) { if (!is.na(.logdir)) { options(warning.length = 8170L, warn = 1L) .fn = file.path(.logdir, sprintf("%05i.log", .i)) .fn = file(.fn, open = "wt") .start.time = as.integer(Sys.time()) sink(.fn) sink(.fn, type = "message") on.exit(sink(NULL)) } # make sure we dont parallelize any further options(parallelMap.on.slave = TRUE) # just make sure, we should not have changed anything on the master # except for BatchJobs / interactive on.exit(options(parallelMap.on.slave = FALSE)) # wrap in try block so we can handle error on master res = try(.fun(...)) # now we cant simply return the error object, because clusterMap would act on it. great... if (BBmisc::is.error(res)) { res = list(try.object = res) class(res) = "parallelMapErrorWrapper" } if (!is.na(.logdir)) { .end.time = as.integer(Sys.time()) print(gc()) message(sprintf("Job time in seconds: %i", .end.time - .start.time)) # I am not sure why i need to do this again, but without i crash in multicore sink(NULL) } return(res) } assignInFunctionNamespace = function(fun, li = list(), env = new.env()) { # copy exported objects in PKG_LOCAL_ENV to env of fun so we can find them in any case in call ee = environment(fun) ns = ls(env) for (n in ns) { assign(n, get(n, envir = env), envir = ee) } ns = names(li) for (n in ns) { assign(n, li[[n]], envir = ee) } } parallelMap/R/parallelApply.R0000644000176200001440000000263614066072250015643 0ustar liggesusers#' Parallel versions of apply-family functions. #' #' `parallelLapply`: A parallel [lapply()] version.\cr #' `parallelSapply`: A parallel [sapply()] version.\cr #' All functions are simple wrappers for [parallelMap()]. #' #' @param xs (`vector` | `list`)\cr #' `fun` is applied to the elements of this argument. #' @param fun [`function`]\cr #' Function to map over `xs`. #' @param ... (any)\cr #' Further arguments passed to `fun`. #' @param simplify (`logical(1)`)\cr #' See [sapply()]. #' Default is `TRUE`. #' @param use.names (`logical(1)`)\cr #' See [sapply()]. #' Default is `TRUE`. #' @param impute.error (`NULL` | `function(x)`)\cr #' See [parallelMap()]. #' @param level (`character(1)`)\cr #' See [parallelMap()]. #' @return For `parallelLapply` a named list, for `parallelSapply` it depends #' on the return value of `fun` and the settings of `simplify` and #' `use.names`. #' @export parallelLapply = function(xs, fun, ..., impute.error = NULL, level = NA_character_) { parallelMap(fun, xs, more.args = list(...), simplify = FALSE, use.names = TRUE, impute.error = impute.error, level = level) } #' @rdname parallelLapply #' @export parallelSapply = function(xs, fun, ..., simplify = TRUE, use.names = TRUE, impute.error = NULL, level = NA_character_) { parallelMap(fun, xs, more.args = list(...), simplify = simplify, use.names = use.names, impute.error = impute.error, level = level) } parallelMap/R/clusterMapLB.R0000644000176200001440000000067014066072250015372 0ustar liggesusersclusterMapLB = function(cl, fun, ..., MoreArgs = NULL) { force(fun) args = list(...) force(MoreArgs) if (length(args) == 0L) { stop("need at least one argument") } n = viapply(args, length) vlen = max(n) ind = which(n != vlen) args[ind] = lapply(args[ind], rep, length = vlen) wrapper = function(i) do.call(fun, args = c(lapply(args, function(x) x[[i]]), MoreArgs)) clusterApplyLB(cl, seq_len(vlen), wrapper) } parallelMap/R/getOption.R0000644000176200001440000000624214066072250015006 0ustar liggesusersgetPMOption = function(opt, def) { getOption(sprintf("parallelMap.%s", opt), def) } getPMDefOption = function(opt, def) { getOption(sprintf("parallelMap.default.%s", opt), def) } getPMOptStatus = function() { getPMOption("status") } ##### PM current options ##### getPMOptMode = function() { getPMOption("mode") } getPMOptCpus = function() { getPMOption("cpus") } getPMOptSocketHosts = function() { getPMOption("socket.hosts") } getPMOptLogging = function() { getPMOption("logging") } getPMOptLevel = function() { getPMOption("level") } getPMOptLoadBalancing = function() { getPMOption("load.balancing") } getPMOptShowInfo = function() { getPMOption("show.info") } getPMOptStorageDir = function() { getPMOption("storagedir") } getPMOptBatchJobsResources = function() { getPMOption("bj.resources", list()) } getPMOptBatchtoolsResources = function() { getPMOption("bt.resources", list()) } getPMOptNextMap = function() { getPMOption("nextmap") } getPMOptOnSlave = function() { getPMOption("on.slave") } getPMOptReproducible = function() { getPMOption("reproducible") } ##### PM default options ##### getPMDefOptMode = function(mode) { if (missing(mode)) { mode = getPMDefOption("mode", MODE_LOCAL) } assertChoice(mode, MODES) return(mode) } getPMDefOptCpus = function(cpus) { # NA means "do autodetect" if (missing(cpus)) { cpus = getPMDefOption("cpus", NA_integer_) } cpus = asInt(cpus, na.ok = TRUE, lower = 1L) return(cpus) } getPMDefOptSocketHosts = function(socket.hosts) { if (missing(socket.hosts)) { socket.hosts = getPMDefOption("socket.hosts", NULL) } if (!is.null(socket.hosts)) { assertCharacter(socket.hosts, min.len = 1L, any.missing = FALSE) } return(socket.hosts) } getPMDefOptLogging = function(logging) { if (missing(logging)) { logging = getPMDefOption("logging", FALSE) } assertFlag(logging) return(logging) } getPMDefOptLevel = function(level) { if (missing(level)) { level = getPMDefOption("level", NA_character_) } assertString(level, na.ok = TRUE) return(level) } getPMDefOptShowInfo = function(show.info) { if (missing(show.info)) { show.info = getPMDefOption("show.info", TRUE) } assertFlag(show.info) return(show.info) } getPMDefOptStorageDir = function(storagedir) { if (missing(storagedir)) { storagedir = getPMDefOption("storagedir", getwd()) } assertString(storagedir) return(storagedir) } getPMDefOptReproducible = function(reproducible) { if (missing(reproducible)) { reproducible = getPMDefOption("reproducible", TRUE) } assertFlag(reproducible) return(reproducible) } ##### modes ##### isModeLocal = function() { getPMOptMode() == MODE_LOCAL } isModeMulticore = function() { getPMOptMode() == MODE_MULTICORE } isModeSocket = function() { getPMOptMode() == MODE_SOCKET } isModeMPI = function() { getPMOptMode() == MODE_MPI } isModeBatchJobs = function() { getPMOptMode() == MODE_BATCHJOBS } isModeBatchtools = function() { getPMOptMode() == MODE_BATCHTOOLS } ##### status ##### isStatusStarted = function() { getPMOptStatus() == STATUS_STARTED } isStatusStopped = function() { getPMOptStatus() == STATUS_STOPPED } parallelMap/R/parallelShowOptions.R0000644000176200001440000000344014066072250017044 0ustar liggesusers#' @title Retrieve the configured package options. #' #' @description #' Returned are current and default settings, both as lists. #' The return value has slots elements `settings` and `defaults`, #' which are both lists of the same structure, named by option names. #' #' A printer exists to display this object. #' #' For details on the configuration procedure please read #' [parallelStart()] and . #' #' @return `ParallelMapOptions`. See above. #' @export parallelGetOptions = function() { opts = c("mode", "cpus", "level", "logging", "show.info", "storagedir", "bj.resources", "reproducible") settings = setNames(lapply(opts, getPMOption), opts) defaults = setNames(lapply(opts, getPMDefOption), opts) makeS3Obj("ParallelMapOptions", settings = settings, defaults = defaults) } #' @export print.ParallelMapOptions = function(x, ...) { mycat = function(opt) { opt1val = x$settings[[opt]] opt2val = x$defaults[[opt]] if (opt == "bj.resources") { opt1val = ifelse(length(opt1val) == 0L, "(defaults from BatchJobs config)", convertToShortString(opt1val)) if (!is.null(opt2val)) { opt2val = convertToShortString(opt2val) } } if (is.null(opt2val)) { opt2val = "not set" } if (opt %nin% c("bj.resources", "storagedir")) { catf("%-20s: %-10s (%s)", opt, opt1val, opt2val) } else { catf("%-20s: %-10s\n (%s)", opt, opt1val, opt2val) } } catf("%-20s: %-10s (%s)", "parallelMap options", "value", "default") catf("") mycat("mode") mycat("cpus") mycat("level") mycat("logging") mycat("show.info") mycat("storagedir") mycat("reproducible") if (isModeBatchJobs() || identical(getPMDefOptMode(), MODE_BATCHJOBS)) { mycat("bj.resources") } } parallelMap/R/parallelRegisterLevels.R0000644000176200001440000000574214066072250017516 0ustar liggesusers#' @title Register a parallelization level #' #' @description #' Package developers should call this function in their packages' #' `base::.onLoad()`. This enables the user to query available levels and bind #' parallelization to specific levels. This is especially helpful for nested #' calls to [parallelMap()], e.g. where the inner call should be parallelized #' instead of the outer one. #' #' To avoid name clashes, we encourage developers to always specify the argument #' `package`. This will prefix the specified levels with the string containing #' the package name, e.g. `parallelRegisterLevels(package="foo", #' levels="dummy")` will register the level \dQuote{foo.dummy} and users can #' start parallelization for this level with `parallelStart(, level = #' "parallelMap.dummy")`. If you do not provide `package`, the level names will #' be associated with category \dQuote{custom} and can there be later referred #' to with \dQuote{custom.dummy}. #' #' @param package (`character(1)`)\cr #' Name of your package. #' Default is \dQuote{custom} (we are not in a package). #' @param levels (`character(1)`)\cr #' Available levels that are used in the [parallelMap()] operations of your #' package or code. If `package` is not missing, all levels will be prefixed #' with \dQuote{package.}. #' @return Nothing. #' @export parallelRegisterLevels = function(package = "custom", levels) { assertString(package) assertCharacter(levels, min.len = 1L, any.missing = FALSE) reg.levs = getPMOption("registered.levels", list()) if (is.na(package)) { reg.levs[["custom"]] = union(reg.levs[["custom"]], levels) } else { reg.levs[[package]] = union(reg.levs[[package]], sprintf("%s.%s", package, levels)) } options(parallelMap.registered.levels = reg.levs) invisible(NULL) } #' @title Get registered parallelization levels for all currently loaded packages. #' #' @description #' With `flatten = FALSE`, a structured S3 object is returned. The S3 object #' only has one slot, which is called `levels`. This contains a named list. Each #' name refers to `package` from the call to [parallelRegisterLevels()], while #' the entries are character vectors of the form \dQuote{package.level}. With #' `flatten = TRUE`, a simple character vector is returned that contains all #' concatenated entries of `levels` from above. #' #' @param flatten (`logical(1)`)\cr #' Flatten to character vector or not? See description. #' Default is `FALSE`. #' @return `RegisteredLevels` | `character`. See above. #' @export parallelGetRegisteredLevels = function(flatten = FALSE) { assertFlag(flatten) lvls = getPMOption("registered.levels", list()) if (flatten) { return(as.character(unlist(lvls))) } else { return(makeS3Obj("RegisteredLevels", levels = lvls)) } } #' @export print.RegisteredLevels = function(x, ...) { levs = parallelGetRegisteredLevels()$levels ns = names(levs) for (i in seq_along(levs)) { catf("%s: %s", ns[i], collapse(levs[[i]], sep = ", ")) } invisible(NULL) } parallelMap/R/parallelLibrary.R0000644000176200001440000000740614066072250016162 0ustar liggesusers#' @title Load packages for parallelization. #' #' @description Makes sure that the packages are loaded in slave process so that #' they can be used in a job function which is later run with [parallelMap()]. #' #' For all modes, the packages are also (potentially) loaded on the master. #' #' @param ... [character]\cr #' Names of packages to load. #' @param packages (`character(1)`)\cr #' Names of packages to load. #' Alternative way to pass arguments. #' @param master (`logical(1)`)\cr #' Load packages also on master for any mode? #' Default is `TRUE`. #' @param level (`character(1)`)\cr #' If a (non-missing) level is specified in [parallelStart()], #' the function only loads the packages if the level specified here matches. #' See [parallelMap()]. #' Useful if this function is used in a package. #' Default is `NA`. #' @param show.info (`logical(1)`)\cr #' Verbose output on console? #' Can be used to override setting from options / [parallelStart()]. #' Default is NA which means no overriding. #' @return Nothing. #' @export parallelLibrary = function(..., packages, master = TRUE, level = NA_character_, show.info = NA) { args = list(...) assertList(args, types = "character") if (!missing(packages)) { assertCharacter(packages, any.missing = FALSE) packages = c(as.character(args), packages) } else { packages = as.character(args) } assertFlag(master) assertString(level, na.ok = TRUE) assertFlag(show.info, na.ok = TRUE) mode = getPMOptMode() level = getPMOptLevel() # remove duplicates packages = unique(packages) if (length(packages) > 0L) { if (master) { requirePackages(packages, why = "parallelLibrary") } # if level matches, load on slaves if (isParallelizationLevel(level)) { # only load when we have not already done on master if (mode %in% c(MODE_LOCAL)) { if (master) { showInfoMessage("Packages already available on the slaves") } else { showInfoMessage("Loading packages on master (to be available on slaves for mode %s): %s", mode, collapse(packages), show.info = show.info) requirePackages(packages, why = "parallelLibrary") } } else if (mode %in% c(MODE_SOCKET, MODE_MPI)) { showInfoMessage("Loading packages on slaves for mode %s: %s", mode, collapse(packages), show.info = show.info) .parallelMap.pkgs = packages exportToSlavePkgParallel(".parallelMap.pkgs", .parallelMap.pkgs) # oks is a list (slaves) of logical vectors (pkgs) oks = clusterEvalQ(cl = NULL, { vapply(.parallelMap.pkgs, require, character.only = TRUE, USE.NAMES = TRUE, FUN.VALUE = NA) }) # get not loaded pkgs not.loaded = lapply(oks, function(v) { names(v)[!v] }) not.loaded = unique(unlist(not.loaded)) if (length(not.loaded) > 0L) { stopf("Packages could not be loaded on all slaves: %s.", collapse(not.loaded)) } } else if (mode %in% c(MODE_BATCHJOBS)) { showInfoMessage("Storing package info for BatchJobs slave jobs: %s", collapse(packages), show.info = show.info) # collect in R option, add new packages to old ones suppressMessages({ reg = getBatchJobsReg() BatchJobs::addRegistryPackages(reg, packages) }) } else if (mode %in% c(MODE_BATCHTOOLS)) { showInfoMessage("Storing package info for batchtools slave jobs: %s", collapse(packages), show.info = show.info) # collect in R option, add new packages to old ones suppressMessages({ reg = getBatchtoolsReg() reg$packages <- c(reg$packages, packages) batchtools::saveRegistry(reg = reg) }) } } } invisible(NULL) } parallelMap/R/logging.R0000644000176200001440000000111314066072250014454 0ustar liggesusers# get all log dirs under storage dir getLogDirs = function() { # FIXME why would someone restrict here to 999 maps? list.files(getPMOptStorageDir(), pattern = "parallelMap_logs_???", full.names = TRUE) } # delete all log dirs under storage dir deleteAllLogDirs = function() { fns = getLogDirs() n = length(fns) if (n > 0L) { showInfoMessage("Deleting %i log dirs in storage dir.", n) } } getNextLogDir = function() { ld = file.path(getPMOptStorageDir(), sprintf("parallelMap_logs_%03i", getPMOptNextMap())) if (!file.exists(ld)) { dir.create(ld) } ld } parallelMap/R/makeMulticoreCluster.R0000644000176200001440000000144214066072250017176 0ustar liggesusers# fake cluster constructor mimicking makeCluster to store some settings. makeMulticoreCluster = function(mc.preschedule = FALSE, mc.set.seed = TRUE, mc.silent = FALSE, mc.cleanup = TRUE) { assertFlag(mc.preschedule) assertFlag(mc.set.seed) assertFlag(mc.silent) assertFlag(mc.cleanup) x = get(".MulticoreCluster", envir = getNamespace("parallelMap")) x$mc.preschedule = mc.preschedule x$mc.set.seed = mc.set.seed x$mc.silent = mc.silent x$mc.cleanup = mc.cleanup invisible(TRUE) } MulticoreClusterMap = function(FUN, ...) { opts = as.list(get(".MulticoreCluster", envir = getNamespace("parallelMap"))) mcmapply_fixed(FUN, ..., mc.preschedule = opts$mc.preschedule, mc.set.seed = opts$mc.set.seed, mc.silent = opts$mc.silent, mc.cleanup = opts$mc.cleanup) } parallelMap/R/parallelStart.R0000644000176200001440000002662414066072250015656 0ustar liggesusers#' Parallelization setup for parallelMap. #' #' Defines the underlying parallelization mode for [parallelMap()]. Also allows #' to set a \dQuote{level} of parallelization. Only calls to [parallelMap()] #' with a matching level are parallelized. The defaults of all settings are #' taken from your options, which you can also define in your R profile. For an #' introductory tutorial and information on the options configuration, please go #' to the project's github page at https://github.com/mlr-org/parallelMap. #' #' Currently the following modes are supported, which internally dispatch the #' mapping operation to functions from different parallelization packages: #' #' - **local**: No parallelization with [mapply()] #' - **multicore**: Multicore execution on a single machine with `parallel::mclapply()`. #' - **socket**: Socket cluster on one or multiple machines with `parallel::makePSOCKcluster()` and `parallel::clusterMap()`. #' - **mpi**: Snow MPI cluster on one or multiple machines with [parallel::makeCluster()] and `parallel::clusterMap()`. #' - **BatchJobs**: Parallelization on batch queuing HPC clusters, e.g., Torque, SLURM, etc., with [BatchJobs::batchMap()]. #' #' For BatchJobs mode you need to define a storage directory through the #' argument `storagedir` or the option `parallelMap.default.storagedir`. #' #' @param mode (`character(1)`)\cr #' Which parallel mode should be used: \dQuote{local}, \dQuote{multicore}, #' \dQuote{socket}, \dQuote{mpi}, \dQuote{BatchJobs}. Default is the option #' `parallelMap.default.mode` or, if not set, \dQuote{local} without parallel #' execution. #' @param cpus (`integer(1)`)\cr #' Number of used cpus. For local and BatchJobs mode this argument is ignored. #' For socket mode, this is the number of processes spawned on localhost, if #' you want processes on multiple machines use `socket.hosts`. Default is the #' option `parallelMap.default.cpus` or, if not set, [parallel::detectCores()] #' for multicore mode, `max(1, [mpi.universe.size][Rmpi::mpi.universe.size] - #' 1)` for mpi mode and 1 for socket mode. #' @param socket.hosts [character]\cr #' Only used in socket mode, otherwise ignored. Names of hosts where parallel #' processes are spawned. Default is the option #' `parallelMap.default.socket.hosts`, if this option exists. #' @param bj.resources [list]\cr #' Resources like walltime for submitting jobs on HPC clusters via BatchJobs. #' See [BatchJobs::submitJobs()]. Defaults are taken from your BatchJobs #' config file. #' @param bt.resources [list]\cr #' Analog to `bj.resources`. #' See [batchtools::submitJobs()]. #' @param logging (`logical(1)`)\cr #' Should slave output be logged to files via [sink()] under the `storagedir`? #' Files are named `.log` and put into unique subdirectories #' named `parallelMap_log_` for each subsequent [parallelMap()] #' operation. Previous logging directories are removed on `parallelStart` if #' `logging` is enabled. Logging is not supported for local mode, because you #' will see all output on the master and can also run stuff like [traceback()] #' in case of errors. Default is the option `parallelMap.default.logging` or, #' if not set, `FALSE`. #' @param storagedir (`character(1)`)\cr #' Existing directory where log files and intermediate objects for BatchJobs #' mode are stored. Note that all nodes must have write access to exactly this #' path. Default is the current working directory. #' @param level (`character(1)`)\cr #' You can set this so only calls to [parallelMap()] that have exactly the #' same level are parallelized. Default is the option #' `parallelMap.default.level` or, if not set, `NA` which means all calls to #' [parallelMap()] are are potentially parallelized. #' @param load.balancing (`logical(1)`)\cr #' Enables load balancing for multicore, socket and mpi. #' Set this to `TRUE` if you have heterogeneous runtimes. #' Default is `FALSE` #' @param show.info (`logical(1)`)\cr #' Verbose output on console for all further package calls? Default is the #' option `parallelMap.default.show.info` or, if not set, `TRUE`. #' @param suppress.local.errors (`logical(1)`)\cr #' Should reporting of error messages during function evaluations in local #' mode be suppressed? Default ist FALSE, i.e. every error message is shown. #' @param reproducible (`logical(1)`)\cr #' Should parallel jobs produce reproducible results when setting a seed? #' With this option, `parallelMap()` calls will be reproducible when using #' `set.seed()` with the default RNG kind. This is not the case by default #' when parallelizing in R, since the default RNG kind "Mersenne-Twister" is #' not honored by parallel processes. Instead RNG kind `"L'Ecuyer-CMRG"` needs #' to be used to ensure paralllel reproducibility. #' Default is the option `parallelMap.default.reproducible` or, if not set, #' `TRUE`. #' @param ... (any)\cr #' Optional parameters, for socket mode passed to #' `parallel::makePSOCKcluster()`, for mpi mode passed to #' [parallel::makeCluster()] and for multicore passed to #' `parallel::mcmapply()` (`mc.preschedule` (overwriting `load.balancing`), #' `mc.set.seed`, `mc.silent` and `mc.cleanup` are supported for multicore). #' @return Nothing. #' @export parallelStart = function(mode, cpus, socket.hosts, bj.resources = list(), bt.resources = list(), logging, storagedir, level, load.balancing = FALSE, show.info, suppress.local.errors = FALSE, reproducible, ...) { # if stop was not called, warn and do it now if (isStatusStarted() && !isModeLocal()) { warningf("Parallelization was not stopped, doing it now.") parallelStop() } # FIXME: what should we do onexit if an error happens in this function? mode = getPMDefOptMode(mode) cpus = getPMDefOptCpus(cpus) socket.hosts = getPMDefOptSocketHosts(socket.hosts) reproducible = getPMDefOptReproducible(reproducible) level = getPMDefOptLevel(level) rlevls = parallelGetRegisteredLevels(flatten = TRUE) if (!is.na(level) && level %nin% rlevls) { warningf( "Selected level='%s' not registered! This is likely an error! Note that you can also register custom levels yourself to get rid of this warning, see ?parallelRegisterLevels.R", level) } logging = getPMDefOptLogging(logging) storagedir = getPMDefOptStorageDir(storagedir) # defaults are in batchjobs conf assertList(bj.resources) assertList(bt.resources) assertFlag(load.balancing) show.info = getPMDefOptShowInfo(show.info) # multicore not supported on windows if (mode == MODE_MULTICORE && .Platform$OS.type == "windows") { stop("Multicore mode not supported on windows!") } assertDirectoryExists(storagedir, access = "w") # store options for session, we already need them for helper funs below options(parallelMap.mode = mode) options(parallelMap.level = level) options(parallelMap.logging = logging) options(parallelMap.storagedir = storagedir) options(parallelMap.bj.resources = bj.resources) options(parallelMap.bt.resources = bt.resources) options(parallelMap.load.balancing = load.balancing) options(parallelMap.show.info = show.info) options(parallelMap.status = STATUS_STARTED) options(parallelMap.nextmap = 1L) options(parallelMap.suppress.local.errors = suppress.local.errors) options(parallelMap.reproducible = reproducible) # try to autodetect cpus if not set if (is.na(cpus) && mode %in% c(MODE_MULTICORE, MODE_MPI)) { cpus = autodetectCpus(mode) } if (isModeSocket()) { if (!is.na(cpus) && !is.null(socket.hosts)) { stopf("You cannot set both cpus and socket.hosts in socket mode!") } if (is.na(cpus) && is.null(socket.hosts)) { cpus = 1L } } if (isModeLocal()) { if (!is.na(cpus)) { stopf("Setting %i cpus makes no sense for local mode!", cpus) } } options(parallelMap.cpus = cpus) showStartupMsg(mode, cpus, socket.hosts) # now load extra packs we need requirePackages(getExtraPackages(mode), why = "parallelStart") # delete log dirs from previous runs if (logging) { if (isModeLocal()) { stop("Logging not supported for local mode!") } deleteAllLogDirs() } # init parallel packs / modes, if necessary if (isModeMulticore()) { args = list(...) args$mc.preschedule = args$mc.preschedule %??% !load.balancing cl = do.call(makeMulticoreCluster, args) } else if (isModeSocket()) { # set names from cpus or socket.hosts, only 1 can be defined here if (is.na(cpus)) { names = socket.hosts } else { names = cpus } cl = makePSOCKcluster(names = names, ...) if (reproducible) { clusterSetRNGStream(cl, iseed = sample(1:100000, 1)) } setDefaultCluster(cl) } else if (isModeMPI()) { cl = makeCluster(spec = cpus, type = "MPI", ...) if (reproducible) { clusterSetRNGStream(cl, iseed = sample(1:100000, 1)) } setDefaultCluster(cl) } else if (isModeBatchJobs()) { # create registry in selected directory with random, unique name fd = getBatchJobsNewRegFileDir() suppressMessages({ BatchJobs::makeRegistry(id = basename(fd), file.dir = fd, work.dir = getwd()) }) } else if (isModeBatchtools()) { fd = getBatchtoolsNewRegFileDir() old = getOption("batchtools.verbose") options(batchtools.verbose = FALSE) on.exit(options(batchtools.verbose = old)) reg = batchtools::makeRegistry(file.dir = fd, work.dir = getwd()) } invisible(NULL) } #' @export #' @rdname parallelStart parallelStartLocal = function(show.info, suppress.local.errors = FALSE, ...) { parallelStart( mode = MODE_LOCAL, cpus = NA_integer_, level = NA_character_, logging = FALSE, show.info = show.info, suppress.local.errors = suppress.local.errors, ...) } #' @export #' @rdname parallelStart parallelStartMulticore = function(cpus, logging, storagedir, level, load.balancing = FALSE, show.info, reproducible, ...) { parallelStart( mode = MODE_MULTICORE, cpus = cpus, level = level, logging = logging, storagedir = storagedir, load.balancing = load.balancing, show.info = show.info, reproducible = reproducible, ...) } #' @export #' @rdname parallelStart parallelStartSocket = function(cpus, socket.hosts, logging, storagedir, level, load.balancing = FALSE, show.info, reproducible, ...) { parallelStart( mode = MODE_SOCKET, cpus = cpus, socket.hosts = socket.hosts, level = level, logging = logging, storagedir = storagedir, load.balancing = load.balancing, show.info = show.info, reproducible = reproducible, ...) } #' @export #' @rdname parallelStart parallelStartMPI = function(cpus, logging, storagedir, level, load.balancing = FALSE, show.info, reproducible, ...) { parallelStart( mode = MODE_MPI, cpus = cpus, level = level, logging = logging, storagedir = storagedir, load.balancing = load.balancing, show.info = show.info, reproducible = reproducible, ...) } #' @export #' @rdname parallelStart parallelStartBatchJobs = function(bj.resources = list(), logging, storagedir, level, show.info, ...) { parallelStart( mode = MODE_BATCHJOBS, level = level, logging = logging, storagedir = storagedir, bj.resources = bj.resources, show.info = show.info, ...) } #' @export #' @rdname parallelStart parallelStartBatchtools = function(bt.resources = list(), logging, storagedir, level, show.info, ...) { parallelStart( mode = MODE_BATCHTOOLS, level = level, logging = logging, storagedir = storagedir, bt.resources = bt.resources, show.info = show.info, ...) } parallelMap/NEWS.md0000644000176200001440000000623414066074706013621 0ustar liggesusers # parallelMap 1.5.1 - Removed "LazyDate" field from DESCRIPTION. - Fixed broken URLs. # parallelMap 1.5.0 - `parallelLapply()` does not drop list element names anymore (#58) - `parallelStart()` gains argument `reproducible`. This argument ensures reproducibility across parallel workers and is set to `TRUE` by default. Internally, we take care to use the `"L'Ecuyer-CMRG"` RNG kind or `clusterSetRNGStream()` (depending on the parallel mode) to ensure this. This argument works similar to the `future.seed` argument for future-based parallelization which also ensures reproducibility across parallel processes with the standard RNG kind. - `parallelLibrary()`: Respect custom levels when exporting packages (#67) - `parallelLibrary()`: Allow to add packages to a batchtools library (@dagola, #70) - Bugfix: Printing the state of an object holding the current parallelMap options (queried via `parallelGetOptions()`) did not return the object state but instead the global state of the options (#41, @mb706). # parallelMap 1.4.0.9000 - `parallelLapply()` does not drop list element names anymore (#58) - `parallelStart()` gains argument `reproducible`. This argument ensures reproducibility across parallel workers and is set to `TRUE` by default. Internally, we take care to use the `"L'Ecuyer-CMRG"` RNG kind or `clusterSetRNGStream()` (depending on the parallel mode) to ensure this. This argument works similar to the `future.seed` argument for future-based parallelization which also ensures reproducibility across parallel processes with the standard RNG kind. - `parallelLibrary()`: Respect custom levels when exporting packages (#67) - `parallelLibrary()`: Allow to add packages to a batchtools library (@dagola, #70) - Bugfix: Printing the state of an object holding the current parallelMap options (queried via `parallelGetOptions()`) did not return the object state but instead the global state of the options (#41, @mb706). # parallelMap 1.4 - Load balancing for multicore, socket and mpi can now be controlled via the flag "load.balancing" passed to parallelStart(). Note that the default for multicore now defaults to disabled load balancing. - BatchTools mode # parallelMap 1.3 - parallelGetRegisteredLevels has new argument "flatten" - parallelShowOptions was converted to parallelGetOptions (with a printer) # parallelMap 1.2 - Arguments of mcmapply (mc.preschedule, ...) can now be specified via parallelStart - We import package "parallel" now - parallelShowRegisteredLevels was changed to parallelGetRegisteredLevels. The latter returns a structured object, with a printer method. # parallelMap 1.1 - Package in general much more stable now - parallelLibrary was improved a lot - better / more configurable info messages on console - BatchJobs mode: working directory for slave jobs is the current working dir on the master, not the storage.dir - BatchJobs mode: errors are thrown, if jobs expire - parallelMap/Lapply/Sapply: impute.error option - removed autostart option for stability ## new functions - parallelSource - parallelExport # parallelMap 1.0-83 - First submit to CRAN. parallelMap/MD50000644000176200001440000000575214066267304013035 0ustar liggesusers699c860254a7a660c6e3f096472eddb6 *DESCRIPTION 877dea832be8d8f8225718be1ab34ba4 *LICENSE fd96945af86ecf9309bf356ad5d4b071 *NAMESPACE a3d2ce2f073a01711246be3435ecab12 *NEWS.md fb441651c011f2a3296a6eba361f5ac1 *R/autodetectCpus.R 3389162c62a8a7f836bb1efea2b16824 *R/batchjobs.R 3e4c6318ded2ddb8433b11094ca5b908 *R/batchtools.R 83e804c9ae102e837507fca512c3f6e3 *R/clusterMapLB.R 5820f2d78adfc29cdf2d422361a038cb *R/displayErrorMessages.R 825533a2be239457b561464a2115bf8d *R/getExtraPackages.R 4a93ce91113c044151a8c1e4070de4f3 *R/getOption.R d5f07bb44fb2b0556725c248de7c34d2 *R/logging.R 13baacb8e4fbe540bfcc1cab9f7cf98e *R/makeMulticoreCluster.R e0605eefda7d0b7ee2f4761bc3584fbd *R/mclapply_fixed.R f0350498ad492b5cfdefdc8a2a3cf932 *R/parallelApply.R 7f538bd75997e017c1b612a84ab32ee6 *R/parallelExport.R 1f10ba4a769a85396d884386b6ec3b27 *R/parallelLibrary.R c55b17d6768104134e40ada133c59078 *R/parallelMap.R 7d7376c58259a4f54d79a9bb1765a696 *R/parallelRegisterLevels.R fd5aa2b0052f25139ec7b2ccd651c8fd *R/parallelShowOptions.R 183b7c7b7280e15b79294fb96ae7b5b9 *R/parallelSource.R 327cacb5268217643f78659327fea2c0 *R/parallelStart.R 7c4a96bb8a24504e5f160097dcb8a155 *R/parallelStop.R 4726ef7f33f4c3232608948ccd6ed4d3 *R/utils.R 73e3dc04173899d99c14e1f84f0e1ddc *R/zzz.R 6fea27662fd36ef1ec73f029bda85335 *inst/test_source_file.R cfce6663bf863d81f711b7b0bb5d8f91 *man/parallelExport.Rd bcdd9e603c89c436366114b2965149db *man/parallelGetOptions.Rd d8595eaa970ef6a45c8460b7fb6cd6e0 *man/parallelGetRegisteredLevels.Rd 862cd636be5910954fb8de62cb743fa9 *man/parallelLapply.Rd a4a172132b7c5e402bde86d7900cc53d *man/parallelLibrary.Rd c7e4b7009c20507230b42c477d725585 *man/parallelMap.Rd f9aee5d5ce83ebd9d3d32af4c9326206 *man/parallelRegisterLevels.Rd 9d8490fa9d11038e4abbd7d0b50001d3 *man/parallelSource.Rd 855afd1f4ec4dcf3f5497caaaf56600f *man/parallelStart.Rd 78622d96e3d2370c5b81c957a7c4cec6 *man/parallelStop.Rd c4d084ffc664e1711000eedec80d9b3f *tests/run-all.R 424a1a80ce56385be31587546b95dcd6 *tests/testthat/helper_sockettest.R fc806427e42c90d7c1efa8221628dd02 *tests/testthat/helpers.R af490702d5a2466f00f3e73ea50705d0 *tests/testthat/test_01_registerLevels.R 05bdef8eab30d119ac1a3db9d2e674b4 *tests/testthat/test_autodetectCpus.R cdd7754a0db6a793d21d2e083d6fad29 *tests/testthat/test_batchjobs.R ff689b014fb5b6b8e9dcedcfa286fa93 *tests/testthat/test_batchtools.R 3577fed061c70ba60ce83158914d0811 *tests/testthat/test_local.R 724945fa813b4653cb8337fd7f5fb965 *tests/testthat/test_mpi.R 6695d49cec01a79d756f2fec4fb2241f *tests/testthat/test_multicore.R 1d213a6b60d5acd12cea391368b90db5 *tests/testthat/test_parallelApply.R c3986eafcd888ee8b469db2f4163aeb3 *tests/testthat/test_parallelGetOptions.R c3a62c0d71fe4faed306148c78f4bc4d *tests/testthat/test_parallelLibrary.R ade4e16fc924238e6b7706e8b0353e69 *tests/testthat/test_parallelStart.R a160f7c10f8a9f50ec9f413fb5991098 *tests/testthat/test_reproducibility.R e57f9f3c17ceaa9c490122becf7787de *tests/testthat/test_socket.R 17e24e931270cf98068ec56dd25c01f0 *tests/testthat/test_stopWithJobErrorMessages.R parallelMap/inst/0000755000176200001440000000000014066072250013463 5ustar liggesusersparallelMap/inst/test_source_file.R0000644000176200001440000000001214066072250017135 0ustar liggesusersxxx = 123