future/ 0000755 0001762 0000144 00000000000 13240051214 011554 5 ustar ligges users future/inst/ 0000755 0001762 0000144 00000000000 13237670253 012551 5 ustar ligges users future/inst/vignettes-static/ 0000755 0001762 0000144 00000000000 13147540314 016040 5 ustar ligges users future/inst/vignettes-static/future-1-overview.md.rsp.rsp 0000644 0001762 0000144 00000105065 13147540314 023313 0 ustar ligges users <%---------------------------------------------------------------------
This *.md.rsp.rsp file is used to generate the *.md.rsp that
is the copied to vignettes/. The latter contains no dynamic
code. The reason for this is that this RSP file uses
multisession futures and for some unknown reason those gives
errors during R CMD build on Windows. Everywhere else they
work including when building this RSP manually. /HB 2016-04-12
---------------------------------------------------------------------%>
<%%@meta language="R-vignette" content="--------------------------------
%\VignetteIndexEntry{A Future for R: A Comprehensive Overview}
%\VignetteAuthor{Henrik Bengtsson}
%\VignetteKeyword{R}
%\VignetteKeyword{package}
%\VignetteKeyword{vignette}
%\VignetteKeyword{future}
%\VignetteKeyword{promise}
%\VignetteKeyword{lazy evaluation}
%\VignetteKeyword{synchronous}
%\VignetteKeyword{asynchronous}
%\VignetteKeyword{parallel}
%\VignetteKeyword{cluster}
%\VignetteEngine{R.rsp::rsp}
%\VignetteTangle{FALSE}
Do not edit the *.md.rsp file. Instead edit the *.md.rsp.rsp (sic!)
file found under inst/vignettes-static/ of the source package.
--------------------------------------------------------------------"%%>
<%@meta language="R-vignette" content="--------------------------------
%\VignetteIndexEntry{A Future for R: A Comprehensive Overview}
%\VignetteAuthor{Henrik Bengtsson}
%\VignetteKeyword{R}
%\VignetteKeyword{package}
%\VignetteKeyword{vignette}
%\VignetteKeyword{future}
%\VignetteKeyword{promise}
%\VignetteKeyword{lazy evaluation}
%\VignetteKeyword{synchronous}
%\VignetteKeyword{asynchronous}
%\VignetteKeyword{parallel}
%\VignetteKeyword{cluster}
%\VignetteEngine{R.rsp::rsp}
%\VignetteTangle{FALSE}
--------------------------------------------------------------------"%>
<%
R.utils::use("R.utils")
use("future")
`%<-%` <- future::`%<-%`
options("withCapture/newline" = FALSE)
options(mc.cores = 2L)
%>
# <%@meta name="title"%>
## Introduction
The purpose of the [future] package is to provide a very simple and uniform way of evaluating R expressions asynchronously using various resources available to the user.
In programming, a _future_ is an abstraction for a _value_ that may be available at some point in the future. The state of a future can either be _unresolved_ or _resolved_. As soon as it is resolved, the value is available instantaneously. If the value is queried while the future is still unresolved, the current process is _blocked_ until the future is resolved. It is possible to check whether a future is resolved or not without blocking. Exactly how and when futures are resolved depends on what strategy is used to evaluate them. For instance, a future can be resolved using a sequential strategy, which means it is resolved in the current R session. Other strategies may be to resolve futures asynchronously, for instance, by evaluating expressions in parallel on the current machine or concurrently on a compute cluster.
Here is an example illustrating how the basics of futures work. First, consider the following code snippet that uses plain R code:
```r
> v <- {
+ cat("Resolving...\n")
+ 3.14
+ }
Resolving...
> v
[1] 3.14
```
It works by assigning the value of an expression to variable `v` and we then print the value of `v`. Moreover, when the expression for `v` is evaluated we also print a message.
Here is the same code snippet modified to use futures instead:
```r
> library("future")
> v %<-% {
+ cat("Resolving...\n")
+ 3.14
+ }
Resolving...
> v
[1] 3.14
```
The difference is in how `v` is constructed; with plain R we use `<-` whereas with futures we use `%<-%`.
So why are futures useful? Because we can choose to evaluate the future expression in a separate R process asynchronously by simply switching settings as:
```r
> library("future")
> plan(multiprocess)
> v %<-% {
+ cat("Resolving...\n")
+ 3.14
+ }
> v
[1] 3.14
```
With asynchronous futures, the current/main R process does _not_ block, which means it is available for further processing while the futures are being resolved
in separates processes running in the background. In other words, futures provide a simple but yet powerful construct for parallel and / or distributed processing in R.
Now, if you cannot be bothered to read all the nitty-gritty details about futures, but just want to try them out, then skip to the end to play with the Mandelbrot demo using both parallel and non-parallel evaluation.
## Implicit or Explicit Futures
Futures can be created either _implicitly_ or _explicitly_. In the introductory example above we used _implicit futures_ created via the `v %<-% { expr }` construct. An alternative is _explicit futures_ using the `f <- future({ expr })` and `v <- value(f)` constructs. With these, our example could alternatively be written as:
```r
> library("future")
> f <- future({
+ cat("Resolving...\n")
+ 3.14
+ })
Resolving...
> v <- value(f)
> v
[1] 3.14
```
Either style of future construct works equally(*) well. The implicit style is most similar to how regular R code is written. In principle, all you have to do is to replace `<-` with a `%<-%` to turn the assignment into a future assignment. On the other hand, this simplicity can also be deceiving, particularly when asynchronous futures are being used. In contrast, the explicit style makes it much clearer that futures are being used, which lowers the risk for mistakes and better communicates the design to others reading your code.
(*) There are cases where `%<-%` cannot be used without some (small) modifications. We will return to this in Section 'Constraints when using Implicit Futures' near the end of this document.
To summarize, for explicit futures, we use:
* `f <- future({ expr })` - creates a future
* `v <- value(f)` - gets the value of the future (blocks if not yet resolved)
For implicit futures, we use:
* `v %<-% { expr }` - creates a future and a promise to its value
To keep it simple, we will use the implicit style in the rest of this document, but everything discussed will also apply to explicit futures.
## Controlling How Futures are Resolved
The future package implements the following types of futures:
| Name | OSes | Description
|:----------------|:------------|:-----------------------------------------------------
| _synchronous:_ | | _non-parallel:_
| `sequential` | all | sequentially and in the current R process
| `transparent` | all | as sequential w/ early signaling and w/out local (for debugging)
| _asynchronous:_ | | _parallel_:
| `multiprocess` | all | multicore iff supported, otherwise multisession
| `multisession` | all | background R sessions (on current machine)
| `multicore` | not Windows | forked R processes (on current machine)
| `cluster` | all | external R sessions on current, local, and/or remote machines
| `remote` | all | Simple access to remote R sessions
_Note_: Prior to future 1.3.0, `eager` and `lazy` were also options, but has since been deprecated and are both defunct as of 1.6.0. The reason for this is that whether a future should be resolved by lazy evaluation or not has to, in some cases, be in the control of the developer, and if the end user would be able change that, the code may not function as intended.
The future package is designed such that support for additional strategies can be implemented as well. For instance, the [future.batchtools] package provides futures for all types of _cluster functions_ ("backends") that the [batchtools] package supports. Specifically, futures for evaluating R expressions via job schedulers such as Slurm, TORQUE/PBS, Oracle/Sun Grid Engine (SGE) and Load Sharing Facility (LSF) are also available. (_Comment_: The [future.BatchJobs] package provides analogue backends based on the [BatchJobs] package; however the BatchJobs developers have deprecated it in favor of batchtools.)
By default, future expressions are evaluated eagerly (= instantaneously) and synchronously (in the current R session). This evaluation strategy is referred to as "sequential". In this section, we will go through each of these strategies and discuss what they have in common and how they differ.
### Consistent Behavior Across Futures
Before going through each of the different future strategies, it is probably helpful to clarify the objectives the Future API (as defined by the future package). When programming with futures, it should not really matter what future strategy is used for executing code. This is because we cannot really know what computational resources the user has access to so the choice of evaluation strategy should be in the hands of the user and not the developer. In other words, the code should not make any assumptions on the type of futures used, e.g. synchronous or asynchronous.
One of the designs of the Future API was to encapsulate any differences such that all types of futures will appear to work the same. This despite expressions may be evaluated locally in the current R session or across the world in remote R sessions. Another obvious advantage of having a consistent API and behavior among different types of futures is that it helps while prototyping. Typically one would use sequential evaluation while building up a script and, later, when the script is fully developed, one may turn on asynchronous processing.
Because of this, the defaults of the different strategies are such that the results and side effects of evaluating a future expression are as similar as possible. More specifically, the following is true for all futures:
* All _evaluation is done in a local environment_ (i.e. `local({ expr })`) so that assignments do not affect the calling environment. This is natural when evaluating in an external R process, but is also enforced when evaluating in the current R session.
* When a future is constructed, _global variables are identified_. For asynchronous evaluation, globals are exported to the R process/session that will be evaluating the future expression. For sequential futures with lazy evaluation (`lazy = TRUE`), globals are "frozen" (cloned to a local environment of the future). Also, in order to protect against exporting too large objects by mistake, there is a built-in assertion that the total size of all globals is less than a given threshold (controllable via an option, cf. `help("future.options")`). If the threshold is exceeded, an informative error is thrown.
* Future _expressions are only evaluated once_. As soon as the value (or an error) has been collected it will be available for all succeeding requests.
Here is an example illustrating that all assignments are done to a local environment:
```r
<%=withCapture({
plan(sequential)
a <- 1
x %<-% {
a <- 2
2 * a
}
x
a
})%>
```
<%---
And here is an example illustrating that globals are validated already when the future is created:
```r
> rm(b)
> x %<-% { 2 * b }
Error in globalsOf(expr, envir = envir, substitute = FALSE, tweak = tweak, :
Identified a global object via static code inspection ({; 2 * b; }), but
failed to locate the corresponding object in the relevant environments: 'b'
```
We will return to global variables and global functions in Section 'Globals' near the end of this document.
---%>
Now we are ready to explore the different future strategies.
### Synchronous Futures
Synchronous futures are resolved one after another and most commonly by the R process that creates them. When a synchronous future is being resolved it blocks the main process until resolved. There are two types of synchronous futures in the future package, _sequential_ and _transparent_. (In future 1.2.0 and before, there was also _lazy_ futures, which has now been deprecated in favor of `f <- future(..., lazy = TRUE)` and `v %<-% { ... } %lazy% TRUE`.)
#### Sequential Futures
Sequential futures are the default unless otherwise specified. They were designed to behave as similar as possible to regular R evaluation while still fulfilling the Future API and its behaviors. Here is an example illustrating their properties:
```r
<%=withCapture({
plan(sequential)
<%@include file="incl/future-1-overview-example2.R"%>
})%>
```
Since eager sequential evaluation is taking place, each of the three futures is resolved instantaneously in the moment it is created. Note also how `pid` in the calling environment, which was assigned the process ID of the current process, is neither overwritten nor removed. This is because futures are evaluated in a local environment. Since synchronous (uni-)processing is used, future `b` is resolved by the main R process (still in a local environment), which is why the value of `b` and `pid` are the same.
<%
## Sanity checks
stopifnot(b == pid)
%>
#### Transparent Futures
For troubleshooting, _transparent_ futures can be used by specifying `plan(transparent)`. A transparent future is technically a sequential future with instant signaling of conditions (including errors and warnings) and where evaluation, and therefore also assignments, take place in the calling environment. Transparent futures are particularly useful for troubleshooting errors that are otherwise hard to narrow down.
### Asynchronous Futures
Next, we will turn to asynchronous futures, which are futures that are resolved in the background. By design, these futures are non-blocking, that is, after being created the calling process is available for other tasks including creating additional futures. It is only when the calling process tries to access the value of a future that is not yet resolved, or trying to create another asynchronous future when all available R processes are busy serving other futures, that it blocks.
#### Multisession Futures
We start with multisession futures because they are supported by all operating systems. A multisession future is evaluated in a background R session running on the same machine as the calling R process. Here is our example with multisession evaluation:
```r
<%=bfr <- withCapture({
plan(multisession)
<%@include file="incl/future-1-overview-example2.R"%>
})%>
```
The first thing we observe is that the values of `a`, `c` and `pid` are the same as previously. However, we notice that `b` is different from before. This is because future `b` is evaluated in a different R process and therefore it returns a different process ID. Another difference is that the messages, generated by `cat()`, are no longer displayed. This is because they are outputted to the background sessions and not the calling session.
<%
## Sanity checks
if (availableCores() == 1L) {
stopifnot(b == pid)
} else {
stopifnot(b != pid)
}
%>
When multisession evaluation is used, the package launches a set of R sessions in the background that will serve multisession futures by evaluating their expressions as they are created. If all background sessions are busy serving other futures, the creation of the next multisession future is _blocked_ until a background session becomes available again. The total number of background processes launched is decided by the value of `availableCores()`, e.g.
```r
<%=withCapture({
availableCores()
})%>
```
This particular result tells us that the `mc.cores` option was set such that we are allowed to use in total <%= availableCores() %> processes including the main process. In other words, with these settings, there will be <%= availableCores() %> background processes serving the multisession futures. The `availableCores()` is also agile to different options and system environment variables. For instance, if compute cluster schedulers are used (e.g. TORQUE/PBS and Slurm), they set specific environment variable specifying the number of cores that was allotted to any given job; `availableCores()` acknowledges these as well. If nothing else is specified, all available cores on the machine will be utilized, cf. `parallel::detectCores()`. For more details, please see `help("availableCores", package = "future")`.
#### Multicore Futures
On operating systems where R supports _forking_ of processes, which is basically all operating system except Windows, an alternative to spawning R sessions in the background is to fork the existing R process. Forking an R process is considered faster than working with a separate R session running in the background. One reason is that the overhead of exporting large globals to the background session can be greater than when forking is used.
To use multicore futures, we specify:
```r
plan(multicore)
```
<%---
```r
<%=if (supportsMulticore()) {
withCapture({
plan(multicore)
<%@include file="incl/future-1-overview-example2.R"%>
})
} else {
## If this vignette is compiled on a system not supporting
## multicore processing, let's fake the future based on
## the output of the multisession example
bfr <- gsub("multisession", "multicore", bfr)
bfr <- gsub(b, b + 5L, bfr)
bfr
}%>
```
---%>
The only real different between using multicore and multisession futures is that any output written (to standard output or standard error) by a multicore process is instantaneously outputted in calling process. Other than this, the behavior of using multicore evaluation is very similar to that of using multisession evaluation.
Just like for multisession futures, the maximum number of parallel processes running will be decided by `availableCores()`, since in both cases the evaluation is done on the local machine.
#### Multiprocess Futures
Sometimes we do not know whether multicore futures are supported or not, but it might still be that we would like to write platform-independent scripts or instructions that work everywhere. In such cases we can specify that we want to use "multiprocess" futures as in:
```r
plan(multiprocess)
```
A multiprocess future is not a formal class of futures by itself, but rather a convenient alias for either of the two. When this is specified, multisession evaluation will be used unless multicore evaluation is supported.
#### Cluster Futures
Cluster futures evaluate expressions on an ad-hoc cluster (as implemented by the parallel package). For instance, assume you have access to three nodes `n1`, `n2` and `n3`, you can then use these for asynchronous evaluation as:
```r
<%=
## For the vignette we cannot assume nodes n1, n2 and n3 exist.
## The second best we can do is to use the local machine.
hosts <- c("localhost", "localhost", "localhost")
bfr <- withCapture({
plan(cluster, workers = hosts)
<%@include file="incl/future-1-overview-example2.R"%>
})
bfr <- gsub('hosts', 'c("n1", "n2", "n3")', bfr)
bfr
%>
```
Just as for most other asynchronous evaluation strategies, the output from `cat()` is not displayed on the current/calling machine.
<%
## Sanity checks
stopifnot(b != pid)
%>
Any types of clusters that `parallel::makeCluster()` creates can be used for cluster futures. For instance, the above cluster can be explicitly set up as:
```r
cl <- parallel::makeCluster(c("n1", "n2", "n3"))
plan(cluster, workers = cl)
```
Also, it is considered good style to shut down cluster `cl` when it is no longer needed, that is, calling `parallel::stopCluster(cl)`. However, it will shut itself down if the main process is terminated. For more information on how to set up and manage such clusters, see `help("makeCluster", package = "parallel")`.
Clusters created implicitly using `plan(cluster, workers = hosts)` where `hosts` is a character vector will also be shut down when the main R session terminates, or when the future strategy is changed, e.g. by calling `plan(sequential)`.
Note that with automatic authentication setup (e.g. SSH key pairs), there is nothing preventing us from using the same approach for using a cluster of remote machines.
<%---
### Different Strategies for Different Futures
Sometimes one may want to use an alternative evaluation strategy for a specific future. Although one can use `old <- plan(new)` and afterward `plan(old)` to temporarily switch strategies, a simpler approach is to use the `%plan%` operator, e.g.
```r
<%=withCapture({
plan(sequential)
pid <- Sys.getpid()
pid
a %<-% { Sys.getpid() }
b %<-% { Sys.getpid() } %plan% multiprocess
c %<-% { Sys.getpid() } %plan% multiprocess
a
b
c
})%>
```
As seen by the different process IDs, future `a` is evaluated sequentially using the same process as the calling environment whereas the other two are evaluated using multiprocess futures.
<%
## Sanity checks
stopifnot(a == pid)
if (availableCores() == 1L) {
stopifnot(b == pid, c == pid, c == b)
} else if (availableCores() == 2L && !supportsMulticore()) {
stopifnot(b != pid, c != pid, c == b)
} else {
stopifnot(b != pid, c != pid, c != b)
}
%>
However, using different plans to individual futures this way has the drawback of hard coding the evaluation strategy. Doing so may prevent some users from using your script or your package, because they do not have the sufficient resources. It may also prevent users with a lot of resources from utilizing those because you assumed a less-powerful set of hardware. Because of this, we recommend against the use of `%plan%` other than for interactive prototyping.
---%>
### Nested Futures and Evaluation Topologies
This far we have discussed what can be referred to as "flat topology" of futures, that is, all futures are created in and assigned to the same environment. However, there is nothing stopping us from using a "nested topology" of futures, where one set of futures may, in turn, create another set of futures internally and so on.
For instance, here is an example of two "top" futures (`a` and `b`) that uses multiprocess evaluation and where the second future (`b`) in turn uses two internal futures:
```r
<%=withCapture({
plan(multiprocess)
<%@include file="incl/future-1-overview-example3.R"%>
pid
a
b
})%>
```
By inspection the process IDs, we see that there are in total three different processes involved for resolving the futures. There is the main R process (pid <%= pid %>), and there are the two processes used by `a` (pid <%= a %>) and `b` (pid <%= b[1] %>). However, the two futures (`b1` and `b2`) that is nested by `b` are evaluated by the same R process as `b`. This is because nested futures use sequential evaluation unless otherwise specified. There are a few reasons for this, but the main reason is that it protects us from spawning off a large number of background processes by mistake, e.g. via recursive calls.
<%
## Sanity checks
if (availableCores() == 1L) {
stopifnot(a == pid, all(b == pid))
} else if (availableCores() == 2L && !supportsMulticore()) {
stopifnot(a != pid, all(b != pid), !all(b != a), all(b == b[1]))
} else {
stopifnot(a != pid, all(b != pid), all(b != a), all(b == b[1]))
}
%>
To specify a different type of _evaluation topology_, other than the first level of futures being resolved by multiprocess evaluation and the second level by sequential evaluation, we can provide a list of evaluation strategies to `plan()`. First, the same evaluation strategies as above can be explicitly specified as:
```r
plan(list(multiprocess, sequential))
```
We would actually get the same behavior if we try with multiple levels of multiprocess evaluations;
```r
<%=withCapture({
plan(list(multiprocess, multiprocess))
})%>
[...]
<%
<%@include file="incl/future-1-overview-example3.R"%>
%>
<%=withCapture({
pid
a
b
})%>
```
The reason for this is, also here, to protect us from launching more processes than what the machine can support. Internally, this is done by setting `mc.cores = 1` such that functions like `parallel::mclapply()` will fall back to run sequentially. This is the case for both multisession and multicore evaluation.
<%
## Sanity checks
if (availableCores() == 1L) {
stopifnot(a == pid, all(b == pid))
} else if (availableCores() == 2L && !supportsMulticore()) {
stopifnot(a != pid, all(b != pid), !all(b != a), all(b == b[1]))
} else {
stopifnot(a != pid, all(b != pid), all(b != a), all(b == b[1]))
}
%>
Continuing, if we start off by sequential evaluation and then use multiprocess evaluation for any nested futures, we get:
```r
<%=withCapture({
plan(list(sequential, multiprocess))
})%>
[...]
<%
<%@include file="incl/future-1-overview-example3.R"%>
%>
<%=withCapture({
pid
a
b
})%>
```
which clearly show that `a` and `b` are resolved in the calling process (pid <%= pid %>) whereas the two nested futures (`b1` and `b2`) are resolved in two separate R processes (pids <%= b[2] %> and <%= b[3] %>).
<%
## Sanity checks
stopifnot(a == pid, b[1] == pid)
if (availableCores() == 1L) {
stopifnot(all(b == b[1]))
} else if (availableCores() == 2L && !supportsMulticore()) {
stopifnot(all(b[-1] != b[1]), b[2] == b[3])
} else {
stopifnot(all(b[-1] != b[1]), b[2] != b[3])
}
%>
Having said this, it is indeed possible to use nested multiprocess evaluation strategies, if we explicitly specify (read _force_) the number of cores available at each level. In order to do this we need to "tweak" the default settings, which can be done as follows:
```r
<%=withCapture({
plan(list(
tweak(multiprocess, workers = 2L),
tweak(multiprocess, workers = 2L)
))
})%>
[...]
<%
<%@include file="incl/future-1-overview-example3.R"%>
%>
<%=withCapture({
pid
a
b
})%>
```
First, we see that both `a` and `b` are resolved in different processes (pids <%= a %> and <%= b[1] %>) than the calling process (pid <%= pid %>). Second, the two nested futures (`b1` and `b2`) are resolved in yet two other R processes (pids <%= b[2] %> and <%= b[3] %>).
<%
## Sanity checks
stopifnot(a != pid, b[1] != pid)
if (supportsMulticore()) {
} else {
stopifnot(b[2] != b[1], b[3] != b[1])
}
%>
For more details on working with nested futures and different evaluation strategies at each level, see Vignette '[Futures in R: Future Topologies]'.
### Checking A Future without Blocking
It is possible to check whether a future has been resolved or not without blocking. This can be done using the `resolved(f)` function, which takes an explicit future `f` as input. If we work with implicit futures (as in all the examples above), we can use the `f <- futureOf(a)` function to retrieve the explicit future from an implicit one. For example,
```r
<%=withCapture({
plan(multiprocess)
a %<-% {
cat("Resolving 'a' ...")
Sys.sleep(2)
cat("done\n")
Sys.getpid()
}
cat("Waiting for 'a' to be resolved ...\n")
f <- futureOf(a)
count <- 1
while(!resolved(f)) {
cat(count, "\n")
Sys.sleep(0.2)
count <- count + 1
}
cat("Waiting for 'a' to be resolved ... DONE\n")
a
})%>
```
## Failed Futures
Sometimes the future is not what you expected. If an error occurs while evaluating a future, the error is propagated and thrown as an error in the calling environment _when the future value is requested_. For example, if we use lazy evaluation on a future that generates an error, we might see something like
```r
<%=withCapture({
plan(sequential)
a %<-% {
cat("Resolving 'a' ...\n")
stop("Whoops!")
42
} %lazy% TRUE
cat("Everything is still ok although we have created a future that will fail.\n")
})%>
<% captureOutput(try(resolve(a), silent = TRUE)) %>
> a
Resolving 'a' ...
Error in eval(expr, envir, enclos) : Whoops!
```
The error is thrown each time the value is requested, that is, if we try to get the value again will generate the same error:
```r
> a
Error in eval(expr, envir, enclos) : Whoops!
In addition: Warning message:
restarting interrupted promise evaluation
```
To see the list of calls (evaluated expressions) that lead up to the error, we can use the `backtrace()` function(*) on the future, i.e.
```r
<%=withCapture({
backtrace(a)
})%>
```
(*) The commonly used `traceback()` does not provide relevant information in the context of futures.
## Globals
Whenever an R expression is to be evaluated asynchronously (in parallel) or sequentially via lazy evaluation, global (aka "free") objects have to be identified and passed to the evaluator. They need to be passed exactly as they were at the time the future was created, because, for lazy evaluation, globals may otherwise change between when it is created and when it is resolved. For asynchronous processing, the reason globals need to be identified is so that they can be exported to the process that evaluates the future.
The future package tries to automate these tasks as far as possible. It does this with help of the [globals] package, which uses static-code inspection to identify global variables. If a global variable is identified, it is captured and made available to the evaluating process.
<%---
If it identifies a symbol that it believes is a global object, but it fails to locate it in the calling environment (or any the environment accessible from that one), an error is thrown immediately. This minimizing the risk for run-time errors occurring later (sometimes much later) and in a different process (possible in a remote R session), which otherwise may be hard to troubleshoot. For instance,
```r
> rm(a)
> x <- 5.0
> y %<-% { a * x }
Error in globalsOf(expr, envir = envir, substitute = FALSE, tweak = tweak, :
Identified a global object via static code inspection ({; a * x; }), but
failed to locate the corresponding object in the relevant environments: 'a'
> a <- 1.8
> y %<-% { a * x }
> y
[1] 9
```
---%>
Moreover, if a global is defined in a package, then that global is not exported. Instead, it is made sure that the corresponding package is attached when the future is evaluated. This not only better reflects the setup of the main R session, but it also minimizes the need for exporting globals, which saves not only memory but also time and bandwidth, especially when using remote compute nodes.
<%---
As mentioned previously, for consistency across evaluation strategies, all types of futures validate globals upon creation. This is also true for cases where it would not be necessary, e.g. for eager evaluation of multicore futures (which forks the calling process "as-is"). However, in order to make it as easy as possible to switch between strategies without being surprised by slightly different behaviors, the Future API is designed to check for globals the same way regardless of strategy.
---%>
<%---
Having said this, it is possible to disable validation of globals by setting `globals = FALSE`. This could make sense if one know for sure that only sequential or multicore futures will be used. This argument can be tweaked as `plan(tweak(sequential, globals=FALSE))` and `plan(tweak(multicore, globals=FALSE))`. However, it is strongly advised not to do this. Instead, as a best practice, it is always good if the code/script works with any type of futures.
---%>
Finally, it should be clarified that identifying globals from static code inspection alone is a challenging problem. There will always be corner cases where automatic identification of globals fails so that either false globals are identified (less of a concern) or some of the true globals are missing (which will result in a run-time error or possibly the wrong results). Vignette '[Futures in R: Common Issues with Solutions]' provides examples of common cases and explains how to avoid them as well as how to help the package to identify globals or to ignore falsely identified globals. If that does not suffice, it is always possible to manually specify the global variables by their names (e.g. `globals = c("a", "slow_sum")`) or as name-value pairs (e.g. `globals = list(a = 42, slow_sum = my_sum)`).
## Constraints when using Implicit Futures
There is one limitation with implicit futures that does not exist for explicit ones. Because an explicit future is just like any other object in R it can be assigned anywhere/to anything. For instance, we can create several of them in a loop and assign them to a list, e.g.
```r
<%=withCapture({
plan(multiprocess)
f <- list()
for (ii in 1:3) {
f[[ii]] <- future({ Sys.getpid() })
}
v <- lapply(f, FUN = value)
str(v)
})%>
```
This is _not_ possible to do when using implicit futures. This is because the `%<-%` assignment operator _cannot_ be used in all cases where the regular `<-` assignment operator can be used. It can only be used to assign future values to _environments_ (including the calling environment) much like how `assign(name, value, envir)` works. However, we can assign implicit futures to environments using _named indices_, e.g.
```r
<%=withCapture({
plan(multiprocess)
v <- new.env()
for (name in c("a", "b", "c")) {
v[[name]] %<-% { Sys.getpid() }
}
v <- as.list(v)
str(v)
})%>
```
Here `as.list(v)` blocks until all futures in the environment `v` have been resolved. Then their values are collected and returned as a regular list.
If _numeric indices_ are required, then _list environments_ can be used. List environments, which are implemented by the [listenv] package, are regular environments with customized subsetting operators making it possible to index them much like how lists can be indexed. By using list environments where we otherwise would use lists, we can also assign implicit futures to list-like objects using numeric indices. For example,
```r
<%=withCapture({
library("listenv")
plan(multiprocess)
v <- listenv()
for (ii in 1:3) {
v[[ii]] %<-% { Sys.getpid() }
}
v <- as.list(v)
str(v)
})%>
```
As previously, `as.list(v)` blocks until all futures are resolved.
## Demos
To see a live illustration how different types of futures are evaluated, run the Mandelbrot demo of this package. First, try with the sequential evaluation,
```r
library("future")
plan(sequential)
demo("mandelbrot", package = "future", ask = FALSE)
```
which resembles how the script would run if futures were not used. Then, try multiprocess evaluation, which calculates the different Mandelbrot planes using parallel R processes running in the background. Try,
```r
plan(multiprocess)
demo("mandelbrot", package = "future", ask = FALSE)
```
Finally, if you have access to multiple machines you can try to set up a cluster of workers and use them, e.g.
```r
plan(cluster, workers = c("n2", "n5", "n6", "n6", "n9"))
demo("mandelbrot", package = "future", ask = FALSE)
```
## Contributing
The goal of this package is to provide a standardized and unified API for using futures in R. What you are seeing right now is an early but sincere attempt to achieve this goal. If you have comments or ideas on how to improve the 'future' package, I would love to hear about them. The preferred way to get in touch is via the [GitHub repository](https://github.com/HenrikBengtsson/future/), where you also find the latest source code. I am also open to contributions and collaborations of any kind.
[BatchJobs]: https://cran.r-project.org/package=BatchJobs
[batchtools]: https://cran.r-project.org/package=batchtools
[future]: https://cran.r-project.org/package=future
[future.BatchJobs]: https://cran.r-project.org/package=future.BatchJobs
[future.batchtools]: https://cran.r-project.org/package=future.batchtools
[globals]: https://cran.r-project.org/package=globals
[listenv]: https://cran.r-project.org/package=listenv
[Futures in R: Common Issues with Solutions]: future-2-issues.html
[Futures in R: Future Topologies]: future-3-topologies.html
---
Copyright Henrik Bengtsson, 2015-2017
future/inst/vignettes-static/incl/ 0000755 0001762 0000144 00000000000 13111756521 016765 5 ustar ligges users future/inst/vignettes-static/incl/future-1-overview-example3.R 0000644 0001762 0000144 00000000453 13111756521 024142 0 ustar ligges users pid <- Sys.getpid()
a %<-% {
cat("Resolving 'a' ...\n")
Sys.getpid()
}
b %<-% {
cat("Resolving 'b' ...\n")
b1 %<-% {
cat("Resolving 'b1' ...\n")
Sys.getpid()
}
b2 %<-% {
cat("Resolving 'b2' ...\n")
Sys.getpid()
}
c(b.pid = Sys.getpid(), b1.pid = b1, b2.pid = b2)
}
future/inst/vignettes-static/incl/future-1-overview-example2.R 0000644 0001762 0000144 00000000330 12736356726 024152 0 ustar ligges users pid <- Sys.getpid()
pid
a %<-% {
pid <- Sys.getpid()
cat("Resolving 'a' ...\n")
3.14
}
b %<-% {
rm(pid)
cat("Resolving 'b' ...\n")
Sys.getpid()
}
c %<-% {
cat("Resolving 'c' ...\n")
2 * a
}
b
c
a
pid
future/inst/doc/ 0000755 0001762 0000144 00000000000 13237670253 013316 5 ustar ligges users future/inst/doc/future-4-startup.html 0000644 0001762 0000144 00000040356 13237670252 017366 0 ustar ligges users
A Future for R: Controlling Default Future Strategy
A Future for R: Controlling Default Future Strategy
The default is to use synchronous futures, but this default can be overridden via R options, system environment variables and command-line options as explained below as well as in help("future.options", package = "future").
R options
The default strategy for resolving futures can be controlled via R option future.plan. For instance, if we add
options(future.plan = "multiprocess")
to our ~/.Rprofile startup script, the future package will resolve futures in parallel (asynchronously using all available cores), i.e.
Option future.plan is ignored if command-line option --parallel (-p) is specified.
Environment variables
An alternative to using options() for setting option future.plan is to specify system environment variable R_FUTURE_PLAN. If set, then the future package will set future.plan accordingly when loaded. For example,
Environment variable R_FUTURE_PLAN is ignored if either option future.plan or command-line option --parallel (-p) is specified.
Command-line options
When loaded, the future package checks for the command-line option --parallel=ncores (short -p ncores) and sets the future strategy (via option future.plan) and the number of available cores (via option mc.cores) accordingly. This provides a convenient mechanism for specifying parallel future processing from the command line. For example, if we start R with
$ R --quiet --args --parallel=2
then future will interpret this as we wish to resolve futures in parallel using 2 cores. More specifically, we get that
We can use this command-line option also with Rscript, which provides a convenient mechanism for launching future-enhanced R scripts such that they run in parallel, e.g.
$ Rscript analysis.R --parallel=4
This does, of course, require that the script uses futures and the future package.
If --parallel=1 is specified, or equivalently -p 1, then futures are resolved using a single process.
Specifying these command-line options override any other startup settings.
Copyright Henrik Bengtsson, 2015-2017
future/inst/doc/future-3-topologies.html 0000644 0001762 0000144 00000051440 13237670252 020043 0 ustar ligges users
A Future for R: Future Topologies
A Future for R: Future Topologies
Futures can be nested in R such that one future creates another set of futures and so on. This may, for instance, occur within nested for loops, e.g.
library("future")
library("listenv")
x <- listenv()
for (ii in 1:3) {
x[[ii]] %<-% {
y <- listenv()
for (jj in 1:3) {
y[[jj]] %<-% { ii + jj / 10 }
}
y
}
}
unlist(x)
## [1] 1.1 1.2 1.3 2.1 2.2 2.3 3.1 3.2 3.3
The default is to use synchronous futures unless otherwise specified, which is also true for nested futures. If we for instance specify, plan(multiprocess), the first layer of futures (x[[ii]] %<-% { expr }) will be processed asynchronously in background R processes, and the futures in the second layer of futures (y[[jj]] %<-% { expr }) will be processed synchronously in the separate background R processes. If we wish to be explicit about this, we can specify plan(list(multiprocess, sequential)).
Example: High-Throughput Sequencing
Consider a high-throughput sequencing (HT-Seq) project with 50 human DNA samples where we have one FASTQ file per sample containing the raw sequence reads as they come out of the sequencing machine. With this data, we wish to align each FASTQ to a reference genome such that we generate 24 individual BAM files per sample - one per chromosome.
Here is the layout of what such an analysis could look like in R using futures.
The default is to use synchronous futures, so without further specifications, the above will process each sample and each chromosome sequentially. Next, we will consider what can be done with the following two computer setups:
A single machine with 8 cores
A compute cluster with 3 machines each with 16 cores
One multi-core machine
With a single machine of 8 cores, we could choose to process multiple samples at the same time while processing chromosomes sequentially. In other words, we would like to evaluate the outer layer of futures using multiprocess futures and the inner ones as sequential futures. This can be specified as:
plan(list(multiprocess, sequential))
The internals for processing multiprocess future queries availableCores() to infer how many cores can be used simultaneously, so there is no need to explicitly specify that there are 8 cores available.
Comment: Since synchronous is the default future, we could skip trailing sequential futures in the setup, e.g. plan(list(multiprocess)) or just plan(multiprocess). However, it does not hurt to be explicit.
If we instead would like to process the sample sequentially and the chromosomes in parallel, we can use:
plan(list(sequential, multiprocess))
We could also process the data such that we allocate two cores for processing two samples in parallel each using four cores for processing four chromosomes in parallel:
With a compute cluster of 3 machines each with 16 cores, we can run up to 48 alignment processes in parallel. A natural setup is to have one machine process one sample in parallel. We could specify this as:
Comment: Multiprocess futures are agile to its environment, that is, they will query the machine they are running on to find out how many parallel processes it can run at the same time.
One possible downside to the above setup is that we might not utilize all available cores all the time. This is because the alignment of the shorter chromosomes will finish sooner than the longer ones, which means that we might at the end of each sample have only a few alignment processes running on each machine leaving the remaining cores idle/unused. An alternative set up is then to use the following setup:
This will cause up to 24 (= 3*8) samples to be processed in parallel each processing two chromosomes at the same time.
Example: A remote compute cluster
Imagine we have access to a remote compute cluster, with login node remote.server.org, and that the cluster has three nodes n1, n2, and n3. Also, let us assume we have already set up the cluster such that we can log in via public key authentication via SSH, i.e. when we do ssh remote.server.org authentication is done automatically.
With the above setup, we can use nested futures in our local R session to evaluate R expression on the remote compute cluster and its three nodes. Here is a proof of concept illustrating how the different nested futures are evaluated on different machines.
library("future")
library("listenv")
## Set up access to remote login node
login <- tweak(remote, workers = "remote.server.org")
plan(login)
## Set up cluster nodes on login node
nodes %<-% { .keepme <- parallel::makeCluster(c("n1", "n2", "n3")) }
## Specify future topology
## login node -> { cluster nodes } -> { multiple cores }
plan(list(
login,
tweak(cluster, workers = nodes),
multiprocess
))
## (a) This will be evaluated on the cluster login computer
x %<-% {
thost <- Sys.info()[["nodename"]]
tpid <- Sys.getpid()
y <- listenv()
for (task in 1:4) {
## (b) This will be evaluated on a compute node on the cluster
y[[task]] %<-% {
mhost <- Sys.info()[["nodename"]]
mpid <- Sys.getpid()
z <- listenv()
for (jj in 1:2) {
## (c) These will be evaluated in separate processes on the same compute node
z[[jj]] %<-% data.frame(task = task,
top.host = thost, top.pid = tpid,
mid.host = mhost, mid.pid = mpid,
host = Sys.info()[["nodename"]],
pid = Sys.getpid())
}
Reduce(rbind, z)
}
}
Reduce(rbind, y)
}
print(x)
## task top.host top.pid mid.host mid.pid host pid
## 1 1 login 391547 n1 391878 n1 393943
## 2 1 login 391547 n1 391878 n1 393951
## 3 2 login 391547 n2 392204 n2 393971
## 4 2 login 391547 n2 392204 n2 393978
## 5 3 login 391547 n3 392527 n3 394040
## 6 3 login 391547 n3 392527 n3 394048
## 7 4 login 391547 n1 391878 n1 393959
## 8 4 login 391547 n1 391878 n1 393966
Try the above x %<-% { ... } future with, say, plan(list(sequential, multiprocess)) and see what the output will be.
Copyright Henrik Bengtsson, 2015-2017
future/inst/doc/future-2-issues.md.rsp 0000644 0001762 0000144 00000035452 13237667011 017434 0 ustar ligges users <%@meta language="R-vignette" content="--------------------------------
%\VignetteIndexEntry{A Future for R: Common Issues with Solutions}
%\VignetteAuthor{Henrik Bengtsson}
%\VignetteKeyword{R}
%\VignetteKeyword{package}
%\VignetteKeyword{vignette}
%\VignetteKeyword{future}
%\VignetteKeyword{promise}
%\VignetteEngine{R.rsp::rsp}
%\VignetteTangle{FALSE}
--------------------------------------------------------------------"%>
# <%@meta name="title"%>
In the ideal case, all it takes to start using futures in R is to replace select standard assignments (`<-`) in your R code with future assignments (`%<-%`) and make sure the right-hand side (RHS) expressions are within curly brackets (`{ ... }`). Also, if you assign these to lists (e.g. in a for loop), you need to use a list environment (`listenv`) instead of a plain list.
However, as show below, there are few cases where you might run into some hurdles, but, as also shown, they are often easy to overcome. These are often related to global variables.
_If you identify other cases, please consider [reporting](https://github.com/HenrikBengtsson/future/issues/) them so they can be documented here and possibly even be fixed._
## Issues with globals and packages
### Missing globals (false negatives)
If a global variable is used in a future expression that _conditionally_ overrides this global variable with a local one, the future framework fails to identify the global variable and therefore fails to export it, resulting in a run-time error. For example, although this works:
```r
library(multisession)
reset <- TRUE
x <- 1
y %<-% { if (reset) x <- 0; x + 1 }
y
## [1] 1
```
the following does _not_ work:
```r
reset <- FALSE
x <- 1
y %<-% { if (reset) x <- 0; x + 1 }
y
## Error: object 'x' not found
```
_Comment:_ The goal is to in a future version of the package detect globals also in expression where the local-global state of a variable is only known at run time.
#### do.call() - function not found
When calling a function using `do.call()` make sure to specify the function as the object itself and not by name. This will help identify the function as a global object in the future expression. For instance, use
```r
do.call(file_ext, list("foo.txt"))
```
instead of
```r
do.call("file_ext", list("foo.txt"))
```
so that `file_ext()` is properly located and exported. Although you may not notice a difference when evaluating futures in the same R session, it may become a problem if you use a character string instead of a function object when futures are evaluated in external R sessions, such as on a cluster.
It may also become a problem with futures evaluated with lazy evaluation if the intended function is redefined after the future is resolved. For example,
```r
> library("future")
> library("listenv")
> library("tools")
> plan(sequential)
> pathnames <- c("foo.txt", "bar.png", "yoo.md")
> res <- listenv()
> for (ii in seq_along(pathnames)) {
+ res[[ii]] %<-% do.call("file_ext", list(pathnames[ii])) %lazy% TRUE
+ }
> file_ext <- function(...) "haha!"
> unlist(res)
[1] "haha!" "haha!" "haha!"
```
### Missing packages (false negatives)
Occassionally, the static-code inspection of the future expression fails to identify packages needed to evaluated the expression. This may occur when a expression uses S3 generic functions part of one package whereas the required S3 method is in another package. For example, in the below future generic function `[` is used on data.table object `DT`, which requires S3 method `[.data.table` from the data.table package. However, the future and global packages fail to identify data.table as a required package, which results in an evaluation error:
```r
> library("future")
> plan(multisession)
> library("data.table")
> DT <- data.table(a = LETTERS[1:3], b = 1:3)
> y %<-% DT[, sum(b)]
> y
Error: object 'b' not found
```
The above error occurs because, contrary to the master R process, the R worker that evaluated the future expression does not have data.table loaded. Instead the evaluation falls back to the `[.data.frame` method, which is not what we want.
Until the future framework manages to identify data.table as a required package (which is the goal), we can guide future by specifying additional packages needed:
```r
> y %<-% DT[, sum(b)] %packages% "data.table"
> y
[1] 6
```
or equivalently
```r
> f <- future(DT[, sum(b)], packages = "data.table")
> value(f)
[1] 6
```
## Non-exportable objects
Certain types of objects are tied to a given R session and cannot be passed along to another R process (a "worker"). For instance, if you create a file connection,
```r
> con <- file("output.log", open = "wb")
> cat("hello ", file = con)
> flush(con)
> readLines("output.log", warn = FALSE)
[1] "hello "
```
it will not work when used in another R process(*). If we try, the result is "unknown", e.g.
```r
> library("future")
> plan(multisession)
> f <- future({ cat("world!", file = con); flush(con) })
> value(f)
NULL
> close(con)
> readLines("output.log", warn = FALSE)
[1] "hello "
```
In other words, the output `"world!"` written by the R worker is completely lost.
The culprit here is that the connection uses a so called _external pointer_:
```r
> str(con)
Classes 'file', 'connection' atomic [1:1] 3
..- attr(*, "conn_id")=
```
which is bound to the main R process and makes no sense to the worker. Ideally, the R process of the worker would detect this and produce an informative error message, but as seen here, that does not always occur.
To help avoiding these mistakes, the future framework provides a mechanism (_in beta_) can help detect non-exportable objects by setting:
```r
> options(future.globals.onReference = "warning")
```
which will result in a warning in the above example:
```r
> f <- future({ cat("world!", file = con); flush(con) })
Warning in FALSE :
Detected a non-exportable reference ('externalptr') in one of the globals
('con' of class 'file') used in the future expression
```
To be more conservative, we can also tell it to produce an error:
```r
> options(future.globals.onReference = "error")
> f <- future({ cat("world!", file = con); flush(con) })
Error in FALSE :
Detected a non-exportable reference ('externalptr') in one of the globals
('con' of class 'file') used in the future expression
```
Another example is XML objects of the xml2 package, which may produce evaluation errors (or just invalid results depending on how they are used), e.g.
```r
> library("future")
> plan(multisession)
> library("xml2")
> xml <- read_xml("")
> f <- future(xml_children(xml))
> value(f)
Error: external pointer is not valid
```
The future framework can help detect this _before_ sending off the future to the worker;
```r
> options(future.globals.onReference = "error")
> f <- future(xml_children(xml))
Error in FALSE :
Detected a non-exportable reference ('externalptr') in one of the globals
('xml' of class 'xml_document') used in the future expression
```
A third example is Rcpp which allow us to easily create R functions that are implemented in C++, e.g.
```r
Rcpp::sourceCpp(code = "
#include
using namespace Rcpp;
// [[Rcpp::export]]
int my_length(NumericVector x) {
return x.size();
}
")
```
so that:
```
> x <- 1:10
> my_length(x)
[1] 10
```
However, since this function uses an external pointer internally, we cannot pass it to another R process:
```r
> library("future")
> plan(multisession)
> n %<-% my_length(x)
> n
Error: NULL value passed as symbol address
```
We can detect / protect against this using:
```r
> options(future.globals.onReference = "error")
> n %<-% my_length(x)
Error in FALSE :
Detected a non-exportable reference ('externalptr' of class 'NativeSymbol')
in one of the globals ('my_length' of class 'function') used in the future
expression
```
(*) The exception _may_ be if you use forked R processes, e.g. `plan(multicore)`.
## Trying to pass an unresolved future to another future
It is not possible for a future to resolve another one unless it was created by the future trying to resolve it. For instance, the following gives an error:
```r
> library("future")
> plan(multiprocess)
> f1 <- future({ Sys.getpid() })
> f2 <- future({ value(f1) })
> v1 <- value(f1)
[1] 7464
> v2 <- value(f2)
Error: Invalid usage of futures: A future whose value has not yet been collected
can only be queried by the R process (cdd013cb-e045-f4a5-3977-9f064c31f188; pid
1276 on MyMachine) that created it, not by any other R processes (5579f789-e7b6
-bace-c50d-6c7a23ddb5a3; pid 2352 on MyMachine): {; Sys.getpid(); }
```
This is because the main R process creates two futures, but then the second future tries to retrieve the value of the first one. This is an invalid request because the second future has no channel to communicate with the first future; it is only the process that created a future who can communicate with it(*).
Note that it is only _unresolved_ futures that cannot be queried this way. Thus, the solution to the above problem is to make sure all futures are resolved before they are passed to other futures, e.g.
```r
> f1 <- future({ Sys.getpid() })
> v1 <- value(f1)
> v1
[1] 7464
> f2 <- future({ value(f1) })
> v2 <- value(f2)
> v2
[1] 7464
```
This works because the value has already been collected and stored inside future `f1` before future `f2` is created. Since the value is already stored internally, `value(f1)` is readily available everywhere. Of course, instead of using `value(f1)` for the second future, it would be more readable and cleaner to simply use `v1`.
The above is typically not a problem when future assignments are used. For example:
```r
> v1 %<-% { Sys.getpid() })
> v2 %<-% { v1 }
> v1
[1] 2352
> v2
[1] 2352
```
The reason that this approach works out of the box is because in the second future assignment `v1` is identified as a global variable, which is retrieved. Up to this point, `v1` is a promise ("delayed assignment" in R), but when it is retrieved as a global variable its value is resolved and `v1` becomes a regular variable.
However, there are cases where future assignments can be passed via global variables without being resolved. This can happen if the future assignment is done to an element of an environment (including list environments). For instance,
```r
> library("listenv")
> x <- listenv()
> x$a %<-% { Sys.getpid() }
> x$b %<-% { x$a }
> x$a
[1] 2352
> x$b
Error: Invalid usage of futures: A future whose value has not yet been collected
can only be queried by the R process (cdd013cb-e045-f4a5-3977-9f064c31f188; pid
1276 on localhost) that created it, not by any other R processes (2ce86ccd-5854
-7a05-1373-e1b20022e4d8; pid 7464 on localhost): {; Sys.getpid(); }
```
As previously, this can be avoided by making sure `x$a` is resolved first, which can be one in various ways, e.g. `dummy <- x$a`, `resolve(x$a)` and `force(x$a)`.
_Footnote_: (*) Although sequential futures could be passed on to other futures part of the same R process and be resolved there because they share the same evaluation process, by definition of the Future API it is invalid to do so regardless of future type. This conservative approach is taken in order to make future expressions behave consistently regardless of the type of future used.
## Miscellaneous
### Clashes with other packages
Sometimes other packages have functions or operators with the same name as the future package, and if those packages are attached _after_ the future package, their objects will mask the ones of the future package. For instance, the igraph package also defines a `%<-%` operator which clashes with the one in future _if used at the prompt or in a script_ (it is not a problem inside package because there we explicitly import objects in a known order). Here is what we might get:
```r
> library("future")
> library("igraph")
Attaching package: 'igraph'
The following objects are masked from 'package:future':
%<-%, %->%
The following objects are masked from 'package:stats':
decompose, spectrum
The following object is masked from 'package:base':
union
> y %<-% { 42 }
Error in get(".igraph.from", parent.frame()) :
object '.igraph.from' not found
```
Here we get an error because `%<-%` is from igraph and not the future assignment operator as we wanted. This can be confirmed as:
```r
> environment(`%<-%`)
```
To avoid this problem, attach the two packages in opposite order such that future comes last and thereby overrides igraph, i.e.
```r
> library("igraph")
> library("future")
Attaching package: 'future'
The following objects are masked from 'package:igraph':
%<-%, %->%
> y %<-% { 42 }
> y
[1] 42
```
An alternative is to detach the future package and re-attach it, which will achieve the same thing:
```r
> detach("package:future")
> library("future")
```
Yet another alternative is to explicitly override the object by importing it to the global environment, e.g.
```r
> `%<-%` <- future::`%<-%`
> y %<-% { 42 }
> y
[1] 42
```
In this case, it does not matter in what order the packages are attached because we will always use the copy of `` future::`%<-%` ``.
### Syntax error: "non-numeric argument to binary operator"
The future assignment operator `%<-%` is a _binary infix operator_, which means it has higher precedence than most other binary operators but also higher than some of the unary operators in R. For instance, this explains why we get the following error:
```r
> x %<-% 2 * runif(1)
Error in x %<-% 2 * runif(1) : non-numeric argument to binary operator
```
What effectively is happening here is that because of the higher priority of `%<-%`, we first create a future `x %<-% 2` and then we try to multiply the future (not its value) with the value of `runif(1)` - which makes no sense. In order to properly assign the future variable, we need to put the future expression within curly brackets;
```r
> x %<-% { 2 * runif(1) }
> x
[1] 1.030209
```
Parentheses will also do. For details on precedence on operators in R, see Section 'Infix and prefix operators' in the 'R Language Definition' document.
### R CMD check NOTEs
The code inspection run by `R CMD check` will not recognize the future assignment operator `%<-%` as an assignment operator, which is not surprising because `%<-%` is technically an infix operator. This means that if you for instance use the following code in your package:
```r
foo <- function() {
b <- 3.14
a %<-% { b + 1 }
a
}
```
then `R CMD check` will produce a NOTE saying:
```sh
* checking R code for possible problems ... NOTE
foo: no visible binding for global variable 'a'
Undefined global functions or variables:
a
```
In order to avoid this, we can add a dummy assignment of the missing global at the top of the function, i.e.
```r
foo <- function() {
a <- NULL ## To please R CMD check
b <- 3.14
a %<-% { b + 1 }
a
}
```
[future]: https://cran.r-project.org/package=future
[globals]: https://cran.r-project.org/package=globals
[listenv]: https://cran.r-project.org/package=listenv
---
Copyright Henrik Bengtsson, 2015-2018
future/inst/doc/future-1-overview.md.rsp 0000644 0001762 0000144 00000072313 13237667011 017763 0 ustar ligges users <%@meta language="R-vignette" content="--------------------------------
%\VignetteIndexEntry{A Future for R: A Comprehensive Overview}
%\VignetteAuthor{Henrik Bengtsson}
%\VignetteKeyword{R}
%\VignetteKeyword{package}
%\VignetteKeyword{vignette}
%\VignetteKeyword{future}
%\VignetteKeyword{promise}
%\VignetteKeyword{lazy evaluation}
%\VignetteKeyword{synchronous}
%\VignetteKeyword{asynchronous}
%\VignetteKeyword{parallel}
%\VignetteKeyword{cluster}
%\VignetteEngine{R.rsp::rsp}
%\VignetteTangle{FALSE}
Do not edit the *.md.rsp file. Instead edit the *.md.rsp.rsp (sic!)
file found under inst/vignettes-static/ of the source package.
--------------------------------------------------------------------"%>
# A Future for R: A Comprehensive Overview
## Introduction
The purpose of the [future] package is to provide a very simple and uniform way of evaluating R expressions asynchronously using various resources available to the user.
In programming, a _future_ is an abstraction for a _value_ that may be available at some point in the future. The state of a future can either be _unresolved_ or _resolved_. As soon as it is resolved, the value is available instantaneously. If the value is queried while the future is still unresolved, the current process is _blocked_ until the future is resolved. It is possible to check whether a future is resolved or not without blocking. Exactly how and when futures are resolved depends on what strategy is used to evaluate them. For instance, a future can be resolved using a sequential strategy, which means it is resolved in the current R session. Other strategies may be to resolve futures asynchronously, for instance, by evaluating expressions in parallel on the current machine or concurrently on a compute cluster.
Here is an example illustrating how the basics of futures work. First, consider the following code snippet that uses plain R code:
```r
> v <- {
+ cat("Resolving...\n")
+ 3.14
+ }
Resolving...
> v
[1] 3.14
```
It works by assigning the value of an expression to variable `v` and we then print the value of `v`. Moreover, when the expression for `v` is evaluated we also print a message.
Here is the same code snippet modified to use futures instead:
```r
> library("future")
> v %<-% {
+ cat("Resolving...\n")
+ 3.14
+ }
Resolving...
> v
[1] 3.14
```
The difference is in how `v` is constructed; with plain R we use `<-` whereas with futures we use `%<-%`.
So why are futures useful? Because we can choose to evaluate the future expression in a separate R process asynchronously by simply switching settings as:
```r
> library("future")
> plan(multiprocess)
> v %<-% {
+ cat("Resolving...\n")
+ 3.14
+ }
> v
[1] 3.14
```
With asynchronous futures, the current/main R process does _not_ block, which means it is available for further processing while the futures are being resolved
in separate processes running in the background. In other words, futures provide a simple but yet powerful construct for parallel and / or distributed processing in R.
Now, if you cannot be bothered to read all the nitty-gritty details about futures, but just want to try them out, then skip to the end to play with the Mandelbrot demo using both parallel and non-parallel evaluation.
## Implicit or Explicit Futures
Futures can be created either _implicitly_ or _explicitly_. In the introductory example above we used _implicit futures_ created via the `v %<-% { expr }` construct. An alternative is _explicit futures_ using the `f <- future({ expr })` and `v <- value(f)` constructs. With these, our example could alternatively be written as:
```r
> library("future")
> f <- future({
+ cat("Resolving...\n")
+ 3.14
+ })
Resolving...
> v <- value(f)
> v
[1] 3.14
```
Either style of future construct works equally(*) well. The implicit style is most similar to how regular R code is written. In principle, all you have to do is to replace `<-` with a `%<-%` to turn the assignment into a future assignment. On the other hand, this simplicity can also be deceiving, particularly when asynchronous futures are being used. In contrast, the explicit style makes it much clearer that futures are being used, which lowers the risk for mistakes and better communicates the design to others reading your code.
(*) There are cases where `%<-%` cannot be used without some (small) modifications. We will return to this in Section 'Constraints when using Implicit Futures' near the end of this document.
To summarize, for explicit futures, we use:
* `f <- future({ expr })` - creates a future
* `v <- value(f)` - gets the value of the future (blocks if not yet resolved)
For implicit futures, we use:
* `v %<-% { expr }` - creates a future and a promise to its value
To keep it simple, we will use the implicit style in the rest of this document, but everything discussed will also apply to explicit futures.
## Controlling How Futures are Resolved
The future package implements the following types of futures:
| Name | OSes | Description
|:----------------|:------------|:-----------------------------------------------------
| _synchronous:_ | | _non-parallel:_
| `sequential` | all | sequentially and in the current R process
| `transparent` | all | as sequential w/ early signaling and w/out local (for debugging)
| _asynchronous:_ | | _parallel_:
| `multiprocess` | all | multicore, if supported, otherwise multisession
| `multisession` | all | background R sessions (on current machine)
| `multicore` | not Windows | forked R processes (on current machine)
| `cluster` | all | external R sessions on current, local, and/or remote machines
| `remote` | all | Simple access to remote R sessions
The future package is designed such that support for additional strategies can be implemented as well. For instance, the [future.batchtools] package provides futures for all types of _cluster functions_ ("backends") that the [batchtools] package supports. Specifically, futures for evaluating R expressions via job schedulers such as Slurm, TORQUE/PBS, Oracle/Sun Grid Engine (SGE) and Load Sharing Facility (LSF) are also available. (_Comment_: The [future.BatchJobs] package provides analogue backends based on the [BatchJobs] package; however the BatchJobs developers have deprecated it in favor of batchtools.)
By default, future expressions are evaluated eagerly (= instantaneously) and synchronously (in the current R session). This evaluation strategy is referred to as "sequential". In this section, we will go through each of these strategies and discuss what they have in common and how they differ.
### Consistent Behavior Across Futures
Before going through each of the different future strategies, it is probably helpful to clarify the objectives the Future API (as defined by the future package). When programming with futures, it should not really matter what future strategy is used for executing code. This is because we cannot really know what computational resources the user has access to so the choice of evaluation strategy should be in the hands of the user and not the developer. In other words, the code should not make any assumptions on the type of futures used, e.g. synchronous or asynchronous.
One of the designs of the Future API was to encapsulate any differences such that all types of futures will appear to work the same. This despite expressions may be evaluated locally in the current R session or across the world in remote R sessions. Another obvious advantage of having a consistent API and behavior among different types of futures is that it helps while prototyping. Typically one would use sequential evaluation while building up a script and, later, when the script is fully developed, one may turn on asynchronous processing.
Because of this, the defaults of the different strategies are such that the results and side effects of evaluating a future expression are as similar as possible. More specifically, the following is true for all futures:
* All _evaluation is done in a local environment_ (i.e. `local({ expr })`) so that assignments do not affect the calling environment. This is natural when evaluating in an external R process, but is also enforced when evaluating in the current R session.
* When a future is constructed, _global variables are identified_. For asynchronous evaluation, globals are exported to the R process/session that will be evaluating the future expression. For sequential futures with lazy evaluation (`lazy = TRUE`), globals are "frozen" (cloned to a local environment of the future). Also, in order to protect against exporting too large objects by mistake, there is a built-in assertion that the total size of all globals is less than a given threshold (controllable via an option, cf. `help("future.options")`). If the threshold is exceeded, an informative error is thrown.
* Future _expressions are only evaluated once_. As soon as the value (or an error) has been collected it will be available for all succeeding requests.
Here is an example illustrating that all assignments are done to a local environment:
```r
> plan(sequential)
> a <- 1
> x %<-% {
+ a <- 2
+ 2 * a
+ }
> x
[1] 4
> a
[1] 1
```
Now we are ready to explore the different future strategies.
### Synchronous Futures
Synchronous futures are resolved one after another and most commonly by the R process that creates them. When a synchronous future is being resolved it blocks the main process until resolved. There are two types of synchronous futures in the future package, _sequential_ and _transparent_. (In future 1.2.0 and before, there was also _lazy_ futures, which has now been deprecated in favor of `f <- future(..., lazy = TRUE)` and `v %<-% { ... } %lazy% TRUE`.)
#### Sequential Futures
Sequential futures are the default unless otherwise specified. They were designed to behave as similar as possible to regular R evaluation while still fulfilling the Future API and its behaviors. Here is an example illustrating their properties:
```r
> plan(sequential)
> pid <- Sys.getpid()
> pid
[1] 28518
> a %<-% {
+ pid <- Sys.getpid()
+ cat("Resolving 'a' ...\n")
+ 3.14
+ }
Resolving 'a' ...
> b %<-% {
+ rm(pid)
+ cat("Resolving 'b' ...\n")
+ Sys.getpid()
+ }
Resolving 'b' ...
> c %<-% {
+ cat("Resolving 'c' ...\n")
+ 2 * a
+ }
Resolving 'c' ...
> b
[1] 28518
> c
[1] 6.28
> a
[1] 3.14
> pid
[1] 28518
```
Since eager sequential evaluation is taking place, each of the three futures is resolved instantaneously in the moment it is created. Note also how `pid` in the calling environment, which was assigned the process ID of the current process, is neither overwritten nor removed. This is because futures are evaluated in a local environment. Since synchronous (uni-)processing is used, future `b` is resolved by the main R process (still in a local environment), which is why the value of `b` and `pid` are the same.
#### Transparent Futures
For troubleshooting, _transparent_ futures can be used by specifying `plan(transparent)`. A transparent future is technically a sequential future with instant signaling of conditions (including errors and warnings) and where evaluation, and therefore also assignments, take place in the calling environment. Transparent futures are particularly useful for troubleshooting errors that are otherwise hard to narrow down.
### Asynchronous Futures
Next, we will turn to asynchronous futures, which are futures that are resolved in the background. By design, these futures are non-blocking, that is, after being created the calling process is available for other tasks including creating additional futures. It is only when the calling process tries to access the value of a future that is not yet resolved, or trying to create another asynchronous future when all available R processes are busy serving other futures, that it blocks.
#### Multisession Futures
We start with multisession futures because they are supported by all operating systems. A multisession future is evaluated in a background R session running on the same machine as the calling R process. Here is our example with multisession evaluation:
```r
> plan(multisession)
> pid <- Sys.getpid()
> pid
[1] 28518
> a %<-% {
+ pid <- Sys.getpid()
+ cat("Resolving 'a' ...\n")
+ 3.14
+ }
> b %<-% {
+ rm(pid)
+ cat("Resolving 'b' ...\n")
+ Sys.getpid()
+ }
> c %<-% {
+ cat("Resolving 'c' ...\n")
+ 2 * a
+ }
> b
[1] 28539
> c
[1] 6.28
> a
[1] 3.14
> pid
[1] 28518
```
The first thing we observe is that the values of `a`, `c` and `pid` are the same as previously. However, we notice that `b` is different from before. This is because future `b` is evaluated in a different R process and therefore it returns a different process ID. Another difference is that the messages, generated by `cat()`, are no longer displayed. This is because they are outputted to the background sessions and not the calling session.
When multisession evaluation is used, the package launches a set of R sessions in the background that will serve multisession futures by evaluating their expressions as they are created. If all background sessions are busy serving other futures, the creation of the next multisession future is _blocked_ until a background session becomes available again. The total number of background processes launched is decided by the value of `availableCores()`, e.g.
```r
> availableCores()
mc.cores
2
```
This particular result tells us that the `mc.cores` option was set such that we are allowed to use in total 2 processes including the main process. In other words, with these settings, there will be 2 background processes serving the multisession futures. The `availableCores()` is also agile to different options and system environment variables. For instance, if compute cluster schedulers are used (e.g. TORQUE/PBS and Slurm), they set specific environment variable specifying the number of cores that was allotted to any given job; `availableCores()` acknowledges these as well. If nothing else is specified, all available cores on the machine will be utilized, cf. `parallel::detectCores()`. For more details, please see `help("availableCores", package = "future")`.
#### Multicore Futures
On operating systems where R supports _forking_ of processes, which is basically all operating system except Windows, an alternative to spawning R sessions in the background is to fork the existing R process. Forking an R process is considered faster than working with a separate R session running in the background. One reason is that the overhead of exporting large globals to the background session can be greater than when forking is used.
To use multicore futures, we specify:
```r
plan(multicore)
```
The only real different between using multicore and multisession futures is that any output written (to standard output or standard error) by a multicore process is instantaneously outputted in calling process. Other than this, the behavior of using multicore evaluation is very similar to that of using multisession evaluation.
Just like for multisession futures, the maximum number of parallel processes running will be decided by `availableCores()`, since in both cases the evaluation is done on the local machine.
#### Multiprocess Futures
Sometimes we do not know whether multicore futures are supported or not, but it might still be that we would like to write platform-independent scripts or instructions that work everywhere. In such cases we can specify that we want to use "multiprocess" futures as in:
```r
plan(multiprocess)
```
A multiprocess future is not a formal class of futures by itself, but rather a convenient alias for either of the two. When this is specified, multisession evaluation will be used unless multicore evaluation is supported.
#### Cluster Futures
Cluster futures evaluate expressions on an ad-hoc cluster (as implemented by the parallel package). For instance, assume you have access to three nodes `n1`, `n2` and `n3`, you can then use these for asynchronous evaluation as:
```r
> plan(cluster, workers = c("n1", "n2", "n3"))
> pid <- Sys.getpid()
> pid
[1] 28518
> a %<-% {
+ pid <- Sys.getpid()
+ cat("Resolving 'a' ...\n")
+ 3.14
+ }
> b %<-% {
+ rm(pid)
+ cat("Resolving 'b' ...\n")
+ Sys.getpid()
+ }
> c %<-% {
+ cat("Resolving 'c' ...\n")
+ 2 * a
+ }
> b
[1] 28561
> c
[1] 6.28
> a
[1] 3.14
> pid
[1] 28518
```
Just as for most other asynchronous evaluation strategies, the output from `cat()` is not displayed on the current/calling machine.
Any types of clusters that `parallel::makeCluster()` creates can be used for cluster futures. For instance, the above cluster can be explicitly set up as:
```r
cl <- parallel::makeCluster(c("n1", "n2", "n3"))
plan(cluster, workers = cl)
```
Also, it is considered good style to shut down cluster `cl` when it is no longer needed, that is, calling `parallel::stopCluster(cl)`. However, it will shut itself down if the main process is terminated. For more information on how to set up and manage such clusters, see `help("makeCluster", package = "parallel")`.
Clusters created implicitly using `plan(cluster, workers = hosts)` where `hosts` is a character vector will also be shut down when the main R session terminates, or when the future strategy is changed, e.g. by calling `plan(sequential)`.
Note that with automatic authentication setup (e.g. SSH key pairs), there is nothing preventing us from using the same approach for using a cluster of remote machines.
### Nested Futures and Evaluation Topologies
This far we have discussed what can be referred to as "flat topology" of futures, that is, all futures are created in and assigned to the same environment. However, there is nothing stopping us from using a "nested topology" of futures, where one set of futures may, in turn, create another set of futures internally and so on.
For instance, here is an example of two "top" futures (`a` and `b`) that uses multiprocess evaluation and where the second future (`b`) in turn uses two internal futures:
```r
> plan(multiprocess)
> pid <- Sys.getpid()
> a %<-% {
+ cat("Resolving 'a' ...\n")
+ Sys.getpid()
+ }
> b %<-% {
+ cat("Resolving 'b' ...\n")
+ b1 %<-% {
+ cat("Resolving 'b1' ...\n")
+ Sys.getpid()
+ }
+ b2 %<-% {
+ cat("Resolving 'b2' ...\n")
+ Sys.getpid()
+ }
+ c(b.pid = Sys.getpid(), b1.pid = b1, b2.pid = b2)
+ }
> pid
[1] 28518
> a
[1] 28584
> b
b.pid b1.pid b2.pid
28585 28585 28585
```
By inspection the process IDs, we see that there are in total three different processes involved for resolving the futures. There is the main R process (pid 28518), and there are the two processes used by `a` (pid 28584) and `b` (pid 28585). However, the two futures (`b1` and `b2`) that is nested by `b` are evaluated by the same R process as `b`. This is because nested futures use sequential evaluation unless otherwise specified. There are a few reasons for this, but the main reason is that it protects us from spawning off a large number of background processes by mistake, e.g. via recursive calls.
To specify a different type of _evaluation topology_, other than the first level of futures being resolved by multiprocess evaluation and the second level by sequential evaluation, we can provide a list of evaluation strategies to `plan()`. First, the same evaluation strategies as above can be explicitly specified as:
```r
plan(list(multiprocess, sequential))
```
We would actually get the same behavior if we try with multiple levels of multiprocess evaluations;
```r
> plan(list(multiprocess, multiprocess))
[...]
> pid
[1] 28518
> a
[1] 28586
> b
b.pid b1.pid b2.pid
28587 28587 28587
```
The reason for this is, also here, to protect us from launching more processes than what the machine can support. Internally, this is done by setting `mc.cores = 1` such that functions like `parallel::mclapply()` will fall back to run sequentially. This is the case for both multisession and multicore evaluation.
Continuing, if we start off by sequential evaluation and then use multiprocess evaluation for any nested futures, we get:
```r
> plan(list(sequential, multiprocess))
[...]
Resolving 'a' ...
Resolving 'b' ...
> pid
[1] 28518
> a
[1] 28518
> b
b.pid b1.pid b2.pid
28518 28588 28589
```
which clearly show that `a` and `b` are resolved in the calling process (pid 28518) whereas the two nested futures (`b1` and `b2`) are resolved in two separate R processes (pids 28588 and 28589).
Having said this, it is indeed possible to use nested multiprocess evaluation strategies, if we explicitly specify (read _force_) the number of cores available at each level. In order to do this we need to "tweak" the default settings, which can be done as follows:
```r
> plan(list(tweak(multiprocess, workers = 2L), tweak(multiprocess,
+ workers = 2L)))
[...]
> pid
[1] 28518
> a
[1] 28590
> b
b.pid b1.pid b2.pid
28591 28592 28594
```
First, we see that both `a` and `b` are resolved in different processes (pids 28590 and 28591) than the calling process (pid 28518). Second, the two nested futures (`b1` and `b2`) are resolved in yet two other R processes (pids 28592 and 28594).
For more details on working with nested futures and different evaluation strategies at each level, see Vignette '[Futures in R: Future Topologies]'.
### Checking A Future without Blocking
It is possible to check whether a future has been resolved or not without blocking. This can be done using the `resolved(f)` function, which takes an explicit future `f` as input. If we work with implicit futures (as in all the examples above), we can use the `f <- futureOf(a)` function to retrieve the explicit future from an implicit one. For example,
```r
> plan(multiprocess)
> a %<-% {
+ cat("Resolving 'a' ...")
+ Sys.sleep(2)
+ cat("done\n")
+ Sys.getpid()
+ }
> cat("Waiting for 'a' to be resolved ...\n")
Waiting for 'a' to be resolved ...
> f <- futureOf(a)
> count <- 1
> while (!resolved(f)) {
+ cat(count, "\n")
+ Sys.sleep(0.2)
+ count <- count + 1
+ }
1
2
3
4
5
> cat("Waiting for 'a' to be resolved ... DONE\n")
Waiting for 'a' to be resolved ... DONE
> a
[1] 28595
```
## Failed Futures
Sometimes the future is not what you expected. If an error occurs while evaluating a future, the error is propagated and thrown as an error in the calling environment _when the future value is requested_. For example, if we use lazy evaluation on a future that generates an error, we might see something like
```r
> plan(sequential)
> a %<-% {
+ cat("Resolving 'a' ...\n")
+ stop("Whoops!")
+ 42
+ } %lazy% TRUE
> cat("Everything is still ok although we have created a future that will fail.\n")
Everything is still ok although we have created a future that will fail.
> a
Resolving 'a' ...
Error in eval(expr, envir, enclos) : Whoops!
```
The error is thrown each time the value is requested, that is, if we try to get the value again will generate the same error:
```r
> a
Error in eval(expr, envir, enclos) : Whoops!
In addition: Warning message:
restarting interrupted promise evaluation
```
To see the list of calls (evaluated expressions) that lead up to the error, we can use the `backtrace()` function(*) on the future, i.e.
```r
> backtrace(a)
[[1]]
eval(quote({
cat("Resolving 'a' ...\\n")
stop("Whoops!")
42
}), new.env())
[[2]]
eval(quote({
cat("Resolving 'a' ...\\n")
stop("Whoops!")
42
}), new.env())
[[3]]
stop("Whoops!")
```
(*) The commonly used `traceback()` does not provide relevant information in the context of futures.
## Globals
Whenever an R expression is to be evaluated asynchronously (in parallel) or sequentially via lazy evaluation, global (aka "free") objects have to be identified and passed to the evaluator. They need to be passed exactly as they were at the time the future was created, because, for lazy evaluation, globals may otherwise change between when it is created and when it is resolved. For asynchronous processing, the reason globals need to be identified is so that they can be exported to the process that evaluates the future.
The future package tries to automate these tasks as far as possible. It does this with help of the [globals] package, which uses static-code inspection to identify global variables. If a global variable is identified, it is captured and made available to the evaluating process.
Moreover, if a global is defined in a package, then that global is not exported. Instead, it is made sure that the corresponding package is attached when the future is evaluated. This not only better reflects the setup of the main R session, but it also minimizes the need for exporting globals, which saves not only memory but also time and bandwidth, especially when using remote compute nodes.
Finally, it should be clarified that identifying globals from static code inspection alone is a challenging problem. There will always be corner cases where automatic identification of globals fails so that either false globals are identified (less of a concern) or some of the true globals are missing (which will result in a run-time error or possibly the wrong results). Vignette '[Futures in R: Common Issues with Solutions]' provides examples of common cases and explains how to avoid them as well as how to help the package to identify globals or to ignore falsely identified globals. If that does not suffice, it is always possible to manually specify the global variables by their names (e.g. `globals = c("a", "slow_sum")`) or as name-value pairs (e.g. `globals = list(a = 42, slow_sum = my_sum)`).
## Constraints when using Implicit Futures
There is one limitation with implicit futures that does not exist for explicit ones. Because an explicit future is just like any other object in R it can be assigned anywhere/to anything. For instance, we can create several of them in a loop and assign them to a list, e.g.
```r
> plan(multiprocess)
> f <- list()
> for (ii in 1:3) {
+ f[[ii]] <- future({
+ Sys.getpid()
+ })
+ }
> v <- lapply(f, FUN = value)
> str(v)
List of 3
$ : int 28602
$ : int 28603
$ : int 28605
```
This is _not_ possible to do when using implicit futures. This is because the `%<-%` assignment operator _cannot_ be used in all cases where the regular `<-` assignment operator can be used. It can only be used to assign future values to _environments_ (including the calling environment) much like how `assign(name, value, envir)` works. However, we can assign implicit futures to environments using _named indices_, e.g.
```r
> plan(multiprocess)
> v <- new.env()
> for (name in c("a", "b", "c")) {
+ v[[name]] %<-% {
+ Sys.getpid()
+ }
+ }
> v <- as.list(v)
> str(v)
List of 3
$ a: int 28606
$ b: int 28607
$ c: int 28608
```
Here `as.list(v)` blocks until all futures in the environment `v` have been resolved. Then their values are collected and returned as a regular list.
If _numeric indices_ are required, then _list environments_ can be used. List environments, which are implemented by the [listenv] package, are regular environments with customized subsetting operators making it possible to index them much like how lists can be indexed. By using list environments where we otherwise would use lists, we can also assign implicit futures to list-like objects using numeric indices. For example,
```r
> library("listenv")
> plan(multiprocess)
> v <- listenv()
> for (ii in 1:3) {
+ v[[ii]] %<-% {
+ Sys.getpid()
+ }
+ }
> v <- as.list(v)
> str(v)
List of 3
$ : int 28609
$ : int 28610
$ : int 28611
```
As previously, `as.list(v)` blocks until all futures are resolved.
## Demos
To see a live illustration how different types of futures are evaluated, run the Mandelbrot demo of this package. First, try with the sequential evaluation,
```r
library("future")
plan(sequential)
demo("mandelbrot", package = "future", ask = FALSE)
```
which resembles how the script would run if futures were not used. Then, try multiprocess evaluation, which calculates the different Mandelbrot planes using parallel R processes running in the background. Try,
```r
plan(multiprocess)
demo("mandelbrot", package = "future", ask = FALSE)
```
Finally, if you have access to multiple machines you can try to set up a cluster of workers and use them, e.g.
```r
plan(cluster, workers = c("n2", "n5", "n6", "n6", "n9"))
demo("mandelbrot", package = "future", ask = FALSE)
```
## Contributing
The goal of this package is to provide a standardized and unified API for using futures in R. What you are seeing right now is an early but sincere attempt to achieve this goal. If you have comments or ideas on how to improve the 'future' package, I would love to hear about them. The preferred way to get in touch is via the [GitHub repository](https://github.com/HenrikBengtsson/future/), where you also find the latest source code. I am also open to contributions and collaborations of any kind.
[BatchJobs]: https://cran.r-project.org/package=BatchJobs
[batchtools]: https://cran.r-project.org/package=batchtools
[future]: https://cran.r-project.org/package=future
[future.BatchJobs]: https://cran.r-project.org/package=future.BatchJobs
[future.batchtools]: https://cran.r-project.org/package=future.batchtools
[globals]: https://cran.r-project.org/package=globals
[listenv]: https://cran.r-project.org/package=listenv
[Futures in R: Common Issues with Solutions]: future-2-issues.html
[Futures in R: Future Topologies]: future-3-topologies.html
---
Copyright Henrik Bengtsson, 2015-2017
future/inst/doc/future-2-issues.html 0000644 0001762 0000144 00000074544 13237670252 017203 0 ustar ligges users
A Future for R: Common Issues with Solutions
A Future for R: Common Issues with Solutions
In the ideal case, all it takes to start using futures in R is to replace select standard assignments (<-) in your R code with future assignments (%<-%) and make sure the right-hand side (RHS) expressions are within curly brackets ({ ... }). Also, if you assign these to lists (e.g. in a for loop), you need to use a list environment (listenv) instead of a plain list.
However, as show below, there are few cases where you might run into some hurdles, but, as also shown, they are often easy to overcome. These are often related to global variables.
If you identify other cases, please consider reporting them so they can be documented here and possibly even be fixed.
Issues with globals and packages
Missing globals (false negatives)
If a global variable is used in a future expression that conditionally overrides this global variable with a local one, the future framework fails to identify the global variable and therefore fails to export it, resulting in a run-time error. For example, although this works:
library(multisession)
reset <- TRUE
x <- 1
y %<-% { if (reset) x <- 0; x + 1 }
y
## [1] 1
the following does not work:
reset <- FALSE
x <- 1
y %<-% { if (reset) x <- 0; x + 1 }
y
## Error: object 'x' not found
Comment: The goal is to in a future version of the package detect globals also in expression where the local-global state of a variable is only known at run time.
do.call() - function not found
When calling a function using do.call() make sure to specify the function as the object itself and not by name. This will help identify the function as a global object in the future expression. For instance, use
do.call(file_ext, list("foo.txt"))
instead of
do.call("file_ext", list("foo.txt"))
so that file_ext() is properly located and exported. Although you may not notice a difference when evaluating futures in the same R session, it may become a problem if you use a character string instead of a function object when futures are evaluated in external R sessions, such as on a cluster.
It may also become a problem with futures evaluated with lazy evaluation if the intended function is redefined after the future is resolved. For example,
Occassionally, the static-code inspection of the future expression fails to identify packages needed to evaluated the expression. This may occur when a expression uses S3 generic functions part of one package whereas the required S3 method is in another package. For example, in the below future generic function [ is used on data.table object DT, which requires S3 method [.data.table from the data.table package. However, the future and global packages fail to identify data.table as a required package, which results in an evaluation error:
> library("future")
> plan(multisession)
> library("data.table")
> DT <- data.table(a = LETTERS[1:3], b = 1:3)
> y %<-% DT[, sum(b)]
> y
Error: object 'b' not found
The above error occurs because, contrary to the master R process, the R worker that evaluated the future expression does not have data.table loaded. Instead the evaluation falls back to the [.data.frame method, which is not what we want.
Until the future framework manages to identify data.table as a required package (which is the goal), we can guide future by specifying additional packages needed:
> y %<-% DT[, sum(b)] %packages% "data.table"
> y
[1] 6
Certain types of objects are tied to a given R session and cannot be passed along to another R process (a “worker”). For instance, if you create a file connection,
which is bound to the main R process and makes no sense to the worker. Ideally, the R process of the worker would detect this and produce an informative error message, but as seen here, that does not always occur.
To help avoiding these mistakes, the future framework provides a mechanism (in beta) can help detect non-exportable objects by setting:
> options(future.globals.onReference = "warning")
which will result in a warning in the above example:
> f <- future({ cat("world!", file = con); flush(con) })
Warning in FALSE :
Detected a non-exportable reference ('externalptr') in one of the globals
('con' of class 'file') used in the future expression
To be more conservative, we can also tell it to produce an error:
> options(future.globals.onReference = "error")
> f <- future({ cat("world!", file = con); flush(con) })
Error in FALSE :
Detected a non-exportable reference ('externalptr') in one of the globals
('con' of class 'file') used in the future expression
Another example is XML objects of the xml2 package, which may produce evaluation errors (or just invalid results depending on how they are used), e.g.
> library("future")
> plan(multisession)
> library("xml2")
> xml <- read_xml("<body></body>")
> f <- future(xml_children(xml))
> value(f)
Error: external pointer is not valid
The future framework can help detect this before sending off the future to the worker;
> options(future.globals.onReference = "error")
> f <- future(xml_children(xml))
Error in FALSE :
Detected a non-exportable reference ('externalptr') in one of the globals
('xml' of class 'xml_document') used in the future expression
A third example is Rcpp which allow us to easily create R functions that are implemented in C++, e.g.
Rcpp::sourceCpp(code = "
#include <Rcpp.h>
using namespace Rcpp;
// [[Rcpp::export]]
int my_length(NumericVector x) {
return x.size();
}
")
so that:
> x <- 1:10
> my_length(x)
[1] 10
However, since this function uses an external pointer internally, we cannot pass it to another R process:
> library("future")
> plan(multisession)
> n %<-% my_length(x)
> n
Error: NULL value passed as symbol address
We can detect / protect against this using:
> options(future.globals.onReference = "error")
> n %<-% my_length(x)
Error in FALSE :
Detected a non-exportable reference ('externalptr' of class 'NativeSymbol')
in one of the globals ('my_length' of class 'function') used in the future
expression
(*) The exception may be if you use forked R processes, e.g. plan(multicore).
Trying to pass an unresolved future to another future
It is not possible for a future to resolve another one unless it was created by the future trying to resolve it. For instance, the following gives an error:
> library("future")
> plan(multiprocess)
> f1 <- future({ Sys.getpid() })
> f2 <- future({ value(f1) })
> v1 <- value(f1)
[1] 7464
> v2 <- value(f2)
Error: Invalid usage of futures: A future whose value has not yet been collected
can only be queried by the R process (cdd013cb-e045-f4a5-3977-9f064c31f188; pid
1276 on MyMachine) that created it, not by any other R processes (5579f789-e7b6
-bace-c50d-6c7a23ddb5a3; pid 2352 on MyMachine): {; Sys.getpid(); }
This is because the main R process creates two futures, but then the second future tries to retrieve the value of the first one. This is an invalid request because the second future has no channel to communicate with the first future; it is only the process that created a future who can communicate with it(*).
Note that it is only unresolved futures that cannot be queried this way. Thus, the solution to the above problem is to make sure all futures are resolved before they are passed to other futures, e.g.
This works because the value has already been collected and stored inside future f1 before future f2 is created. Since the value is already stored internally, value(f1) is readily available everywhere. Of course, instead of using value(f1) for the second future, it would be more readable and cleaner to simply use v1.
The above is typically not a problem when future assignments are used. For example:
The reason that this approach works out of the box is because in the second future assignment v1 is identified as a global variable, which is retrieved. Up to this point, v1 is a promise (“delayed assignment” in R), but when it is retrieved as a global variable its value is resolved and v1 becomes a regular variable.
However, there are cases where future assignments can be passed via global variables without being resolved. This can happen if the future assignment is done to an element of an environment (including list environments). For instance,
> library("listenv")
> x <- listenv()
> x$a %<-% { Sys.getpid() }
> x$b %<-% { x$a }
> x$a
[1] 2352
> x$b
Error: Invalid usage of futures: A future whose value has not yet been collected
can only be queried by the R process (cdd013cb-e045-f4a5-3977-9f064c31f188; pid
1276 on localhost) that created it, not by any other R processes (2ce86ccd-5854
-7a05-1373-e1b20022e4d8; pid 7464 on localhost): {; Sys.getpid(); }
As previously, this can be avoided by making sure x$a is resolved first, which can be one in various ways, e.g. dummy <- x$a, resolve(x$a) and force(x$a).
Footnote: (*) Although sequential futures could be passed on to other futures part of the same R process and be resolved there because they share the same evaluation process, by definition of the Future API it is invalid to do so regardless of future type. This conservative approach is taken in order to make future expressions behave consistently regardless of the type of future used.
Miscellaneous
Clashes with other packages
Sometimes other packages have functions or operators with the same name as the future package, and if those packages are attached after the future package, their objects will mask the ones of the future package. For instance, the igraph package also defines a %<-% operator which clashes with the one in future if used at the prompt or in a script (it is not a problem inside package because there we explicitly import objects in a known order). Here is what we might get:
> library("future")
> library("igraph")
Attaching package: 'igraph'
The following objects are masked from 'package:future':
%<-%, %->%
The following objects are masked from 'package:stats':
decompose, spectrum
The following object is masked from 'package:base':
union
> y %<-% { 42 }
Error in get(".igraph.from", parent.frame()) :
object '.igraph.from' not found
Here we get an error because %<-% is from igraph and not the future assignment operator as we wanted. This can be confirmed as:
To avoid this problem, attach the two packages in opposite order such that future comes last and thereby overrides igraph, i.e.
> library("igraph")
> library("future")
Attaching package: 'future'
The following objects are masked from 'package:igraph':
%<-%, %->%
> y %<-% { 42 }
> y
[1] 42
An alternative is to detach the future package and re-attach it, which will achieve the same thing:
> detach("package:future")
> library("future")
Yet another alternative is to explicitly override the object by importing it to the global environment, e.g.
> `%<-%` <- future::`%<-%`
> y %<-% { 42 }
> y
[1] 42
In this case, it does not matter in what order the packages are attached because we will always use the copy of future::`%<-%`.
Syntax error: “non-numeric argument to binary operator”
The future assignment operator %<-% is a binary infix operator, which means it has higher precedence than most other binary operators but also higher than some of the unary operators in R. For instance, this explains why we get the following error:
> x %<-% 2 * runif(1)
Error in x %<-% 2 * runif(1) : non-numeric argument to binary operator
What effectively is happening here is that because of the higher priority of %<-%, we first create a future x %<-% 2 and then we try to multiply the future (not its value) with the value of runif(1) - which makes no sense. In order to properly assign the future variable, we need to put the future expression within curly brackets;
> x %<-% { 2 * runif(1) }
> x
[1] 1.030209
Parentheses will also do. For details on precedence on operators in R, see Section 'Infix and prefix operators' in the 'R Language Definition' document.
R CMD check NOTEs
The code inspection run by R CMD check will not recognize the future assignment operator %<-% as an assignment operator, which is not surprising because %<-% is technically an infix operator. This means that if you for instance use the following code in your package:
foo <- function() {
b <- 3.14
a %<-% { b + 1 }
a
}
then R CMD check will produce a NOTE saying:
* checking R code for possible problems ... NOTE
foo: no visible binding for global variable 'a'
Undefined global functions or variables:
a
In order to avoid this, we can add a dummy assignment of the missing global at the top of the function, i.e.
foo <- function() {
a <- NULL ## To please R CMD check
b <- 3.14
a %<-% { b + 1 }
a
}
Copyright Henrik Bengtsson, 2015-2018
future/inst/doc/future-3-topologies.md.rsp 0000644 0001762 0000144 00000016747 13111756521 020310 0 ustar ligges users <%@meta language="R-vignette" content="--------------------------------
%\VignetteIndexEntry{A Future for R: Future Topologies}
%\VignetteAuthor{Henrik Bengtsson}
%\VignetteKeyword{R}
%\VignetteKeyword{package}
%\VignetteKeyword{vignette}
%\VignetteKeyword{future}
%\VignetteKeyword{promise}
%\VignetteEngine{R.rsp::rsp}
%\VignetteTangle{FALSE}
--------------------------------------------------------------------"%>
<%
library("R.utils")
`%<-%` <- future::`%<-%`
options("withCapture/newline"=FALSE)
%>
# <%@meta name="title"%>
Futures can be nested in R such that one future creates another set of futures and so on. This may, for instance, occur within nested for loops, e.g.
```r
library("future")
library("listenv")
x <- listenv()
for (ii in 1:3) {
x[[ii]] %<-% {
y <- listenv()
for (jj in 1:3) {
y[[jj]] %<-% { ii + jj / 10 }
}
y
}
}
unlist(x)
## [1] 1.1 1.2 1.3 2.1 2.2 2.3 3.1 3.2 3.3
```
The default is to use synchronous futures unless otherwise specified, which is also true for nested futures. If we for instance specify, `plan(multiprocess)`, the first layer of futures (`x[[ii]] %<-% { expr }`) will be processed asynchronously in background R processes, and the futures in the second layer of futures (`y[[jj]] %<-% { expr }`) will be processed synchronously in the separate background R processes. If we wish to be explicit about this, we can specify `plan(list(multiprocess, sequential))`.
## Example: High-Throughput Sequencing
Consider a high-throughput sequencing (HT-Seq) project with 50 human DNA samples where we have one FASTQ file per sample containing the raw sequence reads as they come out of the sequencing machine. With this data, we wish to align each FASTQ to a reference genome such that we generate 24 individual BAM files per sample - one per chromosome.
Here is the layout of what such an analysis could look like in R using futures.
```r
library("future")
library("listenv")
htseq_align <- function(fq, chr) { chr }
fqs <- dir(pattern = "[.]fastq$")
bams <- listenv()
for (ss in seq_along(fqs)) {
fq <- fqs[ss]
bams[[ss]] %<-% {
bams_ss <- listenv()
for (cc in 1:24) {
bams_ss[[cc]] %<-% htseq_align(fq, chr = cc)
}
as.list(bams_ss)
}
}
bams <- as.list(bams)
```
The default is to use synchronous futures, so without further specifications, the above will process each sample and each chromosome sequentially. Next, we will consider what can be done with the following two computer setups:
* A single machine with 8 cores
* A compute cluster with 3 machines each with 16 cores
### One multi-core machine
With a single machine of 8 cores, we could choose to process multiple samples at the same time while processing chromosomes sequentially. In other words, we would like to evaluate the outer layer of futures using multiprocess futures and the inner ones as sequential futures. This can be specified as:
```r
plan(list(multiprocess, sequential))
```
The internals for processing multiprocess future queries `availableCores()` to infer how many cores can be used simultaneously, so there is no need to explicitly specify that there are 8 cores available.
_Comment_: Since synchronous is the default future, we could skip trailing sequential futures in the setup, e.g. `plan(list(multiprocess))` or just `plan(multiprocess)`. However, it does not hurt to be explicit.
If we instead would like to process the sample sequentially and the chromosomes in parallel, we can use:
```r
plan(list(sequential, multiprocess))
```
We could also process the data such that we allocate two cores for processing two samples in parallel each using four cores for processing four chromosomes in parallel:
```r
plan(list(tweak(multiprocess, workers = 2), tweak(multiprocess, workers = 4)))
```
### An ad-hoc compute cluster
With a compute cluster of 3 machines each with 16 cores, we can run up to 48 alignment processes in parallel. A natural setup is to have one machine process one sample in parallel. We could specify this as:
```r
nodes <- c("n1", "n2", "n3")
plan(list(tweak(cluster, workers = nodes), multiprocess))
```
_Comment:_ Multiprocess futures are agile to its environment, that is, they will query the machine they are running on to find out how many parallel processes it can run at the same time.
One possible downside to the above setup is that we might not utilize all available cores all the time. This is because the alignment of the shorter chromosomes will finish sooner than the longer ones, which means that we might at the end of each sample have only a few alignment processes running on each machine leaving the remaining cores idle/unused. An alternative set up is then to use the following setup:
```r
nodes <- rep(c("n1", "n2", "n3"), each = 8)
plan(list(tweak(cluster, workers = nodes), multiprocess))
```
This will cause up to 24 (= 3*8) samples to be processed in parallel each processing two chromosomes at the same time.
## Example: A remote compute cluster
Imagine we have access to a remote compute cluster, with login node `remote.server.org`, and that the cluster has three nodes `n1`, `n2`, and `n3`. Also, let us assume we have already set up the cluster such that we can log in via public key authentication via SSH, i.e. when we do `ssh remote.server.org` authentication is done automatically.
With the above setup, we can use nested futures in our local R session to evaluate R expression on the remote compute cluster and its three nodes. Here is a proof of concept illustrating how the different nested futures are evaluated on different machines.
```r
library("future")
library("listenv")
## Set up access to remote login node
login <- tweak(remote, workers = "remote.server.org")
plan(login)
## Set up cluster nodes on login node
nodes %<-% { .keepme <- parallel::makeCluster(c("n1", "n2", "n3")) }
## Specify future topology
## login node -> { cluster nodes } -> { multiple cores }
plan(list(
login,
tweak(cluster, workers = nodes),
multiprocess
))
## (a) This will be evaluated on the cluster login computer
x %<-% {
thost <- Sys.info()[["nodename"]]
tpid <- Sys.getpid()
y <- listenv()
for (task in 1:4) {
## (b) This will be evaluated on a compute node on the cluster
y[[task]] %<-% {
mhost <- Sys.info()[["nodename"]]
mpid <- Sys.getpid()
z <- listenv()
for (jj in 1:2) {
## (c) These will be evaluated in separate processes on the same compute node
z[[jj]] %<-% data.frame(task = task,
top.host = thost, top.pid = tpid,
mid.host = mhost, mid.pid = mpid,
host = Sys.info()[["nodename"]],
pid = Sys.getpid())
}
Reduce(rbind, z)
}
}
Reduce(rbind, y)
}
print(x)
## task top.host top.pid mid.host mid.pid host pid
## 1 1 login 391547 n1 391878 n1 393943
## 2 1 login 391547 n1 391878 n1 393951
## 3 2 login 391547 n2 392204 n2 393971
## 4 2 login 391547 n2 392204 n2 393978
## 5 3 login 391547 n3 392527 n3 394040
## 6 3 login 391547 n3 392527 n3 394048
## 7 4 login 391547 n1 391878 n1 393959
## 8 4 login 391547 n1 391878 n1 393966
```
Try the above `x %<-% { ... }` future with, say, `plan(list(sequential, multiprocess))` and see what the output will be.
[listenv]: https://cran.r-project.org/package=listenv
[globals]: https://cran.r-project.org/package=globals
[Futures in R: Common issues with solutions]: future-issues.html
---
Copyright Henrik Bengtsson, 2015-2017
future/inst/doc/future-1-overview.html 0000644 0001762 0000144 00000134216 13237670250 017524 0 ustar ligges users
A Future for R: A Comprehensive Overview
A Future for R: A Comprehensive Overview
Introduction
The purpose of the future package is to provide a very simple and uniform way of evaluating R expressions asynchronously using various resources available to the user.
In programming, a future is an abstraction for a value that may be available at some point in the future. The state of a future can either be unresolved or resolved. As soon as it is resolved, the value is available instantaneously. If the value is queried while the future is still unresolved, the current process is blocked until the future is resolved. It is possible to check whether a future is resolved or not without blocking. Exactly how and when futures are resolved depends on what strategy is used to evaluate them. For instance, a future can be resolved using a sequential strategy, which means it is resolved in the current R session. Other strategies may be to resolve futures asynchronously, for instance, by evaluating expressions in parallel on the current machine or concurrently on a compute cluster.
Here is an example illustrating how the basics of futures work. First, consider the following code snippet that uses plain R code:
> v <- {
+ cat("Resolving...\n")
+ 3.14
+ }
Resolving...
> v
[1] 3.14
It works by assigning the value of an expression to variable v and we then print the value of v. Moreover, when the expression for v is evaluated we also print a message.
Here is the same code snippet modified to use futures instead:
> library("future")
> v %<-% {
+ cat("Resolving...\n")
+ 3.14
+ }
Resolving...
> v
[1] 3.14
The difference is in how v is constructed; with plain R we use <- whereas with futures we use %<-%.
So why are futures useful? Because we can choose to evaluate the future expression in a separate R process asynchronously by simply switching settings as:
> library("future")
> plan(multiprocess)
> v %<-% {
+ cat("Resolving...\n")
+ 3.14
+ }
> v
[1] 3.14
With asynchronous futures, the current/main R process does not block, which means it is available for further processing while the futures are being resolved
in separate processes running in the background. In other words, futures provide a simple but yet powerful construct for parallel and / or distributed processing in R.
Now, if you cannot be bothered to read all the nitty-gritty details about futures, but just want to try them out, then skip to the end to play with the Mandelbrot demo using both parallel and non-parallel evaluation.
Implicit or Explicit Futures
Futures can be created either implicitly or explicitly. In the introductory example above we used implicit futures created via the v %<-% { expr } construct. An alternative is explicit futures using the f <- future({ expr }) and v <- value(f) constructs. With these, our example could alternatively be written as:
> library("future")
> f <- future({
+ cat("Resolving...\n")
+ 3.14
+ })
Resolving...
> v <- value(f)
> v
[1] 3.14
Either style of future construct works equally(*) well. The implicit style is most similar to how regular R code is written. In principle, all you have to do is to replace <- with a %<-% to turn the assignment into a future assignment. On the other hand, this simplicity can also be deceiving, particularly when asynchronous futures are being used. In contrast, the explicit style makes it much clearer that futures are being used, which lowers the risk for mistakes and better communicates the design to others reading your code.
(*) There are cases where %<-% cannot be used without some (small) modifications. We will return to this in Section 'Constraints when using Implicit Futures' near the end of this document.
To summarize, for explicit futures, we use:
f <- future({ expr }) - creates a future
v <- value(f) - gets the value of the future (blocks if not yet resolved)
For implicit futures, we use:
v %<-% { expr } - creates a future and a promise to its value
To keep it simple, we will use the implicit style in the rest of this document, but everything discussed will also apply to explicit futures.
Controlling How Futures are Resolved
The future package implements the following types of futures:
Name
OSes
Description
synchronous:
non-parallel:
sequential
all
sequentially and in the current R process
transparent
all
as sequential w/ early signaling and w/out local (for debugging)
asynchronous:
parallel:
multiprocess
all
multicore, if supported, otherwise multisession
multisession
all
background R sessions (on current machine)
multicore
not Windows
forked R processes (on current machine)
cluster
all
external R sessions on current, local, and/or remote machines
remote
all
Simple access to remote R sessions
The future package is designed such that support for additional strategies can be implemented as well. For instance, the future.batchtools package provides futures for all types of cluster functions (“backends”) that the batchtools package supports. Specifically, futures for evaluating R expressions via job schedulers such as Slurm, TORQUE/PBS, Oracle/Sun Grid Engine (SGE) and Load Sharing Facility (LSF) are also available. (Comment: The future.BatchJobs package provides analogue backends based on the BatchJobs package; however the BatchJobs developers have deprecated it in favor of batchtools.)
By default, future expressions are evaluated eagerly (= instantaneously) and synchronously (in the current R session). This evaluation strategy is referred to as “sequential”. In this section, we will go through each of these strategies and discuss what they have in common and how they differ.
Consistent Behavior Across Futures
Before going through each of the different future strategies, it is probably helpful to clarify the objectives the Future API (as defined by the future package). When programming with futures, it should not really matter what future strategy is used for executing code. This is because we cannot really know what computational resources the user has access to so the choice of evaluation strategy should be in the hands of the user and not the developer. In other words, the code should not make any assumptions on the type of futures used, e.g. synchronous or asynchronous.
One of the designs of the Future API was to encapsulate any differences such that all types of futures will appear to work the same. This despite expressions may be evaluated locally in the current R session or across the world in remote R sessions. Another obvious advantage of having a consistent API and behavior among different types of futures is that it helps while prototyping. Typically one would use sequential evaluation while building up a script and, later, when the script is fully developed, one may turn on asynchronous processing.
Because of this, the defaults of the different strategies are such that the results and side effects of evaluating a future expression are as similar as possible. More specifically, the following is true for all futures:
All evaluation is done in a local environment (i.e. local({ expr })) so that assignments do not affect the calling environment. This is natural when evaluating in an external R process, but is also enforced when evaluating in the current R session.
When a future is constructed, global variables are identified. For asynchronous evaluation, globals are exported to the R process/session that will be evaluating the future expression. For sequential futures with lazy evaluation (lazy = TRUE), globals are “frozen” (cloned to a local environment of the future). Also, in order to protect against exporting too large objects by mistake, there is a built-in assertion that the total size of all globals is less than a given threshold (controllable via an option, cf. help("future.options")). If the threshold is exceeded, an informative error is thrown.
Future expressions are only evaluated once. As soon as the value (or an error) has been collected it will be available for all succeeding requests.
Here is an example illustrating that all assignments are done to a local environment:
> plan(sequential)
> a <- 1
> x %<-% {
+ a <- 2
+ 2 * a
+ }
> x
[1] 4
> a
[1] 1
Now we are ready to explore the different future strategies.
Synchronous Futures
Synchronous futures are resolved one after another and most commonly by the R process that creates them. When a synchronous future is being resolved it blocks the main process until resolved. There are two types of synchronous futures in the future package, sequential and transparent. (In future 1.2.0 and before, there was also lazy futures, which has now been deprecated in favor of f <- future(..., lazy = TRUE) and v %<-% { ... } %lazy% TRUE.)
Sequential Futures
Sequential futures are the default unless otherwise specified. They were designed to behave as similar as possible to regular R evaluation while still fulfilling the Future API and its behaviors. Here is an example illustrating their properties:
Since eager sequential evaluation is taking place, each of the three futures is resolved instantaneously in the moment it is created. Note also how pid in the calling environment, which was assigned the process ID of the current process, is neither overwritten nor removed. This is because futures are evaluated in a local environment. Since synchronous (uni-)processing is used, future b is resolved by the main R process (still in a local environment), which is why the value of b and pid are the same.
Transparent Futures
For troubleshooting, transparent futures can be used by specifying plan(transparent). A transparent future is technically a sequential future with instant signaling of conditions (including errors and warnings) and where evaluation, and therefore also assignments, take place in the calling environment. Transparent futures are particularly useful for troubleshooting errors that are otherwise hard to narrow down.
Asynchronous Futures
Next, we will turn to asynchronous futures, which are futures that are resolved in the background. By design, these futures are non-blocking, that is, after being created the calling process is available for other tasks including creating additional futures. It is only when the calling process tries to access the value of a future that is not yet resolved, or trying to create another asynchronous future when all available R processes are busy serving other futures, that it blocks.
Multisession Futures
We start with multisession futures because they are supported by all operating systems. A multisession future is evaluated in a background R session running on the same machine as the calling R process. Here is our example with multisession evaluation:
> plan(multisession)
> pid <- Sys.getpid()
> pid
[1] 28518
> a %<-% {
+ pid <- Sys.getpid()
+ cat("Resolving 'a' ...\n")
+ 3.14
+ }
> b %<-% {
+ rm(pid)
+ cat("Resolving 'b' ...\n")
+ Sys.getpid()
+ }
> c %<-% {
+ cat("Resolving 'c' ...\n")
+ 2 * a
+ }
> b
[1] 28539
> c
[1] 6.28
> a
[1] 3.14
> pid
[1] 28518
The first thing we observe is that the values of a, c and pid are the same as previously. However, we notice that b is different from before. This is because future b is evaluated in a different R process and therefore it returns a different process ID. Another difference is that the messages, generated by cat(), are no longer displayed. This is because they are outputted to the background sessions and not the calling session.
When multisession evaluation is used, the package launches a set of R sessions in the background that will serve multisession futures by evaluating their expressions as they are created. If all background sessions are busy serving other futures, the creation of the next multisession future is blocked until a background session becomes available again. The total number of background processes launched is decided by the value of availableCores(), e.g.
> availableCores()
mc.cores
2
This particular result tells us that the mc.cores option was set such that we are allowed to use in total 2 processes including the main process. In other words, with these settings, there will be 2 background processes serving the multisession futures. The availableCores() is also agile to different options and system environment variables. For instance, if compute cluster schedulers are used (e.g. TORQUE/PBS and Slurm), they set specific environment variable specifying the number of cores that was allotted to any given job; availableCores() acknowledges these as well. If nothing else is specified, all available cores on the machine will be utilized, cf. parallel::detectCores(). For more details, please see help("availableCores", package = "future").
Multicore Futures
On operating systems where R supports forking of processes, which is basically all operating system except Windows, an alternative to spawning R sessions in the background is to fork the existing R process. Forking an R process is considered faster than working with a separate R session running in the background. One reason is that the overhead of exporting large globals to the background session can be greater than when forking is used.
To use multicore futures, we specify:
plan(multicore)
The only real different between using multicore and multisession futures is that any output written (to standard output or standard error) by a multicore process is instantaneously outputted in calling process. Other than this, the behavior of using multicore evaluation is very similar to that of using multisession evaluation.
Just like for multisession futures, the maximum number of parallel processes running will be decided by availableCores(), since in both cases the evaluation is done on the local machine.
Multiprocess Futures
Sometimes we do not know whether multicore futures are supported or not, but it might still be that we would like to write platform-independent scripts or instructions that work everywhere. In such cases we can specify that we want to use “multiprocess” futures as in:
plan(multiprocess)
A multiprocess future is not a formal class of futures by itself, but rather a convenient alias for either of the two. When this is specified, multisession evaluation will be used unless multicore evaluation is supported.
Cluster Futures
Cluster futures evaluate expressions on an ad-hoc cluster (as implemented by the parallel package). For instance, assume you have access to three nodes n1, n2 and n3, you can then use these for asynchronous evaluation as:
Just as for most other asynchronous evaluation strategies, the output from cat() is not displayed on the current/calling machine.
Any types of clusters that parallel::makeCluster() creates can be used for cluster futures. For instance, the above cluster can be explicitly set up as:
Also, it is considered good style to shut down cluster cl when it is no longer needed, that is, calling parallel::stopCluster(cl). However, it will shut itself down if the main process is terminated. For more information on how to set up and manage such clusters, see help("makeCluster", package = "parallel").
Clusters created implicitly using plan(cluster, workers = hosts) where hosts is a character vector will also be shut down when the main R session terminates, or when the future strategy is changed, e.g. by calling plan(sequential).
Note that with automatic authentication setup (e.g. SSH key pairs), there is nothing preventing us from using the same approach for using a cluster of remote machines.
Nested Futures and Evaluation Topologies
This far we have discussed what can be referred to as “flat topology” of futures, that is, all futures are created in and assigned to the same environment. However, there is nothing stopping us from using a “nested topology” of futures, where one set of futures may, in turn, create another set of futures internally and so on.
For instance, here is an example of two “top” futures (a and b) that uses multiprocess evaluation and where the second future (b) in turn uses two internal futures:
By inspection the process IDs, we see that there are in total three different processes involved for resolving the futures. There is the main R process (pid 28518), and there are the two processes used by a (pid 28584) and b (pid 28585). However, the two futures (b1 and b2) that is nested by b are evaluated by the same R process as b. This is because nested futures use sequential evaluation unless otherwise specified. There are a few reasons for this, but the main reason is that it protects us from spawning off a large number of background processes by mistake, e.g. via recursive calls.
To specify a different type of evaluation topology, other than the first level of futures being resolved by multiprocess evaluation and the second level by sequential evaluation, we can provide a list of evaluation strategies to plan(). First, the same evaluation strategies as above can be explicitly specified as:
plan(list(multiprocess, sequential))
We would actually get the same behavior if we try with multiple levels of multiprocess evaluations;
> plan(list(multiprocess, multiprocess))
[...]
> pid
[1] 28518
> a
[1] 28586
> b
b.pid b1.pid b2.pid
28587 28587 28587
The reason for this is, also here, to protect us from launching more processes than what the machine can support. Internally, this is done by setting mc.cores = 1 such that functions like parallel::mclapply() will fall back to run sequentially. This is the case for both multisession and multicore evaluation.
Continuing, if we start off by sequential evaluation and then use multiprocess evaluation for any nested futures, we get:
which clearly show that a and b are resolved in the calling process (pid 28518) whereas the two nested futures (b1 and b2) are resolved in two separate R processes (pids 28588 and 28589).
Having said this, it is indeed possible to use nested multiprocess evaluation strategies, if we explicitly specify (read force) the number of cores available at each level. In order to do this we need to “tweak” the default settings, which can be done as follows:
First, we see that both a and b are resolved in different processes (pids 28590 and 28591) than the calling process (pid 28518). Second, the two nested futures (b1 and b2) are resolved in yet two other R processes (pids 28592 and 28594).
For more details on working with nested futures and different evaluation strategies at each level, see Vignette 'Futures in R: Future Topologies'.
Checking A Future without Blocking
It is possible to check whether a future has been resolved or not without blocking. This can be done using the resolved(f) function, which takes an explicit future f as input. If we work with implicit futures (as in all the examples above), we can use the f <- futureOf(a) function to retrieve the explicit future from an implicit one. For example,
> plan(multiprocess)
> a %<-% {
+ cat("Resolving 'a' ...")
+ Sys.sleep(2)
+ cat("done\n")
+ Sys.getpid()
+ }
> cat("Waiting for 'a' to be resolved ...\n")
Waiting for 'a' to be resolved ...
> f <- futureOf(a)
> count <- 1
> while (!resolved(f)) {
+ cat(count, "\n")
+ Sys.sleep(0.2)
+ count <- count + 1
+ }
1
2
3
4
5
> cat("Waiting for 'a' to be resolved ... DONE\n")
Waiting for 'a' to be resolved ... DONE
> a
[1] 28595
Failed Futures
Sometimes the future is not what you expected. If an error occurs while evaluating a future, the error is propagated and thrown as an error in the calling environment when the future value is requested. For example, if we use lazy evaluation on a future that generates an error, we might see something like
> plan(sequential)
> a %<-% {
+ cat("Resolving 'a' ...\n")
+ stop("Whoops!")
+ 42
+ } %lazy% TRUE
> cat("Everything is still ok although we have created a future that will fail.\n")
Everything is still ok although we have created a future that will fail.
> a
Resolving 'a' ...
Error in eval(expr, envir, enclos) : Whoops!
The error is thrown each time the value is requested, that is, if we try to get the value again will generate the same error:
> a
Error in eval(expr, envir, enclos) : Whoops!
In addition: Warning message:
restarting interrupted promise evaluation
To see the list of calls (evaluated expressions) that lead up to the error, we can use the backtrace() function(*) on the future, i.e.
(*) The commonly used traceback() does not provide relevant information in the context of futures.
Globals
Whenever an R expression is to be evaluated asynchronously (in parallel) or sequentially via lazy evaluation, global (aka “free”) objects have to be identified and passed to the evaluator. They need to be passed exactly as they were at the time the future was created, because, for lazy evaluation, globals may otherwise change between when it is created and when it is resolved. For asynchronous processing, the reason globals need to be identified is so that they can be exported to the process that evaluates the future.
The future package tries to automate these tasks as far as possible. It does this with help of the globals package, which uses static-code inspection to identify global variables. If a global variable is identified, it is captured and made available to the evaluating process.
Moreover, if a global is defined in a package, then that global is not exported. Instead, it is made sure that the corresponding package is attached when the future is evaluated. This not only better reflects the setup of the main R session, but it also minimizes the need for exporting globals, which saves not only memory but also time and bandwidth, especially when using remote compute nodes.
Finally, it should be clarified that identifying globals from static code inspection alone is a challenging problem. There will always be corner cases where automatic identification of globals fails so that either false globals are identified (less of a concern) or some of the true globals are missing (which will result in a run-time error or possibly the wrong results). Vignette 'Futures in R: Common Issues with Solutions' provides examples of common cases and explains how to avoid them as well as how to help the package to identify globals or to ignore falsely identified globals. If that does not suffice, it is always possible to manually specify the global variables by their names (e.g. globals = c("a", "slow_sum")) or as name-value pairs (e.g. globals = list(a = 42, slow_sum = my_sum)).
Constraints when using Implicit Futures
There is one limitation with implicit futures that does not exist for explicit ones. Because an explicit future is just like any other object in R it can be assigned anywhere/to anything. For instance, we can create several of them in a loop and assign them to a list, e.g.
> plan(multiprocess)
> f <- list()
> for (ii in 1:3) {
+ f[[ii]] <- future({
+ Sys.getpid()
+ })
+ }
> v <- lapply(f, FUN = value)
> str(v)
List of 3
$ : int 28602
$ : int 28603
$ : int 28605
This is not possible to do when using implicit futures. This is because the %<-% assignment operator cannot be used in all cases where the regular <- assignment operator can be used. It can only be used to assign future values to environments (including the calling environment) much like how assign(name, value, envir) works. However, we can assign implicit futures to environments using named indices, e.g.
> plan(multiprocess)
> v <- new.env()
> for (name in c("a", "b", "c")) {
+ v[[name]] %<-% {
+ Sys.getpid()
+ }
+ }
> v <- as.list(v)
> str(v)
List of 3
$ a: int 28606
$ b: int 28607
$ c: int 28608
Here as.list(v) blocks until all futures in the environment v have been resolved. Then their values are collected and returned as a regular list.
If numeric indices are required, then list environments can be used. List environments, which are implemented by the listenv package, are regular environments with customized subsetting operators making it possible to index them much like how lists can be indexed. By using list environments where we otherwise would use lists, we can also assign implicit futures to list-like objects using numeric indices. For example,
> library("listenv")
> plan(multiprocess)
> v <- listenv()
> for (ii in 1:3) {
+ v[[ii]] %<-% {
+ Sys.getpid()
+ }
+ }
> v <- as.list(v)
> str(v)
List of 3
$ : int 28609
$ : int 28610
$ : int 28611
As previously, as.list(v) blocks until all futures are resolved.
Demos
To see a live illustration how different types of futures are evaluated, run the Mandelbrot demo of this package. First, try with the sequential evaluation,
which resembles how the script would run if futures were not used. Then, try multiprocess evaluation, which calculates the different Mandelbrot planes using parallel R processes running in the background. Try,
The goal of this package is to provide a standardized and unified API for using futures in R. What you are seeing right now is an early but sincere attempt to achieve this goal. If you have comments or ideas on how to improve the 'future' package, I would love to hear about them. The preferred way to get in touch is via the GitHub repository, where you also find the latest source code. I am also open to contributions and collaborations of any kind.
Copyright Henrik Bengtsson, 2015-2017
future/inst/doc/future-4-startup.md.rsp 0000644 0001762 0000144 00000005775 13111756521 017626 0 ustar ligges users <%@meta language="R-vignette" content="--------------------------------
%\VignetteIndexEntry{A Future for R: Controlling Default Future Strategy}
%\VignetteAuthor{Henrik Bengtsson}
%\VignetteKeyword{R}
%\VignetteKeyword{package}
%\VignetteKeyword{vignette}
%\VignetteKeyword{future}
%\VignetteKeyword{promise}
%\VignetteEngine{R.rsp::rsp}
%\VignetteTangle{FALSE}
--------------------------------------------------------------------"%>
<%
library("R.utils")
`%<-%` <- future::`%<-%`
options("withCapture/newline" = FALSE)
%>
# <%@meta name="title"%>
The default is to use synchronous futures, but this _default_ can be overridden via R options, system environment variables and command-line options as explained below as well as in `help("future.options", package = "future")`.
## R options
The default strategy for resolving futures can be controlled via R option `future.plan`. For instance, if we add
```r
options(future.plan = "multiprocess")
```
to our `~/.Rprofile` startup script, the future package will resolve futures in parallel (asynchronously using all available cores), i.e.
```sh
$ Rscript -e "class(future::plan())"
[1] "multiprocess" "future" "function"
```
Option `future.plan` is ignored if command-line option `--parallel` (`-p`) is specified.
## Environment variables
An alternative to using `options()` for setting option `future.plan` is to specify system environment variable `R_FUTURE_PLAN`. If set, then the future package will set `future.plan` accordingly _when loaded_. For example,
```sh
$ export R_FUTURE_PLAN=multiprocess
$ Rscript -e "class(future::plan())"
[1] "multiprocess" "future" "function"
```
Environment variable `R_FUTURE_PLAN` is ignored if either option `future.plan` or command-line option `--parallel` (`-p`) is specified.
## Command-line options
When loaded, the future package checks for the command-line option `--parallel=ncores` (short `-p ncores`) and sets the future strategy (via option `future.plan`) and the number of available cores (via option `mc.cores`) accordingly. This provides a convenient mechanism for specifying parallel future processing from the command line. For example, if we start R with
```sh
$ R --quiet --args --parallel=2
```
then future will interpret this as we wish to resolve futures in parallel using 2 cores. More specifically, we get that
```r
> future::availableCores()
mc.cores
2
> class(future::plan())
[1] "tweaked" "multiprocess" "future" "function"
```
We can use this command-line option also with `Rscript`, which provides a convenient mechanism for launching future-enhanced R scripts such that they run in parallel, e.g.
```sh
$ Rscript analysis.R --parallel=4
```
This does, of course, require that the script uses futures and the future package.
If `--parallel=1` is specified, or equivalently `-p 1`, then futures are resolved using a single process.
Specifying these command-line options override any other startup settings.
[future]: https://cran.r-project.org/package=future
---
Copyright Henrik Bengtsson, 2015-2017
future/tests/ 0000755 0001762 0000144 00000000000 13237670254 012737 5 ustar ligges users future/tests/as.cluster.R 0000644 0001762 0000144 00000003535 13111756706 015151 0 ustar ligges users source("incl/start.R")
stopCluster <- parallel::stopCluster
message("*** cluster operations ...")
local({
cl0 <- makeClusterPSOCK(1L)
on.exit(stopCluster(cl0))
cl <- cl0
print(cl)
message("*** cluster operations - as.cluster() ...")
cl1 <- as.cluster(cl)
print(cl1)
stopifnot(inherits(cl1, "cluster"), identical(cl1, cl))
node <- cl[[1]]
print(node)
cl2 <- as.cluster(cl)
stopifnot(inherits(cl2, "cluster"), length(cl2) == 1L,
identical(cl2[[1]], node))
node <- cl[[1]]
print(node)
stopifnot(inherits(node, "SOCKnode"))
nodes <- list(node, node)
cl3 <- as.cluster(node)
print(cl3)
stopifnot(inherits(cl3, "cluster"), length(cl3) == 1L,
identical(cl3[[1]], node))
cl4 <- as.cluster(nodes)
print(cl4)
stopifnot(inherits(cl4, "cluster"), length(cl4) == 2L,
identical(cl4[[1]], node), identical(cl4[[2]], node))
message("*** cluster operations - as.cluster() ... DONE")
message("*** cluster operations - c(...) ...")
cl2 <- makeClusterPSOCK("localhost")
on.exit(stopCluster(cl2), add = TRUE)
print(cl2)
cl <- c(cl1, cl2)
print(cl)
stopifnot(inherits(cl, "cluster"), length(cl) == 2L)
stopifnot(identical(cl[1], cl1),
identical(cl[2], cl2[1]))
message("*** cluster operations - c(...) ... DONE")
})
message("*** cluster operations - makeClusterPSOCK(remotes) ...")
remotes <- Sys.getenv("R_FUTURE_TESTS_REMOTES", "")
remotes <- gsub(" ", "", unlist(strsplit(remotes, split = ",")))
remotes <- remotes[nzchar(remotes)]
if (length(remotes) > 0) {
message("Remotes: ", paste(sQuote(remotes), collapse = ", "))
local({
cl <- makeClusterPSOCK(remotes, verbose = TRUE)
on.exit(stopCluster(cl))
print(cl)
})
}
message("*** cluster operations - makeClusterPSOCK(remotes) ... DONE")
message("*** cluster operations ... DONE")
source("incl/end.R")
future/tests/non-exportable,connections.R 0000644 0001762 0000144 00000001746 13237667011 020343 0 ustar ligges users source("incl/start.R")
message("*** Non-exportable globals ...")
options(future.globals.onReference = "warning")
plan(multisession, workers = 2L)
message("* R connections ...")
tmp_file <- tempfile()
con <- file(tmp_file, open = "wb")
cat("hello\n", file = con)
flush(con)
bfr <- readLines(con = tmp_file)
print(bfr)
stopifnot(bfr == "hello")
message("- Run-time error")
## Assert we can detect the reference
res <- tryCatch({
f <- future(cat("world\n", file = con))
}, FutureWarning = identity)
print(res)
stopifnot(inherits(res, "FutureWarning"),
grepl("non-exportable reference", conditionMessage(res)))
f <- future(cat("world\n", file = con))
res <- tryCatch({
v <- value(f)
}, error = identity)
print(res)
stopifnot(inherits(res, "error"))
## Nothing changed
bfr <- readLines(con = tmp_file)
print(bfr)
stopifnot(bfr == "hello")
close(con)
file.remove(tmp_file)
message("* R connections ... DONE")
message("*** Non-exportable globals ... DONE")
source("incl/end.R")
future/tests/fold.R 0000644 0001762 0000144 00000002713 13237667011 014006 0 ustar ligges users source("incl/start,load-only.R")
message("*** fold() ...")
x1s <- list(
a = NULL,
b = 1,
c = c(a = 1, b = 2),
d = 1:10e3
)
x2s <- lapply(x1s, FUN = as.list)
names(x2s) <- toupper(names(x1s))
x3s <- list(
E = data.frame(a = 1:3),
F = data.frame(a = 1:3, b = letters[1:3])
)
xs <- c(x1s, x2s, x3s)
fcns <- list("c" = base::c, "cbind" = base::cbind)
for (kk in seq_along(xs)) {
x_name <- names(xs)[kk]
for (fcn_name in names(fcns)) {
fcn <- fcns[[fcn_name]]
message(sprintf(" - #%d. %s(x[['%s']]) ...", kk, fcn_name, x_name))
x <- xs[[kk]]
str(list(x = x))
y0 <- Reduce(x, f = fcn)
y1 <- fold(x, f = fcn)
y2 <- fold(x, f = fcn, unname = FALSE)
str(list(y0 = y0, y1 = y1, y2 = y2))
stopifnot(all.equal(unname(y1), unname(y0)))
stopifnot(all.equal(unname(y2), unname(y0)))
if (!fcn_name %in% "cbind") {
stopifnot(all.equal(y1, y0))
stopifnot(all.equal(y2, y0))
}
y0 <- Reduce(x, f = fcn, right = TRUE)
y1 <- fold(x, f = fcn, left = FALSE)
y2 <- fold(x, f = fcn, left = FALSE, unname = FALSE)
str(list(y0 = y0, y1 = y1, y2 = y2))
stopifnot(all.equal(unname(y1), unname(y0)))
stopifnot(all.equal(unname(y2), unname(y0)))
if (!fcn_name %in% "cbind") {
stopifnot(all.equal(y1, y0))
stopifnot(all.equal(y2, y0))
}
message(sprintf(" - #%d. %s(x[['%s']]) ... DONE", kk, fcn_name, x_name))
}
}
message("*** fold() ... DONE")
source("incl/end.R")
future/tests/futures.R 0000644 0001762 0000144 00000005015 13111756706 014556 0 ustar ligges users source("incl/start.R")
library("listenv")
## Backward compatibility
if (getRversion() < "3.2.0") {
names <- function(x) {
if (class(x)[1] == "environment") {
ls(envir = x, all.names = TRUE)
} else {
base::names(x)
}
}
}
dims <- list(
NULL,
c(1, 6),
c(2, 3),
c(2, 3, 1),
c(2, 1, 3, 1)
)
message("*** futures() / resolved() / values() ...")
for (cores in 1:availCores) {
message(sprintf("Testing with %d cores ...", cores))
options(mc.cores = cores)
for (type in c("list", "environment", "listenv")) {
message(sprintf("Type of object: %s", type))
for (strategy in supportedStrategies(cores, excl = "multiprocess")) {
message("Type of future: ", strategy)
plan(strategy)
for (dim in dims) {
message("Dimensions: ", deparse(dim))
if (type == "list") {
x <- list()
} else if (type == "listenv") {
x <- listenv()
} else if (type == "environment") {
x <- new.env()
}
x$a <- 1
x$b <- future(2)
x$c <- future(NULL)
if (type != "list") x$d %<-% { 4 }
if (type != "environment") x[[6]] <- 6
str(x)
if (!is.null(dim)) {
if (type != "environment") {
names <- names(x)
dim(x) <- dim
dimnames(x) <- lapply(dim, FUN = function(n) letters[1:n])
names(x) <- names
}
}
f <- futures(x)
str(f)
if (type != "environment") {
stopifnot(length(f) == length(x))
stopifnot(identical(names(f), names(x)))
}
stopifnot(identical(dim(f), dim(x)))
stopifnot(identical(dimnames(f), dimnames(x)))
r <- resolved(x)
str(r)
if (type != "environment") {
stopifnot(length(r) == length(x))
stopifnot(identical(names(r), names(x)))
}
stopifnot(identical(dim(r), dim(x)))
stopifnot(identical(dimnames(r), dimnames(x)))
v <- values(x)
str(v)
if (type != "environment") {
stopifnot(length(v) == length(x))
stopifnot(identical(names(v), names(x)))
}
stopifnot(identical(dim(v), dim(x)))
stopifnot(identical(dimnames(v), dimnames(x)))
} # for (dim ...)
} # for (strategy ...)
message(sprintf("*** futures() - %s ... DONE", type))
} # for (type ...)
message(sprintf("Testing with %d cores ... DONE", cores))
} ## for (cores ...)
message("*** futures() / resolved() / values() ... DONE")
source("incl/end.R")
future/tests/utils.R 0000644 0001762 0000144 00000015365 13237667011 014231 0 ustar ligges users source("incl/start,load-only.R")
message("*** utils ...")
message("*** hpaste() ...")
# Some vectors
x <- 1:6
y <- 10:1
z <- LETTERS[x]
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# Abbreviation of output vector
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
printf("x = %s.\n", hpaste(x))
## x = 1, 2, 3, ..., 6.
printf("x = %s.\n", hpaste(x, maxHead = 2))
## x = 1, 2, ..., 6.
printf("x = %s.\n", hpaste(x), maxHead = 3) # Default
## x = 1, 2, 3, ..., 6.
# It will never output 1, 2, 3, 4, ..., 6
printf("x = %s.\n", hpaste(x, maxHead = 4))
## x = 1, 2, 3, 4, 5 and 6.
# Showing the tail
printf("x = %s.\n", hpaste(x, maxHead = 1, maxTail = 2))
## x = 1, ..., 5, 6.
# Turning off abbreviation
printf("y = %s.\n", hpaste(y, maxHead = Inf))
## y = 10, 9, 8, 7, 6, 5, 4, 3, 2, 1
## ...or simply
printf("y = %s.\n", paste(y, collapse = ", "))
## y = 10, 9, 8, 7, 6, 5, 4, 3, 2, 1
# Change last separator
printf("x = %s.\n", hpaste(x, lastCollapse = " and "))
## x = 1, 2, 3, 4, 5 and 6.
# No collapse
stopifnot(all(hpaste(x, collapse = NULL) == x))
# Empty input
stopifnot(identical(hpaste(character(0)), character(0)))
message("*** hpaste() ...")
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# asIEC()
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
message("*** asIEC() ...")
for (size in c(0, 10 ^ (0:20))) {
cat(sprintf("Size: %.f bytes = %s\n", size, asIEC(size)))
}
message("*** asIEC() ... DONE")
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# .length()
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
message("*** .length() ...")
.length <- future:::.length
objs <- list(
a = 1:3,
b = as.list(1:3),
c = structure(as.list(1:3), class = c("foo", "list")),
d = data.frame(a = 1:3),
e = as.environment(list(a = 1:3))
)
truth <- c(a = 3L, b = 3L, c = 3L, d = 1L, e = 1L)
## Special case: length(x) == 5, but .length(x) == 2
## BUG FIX: https://github.com/HenrikBengtsson/future/issues/164
if (requireNamespace("tools")) {
objs[["f"]] <- structure(list("foo", length = 5L), class = "pdf_doc")
truth["f"] <- 2L
}
for (name in names(objs)) {
obj <- objs[[name]]
len <- length(obj)
.len <- .length(obj)
cat(sprintf("%s: length = %d, .length = %d, expected = %d\n",
name, len, .len, truth[name]))
stopifnot(.len == truth[name])
}
message("*** .length() ... DONE")
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# debug()
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
message("*** mdebug() ...")
mdebug("Hello #1")
options(future.debug = TRUE)
mdebug("Hello #2")
options(future.debug = FALSE)
mdebug("Hello #3")
message("*** mdebug() ... DONE")
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# geval() et al.
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
message("*** geval() et al. ...")
gls <- function(..., envir = .GlobalEnv) ls(..., envir = envir)
message("- gls() ...")
genv <- new.env(parent = globalenv())
vars <- gls(envir = genv)
print(vars)
stopifnot(length(vars) == 0)
message("- gassign() ...")
gassign("a", 1, envir = genv)
vars <- gls(envir = genv)
print(vars)
stopifnot(length(vars) == 1)
message("- grmall() ...")
grmall(envir = genv)
vars <- gls(envir = genv)
print(vars)
stopifnot(length(vars) == 0)
message("- geval() ...")
gassign("a", 1, envir = genv)
res <- geval(substitute(a), envir = genv)
print(res)
vars <- gls(envir = genv)
print(vars)
stopifnot(length(vars) == 1)
message("*** geval() et al. ... DONE")
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# requirePackages()
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
message("*** requirePackages() ...")
res <- requirePackages("future")
res <- requirePackages(c("future", "listenv"))
res <- try(requirePackages(""), silent = TRUE)
stopifnot(inherits(res, "try-error"))
message("*** requirePackages() ... DONE")
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# importParallel()
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
message("*** importParallel() ...")
mclapply <- importParallel("mclapply")
stopifnot(identical(mclapply, parallel::mclapply))
ns <- getNamespace("parallel")
if (exists("sendCall", envir = ns, mode = "function")) {
sendCall <- importParallel("sendCall")
stopifnot(identical(sendCall, parallel:::sendCall))
} else {
res <- try(importParallel("sendCall"), silent = TRUE)
stopifnot(inherits(res, "try-error"))
}
res <- try(importParallel(""), silent = TRUE)
stopifnot(inherits(res, "try-error"))
message("*** importParallel() ... DONE")
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# Random seeds
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
message("*** Random seeds ...")
set_random_seed(seed = NULL)
seed <- get_random_seed()
stopifnot(is.null(seed))
set_random_seed(seed = 42L)
seed <- get_random_seed()
stopifnot(identical(seed, 42L))
res <- tryCatch({
seed <- as_lecyer_cmrg_seed(seed = FALSE)
}, error = identity)
print(res)
stopifnot(inherits(res, "simpleError"))
seed <- as_lecyer_cmrg_seed(seed = 42L)
str(seed)
set_random_seed(seed = seed)
stopifnot(identical(get_random_seed(), seed))
seed2 <- as_lecyer_cmrg_seed(seed = TRUE)
str(seed2)
stopifnot(identical(seed2, seed))
seed3 <- as_lecyer_cmrg_seed(seed = seed)
str(seed3)
stopifnot(identical(seed3, seed))
## Invalid L'Ecuyer seed
seed_invalid <- seed + 1L
res <- tryCatch({
seed <- as_lecyer_cmrg_seed(seed = seed_invalid)
}, error = identity)
print(res)
stopifnot(inherits(res, "simpleError"))
## Invalid seed
res <- tryCatch({
seed <- as_lecyer_cmrg_seed(seed = 1:2)
}, error = identity)
print(res)
stopifnot(inherits(res, "simpleError"))
message("*** Random seeds ... DONE")
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# myInternalIP() and myExternalIP()
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
message("*** myInternalIP() ...")
ips <- myInternalIP(mustWork = FALSE)
message("myInternalIP(): ", paste(ips, collapse = ", "))
message("*** myInternalIP() ... DONE")
ips <- myInternalIP(force = TRUE, which = "first", mustWork = FALSE)
message("myInternalIP(which = 'first'): ", paste(ips, collapse = ", "))
message("*** myInternalIP() ... DONE")
ips <- myInternalIP(force = TRUE, which = "last", mustWork = FALSE)
message("myInternalIP(which = 'last'): ", paste(ips, collapse = ", "))
message("*** myInternalIP() ... DONE")
ips <- myInternalIP(force = TRUE, which = "all", mustWork = FALSE)
message("myInternalIP(which = 'all'): ", paste(ips, collapse = ", "))
message("*** myInternalIP() ... DONE")
message("*** myExternalIP() ...")
ip <- myExternalIP(mustWork = FALSE)
message("myExternalIP(): ", ip)
message("*** myExternalIP() ... DONE")
message("*** utils ... DONE")
source("incl/end.R")
future/tests/globals,manual.R 0000644 0001762 0000144 00000014153 13237667011 015760 0 ustar ligges users source("incl/start.R")
message("*** getGlobalsAndPackages() ...")
FutureGlobals <- future:::FutureGlobals
globals <- structure(list(a = 1), where = list(a = globalenv()))
globals <- FutureGlobals(globals, resolved = TRUE)
gp <- getGlobalsAndPackages(expression(), globals = globals)
message("- getGlobalsAndPackages() - exception ...")
res <- tryCatch({
gp <- getGlobalsAndPackages(expression(), globals = 42)
}, error = identity)
stopifnot(inherits(res, "error"))
message("*** getGlobalsAndPackages() - ... DONE")
message("*** Globals - manually ...")
message("*** Globals manually specified as named list ...")
globals <- list(
a = 1,
b = 2,
sumtwo = function(x) x[1] + x[2]
)
## Assign 'globals' globally
attachLocally(globals)
## Truth
v0 <- local({
x <- 1:10
sumtwo(a + b * x)
})
message("*** Globals & packages - automatic ...")
for (strategy in supportedStrategies()) {
message(sprintf("- Strategy: %s ...", strategy))
plan(strategy)
message("- Globals - automatic ...")
attachLocally(globals)
f <- future({
x <- 1:10
sumtwo(a + b * x)
}, globals = TRUE)
print(f)
rm(list = names(globals))
y <- value(f)
print(y)
stopifnot(all.equal(y, v0))
attachLocally(globals)
f <- futureAssign("y", {
x <- 1:10
sumtwo(a + b * x)
}, globals = TRUE)
print(f)
rm(list = names(globals))
z <- value(f)
print(z)
stopifnot(all.equal(z, y), all.equal(y, v0))
attachLocally(globals)
y %<-% {
x <- 1:10
sumtwo(a + b * x)
} %globals% TRUE
rm(list = names(globals))
print(y)
stopifnot(all.equal(y, v0))
attachLocally(globals)
f <- future({
x <- 1:10
sumtwo(a + b * x)
}, lazy = TRUE, globals = TRUE)
print(f)
rm(list = names(globals))
y <- value(f)
print(y)
stopifnot(all.equal(y, v0))
attachLocally(globals)
f <- futureAssign("y", {
x <- 1:10
sumtwo(a + b * x)
}, lazy = TRUE, globals = TRUE)
print(f)
rm(list = names(globals))
z <- value(f)
print(z)
stopifnot(all.equal(z, y), all.equal(y, v0))
## Same with lazy evaluation
attachLocally(globals)
y %<-% {
x <- 1:10
sumtwo(a + b * x)
} %lazy% TRUE %globals% TRUE
rm(list = names(globals))
print(y)
stopifnot(all.equal(y, v0))
## No need to search for globals
y %<-% { 1 } %globals% FALSE
print(y)
stopifnot(identical(y, 1))
## Same with lazy evaluation
y %<-% { 1 } %lazy% TRUE %globals% FALSE
print(y)
stopifnot(identical(y, 1))
## Exception - missing global
attachLocally(globals)
f <- future({
x <- 1:10
sumtwo(a + b * x)
}, globals = FALSE)
print(f)
rm(list = names(globals))
y <- tryCatch(value(f), error = identity)
if (!inherits(f, c("SequentialFuture", "UniprocessFuture", "MulticoreFuture"))) {
stopifnot(inherits(y, "simpleError"))
}
message("- Globals manually specified as named list ...")
## Make sure globals do not exist
rm(list = names(globals))
f <- future({
x <- 1:10
sumtwo(a + b * x)
}, globals = globals)
print(f)
v <- value(f)
print(v)
stopifnot(all.equal(v, v0))
f <- future({
x <- 1:10
sumtwo(a + b * x)
}, lazy = TRUE, globals = globals)
print(f)
v <- value(f)
print(v)
stopifnot(all.equal(v, v0))
y %<-% {
x <- 1:10
sumtwo(a + b * x)
} %globals% globals
print(y)
stopifnot(all.equal(y, v0))
y %<-% {
x <- 1:10
sumtwo(a + b * x)
} %lazy% TRUE %globals% globals
print(y)
stopifnot(all.equal(y, v0))
message("- Globals manually specified by their names ...")
attachLocally(globals)
f <- future({
x <- 1:10
sumtwo(a + b * x)
}, globals = c("a", "b", "sumtwo"))
print(f)
rm(list = names(globals))
v <- value(f)
print(v)
stopifnot(all.equal(v, v0))
attachLocally(globals)
f <- future({
x <- 1:10
sumtwo(a + b * x)
}, lazy = TRUE, globals = c("a", "b", "sumtwo"))
print(f)
rm(list = names(globals))
v <- value(f)
print(v)
stopifnot(all.equal(v, v0))
attachLocally(globals)
y %<-% {
x <- 1:10
sumtwo(a + b * x)
} %globals% c("a", "b", "sumtwo")
rm(list = names(globals))
print(y)
stopifnot(all.equal(y, v0))
attachLocally(globals)
y %<-% {
x <- 1:10
sumtwo(a + b * x)
} %lazy% TRUE %globals% c("a", "b", "sumtwo")
rm(list = names(globals))
print(y)
stopifnot(all.equal(y, v0))
message("- Globals manually specified as named list - also with '...' ...")
x <- 1:10
y_truth <- x[2:3]
str(y_truth)
## Make sure it's possible to specify '...' as a global
sub <- function(x, ...) value(future(x[...], globals = c("x", "...")))
y <- sub(x, 2:3)
str(y)
stopifnot(identical(y, y_truth))
## Make sure it's possible to specify '...' as a global (not just last)
## Requires globals (> 0.11.0)
sub <- function(x, ...) value(future(x[...], globals = c("...", "x")))
if (packageVersion("globals") > "0.11.0")
y <- sub(x, 2:3)
str(y)
stopifnot(identical(y, y_truth))
## And if '...' is forgotten, it may give an error
sub <- function(x, ...) value(future(x[...], globals = "x"))
y <- tryCatch(sub(x, 2:3), error = identity)
str(y)
stopifnot((strategy %in% c("multisession") && inherits(y, "FutureError")) || identical(y, y_truth))
message("- Packages - manual ...")
## Make sure 'iris', and thereby the 'datasets' package,
## is not picked up as a global
unloadNamespace("datasets")
stopifnot(!"dataset" %in% loadedNamespaces(), !exists("iris", mode = "list"))
ns %<-% {
unloadNamespace("datasets")
loadedNamespaces()
}
print(ns)
stopifnot(!"dataset" %in% ns)
res <- tryCatch({
f <- future({ iris })
v <- value(f)
print(head(v))
}, error = identity)
stopifnot(inherits(res, "error"))
f <- future({ iris }, packages = "datasets")
v <- value(f)
print(head(v))
ns %<-% {
unloadNamespace("datasets")
loadedNamespaces()
}
print(ns)
stopifnot(!"dataset" %in% ns)
res <- tryCatch({
df %<-% { iris }
print(head(df))
}, error = identity)
stopifnot(inherits(res, "error"))
df %<-% { iris } %packages% "datasets"
print(head(df))
message(sprintf("- Strategy: %s ... DONE", strategy))
}
message("*** Globals - manually ... DONE")
source("incl/end.R")
future/tests/future,labels.R 0000644 0001762 0000144 00000001557 13111756706 015641 0 ustar ligges users source("incl/start.R")
message("*** Futures - labels ...")
for (strategy in supportedStrategies()) {
message(sprintf("- plan('%s') ...", strategy))
plan(strategy)
for (label in list(NULL, sprintf("strategy = %s", strategy))) {
fcn <- get(strategy, mode = "function")
stopifnot(inherits(fcn, strategy))
f <- fcn(42, label = label)
print(f)
stopifnot(identical(f$label, label))
v <- value(f)
stopifnot(v == 42)
f <- future(42, label = label)
print(f)
stopifnot(identical(f$label, label))
v <- value(f)
stopifnot(v == 42)
v %<-% { 42 } %label% label
f <- futureOf(v)
print(f)
stopifnot(identical(f$label, label))
stopifnot(v == 42)
} ## for (label ...)
message(sprintf("- plan('%s') ... DONE", strategy))
} ## for (strategy ...)
message("*** Futures - labels ... DONE")
source("incl/end.R")
future/tests/mandelbrot.R 0000644 0001762 0000144 00000000421 13111756521 015177 0 ustar ligges users source("incl/start.R")
message("mandelbrot() ...")
counts <- mandelbrot(xmid = -0.75, ymid = 0.0, side = 3.0, resolution = 100L)
img <- as.raster(counts)
if (getRversion() >= "3.2.0") {
plot(img)
plot(counts)
}
message("mandelbrot() ... DONE")
source("incl/end.R")
future/tests/rng.R 0000644 0001762 0000144 00000005307 13111756521 013646 0 ustar ligges users source("incl/start.R")
message("*** rng ...")
## A valid L'Ecuyer-CMRG RNG seed
seed <- c(407L, 1420090545L, 65713854L, -990249945L,
1780737596L, -1213437427L, 1082168682L)
f <- Future(42, seed = seed)
print(f)
## See Section 6 on 'Random-number generation' in
## vignette("parallel", package = "parallel")
fsample <- function(x, size = 4L, seed = NULL, what = c("future", "%<-%")) {
what <- match.arg(what)
## Must use session-specific '.GlobalEnv' here
.GlobalEnv <- globalenv()
oseed <- .GlobalEnv$.Random.seed
orng <- RNGkind("L'Ecuyer-CMRG")[1L]
on.exit(RNGkind(orng))
if (!is.null(seed)) {
## Reset state of random seed afterwards?
on.exit({
if (is.null(oseed)) {
rm(list = ".Random.seed", envir = .GlobalEnv, inherits = FALSE)
} else {
.GlobalEnv$.Random.seed <- oseed
}
}, add = TRUE)
set.seed(seed)
}
.seed <- .Random.seed
if (what == "future") {
fs <- list()
for (ii in seq_len(size)) {
.seed <- parallel::nextRNGStream(.seed)
fs[[ii]] <- future({ sample(x, size = 1L) }, seed = .seed)
}
res <- values(fs)
} else {
res <- listenv::listenv()
for (ii in seq_len(size)) {
.seed <- parallel::nextRNGStream(.seed)
res[[ii]] %<-% { sample(x, size = 1L) } %seed% .seed
}
res <- as.list(res)
}
res
} # fsample()
dummy <- sample(0:9, size = 1L)
seed0 <- .Random.seed
## Reference sample with fixed random seed
plan("sequential")
y0 <- fsample(0:9, seed = 42L)
## Assert that random seed is reset
stopifnot(identical(.GlobalEnv$.Random.seed, seed0))
for (cores in 1:availCores) {
message(sprintf("Testing with %d cores ...", cores))
options(mc.cores = cores)
for (strategy in supportedStrategies(cores)) {
message(sprintf("%s ...", strategy))
plan(strategy)
for (what in c("future", "%<-%")) {
.GlobalEnv$.Random.seed <- seed0
## Fixed random seed
y1 <- fsample(0:9, seed = 42L, what = what)
print(y1)
stopifnot(identical(y1, y0))
## Assert that random seed is reset
stopifnot(identical(.GlobalEnv$.Random.seed, seed0))
## Fixed random seed
y2 <- fsample(0:9, seed = 42L, what = what)
print(y2)
stopifnot(identical(y2, y1))
stopifnot(identical(y2, y0))
## Assert that random seed is reset
stopifnot(identical(.GlobalEnv$.Random.seed, seed0))
## No seed
y3 <- fsample(0:9, what = what)
print(y3)
## No seed
y4 <- fsample(0:9, what = what)
print(y4)
}
message(sprintf("%s ... done", strategy))
}
message(sprintf("Testing with %d cores ... DONE", cores))
} ## for (cores ...)
message("*** rng ... DONE")
source("incl/end.R")
future/tests/000.sessionDetails.R 0000644 0001762 0000144 00000000146 13111756521 016343 0 ustar ligges users library("future")
sd <- sessionDetails()
print(sd)
print(sd, output = "message")
rm(list = "sd")
future/tests/objectSize.R 0000644 0001762 0000144 00000002440 13237651433 015161 0 ustar ligges users source("incl/start.R")
objectSize <- future:::objectSize
message("objectSize() ...")
env <- new.env()
env$a <- 3.14
env$b <- 1:100
env2 <- new.env()
env2$env <- env
## Namespaces will be skipped
env3 <- getNamespace("utils")
fcn <- function(...) TRUE
objs <- list(
NULL,
TRUE,
1L,
3.14,
"hello",
1:100,
1:100 + 0.1,
letters,
list(a = 3.14, b = 1:100),
list(a = 3.14, b = 1:100, c = list(a = 3.14, b = 1:100)),
env,
env2,
env3,
fcn,
as.FutureGlobals(list(a = 3.14, b = 1:100)),
list(x = as.FutureGlobals(list(a = 3.14, b = 1:100)))
)
for (kk in seq_along(objs)) {
obj <- objs[[kk]]
message(sprintf("objectSize(<%s>) ...", mode(obj)))
str(obj)
size0 <- object.size(obj)
str(size0)
size <- objectSize(obj)
str(size)
message(sprintf("objectSize(<%s>) ... DONE", mode(obj)))
}
message("*** objectSize() - globals with non-trustful length() ...")
length.CantTrustLength <- function(x) length(unclass(x)) + 1L
.length <- future:::.length
x <- structure(as.list(1:3), class = c("CantTrustLength", "list"))
str(list(n = length(x), n_true = .length(x)))
stopifnot(length(x) > .length(x))
size <- objectSize(x)
print(size)
message("*** objectSize() - globals with non-trustful length() ... DONE")
message("objectSize() ... DONE")
source("incl/end.R")
future/tests/nested_futures.R 0000644 0001762 0000144 00000005163 13111756706 016124 0 ustar ligges users source("incl/start.R")
strategies <- supportedStrategies()
message("*** Nested futures ...")
for (strategy1 in strategies) {
for (strategy2 in strategies) {
message(sprintf("- plan(list('%s', '%s')) ...", strategy1, strategy2))
plan(list(a = strategy1, b = strategy2))
nested <- plan("list")
stopifnot(
length(nested) == 2L,
all(names(nested) == c("a", "b")),
inherits(plan(), strategy1)
)
x %<-% {
a <- 1L
## IMPORTANT: Use future::plan() - not just plan() - otherwise
## we're exporting the plan() function including its local stack!
plan_a <- unclass(future::plan("list"))
nested_a <- nested[-1]
stopifnot(
length(nested_a) == 1L,
length(plan_a) == 1L,
inherits(plan_a[[1]], "future"),
inherits(future::plan(), strategy2)
)
## Attribute 'init' is modified at run time
for (kk in seq_along(plan_a)) attr(plan_a[[kk]], "init") <- NULL
for (kk in seq_along(nested_a)) attr(nested_a[[kk]], "init") <- NULL
stopifnot(all.equal(plan_a, nested_a))
y %<-% {
b <- 2L
## IMPORTANT: Use future::plan() - not just plan() - otherwise
## we're exporting the plan() function including its local stack!
plan_b <- future::plan("list")
nested_b <- nested_a[-1]
stopifnot(
length(nested_b) == 0L,
length(plan_b) == 1L,
inherits(plan_b[[1]], "future"),
inherits(future::plan(), getOption("future.default", "sequential"))
)
list(a = a, nested_a = nested_a, plan_a = plan_a,
b = b, nested_b = nested_b, plan_b = plan_b)
}
y
}
str(x)
stopifnot(
length(x) == 3 * length(nested),
all(names(x) == c("a", "nested_a", "plan_a",
"b", "nested_b", "plan_b")),
x$a == 1L,
length(x$nested_a) == 1L,
is.list(x$plan_a),
length(x$plan_a) == 1L,
inherits(x$plan_a[[1]], "future"),
x$b == 2L,
length(x$nested_b) == 0L,
is.list(x$plan_b),
length(x$plan_b) == 1L,
inherits(x$plan_b[[1]], "future"),
inherits(x$plan_b[[1]], getOption("future.default", "sequential"))
)
## Attribute 'init' is modified at run time
for (kk in seq_along(x$plan_a)) attr(x$plan_a[[kk]], "init") <- NULL
for (kk in seq_along(nested)) attr(nested[[kk]], "init") <- NULL
stopifnot(all.equal(x$plan_a, nested[-1L]))
rm(list = c("nested", "x"))
message(sprintf("- plan(list('%s', '%s')) ... DONE", strategy1, strategy2))
}
}
message("*** Nested futures ... DONE")
source("incl/end.R")
future/tests/resolve.R 0000644 0001762 0000144 00000021314 13111756706 014540 0 ustar ligges users source("incl/start.R")
library("listenv")
oopts <- c(oopts, options(future.progress = TRUE))
strategies <- supportedStrategies()
message("*** resolve() ...")
message("*** resolve() for default ...")
x <- 1
y <- resolve(x)
stopifnot(identical(y, x))
message("*** resolve() for default ... DONE")
message("*** resolve() for Future objects ...")
plan(multisession, workers = 2L)
for (value in c(FALSE, TRUE)) {
for (recursive in list(FALSE, TRUE, -1, 0, 1, 2, Inf)) {
message(sprintf("- value = %s, recursive = %s ...", value, recursive))
f <- future({
Sys.sleep(0.5)
list(a = 1, b = 42L)
})
res <- resolve(f, value = value, recursive = recursive)
stopifnot(identical(res, f))
f <- future({
Sys.sleep(0.5)
list(a = 1, b = 42L)
}, lazy = TRUE)
res <- resolve(f, value = value, recursive = recursive)
stopifnot(identical(res, f))
message("- w/ exception ...")
f <- future(list(a = 1, b = 42L, c = stop("Nah!")))
res <- resolve(f, value = value, recursive = recursive)
stopifnot(identical(res, f))
f <- future(list(a = 1, b = 42L, c = stop("Nah!")), lazy = TRUE)
res <- resolve(f, value = value, recursive = recursive)
stopifnot(identical(res, f))
message(sprintf("- value = %s, recursive = %s ... DONE", value, recursive))
} ## for (resolve ...)
} ## for (value ...)
message("- exception ... DONE")
message("*** resolve() for Future objects ... DONE")
message("*** resolve() for lists ...")
for (strategy in strategies) {
message(sprintf("- plan('%s') ...", strategy))
plan(strategy)
x <- list()
y <- resolve(x)
stopifnot(identical(y, x))
x <- list()
x$a <- 1
x$b <- 2
y <- resolve(x)
stopifnot(identical(y, x))
x <- list()
x$a <- future(1)
x$b <- future(2)
x[[3]] <- 3
y <- resolve(x)
stopifnot(identical(y, x))
stopifnot(resolved(x$a))
stopifnot(resolved(x[["b"]]))
x <- list()
x$a <- future(1, lazy = TRUE)
x$b <- future(2)
x[[3]] <- 3
y <- resolve(x)
stopifnot(identical(y, x))
stopifnot(resolved(x$a))
stopifnot(resolved(x[["b"]]))
x <- list()
x$a <- future(1, lazy = TRUE)
x$b <- future(2, lazy = TRUE)
x[[3]] <- 3
y <- resolve(x)
stopifnot(identical(y, x))
stopifnot(resolved(x$a))
stopifnot(resolved(x[["b"]]))
x <- list()
x$a <- future(1)
x$b <- future({Sys.sleep(0.5); 2})
x[[4]] <- 4
dim(x) <- c(2, 2)
y <- resolve(x, idxs = 1)
stopifnot(identical(y, x))
stopifnot(resolved(x[[1]]))
y <- resolve(x, idxs = 2)
stopifnot(identical(y, x))
stopifnot(resolved(x[[2]]))
y <- resolve(x, idxs = 3)
stopifnot(identical(y, x))
y <- resolve(x, idxs = seq_along(x))
stopifnot(identical(y, x))
y <- resolve(x, idxs = names(x))
stopifnot(identical(y, x))
y <- resolve(x, idxs = matrix(c(1, 2), ncol = 2L), value = TRUE)
stopifnot(identical(y, x))
x <- list()
for (kk in 1:3) x[[kk]] <- future({ Sys.sleep(0.1); kk })
y <- resolve(x)
stopifnot(identical(y, x))
x <- list()
for (kk in 1:3) x[[kk]] <- future({ Sys.sleep(0.1); kk }, lazy = TRUE)
y <- resolve(x)
stopifnot(identical(y, x))
## Exceptions
x <- list()
x$a <- 1
x$b <- 2
res <- tryCatch(y <- resolve(x, idxs = 0L), error = identity)
stopifnot(inherits(res, "error"))
res <- tryCatch(y <- resolve(x, idxs = "unknown"), error = identity)
stopifnot(inherits(res, "error"))
x <- list(1, 2)
res <- tryCatch(x <- resolve(x, idxs = "a"), error = identity)
stopifnot(inherits(res, "error"))
message(sprintf("- plan('%s') ...", strategy))
} ## for (strategy ...)
message("*** resolve() for lists ... DONE")
message("*** resolve() for environments ...")
for (strategy in strategies) {
message(sprintf("- plan('%s') ...", strategy))
plan(strategy)
x <- new.env()
y <- resolve(x)
stopifnot(identical(y, x))
x <- new.env()
x$a <- 1
x$b <- 2
y <- resolve(x)
stopifnot(identical(y, x))
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 0L)
x <- new.env()
x$a <- future(1)
x$b <- future(2)
x$c <- 3
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 2L)
y <- resolve(x)
stopifnot(identical(y, x))
stopifnot(resolved(x$a))
stopifnot(resolved(x$b))
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 2L)
x <- new.env()
x$a %<-% { 1 }
x$b %<-% { 2 }
x$c <- 3
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 2L)
y <- resolve(x) ## FIXME: Should not do value()!
stopifnot(identical(y, x))
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 2L)
x <- new.env()
x$a <- future({ 1 })
x$b %<-% { 2 }
x$c <- 3
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 2L)
y <- resolve(x, idxs = "a")
stopifnot(identical(y, x))
stopifnot(resolved(x$a))
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 2L)
y <- resolve(x, idxs = "b")
stopifnot(identical(y, x))
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 2L)
y <- resolve(x, idxs = "c")
stopifnot(identical(y, x))
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 2L)
y <- resolve(x, idxs = names(x), value = TRUE)
stopifnot(identical(y, x))
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 2L)
y <- resolve(x, recursive = TRUE, value = TRUE)
stopifnot(identical(y, x))
## Exceptions
res <- tryCatch(y <- resolve(x, idxs = "unknown"), error = identity)
stopifnot(inherits(res, "error"))
message(sprintf("- plan('%s') ...", strategy))
} ## for (strategy ...)
message("*** resolve() for environments ... DONE")
message("*** resolve() for list environments ...")
for (strategy in strategies) {
message(sprintf("- plan('%s') ...", strategy))
plan(strategy)
options(future.progress = function(done, total) {
msg <- sprintf("Wohoo: %.0f%% (%d/%d)", 100 * done / total, done, total)
if (done < total) {
bs <- paste(rep("\b", times = nchar(msg)), collapse = "")
message(paste(msg, bs, sep = ""), appendLF = FALSE)
} else {
message(msg)
}
})
x <- listenv()
y <- resolve(x)
stopifnot(identical(y, x))
x <- listenv()
x$a <- 1
x$b <- 2
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 0L)
y <- resolve(x)
stopifnot(identical(y, x))
x <- listenv()
x$a <- future(1)
x$b <- future(2)
x$c <- 3
names <- names(x)
dim(x) <- c(1, 3)
names(x) <- names
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 2L)
y <- resolve(x)
stopifnot(identical(y, x))
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 2L)
x <- listenv()
x$a %<-% { 1 }
x$b %<-% { 2 }
x$c <- 3
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 2L)
y <- resolve(x) ## FIXME: Should not do value()!
stopifnot(identical(y, x))
#stopifnot(is.na(futureOf(x$a, mustExist = FALSE)))
#stopifnot(is.na(futureOf(x$b, mustExist = FALSE)))
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 2L)
x <- listenv()
x$a <- future({ 1 })
x$b %<-% { Sys.sleep(0.5); 2 }
x$c %<-% { 3 }
x$d <- 4
names <- names(x)
dim(x) <- c(2, 2)
names(x) <- names
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 3L)
y <- resolve(x, idxs = "a")
stopifnot(identical(y, x))
stopifnot(identical(futureOf(x$a, mustExist = FALSE), x$a))
stopifnot(resolved(x$a))
y <- resolve(x, idxs = "b")
stopifnot(identical(y, x))
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 3L)
idxs <- matrix(c(1, 2), ncol = 2L)
y <- resolve(x, idxs = idxs)
stopifnot(identical(y, x))
#stopifnot(is.na(futureOf(x$c, mustExist = FALSE)))
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 3L)
y <- resolve(x, idxs = 4L)
stopifnot(identical(y, x))
#stopifnot(is.na(futureOf(x[[4L]], mustExist = FALSE)))
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 3L)
y <- resolve(x, idxs = names(x), value = TRUE)
stopifnot(identical(y, x))
stopifnot(length(futureOf(envir = x, drop = TRUE)) == 3L)
y <- resolve(x, recursive = TRUE, value = TRUE)
stopifnot(identical(y, x))
## Exceptions
res <- tryCatch(y <- resolve(x, idxs = 0L), error = identity)
stopifnot(inherits(res, "error"))
res <- tryCatch(y <- resolve(x, idxs = "unknown"), error = identity)
stopifnot(inherits(res, "error"))
message(sprintf("- plan('%s') ...", strategy))
} ## for (strategy ...)
message("*** resolve() for list environments ... DONE")
message("*** resolve() - globals with non-trustful length() ...")
length.CantTrustLength <- function(x) length(unclass(x)) + 1L
.length <- future:::.length
x <- structure(as.list(1:3), class = c("CantTrustLength", "list"))
str(list(n = length(x), n_true = .length(x)))
stopifnot(length(x) > .length(x))
x <- resolve(x)
message("*** resolve() - globals with non-trustful length() ... DONE")
message("*** resolved() - default ...")
res <- resolved(42L)
stopifnot(isTRUE(res))
message("*** resolved() - default ... DONE")
message("*** resolve() ... DONE")
source("incl/end.R")
future/tests/multicore.R 0000644 0001762 0000144 00000004657 13111756521 015072 0 ustar ligges users source("incl/start.R")
library("listenv")
plan(multicore)
message("*** multicore() ...")
for (cores in 1:min(2L, availableCores("multicore"))) {
message(sprintf("Testing with %d cores ...", cores))
options(mc.cores = cores)
if (!supportsMulticore()) {
message(sprintf("Multicore futures are not supporting on '%s'. Falling back to use synchronous sequential futures", .Platform$OS.type))
}
for (globals in c(FALSE, TRUE)) {
message(sprintf("*** multicore(..., globals = %s) without globals", globals))
f <- multicore({
42L
}, globals = globals)
stopifnot(inherits(f, "MulticoreFuture") || ((cores ==1 || !supportsMulticore()) && inherits(f, "SequentialFuture")))
print(resolved(f))
y <- value(f)
print(y)
stopifnot(y == 42L)
message(sprintf("*** multicore(..., globals = %s) with globals", globals))
## A global variable
a <- 0
f <- multicore({
b <- 3
c <- 2
a * b * c
}, globals = globals)
print(f)
## A multicore future is evaluated in a separated
## forked process. Changing the value of a global
## variable should not affect the result of the
## future.
a <- 7 ## Make sure globals are frozen
v <- value(f)
print(v)
stopifnot(v == 0)
message(sprintf("*** multicore(..., globals = %s) with globals and blocking", globals))
x <- listenv()
for (ii in 1:4) {
message(sprintf(" - Creating multicore future #%d ...", ii))
x[[ii]] <- multicore({ ii }, globals = globals)
}
message(sprintf(" - Resolving %d multicore futures", length(x)))
v <- sapply(x, FUN = value)
stopifnot(all(v == 1:4))
message(sprintf("*** multicore(..., globals = %s) and errors", globals))
f <- multicore({
stop("Whoops!")
1
}, globals = globals)
print(f)
v <- value(f, signal = FALSE)
print(v)
stopifnot(inherits(v, "simpleError"))
res <- try(value(f), silent = TRUE)
print(res)
stopifnot(inherits(res, "try-error"))
## Error is repeated
res <- try(value(f), silent = TRUE)
print(res)
stopifnot(inherits(res, "try-error"))
} # for (globals ...)
message("*** multicore(..., workers = 1L) ...")
a <- 2
b <- 3
yTruth <- a * b
f <- multicore({ a * b }, workers = 1L)
rm(list = c("a", "b"))
v <- value(f)
print(v)
stopifnot(v == yTruth)
message("*** multicore(..., workers = 1L) ... DONE")
message(sprintf("Testing with %d cores ... DONE", cores))
} ## for (cores ...)
message("*** multicore() ... DONE")
source("incl/end.R")
future/tests/whichIndex.R 0000644 0001762 0000144 00000004115 13111756521 015146 0 ustar ligges users source("incl/start,load-only.R")
message("*** whichIndex() ...")
dims <- list(
c(1),
c(1, 1),
c(2),
c(2, 1),
c(3, 4, 3)
)
for (dim in dims) {
printf("Dimensions: (%s)\n", paste(dim, collapse = ", "))
dimnames <- lapply(dim, FUN = function(n) letters[seq_len(n)])
X <- array(seq_len(prod(dim)), dim = dim, dimnames = dimnames)
print(X)
idxs <- unique(round(seq(from = 1L, to = length(X), length.out = 5)))
print(idxs)
stopifnot(all(X[idxs] == idxs))
## (a) By matrix and dimindices
I <- arrayInd(idxs, .dim = dim(X))
print(I)
idxs2 <- whichIndex(I, dim = dim(X))
print(idxs2)
stopifnot(all(idxs2 == idxs))
## (b) By matrix and dimnames
N <- array(NA_character_, dim = dim(I))
for (kk in seq_len(ncol(N))) {
N[, kk] <- dimnames[[kk]][I[, kk]]
}
print(N)
idxs3 <- whichIndex(N, dim = dim(X), dimnames = dimnames(X))
print(idxs3)
stopifnot(all(idxs3 == idxs))
## (b) By data.frame
D <- as.data.frame(I)
for (cc in seq(from = 1L, to = ncol(D))) D[[cc]] <- N[, cc]
print(D)
idxs4 <- whichIndex(D, dim = dim(X), dimnames = dimnames(X))
print(idxs4)
stopifnot(all(idxs4 == idxs))
}
## Exceptions
dim <- c(2, 3)
ndim <- length(dim)
dimnames <- lapply(dim, FUN = function(n) letters[seq_len(n)])
I <- matrix(c(1, 1, 2, 4), ncol = ndim)
res <- try(idxs <- whichIndex(I, dim = dim, dimnames = dimnames), silent = TRUE)
stopifnot(inherits(res, "try-error"))
I <- matrix(c(0, 0), ncol = ndim)
res <- try(idxs <- whichIndex(I, dim = dim, dimnames = dimnames), silent = TRUE)
stopifnot(inherits(res, "try-error"))
I <- matrix(c("a", "q"), ncol = ndim)
res <- try(idxs <- whichIndex(I, dim = dim, dimnames = dimnames), silent = TRUE)
stopifnot(inherits(res, "try-error"))
D <- data.frame(a = c(1, 1), b = c(2, 4))
res <- try(idxs <- whichIndex(D, dim = dim, dimnames = dimnames), silent = TRUE)
stopifnot(inherits(res, "try-error"))
D <- data.frame(a = c("a", "q"), b = c(1, 2))
res <- try(idxs <- whichIndex(D, dim = dim, dimnames = dimnames), silent = TRUE)
stopifnot(inherits(res, "try-error"))
message("*** whichIndex() ... DONE")
source("incl/end.R")
future/tests/futureAssign.R 0000644 0001762 0000144 00000004477 13111756521 015546 0 ustar ligges users source("incl/start.R")
message("*** futureAssign() ...")
message("*** futureAssign() - sequential w/ lazy evaluation ...")
delayedAssign("a", {
cat("Delayed assignment evaluated\n")
1
})
futureAssign("b", {
cat("Future assignment evaluated\n")
2
}, lazy = TRUE)
## Because "lazy future" is used, the expression/value
## for 'b' will be resolved at the point. For other
## types of futures, it may already have been resolved
cat(sprintf("b = %s\n", b))
## The expression/value of 'a' is resolved at this point,
## because a delayed assignment (promise) was used.
cat(sprintf("a = %s\n", a))
stopifnot(identical(a, 1))
stopifnot(identical(b, 2))
message("*** futureAssign() - sequential w/ lazy evaluation ... DONE")
message("*** futureAssign() - lazy = TRUE / FALSE ...")
for (cores in 1:availCores) {
message(sprintf("Testing with %d cores ...", cores))
options(mc.cores = cores)
for (strategy in supportedStrategies(cores)) {
message(sprintf("*** futureAssign() with %s futures ...", sQuote(strategy)))
plan(strategy)
## Potential task name clashes
u <- new.env()
v <- new.env()
futureAssign("a", { 2 }, assign.env = u)
futureAssign("a", { 4 }, assign.env = v)
cat(sprintf("u$a = %s\n", u$a))
cat(sprintf("v$a = %s\n", v$a))
stopifnot(identical(u$a, 2))
stopifnot(identical(v$a, 4))
## Global variables
a <- 1
futureAssign("b", { 2 * a })
a <- 2
stopifnot(b == 2)
## Explicit lazy evaluation
for (lazy in c(FALSE, TRUE)) {
a <- 1
f <- futureAssign("b", { 2 * a }, lazy = lazy)
a <- 2
stopifnot(b == 2)
stopifnot(f$lazy == lazy || (strategy %in% c("multisession", "multiprocess") && cores == 1L))
## Set 'lazy' via disposable option
options(future.disposable = list(lazy = lazy))
a <- 1
f <- futureAssign("b", { 2 * a })
a <- 2
stopifnot(b == 2)
stopifnot(f$lazy == lazy || (strategy %in% c("multisession", "multiprocess") && cores == 1L))
}
message(sprintf("*** futureAssign() with %s futures ... DONE", sQuote(strategy)))
} # for (strategy in ...)
message(sprintf("Testing with %d cores ... DONE", cores))
} ## for (cores ...)
message("*** futureAssign() - lazy = TRUE / FALSE ... DONE")
message("*** futureAssign() ... DONE")
source("incl/end.R")
future/tests/sequential.R 0000644 0001762 0000144 00000001115 13111756521 015223 0 ustar ligges users source("incl/start.R")
message("*** sequential() ...")
message("- sequential() w/ required packages ...")
f <- future(median(1:3), lazy = TRUE)
print(f)
## Doesn't work if covr that depends on stats is loaded
try(unloadNamespace("stats"))
v <- value(f)
print(v)
stopifnot(identical(v, 2L))
stopifnot("stats" %in% loadedNamespaces())
message("- SequentialFuture() - exceptions ...")
res <- tryCatch({
f <- SequentialFuture(42, lazy = TRUE, local = FALSE)
}, error = identity)
stopifnot(inherits(res, "error"))
message("*** sequential() ... DONE")
source("incl/end.R")
future/tests/sessionDetails.R 0000644 0001762 0000144 00000000452 13075332474 016053 0 ustar ligges users source("incl/start.R")
message("*** Session details ...")
x <- sessionDetails()
print(x)
x <- sessionDetails(env = FALSE)
print(x)
x <- sessionDetails(env = TRUE)
print(x)
y <- x[1:3]
print(y)
stopifnot(all(class(y) == class(x)))
message("*** Session details ... DONE")
source("incl/end.R")
future/tests/early-signaling.R 0000644 0001762 0000144 00000007114 13111756706 016150 0 ustar ligges users source("incl/start.R")
options(future.debug = FALSE)
message("*** Early signaling of conditions ...")
message("*** Early signaling of conditions with sequential futures ...")
plan(sequential)
f <- future({ stop("bang!") })
r <- resolved(f)
stopifnot(r)
v <- tryCatch(value(f), error = identity)
stopifnot(inherits(v, "error"))
message("- with lazy evaluation ...")
f <- future({ stop("bang!") }, lazy = TRUE)
r <- resolved(f)
stopifnot(r)
v <- tryCatch(value(f), error = identity)
stopifnot(inherits(v, "error"))
plan(sequential, earlySignal = TRUE)
f <- tryCatch(future({ stop("bang!") }), error = identity)
stopifnot(inherits(f, "error"))
message("- with lazy evaluation ...")
## Errors
f <- future({ stop("bang!") }, lazy = TRUE)
r <- tryCatch(resolved(f), error = identity)
stopifnot(inherits(r, "error"))
v <- tryCatch(value(f), error = identity)
stopifnot(inherits(v, "error"))
## Warnings
f <- future({ warning("careful!") }, lazy = TRUE)
res <- tryCatch({
r <- resolved(f)
}, condition = function(w) w)
str(res)
stopifnot(inherits(res, "warning"))
## Messages
f <- future({ message("hey!") }, lazy = TRUE)
res <- tryCatch({
r <- resolved(f)
}, condition = function(w) w)
stopifnot(inherits(res, "message"))
## Condition
f <- future({ signalCondition(simpleCondition("hmm")) }, lazy = TRUE)
res <- tryCatch({
r <- resolved(f)
}, condition = function(w) w)
stopifnot(inherits(res, "condition"))
message("*** Early signaling of conditions with sequential futures ... DONE")
message("Number of available cores: ", availableCores())
message("*** Early signaling of conditions with multisession futures ...")
plan(multisession)
f <- future({ stop("bang!") })
Sys.sleep(0.5)
r <- resolved(f)
stopifnot(r)
v <- tryCatch(value(f), error = identity)
stopifnot(inherits(v, "error"))
plan(multisession, earlySignal = TRUE)
f <- future({ stop("bang!") })
Sys.sleep(0.5)
print(f)
r <- tryCatch(resolved(f), error = identity)
stopifnot(inherits(r, "error") || inherits(f, "SequentialFuture"))
v <- tryCatch(value(f), error = identity)
stopifnot(inherits(v, "error"))
message("*** Early signaling of conditions with multisession futures ... DONE")
message("*** Early signaling of conditions with multiprocess futures ...")
plan(multiprocess)
f <- future({ stop("bang!") })
Sys.sleep(0.5)
r <- resolved(f)
stopifnot(r)
v <- tryCatch(value(f), error = identity)
stopifnot(inherits(v, "error"))
plan(multiprocess, earlySignal = TRUE)
f <- future({ stop("bang!") })
Sys.sleep(0.5)
print(f)
r <- tryCatch(resolved(f), error = identity)
stopifnot(inherits(r, "error") || inherits(f, "SequentialFuture"))
v <- tryCatch(value(f), error = identity)
stopifnot(inherits(v, "error"))
## Errors
f <- future({ stop("bang!") }, earlySignal = TRUE)
Sys.sleep(0.5)
r <- tryCatch(resolved(f), error = identity)
stopifnot(inherits(r, "error"))
v <- tryCatch(value(f), error = identity)
stopifnot(inherits(v, "error"))
## Warnings
f <- future({ warning("careful!") }, earlySignal = TRUE)
Sys.sleep(0.5)
res <- tryCatch({ r <- resolved(f) }, condition = function(w) w)
#stopifnot(inherits(res, "warning"))
## Messages
f <- future({ message("hey!") }, earlySignal = TRUE)
Sys.sleep(0.5)
res <- tryCatch({ r <- resolved(f) }, condition = function(w) w)
#stopifnot(inherits(res, "message"))
## Condition
f <- future({ signalCondition(simpleCondition("hmm")) }, earlySignal = TRUE)
Sys.sleep(0.5)
res <- tryCatch({ r <- resolved(f) }, condition = function(w) w)
#stopifnot(inherits(res, "condition"))
message("*** Early signaling of conditions with multiprocess futures ... DONE")
message("*** Early signaling of conditions ... DONE")
source("incl/end.R")
future/tests/constant.R 0000644 0001762 0000144 00000000763 13111756521 014712 0 ustar ligges users source("incl/start.R")
library("listenv")
message("*** constant() ...")
## No global variables
f <- try(constant(42L), silent = FALSE)
print(f)
stopifnot(inherits(f, "ConstantFuture"))
print(resolved(f))
y <- value(f)
print(y)
stopifnot(y == 42L)
plan(constant)
## No global variables
f <- try(future(42L), silent = FALSE)
print(f)
stopifnot(inherits(f, "ConstantFuture"))
print(resolved(f))
y <- value(f)
print(y)
stopifnot(y == 42L)
message("*** constant() ... DONE")
source("incl/end.R")
future/tests/globals,NSE.R 0000644 0001762 0000144 00000002340 13111756521 015117 0 ustar ligges users source("incl/start.R")
library("listenv")
message("*** Globals w/ non-standard evaluation (NSE) ...")
data <- data.frame(x = 1:5, y = 1:5)
v0 <- subset(data, x < 3)$y
for (strategy in supportedStrategies()) {
message(sprintf("- Strategy: %s ...", strategy))
plan(strategy)
## Assert option is passed on to future
options(future.globals.onMissing = "error")
opt1 %<-% getOption("future.globals.onMissing")
stopifnot(identical(opt1, "error"))
options(future.globals.onMissing = "ignore")
opt2 %<-% getOption("future.globals.onMissing")
stopifnot(identical(opt2, "ignore"))
options(future.globals.onMissing = "error")
res <- try({ v1 %<-% subset(data, x < 3)$y }, silent = TRUE)
stopifnot(inherits(res, "try-error"))
options(future.globals.onMissing = "ignore")
v2 %<-% subset(data, x < 3)$y
stopifnot(identical(v2, v0))
## Nested futures (requires option is passed on to future)
plan(list(sequential, strategy))
options(future.globals.onMissing = "ignore")
v3 %<-% {
a %<-% subset(data, x < 3)$y
a
} %lazy% TRUE
stopifnot(identical(v3, v0))
message(sprintf("- Strategy: %s ... DONE", strategy))
}
message("*** Globals w/ non-standard evaluation (NSE) ... DONE")
source("incl/end.R")
future/tests/globals,subassignment.R 0000644 0001762 0000144 00000006174 13111756706 017372 0 ustar ligges users source("incl/start.R")
oopts <- c(oopts, options(
future.globals.resolve = TRUE,
future.globals.onMissing = "error"
))
message("*** Globals - subassignments ...")
message("*** Globals - subassignments w/ x$a <- value ...")
## Truth:
x <- x0 <- list()
y0 <- list(a = 1)
str(list(x = x, y0 = y0))
y <- local({
x$a <- 1
x
})
stopifnot(identical(y, y0))
y <- local({
x[["a"]] <- 1
x
})
stopifnot(identical(y, y0))
y <- local({
x["a"] <- list(1)
x
})
stopifnot(identical(y, y0))
stopifnot(identical(x, list()))
for (cores in 1:availCores) {
message(sprintf("Testing with %d cores ...", cores))
options(mc.cores = cores)
message("availableCores(): ", availableCores())
for (strategy in supportedStrategies(cores)) {
message(sprintf("- plan('%s') ...", strategy))
plan(strategy)
## Explicit future
x <- list()
f <- future({
x$a <- 1
x
})
rm(list = "x")
y <- value(f)
print(y)
stopifnot(identical(y, y0))
## Explicit future (lazy)
x <- list()
f <- future({
x$a <- 1
x
}, lazy = TRUE)
rm(list = "x")
y <- value(f)
print(y)
stopifnot(identical(y, y0))
## Future assignment
x <- list()
y %<-% {
x$a <- 1
x
}
rm(list = "x")
print(y)
stopifnot(identical(y, y0))
## Same with forced lazy evaluation
x <- list()
y %<-% {
x$a <- 1
x
} %lazy% TRUE
rm(list = "x")
print(y)
stopifnot(identical(y, y0))
## 'x' is _not_ a global variable here
x <- list()
y %<-% {
x <- list(b = 2)
x$a <- 1
x
}
rm(list = "x")
print(y)
stopifnot(identical(y, list(b = 2, a = 1)))
## Explicit future
x <- list()
f <- future({
x[["a"]] <- 1
x
})
rm(list = "x")
y <- value(f)
print(y)
stopifnot(identical(y, y0))
## Explicit future (lazy)
x <- list()
f <- future({
x[["a"]] <- 1
x
}, lazy = TRUE)
rm(list = "x")
y <- value(f)
print(y)
stopifnot(identical(y, y0))
## Future assignment
x <- list()
y %<-% {
x[["a"]] <- 1
x
}
rm(list = "x")
print(y)
stopifnot(identical(y, y0))
## Explicit future
x <- list()
f <- future({
x["a"] <- list(1)
x
})
rm(list = "x")
y <- value(f)
print(y)
stopifnot(identical(y, y0))
## Explicit future (lazy)
x <- list()
f <- future({
x["a"] <- list(1)
x
}, lazy = TRUE)
rm(list = "x")
y <- value(f)
print(y)
stopifnot(identical(y, y0))
## Future assignment
x <- list()
y %<-% {
x["a"] <- list(1)
x
}
rm(list = "x")
print(y)
stopifnot(identical(y, y0))
## Future assignment
x <- list()
name <- "a"
y %<-% {
x[name] <- list(1)
x
}
rm(list = c("x", "name"))
print(y)
stopifnot(identical(y, y0))
} ## for (strategy ...)
message(sprintf("Testing with %d cores ... DONE", cores))
} ## for (cores ...)
message("*** Globals - subassignments w/ x$a <- value ... DONE")
message("*** Globals - subassignments ... DONE")
source("incl/end.R")
future/tests/dotdotdot.R 0000644 0001762 0000144 00000002731 13111756521 015062 0 ustar ligges users source("incl/start.R")
library("listenv")
for (cores in 1:availCores) {
message(sprintf("Testing with %d cores ...", cores))
options(mc.cores = cores)
message("*** Global argument '...' ...")
sum_fcns <- list()
sum_fcns$A <- function(x, ...) {
message("Arguments '...' exists: ", exists("...", inherits = TRUE))
y %<-% { sum(x, ...) }
y
}
sum_fcns$B <- function(x, ...) {
sumt <- function(x) {
message("Arguments '...' exists: ", exists("...", inherits = TRUE))
y %<-% { sum(x, ...) }
y
}
sumt(x)
}
sum_fcns$C <- function(x, y) {
message("Arguments '...' exists: ", exists("...", inherits = TRUE))
y %<-% { sum(x, y) }
y
}
sum_fcns$D <- function(x, y) {
message("Arguments '...' exists: ", exists("...", inherits = TRUE))
y %<-% { sum(x, y, ...) }
y
}
for (strategy in supportedStrategies(cores)) {
message(sprintf("- plan('%s') ...", strategy))
plan(strategy, substitute = FALSE)
for (name in names(sum_fcns)) {
message(sprintf("** Sum function '%s' with plan('%s') ...", name, strategy))
sum_fcn <- sum_fcns[[name]]
print(sum_fcn)
y <- try(sum_fcn(1:2, 3))
print(y)
if (name %in% c("D")) {
stopifnot(inherits(y, "try-error"))
} else {
stopifnot(y == 6)
}
}
}
message(sprintf("Testing with %d cores ... DONE", cores))
} ## for (cores ...)
message("*** Global argument '...' ... DONE")
source("incl/end.R")
future/tests/transparent.R 0000644 0001762 0000144 00000001174 13235416340 015416 0 ustar ligges users source("incl/start.R")
library("listenv")
message("*** transparent() ...")
## No global variables
f <- try(transparent({
42L
}), silent = FALSE)
print(f)
stopifnot(inherits(f, "SequentialFuture"), !f$lazy)
print(resolved(f))
y <- value(f)
print(y)
stopifnot(y == 42L)
plan(transparent)
## No global variables
f <- try(future({
42L
}), silent = FALSE)
print(f)
stopifnot(inherits(f, "SequentialFuture"), !f$lazy)
print(resolved(f))
y <- value(f)
print(y)
stopifnot(y == 42L)
## A global variable
a <- 0
f <- try(future({
b <- 3
c <- 2
a * b * c
}))
print(f)
message("*** transparent() ... DONE")
source("incl/end.R")
future/tests/globals,toolarge.R 0000644 0001762 0000144 00000001743 13155002315 016305 0 ustar ligges users source("incl/start.R")
library("listenv")
message("*** Globals - too large ...")
ooptsT <- options(future.globals.maxSize = object.size(1:1000) - 1L)
limit <- getOption("future.globals.maxSize")
cat(sprintf("Max total size of globals: %g bytes\n", limit))
plan(multisession)
exprs <- list(
A = substitute({ a }, env = list()),
B = substitute({ a * b }, env = list()),
C = substitute({ a * b * c }, env = list()),
D = substitute({ a * b * c * d }, env = list()),
E = substitute({ a * b * c * d * e }, env = list())
)
a <- 1:1000
b <- 1:900
c <- 1:800
d <- 1:700
e <- 1
for (name in names(exprs)) {
message(sprintf("Expression %s:", name))
expr <- exprs[[name]]
print(expr)
res <- tryCatch({
f <- future(expr, substitute = FALSE)
}, error = function(ex) ex)
print(res)
stopifnot(inherits(res, "error"))
msg <- conditionMessage(res)
stopifnot(grepl("exceeds the maximum allowed size", msg))
}
message("*** Globals - too large ... DONE")
source("incl/end.R")
future/tests/backtrace.R 0000644 0001762 0000144 00000003527 13237651433 015006 0 ustar ligges users source("incl/start.R")
message("*** backtrace( ) ...")
message("*** backtrace( ) - explicit future ...")
f <- future({ 42L; stop("Woops") })
v <- value(f, signal = FALSE)
print(v)
calls <- backtrace(f)
print(calls)
message("*** backtrace( ) - explicit future ... DONE")
message("*** backtrace( ) - implicit future ...")
v %<-% { 42L; stop("Woops") }
calls <- backtrace(v)
print(calls)
message("*** backtrace( ) - implicit future ... DONE")
message("*** backtrace( ) - subsetting ...")
env <- new.env()
env[["a"]] %<-% { 42L; stop("Woops") }
env[["b"]] %<-% { 42L; stop("Woops") }
calls <- backtrace(env[["b"]])
print(calls)
stopifnot(is.list(calls))
lenv <- listenv::listenv()
lenv[[1]] %<-% { 42L; stop("Woops") }
lenv[[2]] %<-% { 42L; stop("Woops") }
calls <- backtrace(lenv[[2]])
print(calls)
stopifnot(is.list(calls))
ll <- list()
ll[[1]] <- future({ 42L; stop("Woops") })
ll[[2]] <- future({ 42L; stop("Woops") })
vs <- values(ll, signal = FALSE)
calls <- backtrace(ll[[2]])
print(calls)
stopifnot(is.list(calls))
message("*** backtrace( ) - subsetting ... DONE")
message("*** backtrace( ) - exceptions ...")
message("- No condition ...")
f <- future(42L)
res <- tryCatch(backtrace(f), error = identity)
print(res)
stopifnot(inherits(res, "error"))
message("- No call stack ...")
f <- future({ 42L; stop("Woops") })
v <- value(f, signal = FALSE)
f$value$traceback <- NULL ## Remove call stack
res <- tryCatch(backtrace(f), error = identity)
print(res)
stopifnot(inherits(res, "error"))
if (availableCores() >= 2L) {
message("- Non-resolved future ...")
plan(multiprocess, workers = 2L)
f <- future({ Sys.sleep(3); 42L; stop("Woops") })
res <- tryCatch(backtrace(f), error = identity)
print(res)
stopifnot(inherits(res, "error"))
}
message("*** backtrace( ) - exceptions ... DONE")
message("*** backtrace( ) ... DONE")
source("incl/end.R")
future/tests/remote.R 0000644 0001762 0000144 00000001502 13111756521 014344 0 ustar ligges users source("incl/start.R")
library("listenv")
message("*** remote() ...")
## No global variables
f <- try(remote({
42L
}, workers = "localhost"), silent = FALSE)
print(f)
stopifnot(inherits(f, "ClusterFuture"))
print(resolved(f))
y <- value(f)
print(y)
stopifnot(y == 42L)
plan(remote, workers = "localhost")
## No global variables
f <- try(future({
42L
}), silent = FALSE)
print(f)
stopifnot(inherits(f, "ClusterFuture"))
print(resolved(f))
y <- value(f)
print(y)
stopifnot(y == 42L)
## A global variable
a <- 0
f <- try(future({
b <- 3
c <- 2
a * b * c
}))
print(f)
message("*** remote() - exceptions ...")
res <- try(remote(42L, workers = TRUE), silent = TRUE)
print(res)
stopifnot(inherits(res, "try-error"))
message("*** remote() - exceptions ... DONE")
message("*** remote() ... DONE")
source("incl/end.R")
future/tests/globals,resolve.R 0000644 0001762 0000144 00000002426 13111756706 016163 0 ustar ligges users source("incl/start.R")
library("listenv")
oopts <- c(oopts, options(future.globals.resolve = TRUE))
setTimeLimit(cpu = 10, elapsed = 10, transient = TRUE)
message("*** Tricky use cases related to globals (part 2) ...")
## Allow for two background processes
plan(multisession, workers = 2L)
env <- new.env()
## Create future #1 (consumes background process #1)
env$a %<-% { 5 }
## Create future #2 (consumes background process #2)
b %<-% { "a" }
## Resolve future #2 (frees up background process #2)
message(sprintf("b = %s\n", sQuote(b)))
## Create future #3 (consumes background process #2)
## THIS IS THE TRICKY PART:
## Two globals are identified `env` and `b` and both are resolved.
## However, object `env[[b]]` (here element `a` of environment `env`)
## is not touched and therefore not resolved (since it is a future)
## unless environment `env` is resolved recursively. (Issue #49)
y %<-% { env[[b]] }
## Resolve future #3
message(sprintf("y = %s\n", y))
## Resolve future #1 if not already done
str(as.list(env))
## Create future #4
## Since future #1 is resolved it will work at this point
y %<-% { env[[b]] }
## Resolve future #4
message(sprintf("y = %s\n", y))
message("*** Tricky use cases related to globals (part 2) ... DONE")
## Cleanup
setTimeLimit()
source("incl/end.R")
future/tests/Future-class.R 0000644 0001762 0000144 00000002475 13147540314 015440 0 ustar ligges users source("incl/start.R")
message("*** Future class ...")
message("*** Future class - exception ...")
f <- Future()
print(f)
res <- tryCatch(value(f), error = identity)
print(res)
stopifnot(inherits(res, "error"))
## values() is an alias for value() for Future
res <- tryCatch(values(f), error = identity)
stopifnot(inherits(res, "error"))
## Invalid seed
res <- tryCatch(f <- Future(42, seed = 1:2), error = identity)
stopifnot(inherits(res, "error"))
## When no packages are exported
foo <- structure(function(...) { Future(1) }, class = "future")
plan(foo)
f <- Future()
expr <- getExpression(f)
print(expr)
stopifnot(is.call(expr))
clazzes <- list(
sequential = SequentialFuture,
multisession = function(...) MultisessionFuture(..., workers = 2L),
sequential = SequentialFuture
)
if (supportsMulticore()) {
clazzes$multicore = function(...) MulticoreFuture(..., workers = 2L)
}
for (clazz in clazzes) {
## Calling run() more than once
f <- clazz({ 42L })
print(f)
run(f)
res <- tryCatch(run(f), error = identity)
stopifnot(inherits(res, "error"))
v <- value(f)
print(v)
stopifnot(v == 42L)
## Call value() without run()
f <- clazz({ 42L })
v <- value(f)
print(v)
stopifnot(v == 42L)
}
message("*** Future class - exception ... DONE")
message("*** Future class ... DONE")
source("incl/end.R")
future/tests/future_lapply,globals.R 0000644 0001762 0000144 00000007617 13155002315 017372 0 ustar ligges users source("incl/start.R")
message("*** future_lapply() - globals ...")
plan(cluster, workers = "localhost")
options(future.debug = FALSE)
a <- 1
b <- 2
globals_set <- list(
A = FALSE,
B = TRUE,
C = c("a", "b"),
D = list(a = 2, b = 3)
)
x <- list(1)
y_truth <- list(A = NULL, B = list(1), C = list(1), D = list(2))
str(y_truth)
for (name in names(globals_set)) {
globals <- globals_set[[name]]
message("Globals set ", sQuote(name))
y <- tryCatch({
future_lapply(x, FUN = function(x) {
median(c(x, a, b))
}, future.globals = globals, future.packages = "utils")
}, error = identity)
print(y)
stopifnot((name == "A" && inherits(y, "error")) ||
identical(y, y_truth[[name]]))
}
message("*** future_lapply() - globals ... DONE")
## Test adopted from http://stackoverflow.com/questions/42561088/nested-do-call-within-a-foreach-dopar-environment-cant-find-function-passed-w
message("*** future_lapply() - tricky globals ...")
my_add <- function(a, b) a + b
call_my_add <- function(a, b) {
do.call(my_add, args = list(a = a, b = b))
}
call_my_add_caller <- function(a, b, FUN = call_my_add) {
do.call(FUN, args = list(a = a, b = b))
}
main <- function(x = 1:2, caller = call_my_add_caller,
args = list(FUN = call_my_add)) {
results <- future_lapply(x, FUN = function(i) {
do.call(caller, args = c(list(a = i, b = i + 1L), args))
})
results
}
x <- list(list(1:2))
z_length <- lapply(x, FUN = do.call, what = length)
fun <- function(...) sum(...)
z_fun <- lapply(x, FUN = do.call, what = fun)
y0 <- NULL
for (strategy in supportedStrategies()) {
plan(strategy)
y <- main(1:3)
if (is.null(y0)) y0 <- y
stopifnot(identical(y, y0))
message("- future_lapply(x, FUN = do.call, ...) ...")
z <- future_lapply(x, FUN = do.call, what = length)
stopifnot(identical(z, z_length))
z <- future_lapply(x, FUN = do.call, what = fun)
stopifnot(identical(z, z_fun))
message("- future_lapply(x, ...) - passing arguments via '...' ...")
## typeof() == "list"
obj <- data.frame(a = 1:2)
stopifnot(typeof(obj) == "list")
y <- future_lapply(1L, function(a, b) typeof(b), b = obj)
stopifnot(identical(y[[1]], typeof(obj)))
## typeof() == "environment"
obj <- new.env()
stopifnot(typeof(obj) == "environment")
y <- future_lapply(1L, function(a, b) typeof(b), b = obj)
stopifnot(identical(y[[1]], typeof(obj)))
## typeof() == "S4"
if (requireNamespace("methods")) {
obj <- methods::getClass("MethodDefinition")
stopifnot(typeof(obj) == "S4")
y <- future_lapply(1L, function(a, b) typeof(b), b = obj)
stopifnot(identical(y[[1]], typeof(obj)))
}
}
message("*** future_lapply() - tricky globals ... DONE")
message("*** future_lapply() - missing arguments ...")
## Here 'abc' becomes missing, i.e. missing(abc) is TRUE
foo <- function(x, abc) future_lapply(x, FUN = function(y) y)
y <- foo(1:2)
stopifnot(identical(y, as.list(1:2)))
message("*** future_lapply() - missing arguments ... DONE")
message("*** future_lapply() - false positives ...")
## Here 'abc' becomes a promise, which fails to resolve
## iff 'xyz' does not exist. (Issue #161)
suppressWarnings(rm(list = "xyz"))
foo <- function(x, abc) future_lapply(x, FUN = function(y) y)
y <- foo(1:2, abc = (xyz >= 3.14))
stopifnot(identical(y, as.list(1:2)))
message("*** future_lapply() - false positives ... DONE")
message("*** future_lapply() - globals exceptions ...")
res <- tryCatch({
y <- future_lapply(1, FUN = function(x) x, future.globals = 42)
}, error = identity)
stopifnot(inherits(res, "error"))
res <- tryCatch({
y <- future_lapply(1, FUN = function(x) x, future.globals = list(1))
}, error = identity)
stopifnot(inherits(res, "error"))
res <- tryCatch({
y <- future_lapply(1, FUN = function(x) x, future.globals = "...future.FUN")
}, error = identity)
stopifnot(inherits(res, "error"))
message("*** future_lapply() - globals exceptions ... DONE")
source("incl/end.R")
future/tests/nbrOfWorkers.R 0000644 0001762 0000144 00000005316 13237667011 015507 0 ustar ligges users source("incl/start.R")
message("*** nbrOfWorkers() ...")
strategies <- c("sequential", "transparent")
for (strategy in strategies) {
message("Type of future: ", strategy)
evaluator <- get(strategy, mode = "function")
n <- nbrOfWorkers(evaluator)
message(sprintf("nbrOfWorkers: %d", n))
stopifnot(n == 1L)
plan(strategy)
n <- nbrOfWorkers()
message(sprintf("nbrOfWorkers: %d", n))
stopifnot(n == 1L)
} ## for (strategy ...)
strategies <- c("cluster", "multiprocess", "multisession", "multicore")
strategies <- intersect(strategies, supportedStrategies())
cores <- availableCores()
message("Number of available cores: ", cores)
workers <- availableWorkers()
nworkers <- length(workers)
message(sprintf("Available workers: [n = %d] %s", nworkers, hpaste(sQuote(workers))))
allButOneCore <- function() max(1L, future::availableCores() - 1L)
allButOneWorker <- function() {
w <- future::availableWorkers()
if (length(w) > 1) w[-1] else w
}
for (strategy in strategies) {
message("Type of future: ", strategy)
evaluator <- get(strategy, mode = "function")
n <- nbrOfWorkers(evaluator)
message(sprintf("nbrOfWorkers: %d", n))
stopifnot(n == nworkers)
plan(strategy)
n <- nbrOfWorkers()
message(sprintf("nbrOfWorkers: %d", n))
stopifnot(n == nworkers)
plan(strategy, workers = allButOneCore)
n <- nbrOfWorkers()
message(sprintf("nbrOfWorkers: %d", n))
stopifnot(n == max(1L, nworkers - 1L))
} ## for (strategy ...)
message("Type of future: cluster")
workers <- rep("localhost", times = 2L)
plan(cluster, workers = workers)
n <- nbrOfWorkers()
message(sprintf("nbrOfWorkers: %d", n))
stopifnot(n == length(workers))
plan(cluster, workers = allButOneWorker)
n <- nbrOfWorkers()
message(sprintf("nbrOfWorkers: %d", n))
stopifnot(n == max(1L, nworkers - 1L))
message("Type of future: remote")
workers <- rep("localhost", times = 2L)
plan(remote, workers = workers)
n <- nbrOfWorkers()
message(sprintf("nbrOfWorkers: %d", n))
stopifnot(n == length(workers))
plan(remote, workers = allButOneWorker)
n <- nbrOfWorkers()
message(sprintf("nbrOfWorkers: %d", n))
stopifnot(n == max(1L, nworkers - 1L))
message("Type of future: constant")
n <- nbrOfWorkers(constant)
message(sprintf("nbrOfWorkers: %d", n))
stopifnot(n == 1)
message("Type of future: ")
foo <- structure(function(...) NULL, class = c("future"))
n <- nbrOfWorkers(foo)
message(sprintf("nbrOfWorkers: %g", n))
stopifnot(n >= 0, is.infinite(n))
message("Type of future: cluster with workers = ")
workers <- makeClusterPSOCK(2L)
print(workers)
plan(cluster, workers = workers)
n <- nbrOfWorkers()
message(sprintf("nbrOfWorkers: %g", n))
stopifnot(n == length(workers))
message("*** nbrOfWorkers() ... DONE")
source("incl/end.R")
future/tests/future.R 0000644 0001762 0000144 00000002353 13111756706 014375 0 ustar ligges users source("incl/start.R")
message("*** future() ...")
f <- future({
42L
}, lazy = TRUE)
print(resolved(f))
y <- value(f)
print(y)
stopifnot(y == 42L)
message("*** future() w/ gc = TRUE ...")
f <- future(42L, gc = TRUE, lazy = TRUE)
print(f)
y <- value(f)
print(y)
stopifnot(y == 42L)
message("*** future() w/ gc = TRUE ... DONE")
message("*** future() ... DONE")
message("*** future() ...")
f <- future({
42L
}, lazy = TRUE)
print(resolved(f))
y <- value(f)
print(y)
stopifnot(y == 42L)
message("*** future() w/ gc = TRUE ...")
f <- future(42L, gc = TRUE, lazy = TRUE)
print(f)
y <- value(f)
print(y)
stopifnot(y == 42L)
message("*** future() w/ gc = TRUE ... DONE")
message("*** future() - exceptions ...")
res <- tryCatch(future(42L, evaluator = TRUE, lazy = TRUE), error = identity)
stopifnot(inherits(res, "error"))
res <- tryCatch(future(42L, evaluator = function(...) TRUE, lazy = TRUE), error = identity)
stopifnot(inherits(res, "error"))
target <- list(name = "", envir = new.env(), code = "Yo!", exists = TRUE)
res <- tryCatch(get_future(target, mustExist = TRUE), error = identity)
stopifnot(inherits(res, "error"))
message("*** future() - exceptions ... DONE")
message("*** future() ... DONE")
source("incl/end.R")
future/tests/requestNode.R 0000644 0001762 0000144 00000001552 13111756706 015361 0 ustar ligges users source("incl/start.R")
message("*** requestNode() ...")
message("*** requestNode() - exceptions ...")
workers <- makeClusterPSOCK(2L)
print(workers)
res <- try(requestNode(function() {}, workers = workers, timeout = -1.0), silent = TRUE)
stopifnot(inherits(res, "try-error"))
res <- try(requestNode(function() {}, workers = workers, alpha = 0), silent = TRUE)
stopifnot(inherits(res, "try-error"))
parallel::stopCluster(workers)
message("*** requestNode() - exceptions ... DONE")
message("*** requestNode() - timeout ...")
plan(multisession, workers = 2L)
f <- future({ Sys.sleep(100); 1 })
f2 <- future({ Sys.sleep(100); 2 })
res <- try(requestNode(function() { }, workers = f$workers, timeout = 0.5, delta = 0.1))
stopifnot(inherits(res, "try-error"))
message("*** requestNode() - timeout ... DONE")
message("*** requestNode() ... DONE")
source("incl/end.R")
future/tests/tweak.R 0000644 0001762 0000144 00000007733 13237236474 014211 0 ustar ligges users source("incl/start,load-only.R")
message("*** Tweaking future strategies ...")
message("*** y <- tweak(future::sequential) ...")
sequential2 <- future::tweak(future::sequential)
print(args(sequential2))
stopifnot(identical(sequential2, future::sequential))
stopifnot(!inherits(sequential2, "tweaked"))
message("*** y <- tweak(future::sequential, local = FALSE) ...")
sequential2 <- future::tweak(future::sequential, local = FALSE)
print(args(sequential2))
stopifnot(!identical(sequential2, future::sequential))
stopifnot(inherits(sequential2, "tweaked"))
stopifnot(identical(formals(sequential2)$local, FALSE))
message("*** y <- tweak('sequential', local = FALSE) ...")
sequential2 <- future::tweak("sequential", local = FALSE)
print(args(sequential2))
stopifnot(!identical(sequential2, future::sequential))
stopifnot(inherits(sequential2, "tweaked"))
stopifnot(identical(formals(sequential2)$local, FALSE))
library("future")
message("*** y <- tweak(sequential, local = FALSE) ...")
sequential2 <- future::tweak(sequential, local = FALSE)
print(args(sequential2))
stopifnot(!identical(sequential2, future::sequential))
stopifnot(inherits(sequential2, "tweaked"))
stopifnot(identical(formals(sequential2)$local, FALSE))
message("*** y <- tweak('sequential', local = FALSE) ...")
sequential2 <- future::tweak('sequential', local = FALSE)
print(args(sequential2))
stopifnot(!identical(sequential2, future::sequential))
stopifnot(inherits(sequential2, "tweaked"))
stopifnot(identical(formals(sequential2)$local, FALSE))
message("*** y <- tweak('sequential', local = FALSE, abc = 1, def = TRUE) ...")
res <- tryCatch({
sequential2 <- future::tweak('sequential', local = FALSE, abc = 1, def = TRUE)
}, warning = function(w) {
w
})
stopifnot(inherits(res, "warning"))
sequential2 <- future::tweak('sequential', local = FALSE, abc = 1, def = TRUE)
print(args(sequential2))
stopifnot(!identical(sequential2, future::sequential))
stopifnot(inherits(sequential2, "tweaked"))
stopifnot(identical(formals(sequential2)$local, FALSE))
message("*** y %<-% { expr } %tweak% tweaks ...")
plan(sequential)
a <- 0
x %<-% { a <- 1; a }
print(x)
stopifnot(a == 0, x == 1)
x %<-% { a <- 2; a } %tweak% list(local = FALSE)
print(x)
stopifnot(a == 2, x == 2)
plan(sequential, local = FALSE)
a <- 0
x %<-% { a <- 1; a }
print(x)
stopifnot(a == 1, x == 1)
x %<-% { a <- 2; a } %tweak% list(local = TRUE)
print(x)
stopifnot(a == 1, x == 2)
# Preserve nested futures
plan(list(A = sequential, B = tweak(sequential, local = FALSE)))
a <- 0
x %<-% {
stopifnot(identical(names(plan("list")), "B"))
a <- 1
a
}
print(x)
stopifnot(a == 0, x == 1)
x %<-% {
stopifnot(identical(names(plan("list")), "B"))
a <- 2
a
} %tweak% list(local = FALSE)
print(x)
stopifnot(a == 2, x == 2)
message("*** y %<-% { expr } %tweak% tweaks ... DONE")
message("*** tweak() - gc = TRUE ...")
res <- tryCatch(tweak(multisession, gc = TRUE), condition = identity)
stopifnot(inherits(res, "tweaked"))
## Argument 'gc' is unknown
res <- tryCatch(tweak(sequential, gc = TRUE), condition = identity)
stopifnot(inherits(res, "warning"))
res <- tryCatch(tweak(multicore, gc = TRUE), condition = identity)
stopifnot(inherits(res, "warning"))
message("*** tweak() - gc = TRUE ... DONE")
message("*** tweak() - exceptions ...")
res <- try(tweak(""), silent = TRUE)
stopifnot(inherits(res, "try-error"))
res <- try(tweak(base::eval), silent = TRUE)
stopifnot(inherits(res, "try-error"))
res <- try(tweak(sequential, "unnamed-argument"), silent = TRUE)
stopifnot(inherits(res, "try-error"))
## Arguments that must not be tweaked
res <- try(tweak(sequential, lazy = TRUE), silent = TRUE)
stopifnot(inherits(res, "try-error"))
res <- try(tweak(sequential, asynchronous = FALSE), silent = TRUE)
stopifnot(inherits(res, "try-error"))
res <- try(tweak(sequential, seed = 42L), silent = TRUE)
stopifnot(inherits(res, "try-error"))
message("*** tweak() - exceptions ... DONE")
message("*** Tweaking future strategies ... DONE")
source("incl/end.R")
future/tests/multiprocess.R 0000644 0001762 0000144 00000005647 13155002315 015611 0 ustar ligges users source("incl/start.R")
library("listenv")
plan(multiprocess)
message("*** multiprocess() ...")
for (cores in 1:availCores) {
message(sprintf("Testing with %d cores ...", cores))
options(mc.cores = cores)
## No global variables
f <- multiprocess({
42L
})
print(f)
stopifnot(inherits(f, "MultiprocessFuture") || inherits(f, "SequentialFuture"))
print(resolved(f))
y <- value(f)
print(y)
stopifnot(y == 42L)
## A global variable
a <- 0
f <- multiprocess({
b <- 3
c <- 2
a * b * c
})
print(f)
## A multiprocess future is evaluated in a separate
## R process. Changing the value of a global
## variable should not affect the result of the
## future.
a <- 7 ## Make sure globals are frozen
v <- value(f)
print(v)
stopifnot(v == 0)
message("*** multiprocess() with globals and blocking")
x <- listenv()
for (ii in 1:4) {
message(sprintf(" - Creating multiprocess future #%d ...", ii))
x[[ii]] <- multiprocess({ ii })
}
message(sprintf(" - Resolving %d multiprocess futures", length(x)))
v <- sapply(x, FUN = value)
stopifnot(all(v == 1:4))
message("*** multiprocess() and errors")
f <- multiprocess({
stop("Whoops!")
1
})
print(f)
v <- value(f, signal = FALSE)
print(v)
stopifnot(inherits(v, "simpleError"))
res <- try(value(f), silent = TRUE)
print(res)
stopifnot(inherits(res, "try-error"))
## Error is repeated
res <- try(value(f), silent = TRUE)
print(res)
stopifnot(inherits(res, "try-error"))
message("*** multiprocess() - too large globals ...")
ooptsT <- options(future.globals.maxSize = object.size(1:1014))
limit <- getOption("future.globals.maxSize")
cat(sprintf("Max total size of globals: %g bytes\n", limit))
for (workers in unique(c(1L, availableCores()))) {
message("Max number of processes: ", workers)
## A large object
a <- 1:1014
yTruth <- sum(a)
size <- object.size(a)
cat(sprintf("a: %g bytes\n", size))
f <- multiprocess({ sum(a) }, workers = workers)
print(f)
rm(list = "a")
v <- value(f)
print(v)
stopifnot(v == yTruth)
## A too large object
a <- 1:1019 ## also on 32-bit platforms
yTruth <- sum(a)
size <- object.size(a)
cat(sprintf("a: %g bytes\n", size))
res <- try(f <- multiprocess({ sum(a) }, workers = workers), silent = TRUE)
rm(list = "a")
stopifnot(inherits(res, "try-error"))
}
## Undo options changed in this test
options(ooptsT)
message("*** multiprocess() - too large globals ... DONE")
message("*** multiprocess(..., workers = 1L) ...")
a <- 2
b <- 3
yTruth <- a * b
f <- multiprocess({ a * b }, workers = 1L)
rm(list = c("a", "b"))
v <- value(f)
print(v)
stopifnot(v == yTruth)
message("*** multiprocess(..., workers = 1L) ... DONE")
message(sprintf("Testing with %d cores ... DONE", cores))
} ## for (cores ...)
message("*** multiprocess() ... DONE")
source("incl/end.R")
future/tests/globals,tricky_recursive.R 0000644 0001762 0000144 00000006240 13111756706 020076 0 ustar ligges users source("incl/start.R")
## Test adopted from http://stackoverflow.com/questions/42561088/nested-do-call-within-a-foreach-dopar-environment-cant-find-function-passed-w
options(future.debug = FALSE)
message("*** Tricky globals requiring recursive search ...")
my_add <- function(a, b) a + b
call_my_add <- function(a, b) {
do.call(my_add, args = list(a = a, b = b))
}
call_my_add_caller <- function(a, b, FUN = call_my_add) {
do.call(FUN, args = list(a = a, b = b))
}
main_future <- function(x = 1L, caller = call_my_add_caller,
args = list(FUN = call_my_add)) {
f <- future(caller(a = x, b = x + 1L, FUN = args$FUN))
value(f)
}
main_future_no_FUN <- function(x = 1L, caller = call_my_add_caller,
args = list(FUN = call_my_add)) {
f <- future(caller(a = x, b = x + 1L))
value(f)
}
main_futureCall <- function(x = 1L, caller = call_my_add_caller,
args = list(FUN = call_my_add)) {
f <- futureCall(caller, args = c(list(a = x, b = x+1L), args))
value(f)
}
main_futureCall_no_FUN <- function(x = 1L, caller = call_my_add_caller,
args = list(FUN = call_my_add)) {
f <- futureCall(caller, args = list(a = x, b = x+1L))
value(f)
}
main_lapply <- function(x = 1:2, caller = call_my_add_caller,
args = list(FUN = call_my_add)) {
lapply(x, FUN = function(i) {
do.call(caller, args = c(list(a = i, b = i+1L), args))
})
}
main_lapply_no_FUN <- function(x = 1:2, caller = call_my_add_caller,
args = list(FUN = call_my_add)) {
lapply(x, FUN = function(i) {
do.call(caller, args = list(a = i, b = i+1L))
})
}
main_future_lapply <- function(x = 1:2, caller = call_my_add_caller,
args = list(FUN = call_my_add)) {
future_lapply(x, FUN = function(i) {
do.call(caller, args = c(list(a = i, b = i + 1L), args))
})
}
main_future_lapply_no_FUN <- function(x = 1:2, caller = call_my_add_caller,
args = list(FUN = call_my_add)) {
future_lapply(x, FUN = function(i) {
do.call(caller, args = list(a = i, b = i + 1L))
})
}
x0 <- y0 <- z0 <- NULL
for (strategy in supportedStrategies()) {
message(sprintf("*** strategy = %s ...", sQuote(strategy)))
plan(strategy)
x <- main_future()
str(list(x = x))
if (is.null(x0)) x0 <- x
stopifnot(identical(x, x0))
x2 <- main_future_no_FUN()
str(list(x2 = x2))
stopifnot(identical(x2, x0))
y <- main_futureCall()
str(list(y = y))
if (is.null(y0)) y0 <- y
stopifnot(identical(y, y0))
y2 <- main_futureCall_no_FUN()
str(list(y2 = y2))
stopifnot(identical(y2, y0))
z <- main_lapply()
str(list(z = z))
if (is.null(z0)) z0 <- z
stopifnot(identical(z, z0))
z2 <- main_lapply_no_FUN()
str(list(z2 = z2))
stopifnot(identical(z2, z0))
z3 <- main_future_lapply()
str(list(z3 = z3))
stopifnot(identical(z3, z0))
z4 <- main_future_lapply_no_FUN()
str(list(z4 = z4))
stopifnot(identical(z4, z0))
message(sprintf("*** strategy = %s ... DONE", sQuote(strategy)))
}
message("*** Tricky globals requiring recursive search ... DONE")
source("incl/end.R")
future/tests/futureOf_with_environment.R 0000644 0001762 0000144 00000004001 13111756706 020331 0 ustar ligges users source("incl/start.R")
suppressWarnings(rm(list = c("x", "z")))
message("*** futureOf() with environment ...")
message("*** futureOf() with environment - future assignments ...")
x <- new.env()
x$a %<-% { 1 }
f1 <- futureOf("a", envir = x)
print(f1)
f2 <- futureOf(a, envir = x)
f3 <- futureOf(x[["a"]])
f4 <- futureOf(x$a)
stopifnot(identical(f2, f1), identical(f3, f1), identical(f4, f1))
## Identify all futures
fs <- futureOf(envir = x)
print(fs)
stopifnot(identical(names(fs), c("a")))
stopifnot(identical(fs$a, f1))
fsD <- futureOf(envir = x, drop = TRUE)
print(fsD)
stopifnot(all(sapply(fsD, FUN = inherits, "Future")))
stopifnot(identical(fsD, fs))
message("*** futureOf() with environment - future assignments ... DONE")
message("*** futureOf() with environment - futures ...")
x <- new.env()
x$a <- future({ 1 })
f1 <- futureOf("a", envir = x)
print(f1)
stopifnot(identical(f1, x$a))
f2 <- futureOf(a, envir = x)
stopifnot(identical(f2, x$a))
f3 <- futureOf(x[["a"]])
stopifnot(identical(f3, x$a))
f4 <- futureOf(x$a)
stopifnot(identical(f4, x$a))
## Identify all futures
fs <- futureOf(envir = x)
print(fs)
stopifnot(identical(names(fs), c("a")))
stopifnot(identical(fs$a, f1))
fsD <- futureOf(envir = x, drop = TRUE)
print(fsD)
stopifnot(all(sapply(fsD, FUN = inherits, "Future")))
stopifnot(identical(fsD, fs))
message("*** futureOf() with environment - futures ... DONE")
message("*** futureOf() with environment - exceptions ...")
## Invalid subset
res <- tryCatch(futureOf(x[[0]], mustExist = FALSE), error = identity)
stopifnot(inherits(res, "error"))
res <- tryCatch(futureOf(x[[0]], mustExist = TRUE), error = identity)
stopifnot(inherits(res, "error"))
res <- tryCatch(futureOf(x[[10]], mustExist = TRUE), error = identity)
stopifnot(inherits(res, "error"))
res <- tryCatch(futureOf(x[[1 + 2i]], mustExist = TRUE), error = identity)
stopifnot(inherits(res, "error"))
message("*** futureOf() with environment - exceptions ... DONE")
message("*** futureOf() with environment ... DONE")
source("incl/end.R")
future/tests/availableWorkers.R 0000644 0001762 0000144 00000010122 13111756706 016351 0 ustar ligges users source("incl/start.R")
message("*** availableWorkers() ...")
## The default
w <- availableWorkers()
print(w)
stopifnot(is.character(w), length(w) >= 1)
## Minimium of all known settings (default)
print(availableWorkers(which = "min"))
## Maximum of all known settings (should never be used)
print(availableWorkers(which = "max"))
## All known settings
print(availableWorkers(na.rm = FALSE, which = "all"))
## System settings
w <- availableWorkers(methods = "system")
print(w)
stopifnot(is.character(w), length(w) >= 1)
## Predefined ones for known cluster schedulers
print(availableWorkers(methods = "PBS"))
print(availableWorkers(methods = "SGE"))
print(availableWorkers(methods = "Slurm"))
message("*** HPC related ...")
expand_nodes <- future:::expand_nodes
read_pbs_nodefile <- future:::read_pbs_nodefile
read_pe_hostfile <- future:::read_pe_hostfile
workers0 <- c("n1", "n2", "n3", "n1", "n6", "n3", "n3", "n5")
data0 <- as.data.frame(table(workers0), stringsAsFactors = FALSE)
colnames(data0) <- c("node", "count")
data0 <- data0[order(data0$node, data0$count), ]
message("*** read_pbs_nodefile() ...")
workers <- workers0
pathname <- tempfile()
writeLines(workers, con = pathname)
data <- read_pbs_nodefile(pathname)
str(data)
stopifnot(
c("node") %in% colnames(data),
is.character(data$node),
!anyNA(data$node),
nrow(data$node) == length(workers),
all(sort(data$node) == sort(workers))
)
Sys.setenv(PBS_NODEFILE = pathname)
Sys.setenv(PBS_NP = length(workers),
PBS_NUM_NODES = length(workers) / 2,
PBS_NUM_PPN = 2)
workers <- availableWorkers(methods = "PBS")
print(workers)
stopifnot(length(workers) == length(workers0), all(workers == sort(workers0)))
Sys.setenv(PBS_NUM_PPN = 3)
res <- tryCatch({
workers <- availableWorkers(methods = "PBS")
}, warning = identity)
stopifnot(inherits(res, "warning"))
Sys.setenv(PBS_NP = length(workers) + 1)
res <- tryCatch({
workers <- availableWorkers(methods = "PBS")
}, warning = identity)
stopifnot(inherits(res, "warning"))
## Exceptions
workersE <- c(workers, "n 3")
pathname <- tempfile()
writeLines(workersE, con = pathname)
res <- tryCatch(read_pbs_nodefile(pathname), error = identity)
print(res)
stopifnot(inherits(res, "error"))
Sys.setenv(PBS_NODEFILE = "")
res <- tryCatch({
workers <- availableWorkers(methods = "PBS")
}, warning = identity)
stopifnot(inherits(res, "warning"))
message("*** read_pbs_nodefile() ... DONE")
message("*** read_pe_hostfile() ...")
workers <- workers0
pathname <- tempfile()
write.table(data0, file = pathname, quote = FALSE, row.names = FALSE, col.names = FALSE)
lines <- readLines(pathname)
print(lines)
data <- read_pe_hostfile(pathname)
print(data)
stopifnot(
is.character(data$node),
!anyNA(data$node),
is.integer(data$count),
!anyNA(data$count),
all(is.finite(data$count)),
all(data$count > 0),
nrow(data) == nrow(data0),
all.equal(data[, c("node", "count")], data0[, c("node", "count")])
)
workers <- expand_nodes(data)
stopifnot(length(workers) == length(workers0), all(workers == sort(workers0)))
Sys.setenv(PE_HOSTFILE = pathname)
Sys.setenv(NSLOTS = length(workers0)) ## Use to validate results
workers <- availableWorkers(methods = "SGE")
print(workers)
stopifnot(length(workers) == length(workers0), all(workers == sort(workers0)))
## Test validation
Sys.setenv(NSLOTS = length(workers0) + 1L)
workers <- tryCatch(availableWorkers(methods = "SGE"), warning = identity)
print(workers)
stopifnot(inherits(workers, "warning"))
Sys.setenv(PE_HOSTFILE = "")
res <- tryCatch({
workers <- availableWorkers(methods = "SGE")
}, warning = identity)
stopifnot(inherits(res, "warning"))
message("*** read_pe_hostfile() ... DONE")
message("*** HPC related ... DONE")
## Any R options and system environment variable
print(availableWorkers(methods = c("width", "FOO_BAR_ENV"),
na.rm = FALSE, which = "all"))
## Exception handling
Sys.setenv("FOO_BAR_ENV" = "0")
res <- tryCatch(availableWorkers(methods = "FOO_BAR_ENV"), error = identity)
stopifnot(inherits(res, "error"))
message("*** availableWorkers() ... DONE")
source("incl/end.R")
future/tests/startup.R 0000644 0001762 0000144 00000020054 13111756521 014556 0 ustar ligges users source("incl/start.R")
maxCores <- min(2L, availableCores(methods = "system"))
plan("default")
strategy0 <- plan()
message("*** parseCmdArgs() ...")
args <- parseCmdArgs()
str(args)
options(future.plan = NULL, future.cmdargs = c("-p", 1L))
args <- parseCmdArgs()
str(args)
stopifnot(args$p == 1L)
options(future.plan = NULL, future.cmdargs = c(sprintf("--parallel=%d", maxCores)))
args <- parseCmdArgs()
str(args)
stopifnot(args$p == maxCores)
options(future.plan = NULL, future.cmdargs = c("-p", 1L, sprintf("--parallel=%d", maxCores)))
args <- parseCmdArgs()
str(args)
stopifnot(args$p == maxCores)
options(future.plan = NULL, future.cmdargs = c("-p", 0L))
args <- parseCmdArgs()
stopifnot(is.null(args$p))
res <- tryCatch(parseCmdArgs(), warning = function(w) w)
stopifnot(inherits(res, "warning"))
options(future.plan = NULL, future.cmdargs = c("-p", .Machine$integer.max))
args <- parseCmdArgs()
stopifnot(is.null(args$p))
res <- tryCatch(parseCmdArgs(), warning = function(w) w)
stopifnot(inherits(res, "warning"))
options(future.plan = NULL, future.cmdargs = NULL)
message("*** parseCmdArgs() ... DONE")
message("*** .onLoad() ...")
plan("default")
pkgname <- "future"
message("- .onLoad() w/out command-line options ...")
options(future.plan = NULL, future.cmdargs = NULL)
.onLoad(pkgname, pkgname)
strategy <- plan()
print(strategy)
stopifnot(all(class(strategy) == class(strategy0)))
plan("default")
message("- .onLoad() w/out command-line options ... DONE")
message("- .onLoad() w/ -p 1 ...")
options(future.plan = NULL, future.cmdargs = c("-p", 1))
.onLoad(pkgname, pkgname)
strategy <- plan()
print(strategy)
stopifnot(inherits(strategy, "sequential"))
plan("default")
message("- .onLoad() w/ -p 1 ... DONE")
message("- .onLoad() w/ --parallel=1 ...")
plan("default")
options(future.plan = NULL, future.cmdargs = "-parallel=1")
.onLoad(pkgname, pkgname)
strategy <- plan()
print(strategy)
stopifnot(inherits(strategy, "sequential"))
plan("default")
message("- .onLoad() w/ --parallel=1 ... DONE")
message("- .onLoad() w/ -p 2 ...")
options(future.plan = NULL, future.cmdargs = c("-p", 2))
.onLoad(pkgname, pkgname)
strategy <- plan()
print(strategy)
if (maxCores >= 2) {
stopifnot(inherits(strategy, "multiprocess"))
} else {
stopifnot(all(class(strategy) == class(strategy0)))
}
plan("default")
message("- .onLoad() w/ -p 2 ... DONE")
message("- .onLoad() w/ -p 0 ...")
options(future.plan = NULL, future.cmdargs = c("-p", 0))
.onLoad(pkgname, pkgname)
strategy <- plan()
print(strategy)
stopifnot(all(class(strategy) == class(strategy0)))
plan("default")
message("- .onLoad() w/ -p 0 ... DONE")
message("- .onLoad() w/ -p -1 ...")
options(future.plan = NULL, future.cmdargs = c("-p", -1))
.onLoad(pkgname, pkgname)
strategy <- plan()
print(strategy)
stopifnot(all(class(strategy) == class(strategy0)))
plan("default")
message("- .onLoad() w/ -p -1 ... DONE")
message("- .onLoad() w/ -p foo ...")
options(future.plan = NULL, future.cmdargs = c("-p", "foo"))
.onLoad(pkgname, pkgname)
strategy <- plan()
print(strategy)
stopifnot(all(class(strategy) == class(strategy0)))
plan("default")
message("- .onLoad() w/ -p foo ... DONE")
message("- .onLoad() w/ R_FUTURE_PLAN = 'multisession' ...")
Sys.setenv(R_FUTURE_PLAN = "multisession")
options(future.plan = NULL, future.cmdargs = NULL)
.onLoad(pkgname, pkgname)
strategy <- plan()
print(strategy)
stopifnot(inherits(strategy, "multisession"))
plan("default")
Sys.setenv(R_FUTURE_PLAN = "")
message("- .onLoad() w/ R_FUTURE_PLAN = 'multisession' ... DONE")
message("- .onLoad() w/ future.plan = 'multisession' ...")
options(future.plan = NULL, future.plan = 'multisession', future.cmdargs = NULL)
.onLoad(pkgname, pkgname)
strategy <- plan()
print(strategy)
stopifnot(inherits(strategy, "multisession"))
plan("default")
message("- .onLoad() w/ future.plan = 'multisession' ... DONE")
message("- .onLoad() w/ R_FUTURE_PLAN = 'multisession' & -p 1 ...")
Sys.setenv(R_FUTURE_PLAN = "multisession")
options(future.plan = NULL, future.cmdargs = c("-p", 1))
.onLoad(pkgname, pkgname)
strategy <- plan()
print(strategy)
stopifnot(inherits(strategy, "multisession"))
plan("default")
Sys.setenv(R_FUTURE_PLAN = "")
message("- .onLoad() w/ R_FUTURE_PLAN = 'multisession' & -p 1 ... DONE")
message("- .onLoad() w/ future.plan = 'multisession' & -p 1 ...")
options(future.plan = 'multisession', future.cmdargs = c("-p", "1"))
.onLoad(pkgname, pkgname)
strategy <- plan()
print(strategy)
stopifnot(inherits(strategy, "multisession"))
plan("default")
message("- .onLoad() w/ future.plan = 'multisession' & -p 1 ... DONE")
message("- .onLoad() w/ future.plan = 'multisession' & -p 1 ...")
options(future.plan = multisession, future.cmdargs = c("-p", "1"))
.onLoad(pkgname, pkgname)
strategy <- plan()
print(strategy)
stopifnot(inherits(strategy, "multisession"))
plan("default")
message("- .onLoad() w/ future.plan = 'multisession' & -p 1 ... DONE")
message("- .onLoad() w/ future.availableCores.system = 1L ...")
options(future.availableCores.system = 1L)
.onLoad(pkgname, pkgname)
options(future.availableCores.system = NULL)
message("- .onLoad() w/ future.availableCores.system = 1L ... DONE")
message("- .onLoad() w/ R_FUTURE_AVAILABLECORES_SYSTEM ...")
Sys.setenv(R_FUTURE_AVAILABLECORES_SYSTEM = "1")
.onLoad(pkgname, pkgname)
ncores <- getOption("future.availableCores.system")
print(ncores)
stopifnot(is.integer(ncores), ncores == 1L)
Sys.unsetenv("R_FUTURE_AVAILABLECORES_SYSTEM")
options(future.availableCores.system = NULL)
Sys.setenv(R_FUTURE_AVAILABLECORES_SYSTEM = "NA")
.onLoad(pkgname, pkgname)
ncores <- getOption("future.availableCores.system")
print(ncores)
stopifnot(is.integer(ncores), is.na(ncores))
Sys.unsetenv("R_FUTURE_AVAILABLECORES_SYSTEM")
options(future.availableCores.system = NULL)
Sys.setenv(R_FUTURE_AVAILABLECORES_SYSTEM = "NA_real_")
.onLoad(pkgname, pkgname)
ncores <- getOption("future.availableCores.system")
print(ncores)
stopifnot(is.integer(ncores), is.na(ncores))
Sys.unsetenv("R_FUTURE_AVAILABLECORES_SYSTEM")
options(future.availableCores.system = NULL)
message("- .onLoad() w/ R_FUTURE_AVAILABLECORES_SYSTEM ... DONE")
message("- .onLoad() w/ future.availableCores.fallback = 1L ...")
options(future.availableCores.fallback = 1L)
.onLoad(pkgname, pkgname)
options(future.availableCores.fallback = NULL)
message("- .onLoad() w/ future.availableCores.fallback = 1L ... DONE")
message("- .onLoad() w/ R_FUTURE_AVAILABLECORES_FALLBACK ...")
Sys.setenv(R_FUTURE_AVAILABLECORES_FALLBACK = "1")
.onLoad(pkgname, pkgname)
ncores <- getOption("future.availableCores.fallback")
print(ncores)
stopifnot(is.integer(ncores), ncores == 1L)
Sys.unsetenv("R_FUTURE_AVAILABLECORES_FALLBACK")
options(future.availableCores.fallback = NULL)
Sys.setenv(R_FUTURE_AVAILABLECORES_FALLBACK = "NA")
.onLoad(pkgname, pkgname)
ncores <- getOption("future.availableCores.fallback")
print(ncores)
stopifnot(is.integer(ncores), is.na(ncores))
Sys.unsetenv("R_FUTURE_AVAILABLECORES_FALLBACK")
options(future.availableCores.fallback = NULL)
Sys.setenv(R_FUTURE_AVAILABLECORES_FALLBACK = "NA_real_")
.onLoad(pkgname, pkgname)
ncores <- getOption("future.availableCores.fallback")
print(ncores)
stopifnot(is.integer(ncores), is.na(ncores))
Sys.unsetenv("R_FUTURE_AVAILABLECORES_FALLBACK")
options(future.availableCores.fallback = NULL)
message("- .onLoad() w/ R_FUTURE_AVAILABLECORES_FALLBACK ... DONE")
options(future.plan = NULL, future.cmdargs = NULL, future.availableCores.system = NULL, future.availableCores.fallback = NULL)
message("*** .onLoad() ... DONE")
message("*** .onAttach() ...")
pkgname <- "future"
message("- .onAttach() w/ option future.startup.loadScript ...")
for (value in list(NULL, FALSE, TRUE)) {
options(future.startup.loadScript = value)
.onAttach(pkgname, pkgname)
}
message("- .onAttach() w/ option future.startup.loadScript ... DONE")
message("- .onAttach() with ./.future.R ...")
pathname <- ".future.R"
xyz <- 0L
cat("xyz <- 42L; cat('ping\n')\n", file = pathname)
.onAttach(pkgname, pkgname)
print(xyz)
stopifnot(is.integer(xyz), xyz >= 0, xyz == 42L)
file.remove(pathname)
message("- .onAttach() with ./.future.R ... DONE")
message("*** .onAttach() ... DONE")
source("incl/end.R")
future/tests/availableCores.R 0000644 0001762 0000144 00000003366 13111756706 016004 0 ustar ligges users source("incl/start.R")
message("*** availableCores() ...")
## detectCores() may return NA_integer_
n <- parallel::detectCores()
message(sprintf("detectCores() = %d", n))
stopifnot(length(n) == 1, is.numeric(n))
## Default
n <- availableCores()
message(sprintf("availableCores() = %d", n))
stopifnot(length(n) == 1, is.numeric(n), n >= 1)
## Minimium of all known settings (default)
print(availableCores(which = "min"))
## Maximum of all known settings (should never be used)
print(availableCores(which = "max"))
## All known settings
print(availableCores(na.rm = FALSE, which = "all"))
## System settings
n <- availableCores(methods = "system")
print(n)
stopifnot(length(n) == 1, is.numeric(n), is.finite(n), n >= 1)
## Predefined ones for known cluster schedulers
print(availableCores(methods = "PBS"))
print(availableCores(methods = "SGE"))
print(availableCores(methods = "Slurm"))
## Any R options and system environment variable
print(availableCores(methods = c("width", "FOO_BAR_ENV"),
na.rm = FALSE, which = "all"))
## Exception handling
Sys.setenv("FOO_BAR_ENV" = "0")
res <- try(availableCores(methods = "FOO_BAR_ENV"), silent = TRUE)
stopifnot(inherits(res, "try-error"))
message("*** Internal detectCores() ...")
## Option 'future.availableCores.system'
env <- environment(future:::detectCores)
env$res <- NULL
options(future.availableCores.system = 2L)
n <- detectCores()
print(n)
stopifnot(is.integer(n), is.finite(n), n >= 1, n == 2L)
options(future.availableCores.system = NULL)
## Reset
env <- environment(future:::detectCores)
env$res <- NULL
n <- detectCores()
print(n)
stopifnot(is.integer(n), is.finite(n), n >= 1)
message("*** Internal detectCores() ... DONE")
message("*** availableCores() ... DONE")
source("incl/end.R")
future/tests/FutureRegistry.R 0000644 0001762 0000144 00000006146 13111756706 016072 0 ustar ligges users source("incl/start.R")
message("*** FutureRegistry() ...")
for (where in c("multicore", "rscript")) {
message(sprintf("*** FutureRegistry('%s', 'list') ...", where))
futures <- FutureRegistry(where, action = "list")
print(futures)
stopifnot(length(futures) == 0L)
message(sprintf("*** FutureRegistry('%s', 'add') ...", where))
f <- future({ 1 })
print(f)
FutureRegistry(where, action = "add", future = f)
message(sprintf("*** FutureRegistry('%s', 'list') ...", where))
futures <- FutureRegistry(where, action = "list")
print(futures)
stopifnot(length(futures) == 1L)
message(sprintf("*** FutureRegistry('%s', 'remove') ...", where))
FutureRegistry(where, action = "remove", future = f)
message(sprintf("*** FutureRegistry('%s', 'list') ...", where))
futures <- FutureRegistry(where, action = "list")
print(futures)
stopifnot(length(futures) == 0L)
message(sprintf("*** FutureRegistry('%s', 'add') ...", where))
f <- future({ 2 })
print(f)
FutureRegistry(where, action = "add", future = f)
message(sprintf("*** FutureRegistry('%s', 'list') ...", where))
futures <- FutureRegistry(where, action = "list")
print(futures)
stopifnot(length(futures) == 1L)
message(sprintf("*** FutureRegistry('%s', 'collect-first') ...", where))
FutureRegistry(where, action = "collect-first")
futures <- FutureRegistry(where, action = "list")
print(futures)
stopifnot(length(futures) < 1L)
message(sprintf("*** FutureRegistry('%s', 'add') ...", where))
f <- future({ 2 })
print(f)
FutureRegistry(where, action = "add", future = f)
message(sprintf("*** FutureRegistry('%s', 'reset') ...", where))
FutureRegistry(where, action = "reset")
message(sprintf("*** FutureRegistry('%s', 'list') ...", where))
futures <- FutureRegistry(where, action = "list")
print(futures)
stopifnot(length(futures) == 0L)
}
message("*** FutureRegistry() - exceptions ...")
futures <- FutureRegistry(where = "test", action = "list")
stopifnot(length(futures) == 0)
f <- future(1)
FutureRegistry(where = "test", action = "add", future = f)
futures <- FutureRegistry(where = "test", action = "list")
stopifnot(length(futures) == 1)
res <- tryCatch(FutureRegistry(where = "test", action = "add", future = f), error = identity)
stopifnot(inherits(res, "error"))
futures <- FutureRegistry(where = "test", action = "list")
stopifnot(length(futures) == 1)
FutureRegistry(where = "test", action = "remove", future = f)
futures <- FutureRegistry(where = "test", action = "list")
stopifnot(length(futures) == 0)
res <- tryCatch(FutureRegistry(where = "test", action = "remove", future = f), error = identity)
stopifnot(inherits(res, "error"))
futures <- FutureRegistry(where = "test", action = "list")
stopifnot(length(futures) == 0)
FutureRegistry(where = "test", action = "reset")
futures <- FutureRegistry(where = "test", action = "list")
stopifnot(length(futures) == 0)
res <- tryCatch(FutureRegistry(where = "test", action = ""), error = identity)
stopifnot(inherits(res, "error"))
message("*** FutureRegistry() - exceptions ... DONE")
message("*** FutureRegistry() ... DONE")
source("incl/end.R")
future/tests/incl/ 0000755 0001762 0000144 00000000000 13237670254 013664 5 ustar ligges users future/tests/incl/start,load-only.R 0000644 0001762 0000144 00000004734 13237667011 017044 0 ustar ligges users ## Record original state
ovars <- ls()
oenvs <- oenvs0 <- Sys.getenv()
oopts0 <- options()
covr_testing <- ("covr" %in% loadedNamespaces())
on_solaris <- grepl("^solaris", R.version$os)
## Default options
oopts <- options(
warn = 1L,
mc.cores = 2L,
future.debug = TRUE,
## Reset the following during testing in case
## they are set on the test system
future.availableCores.system = NULL,
future.availableCores.fallback = NULL
)
## Reset the following during testing in case
## they are set on the test system
oenvs2 <- Sys.unsetenv(c(
"R_FUTURE_AVAILABLECORES_SYSTEM",
"R_FUTURE_AVAILABLECORES_FALLBACK",
## SGE
"NSLOTS", "PE_HOSTFILE",
## Slurm
"SLURM_CPUS_PER_TASK",
## TORQUE / PBS
"PBS_NUM_PPN", "PBS_NODEFILE", "PBS_NP", "PBS_NUM_NODES"
))
oplan <- future::plan()
## Use eager futures by default
future::plan("sequential")
## Private future functions
.onLoad <- future:::.onLoad
.onAttach <- future:::.onAttach
asIEC <- future:::asIEC
ClusterRegistry <- future:::ClusterRegistry
constant <- future:::constant
detectCores <- future:::detectCores
fold <- future:::fold
future_lapply <- future:::future_lapply
FutureRegistry <- future:::FutureRegistry
gassign <- future:::gassign
get_future <- future:::get_future
geval <- future:::geval
grmall <- future:::grmall
hpaste <- future:::hpaste
importParallel <- future:::importParallel
mdebug <- future:::mdebug
myExternalIP <- future:::myExternalIP
myInternalIP <- future:::myInternalIP
parseCmdArgs <- future:::parseCmdArgs
requestCore <- future:::requestCore
requestNode <- future:::requestNode
requirePackages <- future:::requirePackages
tweakExpression <- future:::tweakExpression
whichIndex <- future:::whichIndex
get_random_seed <- future:::get_random_seed
set_random_seed <- future:::set_random_seed
as_lecyer_cmrg_seed <- future:::as_lecyer_cmrg_seed
## Local functions for test scripts
printf <- function(...) cat(sprintf(...))
mstr <- function(...) message(paste(capture.output(str(...)), collapse = "\n"))
attachLocally <- function(x, envir = parent.frame()) {
for (name in names(x)) {
assign(name, value = x[[name]], envir = envir)
}
}
supportedStrategies <- function(cores = 1L, excl = c("multiprocess", "cluster"), ...) {
strategies <- future:::supportedStrategies(...)
strategies <- setdiff(strategies, excl)
if (cores > 1) {
strategies <- setdiff(strategies,
c("sequential", "uniprocess", "eager", "lazy"))
}
strategies
}
availCores <- min(2L, future::availableCores())
future/tests/incl/end.R 0000644 0001762 0000144 00000002411 13111756706 014551 0 ustar ligges users ## Undo future strategy
future::plan(oplan)
## Undo options
## (a) Added
added <- setdiff(names(options()), names(oopts0))
opts <- vector("list", length = length(added))
names(opts) <- added
options(opts)
## (b) Modified
options(oopts)
## (c) Assert that everything was undone
stopifnot(identical(options(), oopts0))
## Undo system environment variables
## (a) Added
cenvs <- Sys.getenv()
added <- setdiff(names(cenvs), names(oenvs0))
for (name in added) Sys.unsetenv(name)
## (b) Missing
missing <- setdiff(names(oenvs0), names(cenvs))
if (length(missing) > 0) do.call(Sys.setenv, as.list(oenvs0[missing]))
## (c) Modified?
for (name in intersect(names(cenvs), names(oenvs0))) {
## WORKAROUND: On Linux Wine, base::Sys.getenv() may
## return elements with empty names. /HB 2016-10-06
if (nchar(name) == 0) next
if (!identical(cenvs[[name]], oenvs0[[name]])) {
do.call(Sys.setenv, as.list(oenvs0[name]))
}
}
## (d) Assert that everything was undone
stopifnot(identical(Sys.getenv(), oenvs0))
## Undo variables
rm(list = c(setdiff(ls(), ovars)))
## Travis CI specific: Explicit garbage collection because it
## looks like Travis CI might run out of memory during 'covr'
## testing and we now have so many tests. /HB 2017-01-11
if ("covr" %in% loadedNamespaces()) gc()
future/tests/incl/start.R 0000644 0001762 0000144 00000000063 12736356753 015153 0 ustar ligges users library("future")
source("incl/start,load-only.R")
future/tests/globalsOf,tweaks.R 0000644 0001762 0000144 00000001700 13111756521 016254 0 ustar ligges users source("incl/start.R")
library("globals")
message("*** tweakExpression() ...")
expr <- substitute({ a <<- 1; b <- 2; 3 ->> c }, env = list())
print(expr)
exprT <- tweakExpression(expr)
print(exprT)
b <- 2
exprs <- list(
A = substitute({ a <- b; }, env = list()),
B = substitute({ a <- b; b <- 1 }, env = list()),
C = substitute({ a <- 1; a <- 2 }, env = list()),
D = substitute({ a <<- 1; a <- 2 }, env = list()),
E = substitute({ a <<- 1 }, env = list())
)
truth <- list(
A = "b",
B = "b",
C = character(0L),
D = character(0L),
E = character(0L)
)
for (kk in seq_along(exprs)) {
name <- names(exprs)[kk]
expr <- exprs[[kk]]
cat(sprintf("Expression #%d ('%s'):", kk, name))
print(expr)
globals <- globalsOf(expr, tweak = tweakExpression, recursive = TRUE)
globals <- cleanup(globals)
str(globals)
stopifnot(identical(names(globals), truth[[name]]))
}
message("*** tweakExpression() ... DONE")
source("incl/end.R")
future/tests/cluster.R 0000644 0001762 0000144 00000017434 13237670205 014550 0 ustar ligges users source("incl/start.R")
library("listenv")
options(future.debug = FALSE)
message("*** cluster() ...")
message("Library paths: ", paste(sQuote(.libPaths()), collapse = ", "))
message("Package path: ", sQuote(system.file(package = "future")))
types <- "PSOCK"
if (supportsMulticore()) types <- c(types, "FORK")
## WORKAROUND: covr::package_coverage() -> merge_coverage() -> ... produces
## "Error in readRDS(x) : error reading from connection" for type = "FORK".
## Is this related to mcparallel() comments in help("package_coverage")?
## /HB 2017-05-20
if (covr_testing) types <- setdiff(types, "FORK")
pid <- Sys.getpid()
message("Main PID (original): ", pid)
cl <- NULL
for (type in types) {
message(sprintf("Test set #1 with cluster type %s ...", sQuote(type)))
for (cores in 1:availCores) {
message(sprintf("Testing with %d cores on type = %s ...",
cores, sQuote(type)))
options(mc.cores = cores)
## Set up a cluster with nodes (explicitly)
cl <- parallel::makeCluster(cores, type = type)
print(cl)
plan(cluster, workers = cl)
## No global variables
f <- try(cluster({
42L
}, workers = cl), silent = FALSE)
print(f)
stopifnot(inherits(f, "ClusterFuture"))
print(resolved(f))
y <- value(f)
print(y)
stopifnot(y == 42L)
## Set up a cluster with nodes (implicitly)
plan(cluster, workers = cores)
## No global variables
f <- try(cluster({
42L
}, workers = cl), silent = FALSE)
print(f)
stopifnot(inherits(f, "ClusterFuture"))
print(resolved(f))
y <- value(f)
print(y)
stopifnot(y == 42L)
## A global variable
a <- 0
f <- try(future({
b <- 3
c <- 2
a * b * c
}))
print(f)
## A cluster future is evaluated in a separate
## R session process. Changing the value of a global
## variable should not affect the result of the
## future.
a <- 7 ## Make sure globals are frozen
v <- value(f)
print(v)
stopifnot(v == 0)
message("*** cluster() with globals and blocking")
x <- listenv()
for (ii in 1:3) {
message(sprintf(" - Creating cluster future #%d ...", ii))
x[[ii]] <- future({ ii })
}
message(sprintf(" - Resolving %d cluster futures", length(x)))
v <- sapply(x, FUN = value)
stopifnot(all(v == 1:3))
message("*** cluster() and errors")
f <- future({
stop("Whoops!")
1
})
print(f)
v <- value(f, signal = FALSE)
print(v)
stopifnot(inherits(v, "simpleError"))
res <- tryCatch(value(f), error = identity)
print(res)
stopifnot(inherits(res, "error"))
## Error is repeated
res <- tryCatch(value(f), error = identity)
print(res)
stopifnot(inherits(res, "error"))
message("*** cluster() - too large globals ...")
ooptsT <- options(future.globals.maxSize = object.size(1:1014))
limit <- getOption("future.globals.maxSize")
cat(sprintf("Max total size of globals: %g bytes\n", limit))
## A large object
a <- 1:1014
yTruth <- sum(a)
size <- object.size(a)
cat(sprintf("a: %g bytes\n", size))
f <- future({ sum(a) })
print(f)
rm(list = "a")
v <- value(f)
print(v)
stopifnot(v == yTruth)
## A too large object
a <- 1:1015
yTruth <- sum(a)
size <- object.size(a)
cat(sprintf("a: %g bytes\n", size))
res <- tryCatch(f <- future({ sum(a) }), error = identity)
rm(list = "a")
stopifnot(inherits(res, "error"))
## Undo options changed in this test
options(ooptsT)
message("*** cluster() - too large globals ... DONE")
message("*** cluster() - installed libraries ...")
f <- try(cluster({
list(
libPaths = .libPaths()
)
}, workers = cl), silent = FALSE)
print(f)
stopifnot(inherits(f, "ClusterFuture"))
v <- value(f)
message(paste(capture.output(str(v)), collapse = "\n"))
message("*** cluster() - installed packages ... DONE")
message("*** cluster() - assert covr workaround ...")
f <- try(cluster({
future:::hpaste(1:100)
}, workers = cl), silent = FALSE)
print(f)
stopifnot(inherits(f, "ClusterFuture"))
v <- value(f)
message(v)
stopifnot(v == hpaste(1:100))
message("*** cluster() - assert covr workaround ... DONE")
message(sprintf("Testing with %d cores on type = %s ... DONE",
cores, sQuote(type)))
} ## for (cores ...)
message("*** cluster() - exceptions ...")
res <- tryCatch(cluster(42L, workers = NA), error = identity)
print(res)
stopifnot(inherits(res, "error"))
message("*** cluster() - exceptions ... DONE")
message("*** cluster() - assert registry behavior ...")
## Explicitly created clusters are *not* added to the registry
cl <- parallel::makeCluster(cores, type = type)
plan(cluster, workers = cl)
clR <- ClusterRegistry("get")
stopifnot(is.null(clR))
## ... and therefore changing plans shouldn't change anything
plan(sequential)
clR <- ClusterRegistry("get")
stopifnot(is.null(clR))
message("*** cluster() - assert registry behavior ... DONE")
## Sanity checks
pid2 <- Sys.getpid()
message("Main PID (original): ", pid)
message("Main PID: ", pid2)
stopifnot(pid2 == pid)
## Cleanup
print(cl)
str(cl)
parallel::stopCluster(cl)
plan(sequential)
message(sprintf("Test set #1 with cluster type %s ... DONE", sQuote(type)))
} ## for (type ...)
library("parallel")
for (type in types) {
if (on_solaris) next
message(sprintf("Test set #2 with cluster type %s ...", sQuote(type)))
message("*** cluster() - setDefaultCluster() ...")
cl <- makeCluster(1L, type = type)
print(cl)
setDefaultCluster(cl)
## FIXME: Make plan(cluster, workers = NULL) work such that
## setDefaultCluster() is actually tested.
plan(cluster)
pid <- Sys.getpid()
message(pid)
a %<-% Sys.getpid()
message(a)
setDefaultCluster(NULL)
message("*** cluster() - setDefaultCluster() ... DONE")
## Sanity checks
pid2 <- Sys.getpid()
message("Main PID (original): ", pid)
message("Main PID: ", pid2)
stopifnot(pid2 == pid)
## Cleanup
print(cl)
str(cl)
parallel::stopCluster(cl)
plan(sequential)
message(sprintf("Test set #2 with cluster type %s ... DONE", sQuote(type)))
} ## for (type ...)
for (type in types) {
if (on_solaris) next
message(sprintf("Test set #3 with cluster type %s ...", sQuote(type)))
cl <- parallel::makeCluster(1L, type = type)
print(cl)
## Crashing FORK:ed processes seems to harsh on R (< 3.2.0)
if (type != "FORK" || getRversion() >= "3.2.0") {
message("*** cluster() - crashed worker ...")
plan(cluster, workers = cl)
x %<-% 42L
stopifnot(x == 42L)
## Force R worker to quit
x %<-% quit(save = "no")
res <- tryCatch(y <- x, error = identity)
print(res)
stopifnot(
inherits(res, "simpleError"),
inherits(res, "FutureError")
)
## Cleanup
print(cl)
## FIXME: Why doesn't this work here? It causes the below future to stall.
# parallel::stopCluster(cl)
## Verify that the reset worked
cl <- parallel::makeCluster(1L, type = type)
print(cl)
plan(cluster, workers = cl)
x %<-% 42L
stopifnot(x == 42L)
message("*** cluster() - crashed worker ... DONE")
} ## if (type != "FORK" || getRversion() >= "3.2.0")
## Sanity checks
pid2 <- Sys.getpid()
message("Main PID (original): ", pid)
message("Main PID: ", pid2)
stopifnot(pid2 == pid)
## Cleanup
print(cl)
str(cl)
parallel::stopCluster(cl)
message(sprintf("Test set #3 with cluster type %s ... DONE", sQuote(type)))
} ## for (type ...)
message("*** cluster() ... DONE")
## Sanity checks
pid2 <- Sys.getpid()
message("Main PID (original): ", pid)
message("Main PID: ", pid2)
stopifnot(pid2 == pid)
source("incl/end.R")
future/tests/futureAssign_OP_with_listenv.R 0000644 0001762 0000144 00000003622 13111756521 020732 0 ustar ligges users source("incl/start.R")
library("listenv")
message("*** %<-% to listenv ...")
## - - - - - - - - - - - - - - - - - - - - - - - - - - - -
## Future assignment via infix operator
## - - - - - - - - - - - - - - - - - - - - - - - - - - - -
z <- listenv()
stopifnot(length(names(z)) == 0)
message("*** %<-% to listenv: Assign by index")
z[[1]] %<-% { 2 } %lazy% TRUE
stopifnot(length(z) == 1)
stopifnot(length(names(z)) == 0)
z[[1]] %<-% { 2 } %lazy% TRUE
stopifnot(length(z) == 1)
stopifnot(length(names(z)) == 0)
z[[4]] %<-% { "async!" } %lazy% TRUE
stopifnot(length(z) == 4)
stopifnot(length(names(z)) == 0)
message("*** %<-% to listenv: Update names")
names(z) <- c("A", "B", "C", "D")
stopifnot(identical(names(z), c("A", "B", "C", "D")))
message("*** %<-% to listenv: Assign by name (existing)")
z$B %<-% { TRUE } %lazy% TRUE
stopifnot(length(z) == 4)
stopifnot(identical(names(z), c("A", "B", "C", "D")))
y <- as.list(z)
str(y)
stopifnot(length(y) == 4)
stopifnot(identical(names(y), c("A", "B", "C", "D")))
message("*** %<-% to listenv: Asserting no name clashes among futures")
u <- listenv()
u$a %<-% { 1 } %lazy% TRUE
stopifnot(identical(names(u), "a"))
fu <- futureOf(u$a)
v <- listenv()
v$a %<-% { 2 } %lazy% TRUE
stopifnot(identical(names(v), "a"))
fv <- futureOf(v$a)
stopifnot(!identical(fu, fv))
fu <- futureOf(u$a)
stopifnot(!identical(fu, fv))
stopifnot(identical(u$a, 1))
stopifnot(identical(v$a, 2))
message("*** %<-% to listenv: multiple dimensions ...")
x0 <- list()
length(x0) <- 6
dim(x0) <- c(3, 2)
x <- listenv()
length(x) <- 6
dim(x) <- c(3, 2)
for (cc in 1:ncol(x)) {
for (rr in 1:nrow(x)) {
x0[[rr, cc]] <- sprintf("(%s, %s)", rr, cc)
x[[rr, cc]] %<-% sprintf("(%s, %s)", rr, cc) %lazy% TRUE
}
}
y <- as.list(x)
dim(y) <- dim(x)
stopifnot(identical(y, x0))
message("*** %<-% to listenv: multiple dimensions ... DONE")
message("*** %<-% to listenv ... DONE")
source("incl/end.R")
future/tests/FutureError.R 0000644 0001762 0000144 00000001603 13111756521 015337 0 ustar ligges users source("incl/start.R")
message("*** FutureError class ...")
## Minimal
ex <- FutureError(message = "Woops")
print(ex)
cond <- tryCatch(message("Woops", appendLF = FALSE), message = identity)
ex2 <- FutureError(message = cond)
print(ex2)
stopifnot(conditionMessage(ex2) == conditionMessage(ex))
f <- future({ 42L; stop("woops") })
v <- value(f, signal = FALSE)
print(v)
ex <- FutureError(message = "Woops", future = f, output = c("Darn", "it"))
print(ex)
res <- getOutput(ex)
print(res)
stopifnot(all(res == c("Darn", "it")))
res <- getOutput(ex, head = 1L)
print(res)
stopifnot(res == "Darn")
res <- getOutput(ex, tail = 1L)
print(res)
stopifnot(res == "it")
res <- getOutput(ex, head = 1L, tail = 1L)
print(res)
stopifnot(res == c("Darn", "it"))
res <- getOutput(ex, collapse = "\n")
print(res)
stopifnot(res == "Darn\nit")
message("*** FutureError class ... DONE")
source("incl/end.R")
future/tests/demo.R 0000644 0001762 0000144 00000001537 13111756706 014012 0 ustar ligges users source("incl/start.R")
options(future.demo.mandelbrot.nrow = 2L)
options(future.demo.mandelbrot.resolution = 50L)
options(future.demo.mandelbrot.delay = FALSE)
for (cores in 1:availCores) {
message(sprintf("Testing with %d cores ...", cores))
options(mc.cores = cores)
message("*** Demos ...")
message("*** Mandelbrot demo of the 'future' package ...")
if (getRversion() >= "3.2.0") {
for (strategy in supportedStrategies(cores)) {
message(sprintf("- plan('%s') ...", strategy))
plan(strategy)
demo("mandelbrot", package = "future", ask = FALSE)
message(sprintf("- plan('%s') ... DONE", strategy))
}
} else {
message(" - This demo requires R (>= 3.2.0). Skipping test.")
}
message(sprintf("Testing with %d cores ... DONE", cores))
} ## for (cores ...)
message("*** Demos ... DONE")
source("incl/end.R")
future/tests/globals,tricky.R 0000644 0001762 0000144 00000011043 13237667011 016003 0 ustar ligges users source("incl/start.R")
library("listenv")
oopts <- c(oopts, options(
future.globals.resolve = TRUE,
future.globals.onMissing = "error"
))
message("*** Tricky use cases related to globals ...")
for (cores in 1:availCores) {
message(sprintf("Testing with %d cores ...", cores))
options(mc.cores = cores)
message("availableCores(): ", availableCores())
message("- Local variables with the same name as globals ...")
for (strategy in supportedStrategies(cores)) {
message(sprintf("- plan('%s') ...", strategy))
plan(strategy)
methods <- c("conservative", "ordered")
for (method in methods) {
options(future.globals.method = method)
message(sprintf("Method for identifying globals: '%s' ...", method))
a <- 3
yTruth <- local({
b <- a
a <- 2
a * b
})
y %<-% {
b <- a
a <- 2
a * b
}
rm(list = "a")
res <- try(y, silent = TRUE)
if (method == "conservative" && strategy %in% c("multisession", "cluster")) {
str(list(res = res))
stopifnot(inherits(res, "try-error"))
} else {
message(sprintf("y = %g", y))
stopifnot(identical(y, yTruth))
}
## Same with forced lazy evaluation
a <- 3
y %<-% {
b <- a
a <- 2
a * b
} %lazy% TRUE
rm(list = "a")
res <- try(y, silent = TRUE)
if (method == "conservative") {
str(list(res = res))
stopifnot(inherits(res, "try-error"))
} else {
message(sprintf("y = %g", y))
stopifnot(identical(y, yTruth))
}
res <- listenv()
a <- 1
for (ii in 1:3) {
res[[ii]] %<-% {
b <- a * ii
a <- 0
b
}
}
rm(list = "a")
res <- try(unlist(res), silent = TRUE)
if (method == "conservative" && strategy %in% c("multisession", "cluster")) {
str(list(res = res))
stopifnot(inherits(res, "try-error"))
} else {
print(res)
stopifnot(all(res == 1:3))
}
## Same with forced lazy evaluation
res <- listenv()
a <- 1
for (ii in 1:3) {
res[[ii]] %<-% {
b <- a * ii
a <- 0
b
} %lazy% TRUE
}
rm(list = "a")
res <- try(unlist(res), silent = TRUE)
if (method == "conservative") {
str(list(res = res))
stopifnot(inherits(res, "try-error"))
} else {
print(res)
stopifnot(all(res == 1:3))
}
## Assert that `a` is resolved and turned into a constant future
## at the moment when future `b` is created.
## Requires options(future.globals.resolve = TRUE).
a <- future(1)
b <- future(value(a) + 1)
rm(list = "a")
message(sprintf("value(b) = %g", value(b)))
stopifnot(value(b) == 2)
a <- future(1)
b <- future(value(a) + 1, lazy = TRUE)
rm(list = "a")
message(sprintf("value(b) = %g", value(b)))
stopifnot(value(b) == 2)
a <- future(1, lazy = TRUE)
b <- future(value(a) + 1)
rm(list = "a")
message(sprintf("value(b) = %g", value(b)))
stopifnot(value(b) == 2)
a <- future(1, lazy = TRUE)
b <- future(value(a) + 1, lazy = TRUE)
rm(list = "a")
message(sprintf("value(b) = %g", value(b)))
stopifnot(value(b) == 2)
## BUG FIX: In future (<= 1.0.0) a global 'pkg' would be
## overwritten by the name of the last package attached
## by the future.
pkg <- "foo"
f <- sequential({ pkg })
v <- value(f)
message(sprintf("value(f) = %s", sQuote(v)))
stopifnot(pkg == "foo", v == "foo")
message(sprintf("Method for identifying globals: '%s' ... DONE", method))
}
## BUG FIX: In globals (<= 0.10.3) a global 'x' in LHS of an assignment
## would be missed.
options(future.globals.method = "ordered")
## A local
x <- 1
f <- future({ x <- 0; x <- x + 1; x })
v <- value(f)
message(sprintf("value(f) = %s", sQuote(v)))
stopifnot(v == 1)
## A global
x <- 1
f <- future({ x <- x + 1; x })
v <- value(f)
message(sprintf("value(f) = %s", sQuote(v)))
stopifnot(v == 2)
## A global
x <- function() TRUE
f <- future({ x <- x(); x })
v <- value(f)
message(sprintf("value(f) = %s", sQuote(v)))
stopifnot(v == TRUE)
} ## for (strategy ...)
message(sprintf("Testing with %d cores ... DONE", cores))
} ## for (cores ...)
message("*** Tricky use cases related to globals ... DONE")
source("incl/end.R")
future/tests/requestCore.R 0000644 0001762 0000144 00000001540 13111756706 015361 0 ustar ligges users source("incl/start.R")
message("*** requestCore() ...")
message("*** requestCore() - exceptions ...")
## There are no cores to choose from
res <- try(requestCore(function() {}, workers = 0), silent = TRUE)
stopifnot(inherits(res, "try-error"))
res <- try(requestCore(function() {}, timeout = -1.0), silent = TRUE)
stopifnot(inherits(res, "try-error"))
res <- try(requestCore(function() {}, alpha = 0.0), silent = TRUE)
stopifnot(inherits(res, "try-error"))
message("*** requestCore() - exceptions ... DONE")
message("*** requestCore() - timeout ...")
plan(multicore, workers = 2L)
a %<-% { Sys.sleep(2); 1 }
res <- try(requestCore(function() {}, workers = 1L, timeout = 0.5, delta = 0.1))
stopifnot(inherits(res, "try-error"))
stopifnot(a == 1)
message("*** requestCore() - timeout ... DONE")
message("*** requestCore() ... DONE")
source("incl/end.R")
future/tests/FutureGlobals.R 0000644 0001762 0000144 00000001132 13111756521 015626 0 ustar ligges users source("incl/start.R")
message("*** FutureGlobals() ...")
fg1 <- FutureGlobals()
print(fg1)
fg2 <- FutureGlobals(fg1)
print(fg2)
g <- globals::as.Globals(list(a = 1, b = 1:3))
print(g)
fg3 <- FutureGlobals(g)
print(fg3)
fg4 <- as.FutureGlobals(g)
print(fg4)
print(resolved(fg3))
fg <- fg4
fg_unique <- unique(fg)
print(fg_unique)
fg_resolved <- resolve(fg)
print(fg_resolved)
message("- FutureGlobals() - exceptions ...")
res <- tryCatch(g <- FutureGlobals(NULL), error = identity)
print(res)
stopifnot(inherits(res, "error"))
message("*** FutureGlobals() ... DONE")
source("incl/end.R")
future/tests/futureCall.R 0000644 0001762 0000144 00000001036 13111756521 015161 0 ustar ligges users source("incl/start.R")
message("*** futureCall() ...")
f1 <- future(do.call(rnorm, args = list(n = 100)), lazy = TRUE)
f2 <- futureCall(rnorm, args = list(n = 100), lazy = TRUE)
set.seed(42L)
v0 <- rnorm(n = 100)
str(list(v0 = v0))
set.seed(42L)
v1 <- value(f1)
str(list(v1 = v1))
set.seed(42L)
v2 <- value(f2)
str(list(v2 = v2))
## Because we use lazy futures and set the
## random seed just before they are resolved
stopifnot(all.equal(v1, v0))
stopifnot(all.equal(v1, v2))
message("*** futureCall() ... DONE")
source("incl/end.R")
future/tests/nested_futures,mc.cores.R 0000644 0001762 0000144 00000006103 13111756521 017620 0 ustar ligges users source("incl/start.R")
library("listenv")
options(future.debug = FALSE)
message("*** Nested futures - mc.cores ...")
strategies <- c("multisession")
if (supportsMulticore()) strategies <- c(strategies, "multicore")
pid <- Sys.getpid()
cat(sprintf("Main PID: %d\n", pid))
cat("Available cores on this machine:\n")
cores <- availableCores()
print(cores)
for (mc in 1:2) {
message(sprintf("- mc.cores = %d ...", mc))
options(mc.cores = mc)
mc2 <- min(mc, cores)
for (strategy in strategies) {
message(sprintf("plan(list('sequential', '%s')):", strategy))
plan(list('sequential', strategy))
a %<-% {
b1 %<-% Sys.getpid()
b2 %<-% Sys.getpid()
list(pid = Sys.getpid(), cores = availableCores(), pid1 = b1, pid2 = b2)
}
print(a)
stopifnot(a$pid == pid)
stopifnot((mc2 <= 1 && a$pid1 == pid) || (a$pid1 != pid))
stopifnot((mc2 <= 1 && a$pid2 == pid) || (a$pid2 != pid))
stopifnot(((mc2 <= 1 || a$cores <= 2) && a$pid2 == a$pid1) || (a$pid2 != a$pid1))
message(sprintf("plan(list('sequential', '%s':2)):", strategy))
plan(list('sequential', tweak(strategy, workers = 2)))
a %<-% {
b1 %<-% Sys.getpid()
b2 %<-% Sys.getpid()
list(pid = Sys.getpid(), cores = availableCores(), pid1 = b1, pid2 = b2)
}
print(a)
stopifnot(a$pid == pid)
stopifnot((mc2 <= 1 && a$pid1 == pid) || (a$pid1 != pid))
stopifnot((mc2 <= 1 && a$pid2 == pid) || (a$pid2 != pid))
stopifnot((mc2 <= 1 && a$pid2 == a$pid1) || (a$pid2 != a$pid1))
message(sprintf("plan(list('%s', 'sequential')):", strategy))
plan(list(strategy, 'sequential'))
a %<-% {
b1 %<-% Sys.getpid()
b2 %<-% Sys.getpid()
list(pid = Sys.getpid(), cores = availableCores(), pid1 = b1, pid2 = b2)
}
print(a)
stopifnot((mc2 <= 1 && a$pid == pid) || (a$pid != pid))
stopifnot((mc2 <= 1 && a$pid1 == pid) || (a$pid1 != pid))
stopifnot((mc2 <= 1 && a$pid2 == pid) || (a$pid2 != pid))
stopifnot(a$pid2 == a$pid1)
message(sprintf("plan(list('%s', '%s')):", strategy, strategy))
plan(list(strategy, strategy))
a %<-% {
b1 %<-% { Sys.sleep(1); Sys.getpid() }
b2 %<-% Sys.getpid()
list(pid = Sys.getpid(), cores = availableCores(), pid1 = b1, pid2 = b2)
}
print(a)
stopifnot((mc2 <= 1 && a$pid == pid) || (a$pid != pid))
stopifnot((mc2 <= 1 && a$pid1 == pid) || (a$pid1 != pid))
stopifnot((mc2 <= 1 && a$pid2 == pid) || (a$pid2 != pid))
stopifnot(a$pid2 == a$pid1)
message(sprintf("plan(list('%s':2, '%s':2)):", strategy, strategy))
plan(list(tweak(strategy, workers = 2), tweak(strategy, workers = 2)))
a %<-% {
b1 %<-% Sys.getpid() ## This stalls
b2 %<-% Sys.getpid()
list(pid = Sys.getpid(), cores = availableCores(), pid1 = b1, pid2 = b2)
}
print(a)
stopifnot(a$pid != pid)
stopifnot(a$pid1 != pid)
stopifnot(a$pid2 != pid)
stopifnot(a$pid2 != a$pid1)
} ## for (strategy ...)
message(sprintf(" - mc.cores = %d ... DONE", mc))
} ## for (mc ...)
message("*** Nested futures - mc.cores ... DONE")
source("incl/end.R")
future/tests/ClusterRegistry.R 0000644 0001762 0000144 00000003536 13111756706 016241 0 ustar ligges users source("incl/start.R")
message("*** ClusterRegistry() ... ")
for (cores in 1:availCores) {
message(sprintf("Testing with %d cores ...", cores))
options(mc.cores = cores)
message("Available cores: ", availableCores())
## In case sessions have been created in previous tests
message("Stopping any running cluster ...")
ClusterRegistry("stop")
cluster <- ClusterRegistry("get")
stopifnot(length(cluster) == 0)
message("Stopping any running cluster ... DONE")
message("Starting cluster ...")
res <- tryCatch({
cluster <- ClusterRegistry("set", workers = availableCores() - 1L)
str(cluster)
print(cluster)
}, error = identity)
if (cores == 1) stopifnot(inherits(res, "error"))
message("Starting cluster ... DONE")
message("Starting dual-worker cluster ...")
cluster <- ClusterRegistry(action = "start", workers = 2L)
str(cluster)
print(cluster)
stopifnot(length(cluster) == 2L)
message("Starting duel-worker cluster ... DONE")
message("Starting single-worker cluster ...")
cluster <- ClusterRegistry(action = "start", workers = 1L)
str(cluster)
print(cluster)
stopifnot(length(cluster) == 1L)
message("Starting single-worker cluster ... DONE")
cluster <- ClusterRegistry(action = "get")
print(cluster)
stopifnot(length(cluster) == 1L)
message("Stopping single-worker cluster ...")
cluster <- ClusterRegistry(action = "stop")
print(cluster)
stopifnot(length(cluster) == 0L)
message("Stopping single-worker cluster ... DONE")
message(sprintf("Testing with %d cores ... DONE", cores))
} ## for (cores ...)
message("*** ClusterRegistry() - exceptions ...")
res <- tryCatch(ClusterRegistry(action = "start", workers = TRUE), error = identity)
stopifnot(inherits(res, "error"))
message("*** ClusterRegistry() - exceptions ... DONE")
message("*** ClusterRegistry() ... DONE")
source("incl/end.R")
future/tests/globals,formulas.R 0000644 0001762 0000144 00000007262 13111756706 016337 0 ustar ligges users source("incl/start.R")
library("datasets") ## cars data set
library("stats") ## lm(), poly(), xtabs()
message("*** Globals - formulas ...")
## (i) lm():
## From example("lm", package = "stats")
ctl <- c(4.17, 5.58, 5.18, 6.11, 4.50, 4.61, 5.17, 4.53, 5.33, 5.14)
trt <- c(4.81, 4.17, 4.41, 3.59, 5.87, 3.83, 6.03, 4.89, 4.32, 4.69)
group <- gl(2, 10, 20, labels = c("Ctl", "Trt"))
weight <- c(ctl, trt)
ctl <- trt <- NULL
## Truth:
fit_i <- lm(weight ~ group - 1)
print(fit_i)
## (ii) xtabs(~ x):
x <- c(1, 1, 2, 2, 2)
## Truth:
tbl_ii <- xtabs(~ x)
print(tbl_ii)
## (iii) lm(, data = cars):
exprs <- list(
# "remove-intercept-term" form of no-intercept
a = substitute({ lm(dist ~ . -1, data = cars) }),
# "make-intercept-zero" form of no-intercept
b = substitute({ lm(dist ~ . +0, data = cars) }),
# doesn't do what we want here
c = substitute({ lm(dist ~ speed + speed ^ 2, data = cars) }),
# gets us a quadratic term
d = substitute({ lm(dist ~ speed + I(speed ^ 2), data = cars) }),
# avoid potential multicollinearity
e = substitute({ lm(dist ~ poly(speed, 2), data = cars) })
)
## (iv) Globals - map(x, ~ expr):
## A fake purrr::map() function with limited functionality
map <- function(.x, .f, ...) {
if (inherits(.f, "formula")) {
expr <- .f[[-1]]
.f <- eval(bquote(function(...) {
.(expr)
}))
}
eval(lapply(.x, FUN = .f, ...))
}
inner_function <- function(x) { x + 1 }
outer_function <- function(x) {
map(1:2, ~ inner_function(.x))
}
y_iv <- outer_function(1L)
str(y_iv)
for (cores in 1:availCores) {
message(sprintf("Testing with %d cores ...", cores))
options(mc.cores = cores)
message("availableCores(): ", availableCores())
for (strategy in supportedStrategies(cores)) {
message(sprintf("- plan('%s') ...", strategy))
plan(strategy)
message("- lm() ...")
## Explicit future
f <- future({ lm(weight ~ group - 1) })
fit <- value(f)
print(fit)
stopifnot(all.equal(fit, fit_i))
## Explicit future (lazy)
f <- future({ lm(weight ~ group - 1) }, lazy = TRUE)
fit <- value(f)
print(fit)
stopifnot(all.equal(fit, fit_i))
## Future assignment
fit %<-% { lm(weight ~ group - 1) }
print(fit)
stopifnot(all.equal(fit, fit_i))
## Future assignment (non-lazy)
fit %<-% { lm(weight ~ group - 1) } %lazy% FALSE
print(fit)
stopifnot(all.equal(fit, fit_i))
## Future assignment (lazy)
fit %<-% { lm(weight ~ group - 1) } %lazy% TRUE
print(fit)
stopifnot(all.equal(fit, fit_i))
message("- Globals - one-side formulas, e.g. xtabs(~ x) ...")
## Explicit future
f <- future({ xtabs(~ x) })
tbl <- value(f)
print(tbl)
stopifnot(all.equal(tbl, tbl_ii))
## Future assignment
tbl %<-% { xtabs(~ x) }
print(tbl)
stopifnot(all.equal(tbl, tbl_ii))
message("- Globals - lm(, data = cars) ...")
for (kk in seq_along(exprs)) {
expr <- exprs[[kk]]
name <- names(exprs)[kk]
message(sprintf("- Globals - lm(, data = cars) ...",
kk, sQuote(name)))
fit_iii <- eval(expr)
print(fit_iii)
f <- future(expr, substitute = FALSE)
fit <- value(f)
print(fit)
stopifnot(all.equal(fit, fit_iii))
} ## for (kk ...)
message("- Globals - map(x, ~ expr) ...")
f <- future({ outer_function(1L) })
y <- value(f)
str(y)
stopifnot(all.equal(y, y_iv))
y %<-% { outer_function(1L) }
str(y)
stopifnot(all.equal(y, y_iv))
} ## for (strategy ...)
message(sprintf("Testing with %d cores ... DONE", cores))
} ## for (cores ...)
message("*** Globals - formulas ... DONE")
source("incl/end.R")
future/tests/multisession.R 0000644 0001762 0000144 00000007305 13155002315 015607 0 ustar ligges users source("incl/start.R")
library("listenv")
plan(multisession)
message("*** multisession() ...")
for (cores in 1:availCores) {
message(sprintf("Testing with %d cores ...", cores))
options(mc.cores = cores)
## No global variables
f <- multisession({
42L
})
print(f)
stopifnot(inherits(f, "ClusterFuture") || (inherits(f, "SequentialFuture") && f$lazy))
print(resolved(f))
y <- value(f)
print(y)
stopifnot(y == 42L)
## A global variable
a <- 0
f <- multisession({
b <- 3
c <- 2
a * b * c
})
print(f)
## A multisession future is evaluated in a separate
## R session process. Changing the value of a global
## variable should not affect the result of the
## future.
a <- 7 ## Make sure globals are frozen
v <- value(f)
print(v)
stopifnot(v == 0)
message("*** multisession() with globals and blocking")
x <- listenv()
for (ii in 1:4) {
message(sprintf(" - Creating multisession future #%d ...", ii))
x[[ii]] <- multisession({ ii })
}
message(sprintf(" - Resolving %d multisession futures", length(x)))
v <- sapply(x, FUN = value)
stopifnot(all(v == 1:4))
message("*** multisession() and errors")
f <- multisession({
stop("Whoops!")
1
})
print(f)
v <- value(f, signal = FALSE)
print(v)
stopifnot(inherits(v, "simpleError"))
res <- try(value(f), silent = TRUE)
print(res)
stopifnot(inherits(res, "try-error"))
## Error is repeated
res <- try(value(f), silent = TRUE)
print(res)
stopifnot(inherits(res, "try-error"))
message("*** multisession() - too large globals ...")
ooptsT <- options(future.globals.maxSize = object.size(1:1014))
limit <- getOption("future.globals.maxSize")
cat(sprintf("Max total size of globals: %g bytes\n", limit))
for (workers in unique(c(1L, availableCores()))) {
message("Max number of sessions: ", workers)
## A large object
a <- 1:1014
yTruth <- sum(a)
size <- object.size(a)
cat(sprintf("a: %g bytes\n", size))
f <- multisession({ sum(a) }, workers = workers)
print(f)
rm(list = "a")
v <- value(f)
print(v)
stopifnot(v == yTruth)
## A too large object
a <- 1:1015
yTruth <- sum(a)
size <- object.size(a)
cat(sprintf("a: %g bytes\n", size))
res <- try(f <- multisession({ sum(a) }, workers = workers), silent = TRUE)
rm(list = "a")
stopifnot(inherits(res, "try-error"))
}
## Undo options changed in this test
options(ooptsT)
message("*** multisession() - too large globals ... DONE")
message("*** multisession(..., workers = 1L) ...")
a <- 2
b <- 3
yTruth <- a * b
f <- multisession({ a * b }, workers = 1L)
rm(list = c("a", "b"))
v <- value(f)
print(v)
stopifnot(v == yTruth)
message("*** multisession(..., workers = 1L) ... DONE")
message("*** multisession(..., gc = TRUE) ...")
plan(multisession, workers = 2L)
f <- future({ gc() })
v <- value(f)
print(v)
f <- future({ integer(10e6) })
v <- value(f)
str(v)
f <- future({ gc() })
v <- value(f)
print(v)
f <- future({ integer(10e6) }, gc = TRUE)
v <- value(f)
str(v)
f <- future({ gc() })
v <- value(f)
print(v)
message("*** multisession(..., gc = TRUE) ... TRUE")
message("*** multisession(...) - stopping with plan() change ...")
plan(multisession, workers = 2L)
f <- future(1L)
cl <- ClusterRegistry("get")
stopifnot(inherits(cl, "cluster"), length(cl) >= 1L)
plan(sequential)
cl <- ClusterRegistry("get")
stopifnot(is.null(cl), length(cl) == 0L)
message("*** multisession(...) - stopping with plan() change ... DONE")
message(sprintf("Testing with %d cores ... DONE", cores))
} ## for (cores ...)
message("*** multisession() ... DONE")
source("incl/end.R")
future/tests/uuid.R 0000644 0001762 0000144 00000002744 13237667011 014034 0 ustar ligges users source("incl/start.R")
session_uuid <- future:::session_uuid
add_cluster_uuid <- future:::add_cluster_uuid
message("*** session_uuid() ...")
id0 <- session_uuid()
print(id0)
## Reset session UUID (hack)
environment(session_uuid)$uuids <- list()
id <- session_uuid()
print(id)
stopifnot(id != id0)
## Assert that forked child processes get a unique session id
## Issue: https://github.com/HenrikBengtsson/future/issues/187
if (supportsMulticore()) {
plan(multicore, workers = 2L)
fs <- lapply(1:2, FUN = function(i) {
future({
Sys.sleep(0.2)
session_uuid()
})
})
ids <- unlist(values(fs))
print(ids)
stopifnot(all(ids != id), length(unique(ids)) == 2L)
}
message("*** session_uuid() ... DONE")
message("*** add_cluster_uuid() ...")
cl <- parallel::makeCluster(1L, type = "PSOCK")
cl <- add_cluster_uuid(cl)
str(cl)
parallel::stopCluster(cl)
uuid <- as.vector(attr(cl[[1]]$con, "uuid"))
print(uuid)
stopifnot(is.character(uuid), nzchar(uuid))
if (supportsMulticore()) {
cl <- parallel::makeCluster(1L, type = "FORK")
cl <- add_cluster_uuid(cl)
str(cl)
parallel::stopCluster(cl)
uuid <- as.vector(attr(cl[[1]]$con, "uuid"))
print(uuid)
stopifnot(is.character(uuid), nzchar(uuid))
}
cl <- structure(list(
structure(list(
rank = 1L, RECVTAG = 33, SENDTAG = 22, comm = 1
), class = "MPInode")
), class = c("spawnedMPIcluster", "MPIcluster", "cluster")
)
cl <- add_cluster_uuid(cl)
message("*** add_cluster_uuid() ... DONE")
source("incl/end.R")
future/tests/future_lapply,RNG.R 0000644 0001762 0000144 00000013335 13237667011 016402 0 ustar ligges users source("incl/start.R")
message("*** future_lapply() and RNGs ...")
options(future.debug = FALSE)
message("* future_lapply(x, ..., future.seed = ) ...")
res <- tryCatch({
y <- future_lapply(1:3, FUN = identity, future.seed = as.list(1:2))
}, error = identity)
print(res)
stopifnot(inherits(res, "simpleError"))
res <- tryCatch({
y <- future_lapply(1:3, FUN = identity, future.seed = list(1, 2, 3:4))
}, error = identity)
print(res)
stopifnot(inherits(res, "simpleError"))
res <- tryCatch({
y <- future_lapply(1:3, FUN = identity, future.seed = as.list(1:3))
}, error = identity)
print(res)
stopifnot(inherits(res, "simpleError"))
seeds <- lapply(1:3, FUN = as_lecyer_cmrg_seed)
res <- tryCatch({
y <- future_lapply(1:3, FUN = identity, future.seed = lapply(seeds, FUN = as.numeric))
}, error = identity)
print(res)
stopifnot(inherits(res, "simpleError"))
seeds[[1]][1] <- seeds[[1]][1] + 1L
res <- tryCatch({
y <- future_lapply(1:3, FUN = identity, future.seed = seeds)
}, error = identity)
print(res)
stopifnot(inherits(res, "simpleError"))
message("* future_lapply(x, ..., future.seed = ) ... DONE")
## Iterate of the same set in all tests
x <- 1:5
message("* future_lapply(x, ..., future.seed = FALSE) ...")
y0 <- y0_nested <- seed00 <- NULL
for (cores in 1:availCores) {
message(sprintf(" - Testing with %d cores ...", cores))
options(mc.cores = cores)
for (strategy in supportedStrategies(cores)) {
message(sprintf("* plan('%s') ...", strategy))
plan(strategy)
set.seed(0xBEEF)
seed0 <- get_random_seed()
y <- future_lapply(x, FUN = function(i) i, future.seed = FALSE)
y <- unlist(y)
seed <- get_random_seed()
if (is.null(y0)) {
y0 <- y
seed00 <- seed
}
str(list(y = y))
stopifnot(identical(seed, seed0), identical(seed, seed00))
## NOTE: We cannot guarantee the same random numbers, because
## future.seed = FALSE.
message(sprintf("* plan('%s') ... DONE", strategy))
} ## for (strategy ...)
message(sprintf(" - Testing with %d cores ... DONE", cores))
} ## for (core ...)
message("* future_lapply(x, ..., future.seed = FALSE) ... DONE")
seed_sets <- list(
A = TRUE,
B = NA,
C = 42L,
D = future:::as_lecyer_cmrg_seed(42L),
E = list(),
F = vector("list", length = length(x))
)
## Generate sequence of seeds of the current RNGkind()
## NOTE: This is NOT a good way to generate random seeds!!!
seeds <- lapply(seq_along(x), FUN = function(i) {
set.seed(i)
globalenv()$.Random.seed
})
seed_sets$E <- seeds
## Generate sequence of L'Ecyer CMRG seeds
seeds <- seed_sets$F
seeds[[1]] <- seed_sets$D
for (kk in 2:length(x)) seeds[[kk]] <- parallel::nextRNGStream(seeds[[kk - 1]])
seed_sets$F <- seeds
rm(list = "seeds")
for (name in names(seed_sets)) {
future.seed <- seed_sets[[name]]
if (is.list(future.seed)) {
label <- sprintf("",
length(future.seed), length(future.seed[[1]]))
} else {
label <- hpaste(future.seed)
}
message(sprintf("* future_lapply(x, ..., future.seed = %s) ...", label))
set.seed(0xBEEF)
y0 <- seed00 <- NULL
for (cores in 1:availCores) {
message(sprintf(" - Testing with %d cores ...", cores))
options(mc.cores = cores)
for (strategy in supportedStrategies(cores)) {
message(sprintf("* plan('%s') ...", strategy))
plan(strategy)
set.seed(0xBEEF)
seed0 <- get_random_seed()
y <- future_lapply(x, FUN = function(i) {
rnorm(1L)
}, future.seed = future.seed)
y <- unlist(y)
seed <- get_random_seed()
if (is.null(y0)) {
y0 <- y
seed00 <- seed
}
str(list(y = y))
stopifnot(!identical(seed, seed0), identical(seed, seed00),
identical(y, y0))
## RNG-based results should also be identical regardless of
## load-balance scheduling.
for (scheduling in list(FALSE, TRUE, 0, 0.5, 2.0, Inf)) {
set.seed(0xBEEF)
seed0 <- get_random_seed()
y <- future_lapply(x, FUN = function(i) {
rnorm(1L)
}, future.seed = future.seed, future.scheduling = scheduling)
seed <- get_random_seed()
y <- unlist(y)
str(list(y = y))
stopifnot(!identical(seed, seed0), identical(seed, seed00),
identical(y, y0))
}
## Nested future_lapply():s
for (scheduling in list(FALSE, TRUE)) {
y <- future_lapply(x, FUN = function(i) {
.seed <- globalenv()$.Random.seed
z <- future_lapply(1:3, FUN = function(j) {
list(j = j, seed = globalenv()$.Random.seed)
}, future.seed = .seed)
## Assert that all future seeds are unique
seeds <- lapply(z, FUN = function(x) x$seed)
for (kk in 2:length(seeds)) stopifnot(!all(seeds[[kk]] == seeds[[1]]))
list(i = i, seed = .seed, sample = rnorm(1L), z = z)
}, future.seed = 42L, future.scheduling = scheduling)
if (is.null(y0_nested)) y0_nested <- y
str(list(y = y))
## Assert that all future seeds (also nested ones) are unique
seeds <- Reduce(c, lapply(y, FUN = function(x) {
c(list(seed = x$seed), lapply(x$z, FUN = function(x) x$seed))
}))
for (kk in 2:length(seeds)) stopifnot(!all(seeds[[kk]] == seeds[[1]]))
stopifnot(identical(y, y0_nested))
}
message(sprintf("* plan('%s') ... DONE", strategy))
} ## for (strategy ...)
message(sprintf(" - Testing with %d cores ... DONE", cores))
} ## for (cores ...)
message(sprintf("* future_lapply(x, ..., future.seed = %s) ... DONE", label))
} ## for (name ...)
message("*** future_lapply() and RNGs ... DONE")
source("incl/end.R")
future/tests/future_lapply.R 0000644 0001762 0000144 00000004363 13237667011 015760 0 ustar ligges users source("incl/start.R")
library("listenv")
message("*** future_lapply() ...")
x_a <- list(a = "integer", b = "numeric", c = "character", c = "list")
str(list(x_a = x_a))
y_a <- lapply(x_a, FUN = base::vector, length = 2L)
str(list(y_a = y_a))
x_b <- list(a = c("hello", b = 1:100))
str(list(x_b = x_b))
y_b <- lapply(x_b, FUN = future:::hpaste, collapse = "; ", maxHead = 3L)
str(list(y_b = y_b))
x_c <- list()
y_c <- listenv()
y_c$A <- 3L
x_c$a <- y_c
y_c<- listenv()
y_c$A <- 3L
y_c$B <- c("hello", b = 1:100)
x_c$b <- y_c
print(x_c)
y_c <- lapply(x_c, FUN = listenv::map)
str(list(y_c = y_c))
for (cores in 1:availCores) {
message(sprintf("Testing with %d cores ...", cores))
options(mc.cores = cores)
strategies <- supportedStrategies(cores)
if (cores == 1) strategies <- c("transparent", strategies)
for (strategy in strategies) {
message(sprintf("- plan('%s') ...", strategy))
plan(strategy)
for (scheduling in list(FALSE, TRUE)) {
message("- future_lapply(x, FUN = vector, ...) ...")
y <- future_lapply(x_a, FUN = vector, length = 2L, future.scheduling = scheduling)
str(list(y = y))
stopifnot(identical(y, y_a))
message("- future_lapply(x, FUN = base::vector, ...) ...")
y <- future_lapply(x_a, FUN = base::vector, length = 2L, future.scheduling = scheduling)
str(list(y = y))
stopifnot(identical(y, y_a))
message("- future_lapply(x, FUN = future:::hpaste, ...) ...")
y <- future_lapply(x_b, FUN = future:::hpaste, collapse = "; ", maxHead = 3L, future.scheduling = scheduling)
str(list(y = y))
stopifnot(identical(y, y_b))
message("- future_lapply(x, FUN = listenv::listenv, ...) ...")
y <- future_lapply(x_c, FUN = listenv::map, future.scheduling = scheduling)
str(list(y = y))
stopifnot(identical(y, y_c))
} ## for (scheduling ...)
message("- future_lapply(x, FUN, ...) for large length(x) ...")
a <- 3.14
x_d <- 1:1e4
y <- future_lapply(x_d, FUN = function(z) sqrt(z + a))
y <- unlist(y, use.names = FALSE)
stopifnot(all.equal(y, sqrt(x_d + a)))
} ## for (strategy ...)
message(sprintf("Testing with %d cores ... DONE", cores))
} ## for (cores ...)
message("*** future_lapply() ... DONE")
source("incl/end.R")
future/tests/futureOf_with_listenv.R 0000644 0001762 0000144 00000005440 13111756706 017461 0 ustar ligges users source("incl/start.R")
library("listenv")
message("*** futureOf() with listenv ...")
message("*** futureOf() with listenv - future assignments ...")
x <- listenv()
x$a %<-% { 1 } %lazy% TRUE
f1 <- futureOf("a", envir = x)
print(f1)
f2 <- futureOf(a, envir = x)
f3 <- futureOf(1, envir = x)
f4 <- futureOf(x[["a"]])
f5 <- futureOf(x$a)
f6 <- futureOf(x[[1]])
stopifnot(identical(f2, f1), identical(f3, f2), identical(f4, f3),
identical(f5, f4), identical(f6, f5))
x[[3]] %<-% { 3 } %lazy% TRUE
x$d %<-% { 4 } %lazy% TRUE
x[[5]] <- 5
## Identify all futures
fs <- futureOf(envir = x)
print(fs)
stopifnot(identical(names(fs), names(x)))
stopifnot(identical(fs$a, f1))
stopifnot(identical(fs[[3]], futureOf(3L, envir = x)))
stopifnot(identical(fs$d, futureOf("d", envir = x)))
fsD <- futureOf(envir = x, drop = TRUE)
print(fsD)
stopifnot(all(sapply(fsD, FUN = inherits, "Future")))
stopifnot(!identical(fsD, fs))
message("*** futureOf() with listenv - future assignments ... DONE")
message("*** futureOf() with listenv - futures ...")
x <- listenv()
x$a <- future({ 1 }, lazy = TRUE)
f1 <- futureOf("a", envir = x)
print(f1)
stopifnot(identical(f1, x$a))
f2 <- futureOf(a, envir = x)
stopifnot(identical(f2, x$a))
f3 <- futureOf(1, envir = x)
stopifnot(identical(f3, x$a))
f4 <- futureOf(x[["a"]])
stopifnot(identical(f4, x$a))
f5 <- futureOf(x$a)
stopifnot(identical(f5, x$a))
f6 <- futureOf(x[[1]])
stopifnot(identical(f6, x$a))
x[[3]] <- future({ 3 }, lazy = TRUE)
x$d <- future({ 4 }, lazy = TRUE)
x[[5]] <- 5
## Identify all futures
fs <- futureOf(envir = x)
print(fs)
stopifnot(identical(names(fs), names(x)))
stopifnot(identical(fs$a, f1))
stopifnot(identical(fs[[3]], futureOf(3L, envir = x)))
stopifnot(identical(fs$d, futureOf("d", envir = x)))
fsD <- futureOf(envir = x, drop = TRUE)
print(fsD)
stopifnot(all(sapply(fsD, FUN = inherits, "Future")))
stopifnot(!identical(fsD, fs))
message("*** futureOf() with listenv - futures ... DONE")
message("*** futureOf() with listenv - exceptions ...")
## Invalid subset
res <- tryCatch(futureOf(x[[0]], mustExist = FALSE), error = identity)
stopifnot(inherits(res, "error"))
res <- tryCatch(futureOf(x[[0]], mustExist = TRUE), error = identity)
stopifnot(inherits(res, "error"))
## Out-of-bound subscript, cf lists
stopifnot(is.na(futureOf(x[[10]], mustExist = FALSE)))
res <- tryCatch(futureOf(x[[10]], mustExist = TRUE), error = identity)
stopifnot(inherits(res, "error"))
## Invalid subscript
res <- tryCatch(futureOf(x[[1 + 2i]], mustExist = TRUE), error = identity)
stopifnot(inherits(res, "error"))
## Non-existing object
res <- tryCatch(futureOf(z[[1]], mustExist = TRUE), error = identity)
stopifnot(inherits(res, "error"))
message("*** futureOf() with listenv - exceptions ... DONE")
message("*** futureOf() with listenv ... DONE")
source("incl/end.R")
future/tests/futureAssign_OP_with_environment.R 0000644 0001762 0000144 00000002450 13111756521 021610 0 ustar ligges users source("incl/start.R")
## BACKWARD COMPATIBILITY
if (getRversion() < "3.2.0") {
names <- function(x) if (is.environment(x)) ls(envir = x) else base::names(x)
}
message("*** %<-% to environment ...")
## - - - - - - - - - - - - - - - - - - - - - - - - - - - -
## Async delayed assignment (infix operator)
## - - - - - - - - - - - - - - - - - - - - - - - - - - - -
z <- new.env()
stopifnot(length(names(z)) == 0L)
message("*** %<-% to environment: Assign by index (not allowed)")
res <- try(z[[1]] %<-% { 2 } %lazy% TRUE, silent = TRUE)
stopifnot(inherits(res, "try-error"))
message("*** %<-% to environment: Assign by name (new)")
z$B %<-% { TRUE } %lazy% TRUE
stopifnot(length(z) == 2) # sic!
stopifnot("B" %in% ls(z))
y <- as.list(z)
str(y)
stopifnot(length(y) == 1)
stopifnot(identical(names(y), "B"))
message("*** %<-% to environment: Potential task name clashes")
u <- new.env()
u$a %<-% { 1 } %lazy% TRUE
stopifnot(length(u) == 2)
stopifnot("a" %in% names(u))
fu <- futureOf(u$a)
v <- new.env()
v$a %<-% { 2 } %lazy% TRUE
stopifnot(length(v) == 2)
stopifnot("a" %in% names(v))
fv <- futureOf(v$a)
stopifnot(!identical(fu, fv))
fu <- futureOf(u$a)
stopifnot(!identical(fu, fv))
stopifnot(identical(u$a, 1))
stopifnot(identical(v$a, 2))
message("*** %<-% to environment ... DONE")
source("incl/end.R")
future/tests/makeClusterPSOCK.R 0000644 0001762 0000144 00000004375 13111756521 016143 0 ustar ligges users source("incl/start.R")
is_fqdn <- future:::is_fqdn
is_ip_number <- future:::is_ip_number
is_localhost <- future:::is_localhost
find_rshcmd <- future:::find_rshcmd
message("*** makeClusterPSOCK() ...")
message("- makeClusterPSOCK() - internal utility functions")
stopifnot(
is_fqdn("a.b"),
is_fqdn("a.b.c"),
!is_fqdn("a")
)
stopifnot(
is_ip_number("1.2.3.4"),
!is_ip_number("a"),
!is_ip_number("1.2.3"),
!is_ip_number("1.2.3.256"),
!is_ip_number("1.2.3.-1"),
!is_ip_number("1.2.3.a")
)
## Reset internal cache
stopifnot(is.na(is_localhost(worker = NULL, hostname = NULL)))
stopifnot(
is_localhost("localhost"),
is_localhost("127.0.0.1"),
is_localhost(Sys.info()[["nodename"]]),
is_localhost(Sys.info()[["nodename"]]), ## cache hit
!is_localhost("not.a.localhost.hostname")
)
cmd <- find_rshcmd(must_work = FALSE)
print(cmd)
message("- makeClusterPSOCK()")
cl <- makeClusterPSOCK("", user = "johndoe", master = NULL, revtunnel = FALSE, rshcmd = "my_ssh", renice = TRUE, manual = TRUE, dryrun = TRUE)
print(cl)
cl <- makeClusterPSOCK(1L, port = "random", dryrun = TRUE)
print(cl)
cl <- makeClusterPSOCK(1L)
print(cl)
parallel::stopCluster(cl)
message("- makeClusterPSOCK() - exceptions")
res <- tryCatch({
cl <- makeClusterPSOCK(1:2)
}, error = identity)
print(res)
stopifnot(inherits(res, "error"))
res <- tryCatch({
cl <- makeClusterPSOCK(0L)
}, error = identity)
print(res)
stopifnot(inherits(res, "error"))
res <- tryCatch({
cl <- makeClusterPSOCK(1L, rshcmd = character(0L))
}, error = identity)
print(res)
stopifnot(inherits(res, "error"))
res <- tryCatch({
cl <- makeClusterPSOCK(1L, port = integer(0L))
}, error = identity)
print(res)
stopifnot(inherits(res, "error"))
res <- tryCatch({
cl <- makeClusterPSOCK(1L, port = NA_integer_)
}, error = identity)
print(res)
stopifnot(inherits(res, "error"))
message("- makeClusterPSOCK() - exceptions")
res <- tryCatch({
cl <- makeNodePSOCK("localhost", port = NA_integer_)
}, error = identity)
print(res)
stopifnot(inherits(res, "error"))
res <- tryCatch({
cl <- makeNodePSOCK("not.a.localhost.hostname", revtunnel = TRUE)
}, error = identity)
print(res)
stopifnot(inherits(res, "error"))
message("*** makeClusterPSOCK() ... DONE")
source("incl/end.R")
future/tests/plan.R 0000644 0001762 0000144 00000011454 13237651433 014017 0 ustar ligges users source("incl/start,load-only.R")
message("*** plan() ...")
message("*** Set strategy via future::plan(future::multisession)")
oplan <- future::plan(future::multisession)
print(future::plan())
future::plan(oplan)
print(future::plan())
message("*** Set strategy via future::plan(future::multisession, globals = FALSE)")
oplan <- future::plan(future::multisession, globals = FALSE)
print(future::plan())
future::plan(oplan)
print(future::plan())
message("*** Set strategy via future::plan(future::multisession(globals = FALSE)")
oplan <- future::plan(future::multisession(globals = FALSE))
print(future::plan())
future::plan(oplan)
print(future::plan())
message("*** Set strategy via future::plan('multisession')")
oplan <- future::plan("multisession")
print(future::plan())
future::plan(oplan)
print(future::plan())
message("*** plan('default')")
oplan <- future::plan("default")
print(future::plan())
future::plan(oplan)
print(future::plan())
library("future")
message("*** plan('unknown strategy')")
res <- try(plan('unknown strategy'), silent = TRUE)
print(res)
stopifnot(inherits(res, "try-error"))
message("*** plan(sequential)")
plan(sequential)
a <- 0
f <- future({
b <- 3
c <- 2
a * b * c
})
a <- 7
v <- value(f)
print(v)
stopifnot(v == 0)
message("*** plan('sequential')")
## Setting strategy by name
plan("multisession")
print(plan())
message("*** plan() and overriding defaults")
message("*** plan(sequential)")
plan(sequential)
fcn <- plan()
print(fcn)
stopifnot(formals(fcn)$local == TRUE)
x <- 0
f <- future({ x <- 1 })
print(value(f))
stopifnot(x == 0)
message("*** plan(sequential, local = FALSE)")
plan(sequential, local = FALSE)
fcn <- plan()
print(fcn)
stopifnot(formals(fcn)$local == FALSE)
x <- 0
f <- future({ x <- 1 })
print(value(f))
stopifnot(x == 1)
message("*** plan(sequential, local = FALSE, abc = 1, def = TRUE)")
plan(sequential, local = FALSE, abc = 1, def = TRUE)
fcn <- plan()
print(fcn)
stopifnot(formals(fcn)$local == FALSE)
message("*** plan(sequential(local = FALSE))")
plan(multisession)
plan(sequential(local = FALSE))
fcn <- plan()
print(fcn)
stopifnot(formals(fcn)$local == FALSE)
message("*** plan(tweak(sequential, local = FALSE))")
plan(multisession)
plan(tweak(sequential, local = FALSE))
fcn <- plan()
print(fcn)
stopifnot(formals(fcn)$local == FALSE)
message("*** old <- plan(new)")
truth <- plan()
old <- plan(multisession, globals = FALSE)
stopifnot(identical(unclass(old), unclass(truth)))
curr <- plan() ## curr == multisession(globals = FALSE)
prev <- plan(old) ## prev == sequential(local = FALSE)
stopifnot(identical(unclass(curr), unclass(prev)))
curr <- plan() ## curr == old
stopifnot(identical(unclass(curr), unclass(old)))
stopifnot(identical(unclass(curr), unclass(truth)))
message("*** %plan% 'sequential'")
plan(multisession)
x %<-% { a <- 1 } %plan% "sequential"
stopifnot(identical(body(plan()), body(multisession)))
message("*** %plan% sequential")
plan(multisession)
## %plan% can operate on any expression, so it
## works just as an withPlan({ ... }, plan = ...)
fun <- { plan() } %plan% sequential
f <- fun(1)
stopifnot(inherits(f, "SequentialFuture"), !f$lazy, inherits(f, "SequentialFuture"))
x %<-% { a <- 1 } %plan% sequential
stopifnot(identical(body(plan()), body(multisession)))
message("*** %plan% sequential(local = FALSE) ")
plan(multisession)
a <- 0
x %<-% { a } %plan% sequential(local = FALSE)
a <- 42
print(x)
stopifnot(x == 0)
stopifnot(identical(body(plan()), body(multisession)))
message("*** Nested futures with different plans")
c %<-% {
message("Resolving 'c'")
a %<-% {
message("Resolving 'a'")
2
} %plan% sequential
b %<-% {
message("Resolving 'b'")
-9 * a
}
message("Local variable 'x'")
x <- b / 3
abs(x)
} %lazy% TRUE
d <- 42
print(d)
print(c)
stopifnot(c == 6)
message("*** plan() by functions and character names ... ")
plan(sequential)
a %<-% 42
stopifnot(a == 42)
plan("sequential")
a %<-% 42
stopifnot(a == 42)
plan(list(sequential))
a %<-% 42
stopifnot(a == 42)
plan(list("sequential"))
a %<-% 42
stopifnot(a == 42)
plan(list(sequential, sequential))
a %<-% { b %<-% 42; b }
stopifnot(a == 42)
plan(list("sequential", sequential))
a %<-% { b %<-% 42; b }
stopifnot(a == 42)
plan(list(sequential, "sequential"))
a %<-% { b %<-% 42; b }
stopifnot(a == 42)
plan(list("sequential", "sequential"))
a %<-% { b %<-% 42; b }
stopifnot(a == 42)
plan(list("future::sequential", "sequential"))
a %<-% { b %<-% 42; b }
stopifnot(a == 42)
message("*** plan() by functions and character names ... DONE")
message("*** plan() w/ commands ...")
plan(list(sequential, sequential))
res <- plan("list")
print(res)
stopifnot(length(res) == 2)
plan("pop")
res <- plan("list")
print(res)
stopifnot(length(res) == 1)
plan("reset")
print(plan())
message("*** plan() w/ commands ... DONE")
message("*** plan() ... DONE")
source("incl/end.R")
future/tests/futureOf.R 0000644 0001762 0000144 00000000637 13111756706 014665 0 ustar ligges users source("incl/start.R")
library("listenv")
message("*** futureOf() ...")
a %<-% { 1 } %lazy% TRUE
f1 <- futureOf("a")
print(f1)
f2 <- futureOf(a)
print(f2)
stopifnot(identical(f2, f1))
fs <- futureOf()
print(fs)
## Non-existing object
res <- tryCatch(futureOf("non-exiting-object", mustExist = TRUE), error = identity)
stopifnot(inherits(res, "error"))
message("*** futureOf() ... DONE")
source("incl/end.R")
future/tests/invalid-owner.R 0000644 0001762 0000144 00000005071 13227203046 015631 0 ustar ligges users source("incl/start.R")
## Local functions
usedNodes <- function(future) {
## Number of unresolved cluster futures
workers <- future$workers
reg <- sprintf("workers-%s", attr(workers, "name"))
c(used = length(future:::FutureRegistry(reg, action = "list")), total = length(workers))
}
## This test requires at least two background processes
plan(multisession, workers = 2L)
message("*** future() - invalid ownership ...")
## This R process
session_uuid <- future:::session_uuid(attributes = TRUE)
cat(sprintf("Main R process: %s\n", session_uuid))
message("- Asserting ownership ...")
message("Creating future #1:")
f1 <- future({ future:::session_uuid(attributes = TRUE) })
stopifnot(inherits(f1, "MultisessionFuture"))
cat(sprintf("Future #1 session: %d\n", f1$node))
v1 <- value(f1)
cat(sprintf("Future #1 R process: %s\n", v1))
stopifnot(v1 != session_uuid)
message("Creating future #2:")
f2 <- future({ future:::session_uuid(attributes = TRUE) })
stopifnot(inherits(f2, "MultisessionFuture"))
cat(sprintf("Future #2 session: %d\n", f2$node))
v2 <- value(f2)
cat(sprintf("Future #2 R process: %s\n", v2))
stopifnot(v2 != session_uuid)
message("Creating future #3:")
f3 <- future({ f1$owner })
stopifnot(inherits(f3, "MultisessionFuture"))
cat(sprintf("Future #3 session: %d\n", f3$node))
v3 <- value(f3)
cat(sprintf("Future #3 owner: %s\n", v3))
stopifnot(v3 == session_uuid)
message("Creating future #4:")
f4 <- future({ f1$owner })
stopifnot(inherits(f4, "MultisessionFuture"))
cat(sprintf("Future #4 session: %d\n", f4$node))
v4 <- value(f4)
cat(sprintf("Future #4 owner: %s\n", v4))
stopifnot(v4 == session_uuid)
message("Creating future #5:")
f5 <- future({ stopifnot(f1$owner != future:::session_uuid(attributes = TRUE)); "not-owner" })
stopifnot(inherits(f5, "MultisessionFuture"))
v5 <- value(f5)
stopifnot(v5 == "not-owner")
message("- Asserting ownership ... DONE")
message("- Trying with invalid ownership ...")
message("Creating future #1:")
f1 <- future({ 42L })
print(f1)
cat(sprintf("Future #1 session: %d\n", f1$node))
stopifnot(identical(f1$owner, session_uuid))
print(usedNodes(f1))
message("Creating future #2:")
f2 <- future({ value(f1) })
print(f2)
cat(sprintf("Future #2 session: %d\n", f2$node))
stopifnot(identical(f2$owner, session_uuid))
print(usedNodes(f2))
message("Getting value of future #2:")
res <- tryCatch(value(f2), error = identity)
print(res)
stopifnot(inherits(res, "error"))
v1 <- value(f1)
print(v1)
stopifnot(v1 == 42L)
message("- Trying with invalid ownership ... DONE")
message("*** future() - invalid ownership ... DONE")
source("incl/end.R")
future/tests/futureAssign_OP.R 0000644 0001762 0000144 00000004772 13111756706 016147 0 ustar ligges users source("incl/start.R")
message("*** %<-% ...")
for (cores in 1:availCores) {
message(sprintf("Testing with %d cores ...", cores))
options(mc.cores = cores)
for (strategy in supportedStrategies(cores)) {
message(sprintf("*** %%<-%% with %s futures ...", sQuote(strategy)))
plan(strategy)
rm(list = intersect(c("x", "y"), ls()))
message("** Future evaluation without globals")
v1 %<-% { x <- 1 }
stopifnot(!exists("x", inherits = FALSE), identical(v1, 1))
message("** Future evaluation with globals")
a <- 2
v2 %<-% { x <- a }
stopifnot(!exists("x", inherits = FALSE), identical(v2, a))
message("** Future evaluation with errors")
v3 %<-% {
x <- 3
stop("Woops!")
x
}
stopifnot(!exists("x", inherits = FALSE))
res <- tryCatch(identical(v3, 3), error = identity)
stopifnot(inherits(res, "error"))
y <- listenv::listenv()
for (ii in 1:3) {
y[[ii]] %<-% {
if (ii %% 2 == 0) stop("Woops!")
ii
}
}
res <- tryCatch(as.list(y), error = identity)
stopifnot(inherits(res, "error"))
z <- y[c(1, 3)]
z <- unlist(z)
stopifnot(all(z == c(1, 3)))
res <- tryCatch(y[[2]], error = identity)
stopifnot(inherits(res, "error"))
res <- tryCatch(y[1:2], error = identity)
stopifnot(inherits(res, "error"))
message("** Future evaluation with progress bar")
v4 %<-% {
cat("Processing: ")
for (ii in 1:10) { cat(".") }
cat(" [100%]\n")
4
}
message("** Collecting results")
printf("v1 = %s\n", v1)
stopifnot(v1 == 1)
printf("v2 = %s\n", v2)
stopifnot(v2 == a)
stopifnot(tryCatch({
printf("v3 = %s\n", v3)
}, error = function(ex) {
printf("v3: <%s> (as expect)\n", class(ex)[1])
TRUE
}))
printf("v4 = %s\n", v4)
#stopifnot(v4 == 4)
message("** Left-to-right and right-to-left future assignments")
c %<-% 1
printf("c = %s\n", c)
1 %->% d
printf("d = %s\n", d)
stopifnot(d == c)
message("** Nested future assignments")
a %<-% {
b <- 1
c %<-% 2
3 -> d
4 %->% e
b + c + d + e
}
printf("a = %s\n", a)
stopifnot(a == 10)
{ a + 1 } %->% b
printf("b = %s\n", b)
stopifnot(b == a + 1)
message(sprintf("*** %%<-%% with %s futures ... DONE", sQuote(strategy)))
} # for (strategy in ...)
message(sprintf("Testing with %d cores ... DONE", cores))
} ## for (cores ...)
message("*** %<-% ... DONE")
source("incl/end.R")
future/NAMESPACE 0000644 0001762 0000144 00000010017 13237667011 013010 0 ustar ligges users # Generated by roxygen2: do not edit by hand
S3method("[",FutureGlobals)
S3method("[",sessionDetails)
S3method(as.FutureGlobals,FutureGlobals)
S3method(as.FutureGlobals,Globals)
S3method(as.FutureGlobals,list)
S3method(as.cluster,SOCK0node)
S3method(as.cluster,SOCKnode)
S3method(as.cluster,cluster)
S3method(as.cluster,list)
S3method(as.raster,Mandelbrot)
S3method(c,FutureGlobals)
S3method(c,cluster)
S3method(futures,environment)
S3method(futures,list)
S3method(futures,listenv)
S3method(getExpression,Future)
S3method(getExpression,MulticoreFuture)
S3method(getExpression,MultisessionFuture)
S3method(getOutput,FutureError)
S3method(getOutput,FutureEvaluationCondition)
S3method(mandelbrot,matrix)
S3method(mandelbrot,numeric)
S3method(nbrOfWorkers,"NULL")
S3method(nbrOfWorkers,cluster)
S3method(nbrOfWorkers,future)
S3method(nbrOfWorkers,multiprocess)
S3method(nbrOfWorkers,uniprocess)
S3method(plot,Mandelbrot)
S3method(print,Future)
S3method(print,FutureCondition)
S3method(print,FutureEvaluationCondition)
S3method(print,FutureStrategy)
S3method(print,FutureStrategyList)
S3method(print,future)
S3method(print,sessionDetails)
S3method(resolve,Future)
S3method(resolve,FutureGlobals)
S3method(resolve,default)
S3method(resolve,environment)
S3method(resolve,list)
S3method(resolve,listenv)
S3method(resolved,ClusterFuture)
S3method(resolved,Future)
S3method(resolved,FutureGlobals)
S3method(resolved,MulticoreFuture)
S3method(resolved,UniprocessFuture)
S3method(resolved,default)
S3method(resolved,environment)
S3method(resolved,list)
S3method(run,ClusterFuture)
S3method(run,Future)
S3method(run,MulticoreFuture)
S3method(run,UniprocessFuture)
S3method(tweak,"function")
S3method(tweak,character)
S3method(tweak,future)
S3method(unique,FutureGlobals)
S3method(value,ClusterFuture)
S3method(value,Future)
S3method(value,MulticoreFuture)
S3method(values,Future)
S3method(values,environment)
S3method(values,list)
S3method(values,listenv)
export("%->%")
export("%<-%")
export("%globals%")
export("%label%")
export("%lazy%")
export("%packages%")
export("%plan%")
export("%seed%")
export("%tweak%")
export(ClusterFuture)
export(ConstantFuture)
export(Future)
export(FutureCondition)
export(FutureError)
export(FutureEvaluationCondition)
export(FutureEvaluationError)
export(FutureEvaluationMessage)
export(FutureEvaluationWarning)
export(FutureGlobals)
export(FutureMessage)
export(FutureWarning)
export(MulticoreFuture)
export(MultiprocessFuture)
export(MultisessionFuture)
export(SequentialFuture)
export(UniprocessFuture)
export(as.FutureGlobals)
export(as.cluster)
export(availableCores)
export(availableWorkers)
export(backtrace)
export(cluster)
export(future)
export(futureAssign)
export(futureCall)
export(futureOf)
export(future_lapply)
export(futures)
export(getExpression)
export(getGlobalsAndPackages)
export(getOutput)
export(makeClusterPSOCK)
export(makeNodePSOCK)
export(mandelbrot)
export(mandelbrot_tiles)
export(multicore)
export(multiprocess)
export(multisession)
export(nbrOfWorkers)
export(plan)
export(remote)
export(resolve)
export(resolved)
export(run)
export(sequential)
export(sessionDetails)
export(supportsMulticore)
export(transparent)
export(tweak)
export(value)
export(values)
importFrom(digest,digest)
importFrom(globals,Globals)
importFrom(globals,as.Globals)
importFrom(globals,cleanup)
importFrom(globals,globalsByName)
importFrom(globals,globalsOf)
importFrom(globals,packagesOf)
importFrom(globals,walkAST)
importFrom(grDevices,as.raster)
importFrom(grDevices,hsv)
importFrom(graphics,par)
importFrom(graphics,plot)
importFrom(listenv,get_variable)
importFrom(listenv,listenv)
importFrom(listenv,map)
importFrom(listenv,parse_env_subset)
importFrom(parallel,clusterCall)
importFrom(parallel,clusterExport)
importFrom(parallel,nextRNGStream)
importFrom(parallel,nextRNGSubStream)
importFrom(parallel,splitIndices)
importFrom(parallel,stopCluster)
importFrom(utils,capture.output)
importFrom(utils,file_test)
importFrom(utils,getS3method)
importFrom(utils,head)
importFrom(utils,object.size)
importFrom(utils,read.table)
importFrom(utils,sessionInfo)
importFrom(utils,str)
future/demo/ 0000755 0001762 0000144 00000000000 13111756706 012517 5 ustar ligges users future/demo/fibonacci.R 0000644 0001762 0000144 00000001156 13111756521 014555 0 ustar ligges users library("future")
library("listenv")
## Defines the first 100 Fibonacci numbers
## (0, 1, 1, 2, 3, 5, 8, ...)
## but calculate only the ones need when
## a number is actually requested.
oplan <- plan(sequential)
x <- listenv()
x[[1]] <- 0
x[[2]] <- 1
for (i in 3:100) {
x[[i]] %<-% { x[[i - 2]] + x[[i - 1]] } %lazy% TRUE
}
## At this point nothing has been calculated,
## because lazy evaluation is in place.
## Get the 7:th Fibonnaci numbers (should be 8)
print(x[[7]])
## At this point x[1:7] have been calculated,
## but nothing beyond.
## Let's get the 50:th number.
print(x[[50]])
## Reset plan
plan(oplan)
future/demo/mandelbrot.R 0000644 0001762 0000144 00000004533 13111756706 014776 0 ustar ligges users library("future")
library("graphics")
plot_what_is_done <- function(counts) {
for (kk in seq_along(counts)) {
f <- counts[[kk]]
## Already plotted?
if (!inherits(f, "Future")) next
## Not resolved?
if (!resolved(f)) next
cat(sprintf("Plotting tile #%d of %d ...\n", kk, n))
counts[[kk]] <- value(counts[[kk]])
screen(kk)
plot(counts[[kk]])
}
counts
}
## Options
region <- getOption("future.demo.mandelbrot.region", 1L)
if (!is.list(region)) {
if (region == 1L) {
region <- list(xmid = -0.75, ymid = 0.0, side = 3.0)
} else if (region == 2L) {
region <- list(xmid = 0.283, ymid = -0.0095, side = 0.00026)
} else if (region == 3L) {
region <- list(xmid = 0.282989, ymid = -0.01, side = 3e-8)
}
}
nrow <- getOption("future.demo.mandelbrot.nrow", 3L)
resolution <- getOption("future.demo.mandelbrot.resolution", 400L)
delay <- getOption("future.demo.mandelbrot.delay", interactive())
if (isTRUE(delay)) {
delay <- function(counts) Sys.sleep(rexp(1, rate = 2))
} else if (!is.function(delay)) {
delay <- function(counts) {}
}
## Generate Mandelbrot tiles to be computed
Cs <- mandelbrot_tiles(xmid = region$xmid, ymid = region$ymid,
side = region$side, nrow = nrow,
resolution = resolution)
if (interactive()) {
if (.Platform$GUI == "RStudio") {
if (!"RStudioGD" %in% names(dev.list())) dev.new()
}
dev.new()
plot.new()
split.screen(dim(Cs))
for (ii in seq_along(Cs)) {
screen(ii)
par(mar = c(0, 0, 0, 0))
text(x = 1 / 2, y = 1 / 2, sprintf("Future #%d\nunresolved", ii), cex = 2)
}
} else {
split.screen(dim(Cs))
}
counts <- list()
n <- length(Cs)
for (ii in seq_len(n)) {
cat(sprintf("Mandelbrot tile #%d of %d ...\n", ii, n))
C <- Cs[[ii]]
counts[[ii]] <- future({
cat(sprintf("Calculating tile #%d of %d ...\n", ii, n))
fit <- mandelbrot(C)
## Emulate slowness
delay(fit)
cat(sprintf("Calculating tile #%d of %d ... done\n", ii, n))
fit
})
## Plot tiles that are already resolved
counts <- plot_what_is_done(counts)
}
## Plot remaining tiles
repeat {
counts <- plot_what_is_done(counts)
if (!any(sapply(counts, FUN = inherits, "Future"))) break
}
close.screen()
message("SUGGESTION: Try to rerun this demo after changing strategy for how futures are resolved, e.g. plan(multiprocess).\n")
future/demo/00Index 0000644 0001762 0000144 00000000156 12726157517 013661 0 ustar ligges users fibonacci Lazy definition of the first 100 Fibonacci numbers
mandelbrot Mandelbrot set images using futures
future/NEWS 0000644 0001762 0000144 00000103443 13237670173 012301 0 ustar ligges users Package: future
===============
Version: 1.7.0 [2018-02-10]
SIGNIFICANT CHANGES:
o future_lapply() has moved to the future.apply package available on CRAN.
NEW FEATURES:
o Argument 'workers' of future strategies may now also be a function, which
is called without argument when the future strategy is set up and used as
is. For instance, plan(multiprocess, workers = halfCores) where
halfCores <- function() { max(1, round(availableCores() / 2)) } will use
half of the number of available cores. This is useful when using nested
future strategies with remote machines.
o On Windows, makeClusterPSOCK(), and therefore plan(multisession) and
plan(multiprocess), will use the SSH client distributed with RStudio as
a fallback if neither 'ssh' nor 'plink' is available on the system PATH.
o Now plan() makes sure that nbrOfWorkers() will work for the new strategy.
This will help catch mistakes such as plan(cluster, workers = cl) where
'cl' is a basic R list rather than a 'cluster' list early on.
o Added %packages% to explicitly control packages to be attached when a
future is resolved, e.g. y %<-% { DT[2] } %packages% "data.table".
Note, this is only needed in cases where the automatic identification of
global and package dependencies is not sufficient.
o Added condition classes FutureCondition, FutureMessage, FutureWarning, and
FutureError representing conditions that occur while a future is setup,
launched, queried, or retrieved. They do *not* represent conditions that
occur while evaluating the future expression. For those conditions, new
classes FutureEvaluationCondition, FutureEvaulationMessage,
FutureEvaluationWarning, and FutureEvaluationError exists.
DOCUMENTATION:
o Vignette 'Common Issues with Solutions' now documents the case where the
future framework fails to identify a variable as being global because it
is only so conditionally, e.g. 'if (runif(1) < 1/2) x <- 0; y <- 2 * x'.
BETA FEATURES:
o Added mechanism for detecting globals that _may_ not be exportable to an
external R process (a "worker"). Typically, globals that carry connections
and external pointers ("externalptr") can not be exported, but there are
exceptions. By setting options 'future.globals.onReference' to "warning",
a warning is produced informing the user about potential problems. If
"error", an error is produced. Because there might be false positive, the
default is "ignore", which will cause above scans to be skipped. If there
are non-exportable globals and these tests are skipped, a run-time error
may be produced only when the future expression is evaluated.
BUG FIXES:
o The total size of global variables was overestimated, and dramatically so
if defined in the global environment and there were are large objects there
too. This would sometimes result in a false error saying that the total
size is larger than the allowed limit.
o An assignment such as 'x <- x + 1' where the left-hand side (LHS) 'x' is a
global failed to identify 'x' as a global because the right-hand side (RHS)
'x' would override it as a local variable. Updates to the globals package
fixed this problem.
o makeClusterPSOCK(..., renice = 19) would launch each PSOCK worker via
'nice +19' resulting in the error "nice: '+19': No such file or directory".
This bug was inherited from parallel::makePSOCKcluster(). Now using
'nice --adjustment=19' instead.
o Protection against passing future objects to other futures did not work
for future strategy 'multicore'.
DEPRECATED AND DEFUNCT:
o future_lapply() has moved to the new future.apply package available on CRAN.
The future::future_lapply() function will soon be deprecated, then defunct,
and eventually be removed from the future package. Please update your code
to make use of future.apply::future_lapply() instead.
o Dropped defunct 'eager' and 'lazy' futures; use 'sequential' instead.
o Dropped defunct arguments 'cluster' and 'maxCores'; use 'workers' instead.
o In previous version of the future package the FutureError class was used
to represent both orchestration errors (now FutureError) and evaluation
errors (now FutureEvaluationError). Any usage of class FutureError for
the latter type of errors is deprecated and should be updated to
FutureEvaluationError.
Version: 1.6.2 [2017-10-16]
NEW FEATURES:
o Now plan() accepts also strings such as "future::cluster".
o Now backtrace(x[[el]]) works also for non-environment 'x':s, e.g. lists.
BUG FIXES:
o When measuring the size of globals by scanning their content, for certain
types of classes the inferred lengths of these objects were incorrect
causing internal subset out-of-range issues.
o print() for Future would output one global per line instead of
concatenating the information with commas.
Version: 1.6.1 [2017-09-08]
NEW FEATURES:
o Now exporting getGlobalsAndPackages().
BUG FIXES:
o future_lapply() would give "Error in objectSize.env(x, depth = depth - 1L):
object 'nnn' not found" when for instance 'nnn' is part of an unresolved
expression that is an argument value.
SOFTWARE QUALITY:
o FIX: Some of the package assertion tests made too precise assumptions about
the object sizes, which fails with the introduction of ALTREP in R-devel
which causes the R's SEXP header size to change.
Version: 1.6.0 [2017-08-11]
NEW FEATURES:
o Now tweak(), and hence plan(), generates a more informative error message if
a non-future function is specified by mistake, e.g. calling plan(cluster)
with the 'survival' package attached after 'future' is equivalent to calling
plan(survival::cluster) when plan(future::cluster) was intended.
BUG FIXES:
o nbrOfWorkers() gave an error with plan(remote). Fixed by making the 'remote'
future inherit 'cluster' (as it should).
SOFTWARE QUALITY:
o TESTS: No longer testing forced termination of forked cluster workers when
running on Solaris. The termination was done by launching a future that
called quit(), but that appeared to have corrupted the main R session when
running on Solaris.
DEPRECATED AND DEFUNCT:
o Formally defunct 'eager' and 'lazy' futures; use 'sequential' instead.
o Dropped previously defunct %<=% and %=>% operators.
Version: 1.5.0 [2017-05-24]
SIGNIFICANT CHANGES:
o Multicore and multisession futures no longer reserve one core for the
main R process, which was done to lower the risk for producing a higher
CPU load than the number of cores available for the R session.
NEW FEATURES:
o makeClusterPSOCK() now defaults to use the Windows PuTTY software's SSH
client 'plink -ssh', if 'ssh' is not found.
o Argument 'homogeneous' of makeNodePSOCK(), a helper function of
makeClusterPSOCK(), will default to FALSE also if the hostname is a
fully qualified domain name (FQDN), that is, it "contains periods".
For instance, c('node1', 'node2.server.org') will use homogeneous = TRUE
for the first worker and homogeneous = FALSE for the second.
o makeClusterPSOCK() now asserts that each cluster node is functioning by
retrieving and recording the node's session information including the
process ID of the corresponding R process.
o Nested futures sets option 'mc.cores' to prevent spawning of recursive
parallel processes by mistake. Because 'mc.cores' controls _additional_
processes, it was previously set to zero. However, since some functions
such as mclapply() does not support that, it is now set to one instead.
DOCUMENTATION:
o Help on makeClusterPSOCK() gained more detailed descriptions on arguments
and what their defaults are.
DEPRECATED AND DEFUNCT:
o Formally deprecated eager futures; use sequential instead.
BUG FIXES:
o future_lapply() with multicore / multisession futures, would use a
suboptimal workload balancing where it split up the data in one chunk too
many. This is no longer a problem because of how argument 'workers' is
now defined for those type of futures (see note on top).
o future_lapply(), as well as lazy multicore and lazy sequential futures, did
not respect option 'future.globals.resolve', but was hardcoded to always
resolve globals (future.globals.resolve = TRUE).
o When globals larger than the allowed size (option 'future.globals.maxSize')
are detected an informative error message is generated. Previous version
introduced a bug causing the error to produce another error.
o Lazy sequential futures would produce an error when resolved if required
packages had been detached.
o print() would not display globals gathered for lazy sequential futures.
SOFTWARE QUALITY:
o Added package tests for globals part of formulas part of other globals,
e.g. purrr::map(x, ~ rnorm(.)), which requires globals (>= 0.10.0).
o Now package tests with parallel::makeCluster() not only test for
type = 'PSOCK' clusters but also 'FORK' (when supported).
o TESTS: Cleaned up test scripts such that the overall processing time for
the tests was roughly halved, while preserving the same test coverage.
Version: 1.4.0 [2017-03-12]
SIGNIFICANT CHANGES:
o The default for future_lapply() is now to _not_ generate RNG seeds
(future.seed = FALSE). If proper random number generation is needed,
use future.seed = TRUE. For more details, see help page.
NEW FEATURES:
o future() and future_lapply() gained argument 'packages' for explicitly
specifying packages to be attached when the futures are evaluated.
Note that the default throughout the future package is that all globals
and all required packages are automatically identified and gathered, so
in most cases those do not have to be specified manually.
o The default values for arguments 'connectTimeout' and 'timeout' of
makeNodePSOCK() can now be controlled via global options.
RANDOM NUMBER GENERATION:
o Now future_lapply() guarantees that the RNG state of the calling R process
after returning is updated compared to what it was before and in the exact
same way regardless of 'future.seed' (except FALSE), 'future.scheduling'
and future strategy used. This is done in order to guarantee that an R
script calling future_lapply() multiple times should be numerically
reproducible given the same initial seed.
o It is now possible to specify a pre-generated sequence of .Random.seed
seeds to be used for each FUN(x[i], ...) call in future_lapply(x, FUN, ...).
PERFORMANCE:
o future_lapply() scans global variables for non-resolved futures (to resolve
them) and calculate their total size once. Previously, each chunk
(a future) would redo this.
BUG FIX:
o Now future_lapply(x, FUN, ...) identifies global objects among 'x', 'FUN'
and '...' recursively until no new globals are found. Previously, only
the first level of globals were scanned. This is mostly thanks to a bug
fix in globals 0.9.0.
o A future that used a global object 'x' of a class that overrides length()
would produce an error if length(x) reports more elements than what can
be subsetted.
o nbrOfWorkers() gave an error with plan(cluster, workers = cl) where
'cl' is a cluster object created by parallel::makeCluster() etc.
This prevented for instance future_lapply() to work with such setups.
o plan(cluster, workers = cl) where cl <- makeCluster(..., type = MPI")
would give an instant error due to an invalid internal assertion.
DEPRECATED AND DEFUNCT:
o Previously deprecated arguments 'maxCores' and 'cluster' are now defunct.
o Previously deprecated assignment operators %<=% and %=>% are now defunct.
o availableCores(method = "mc.cores") is now defunct in favor of "mc.cores+1".
Version: 1.3.0 [2017-01-18]
SIGNIFICANT CHANGES:
o Where applicable, workers are now initiated when calling plan(), e.g.
plan(cluster) will set up workers on all cluster nodes. Previously,
this only happened when the first future was created.
NEW FEATURES:
o Renamed 'eager' futures to 'sequential', e.g. plan(sequential).
The 'eager' futures will be deprecated in an upcoming release.
o Added support for controlling whether a future is resolved eagerly
or lazily when creating the future, e.g. future(..., lazy = TRUE)
futureAssign(..., lazy = TRUE), and x %<-% { ... } %lazy% TRUE.
o future(), futureAssign() and futureCall() gained argument 'seed',
which specifies a L'Ecuyer-CMRG random seed to be used by the future.
The seed for future assignment can be specified via %seed%.
o futureAssign() now passes all additional arguments to future().
o Added future_lapply() which supports load balancing ("chunking")
and perfect reproducibility (regardless of type of load balancing
and how futures are resolved) via initial random seed.
o Added availableWorkers(). By default it returns localhost workers
according to availableCores(). In addition, it detects common
HPC allocations given in environment variables set by the HPC
scheduler.
o The default for plan(cluster) is now workers = availableWorkers().
o Now plan() stops any clusters that were implicitly created. For
instance, a multisession cluster created by plan(multisession) will
be stopped when plan(eager) is called.
o makeClusterPSOCK() treats workers that refer to a local machine by
its local or canonical hostname as "localhost". This avoids having
to launch such workers over SSH, which may not be supported on all
systems / compute cluster.
o Option 'future.debug' = TRUE also reports on total size of
globals identified and for cluster futures also the size of the
individual global variables exported.
o Option 'future.wait.timeout' (replaces 'future.wait.times') specifies
the maximum waiting time for a free workers (e.g. a core or a
compute node) before generating a timeout error.
o Option 'future.availableCores.fallback', which defaults to environment
variable 'R_FUTURE_AVAILABLECORES_FALLBACK' can now be used to specify
the default number of cores / workers returned by availableCores()
and availableWorkers() when no other settings are available. For
instance, if R_FUTURE_AVAILABLECORES_FALLBACK=1 is set system wide
in an HPC environment, then all R processes that uses availableCores()
to detect how many cores can be used will run as single-core processes.
Without this fallback setting, and without other core-specifying settings,
the default will be to use all cores on the machine, which does not play
well on multi-user systems.
GLOBALS:
o Globals part of locally defined functions are now also identified
thanks to globals (>= 0.8.0) updates.
DEPRECATED AND DEFUNCT:
o Lazy futures and plan(lazy) are now deprecated. Instead, use plan(eager)
and then f <- future(..., lazy = TRUE) or x %<-% { ... } %lazy% TRUE.
The reason behind this is that in some cases code that uses futures
only works under eager evaluation (lazy = FALSE; the default), or vice
verse. By removing the "lazy" future strategy, the user can no longer
override the lazy = TRUE / FALSE that the developer is using.
BUG FIXES:
o Creation of cluster futures (including multisession ones) would
time out already after 40 seconds if all workers were busy.
New default timeout is 30 days (option 'future.wait.timeout').
o nbrOfWorkers() gave an error for plan(cluster, workers) where 'workers'
was a character vector or a 'cluster' object of the parallel package.
Because of this, future_lapply() gave an error with such setups.
o availableCores(methods = "_R_CHECK_LIMIT_CORES_") would give an error
if not running R CMD check.
Version: 1.2.0 [2016-11-12]
NEW FEATURES:
o Added makeClusterPSOCK() - a version of parallel::makePSOCKcluster()
that allows for more flexible control of how PSOCK cluster workers
are set up and how they are launched and communicated with if running
on external machines.
o Added generic as.cluster() for coercing objects to cluster objects
to be used as in plan(cluster, workers = as.cluster(x)). Also added
a c() implementation for cluster objects such that multiple cluster
objects can be combined into a single one.
o Added sessionDetails() for gathering details of the current R session.
o plan() and plan("list") now prints more user-friendly output.
o On Unix, internal myInternalIP() tries more alternatives for finding
the local IP number.
DEPRECATED AND DEFUNCT:
o %<=% is deprecated. Use %<-% instead. Same for %=>%.
BUG FIXES:
o values() for lists and list environments of futures where one or more
of the futures resolved to NULL would give an error.
o value() for ClusterFuture would give cryptic error message
"Error in stop(ex) : bad error message" if the cluster worker had
crashed / terminated. Now it will instead give an error message like
"Failed to retrieve the value of ClusterFuture from cluster node #1 on
'localhost'. The reason reported was "error reading from connection".
o Argument 'user' to remote() was ignored (since 1.1.0).
Version: 1.1.1 [2016-10-10]
BUG FIXES:
o For the special case where 'remote' futures use workers = "localhost"
they (again) use the exact same R executable as the main / calling R
session (in all other cases it uses whatever 'Rscript' is found in the
PATH). This was already indeed implemented in 1.0.1, but with the
added support for reverse SSH tunnels in 1.1.0 this default behavior
was lost.
Version: 1.1.0 [2016-10-09]
NEW FEATURES:
o REMOTE CLUSTERS: It is now very simple to use cluster() and remote()
to connect to remote clusters / machines. As long as you can connect
via ssh to those machines, it works also with these future. The new
code completely avoids incoming firewall and incoming port forwarding
issues previously needed. This is done by using reverse SSH tunneling.
There is also no need to worry about internal or external IP numbers.
o Added optional argument 'label' to all futures, e.g.
f <- future(42, label="answer") and v %<-% { 42 } %label% "answer".
o Added argument 'user' to cluster() and remote().
o Now all Future classes supports run() for launching the future and
value() calls run() if the future has not been launched.
o MEMORY: Now plan(cluster, gc=TRUE) causes the background
R session to be garbage collected immediately after the value
is collected. Since multisession and remote futures are special
cases of cluster futures, the same is true for these as well.
o ROBUSTNESS: Now the default future strategy is explicitly set when
no strategies are set, e.g. when used nested futures. Previously,
only mc.cores was set so that only a single core was used, but now
also plan("default") set.
o WORKAROUND: resolved() on cluster futures would block on Linux until
future was resolved. This is due to a bug in R. The workaround is
to use round the timeout (in seconds) to an integer, which seems to
always work / be respected.
GLOBALS:
o Global variables part of subassignments in future expressions
are recognized and exported (iff found), e.g. x$a <- value,
x[["a"]] <- value, and x[1,2,3] <- value.
o Global variables part of formulae in future expressions are
recognized and exported (iff found), e.g. y ~ x | z.
o As an alternative to the default automatic identification of
globals, it is now also possible to explicitly specify them either
by their names (as a character vector) or by their names and values
(as a named list), e.g. f <- future({ 2*a }, globals=c("a")) or
f <- future({ 2*a }, globals=list(a=42)). For future assignments one
can use the %globals% operator, e.g. y %<-% { 2*a } %globals% c("a").
DOCUMENTATION:
o Added vignette on command-line options and other methods for
controlling the default type of futures to use.
Version: 1.0.1 [2016-07-04]
o ROBUSTNESS: For the special case where 'remote' futures use
workers = "localhost" they now use the exact same R executable
as the main / calling R session (in all other cases it uses
whatever 'Rscript' is found in the PATH).
o FutureError now extends simpleError and no longer the error
class of captured errors.
DOCUMENTATION:
o Adding section to vignette on globals in formulas describing how
they are currently not automatically detected and how to explicitly
export them.
BUG FIXES:
o Since future 0.13.0, a global 'pkg' would be overwritten by the
name of the last package attached in future.
o Futures that generated R.oo::Exception errors, they triggered
another internal error.
Version: 1.0.0 [2016-06-24]
NEW FEATURES:
o Add support for remote(..., myip=""), which now
queries a set of external lookup services in case one of them
fails.
o Add mandelbrot() function used in demo to the API for convenience.
o ROBUSTNESS: If .future.R script, which is sourced when the future
package is attached, gives an error, then the error is ignored
with a warning.
o TROUBLESHOOTING: If the future requires attachment of packages,
then each namespace is loaded separately and before attaching
the package. This is done in order to see the actual error
message in case there is a problem while loading the namespace.
With require()/library() this error message is otherwise suppressed
and replaced with a generic one.
GLOBALS:
o Falsely identified global variables no longer generate
an error when the future is created. Instead, we leave it to R
and the evaluation of the individual futures to throw an error
if the a global variable is truly missing. This was done in order
to automatically handle future expressions that use non-standard
evaluation (NSE), e.g. subset(df, x < 3) where 'x' is falsely
identified as a global variable.
o Dropped support for system environment variable 'R_FUTURE_GLOBALS_MAXSIZE'.
DOCUMENTATION:
o DEMO: Now the Mandelbrot demo tiles a single Mandelbrot region
with one future per tile. This better illustrates parallelism.
o Documented R options used by the future package.
BUG FIXES:
o Custom futures based on a constructor function that
is defined outside a package gave an error.
o plan("default") assumed that the 'future.plan' option
was a string; gave an error if it was a function.
o Various future options were not passed on to futures.
o A startup .future.R script is no longer sourced if the
future package is attached by a future expression.
Version: 0.15.0 [2016-06-13]
NEW FEATURES:
o Added remote futures, which are cluster futures with convenient
default arguments for simple remote access to R, e.g.
plan(remote, workers="login.my-server.org").
o Now .future.R (if found in the current directory or otherwise in
the user's home directory) is sourced when the future package is
attach (but not loaded). This helps separating scripts from
configuration of futures.
o Added support for plan(cluster, workers=c("n1", "n2", "n2", "n4")),
where 'workers' (also for ClusterFuture()) is a set of host names
passed to parallel::makeCluster(workers). It can also be the number
of localhost workers.
o Added command line option --parallel=
, which is long for -p
.
o Now command line option -p
also set the default future strategy
to multiprocessing (if p >= 2 and eager otherwise), unless another
strategy is already specified via option 'future.plan' or system
environment variable R_FUTURE_PLAN.
o Now availableCores() also acknowledges environment variable NSLOTS
set by Sun/Oracle Grid Engine (SGE).
o MEMORY: Added argument 'gc=FALSE' to all futures. When TRUE, the
garbage collector will run at the very end in the process that
evaluated the future (just before returning the value). This may
help lowering the overall memory footprint when running multiple
parallel R processes. The user can enable this by specifying
plan(multiprocess, gc=TRUE). The developer can control this using
future(expr, gc=TRUE) or v %<-% { expr } %tweak% list(gc=TRUE).
PERFORMANCE:
o Significantly decreased the overhead of creating a future, particularly
multicore futures.
BUG FIXES:
o Future would give an error with plan(list("eager")),
whereas it did work with plan("eager") and plan(list(eager)).
Version: 0.14.0 [2016-05-16]
NEW FEATURES:
o Added nbrOfWorkers().
o Added informative print() method for the Future class.
o values() passes arguments '...' to value() of each Future.
o Added FutureError class.
DEPRECATED AND DEFUNCT:
o Renamed arguments 'maxCores' and 'cluster' to 'workers'. If using
the old argument names a deprecation warning will be generated, but
it will still work until made defunct in a future release.
BUG FIXES:
o resolve() for lists and environments did not work
properly when the set of futures was not resolved in order,
which could happen with asynchronous futures.
Version: 0.13.0 [2016-04-13]
NEW FEATURES:
o Add support to plan() for specifying different future strategies for
the different levels of nested futures.
o Add backtrace() for listing the trace the expressions evaluated (the
calls made) before a condition was caught.
o Add transparent futures, which are eager futures with early signaling
of conditioned enabled and whose expression is evaluated in the calling
environment. This makes the evaluation of such futures as similar
as possible to how R evaluates expressions, which in turn simplifies
troubleshooting errors etc.
o Add support for early signaling of conditions. The default is
(as before) to signal conditions when the value is queried.
In addition, they may be signals as soon as possible, e.g. when
checking whether a future is resolved or not.
o Signaling of conditions when calling value() is now controlled by
argument 'signal' (previously 'onError').
o Now UniprocessFuture:s captures the call stack for errors occurring
while resolving futures.
o ClusterFuture gained argument 'persistent=FALSE'. With persistent=TRUE,
any objects in the cluster R session that was created during the
evaluation of a previous future is available for succeeding futures
that are evaluated in the same session. Moreover, globals are
still identified and exported but "missing" globals will not give
an error - instead it is assumed such globals are available in the
environment where the future is evaluated.
o OVERHEAD: Utility functions exported by ClusterFuture are now much
smaller; previously they would export all of the package environment.
BUG FIXES:
o f <- multicore(NA, maxCores=2) would end up in an endless
waiting loop for a free core if availableCores() returned one.
o ClusterFuture would ignore local=TRUE.
Version: 0.12.0 [2016-02-23]
NEW FEATURES:
o Added multiprocess futures, which are multicore futures if supported,
otherwise multisession futures. This makes it possible to use
plan(multiprocess) everywhere regardless of operating system.
o Future strategy functions gained class attributes such that it is
possible to test what type of future is currently used, e.g.
inherits(plan(), "multicore").
o ROBUSTNESS: It is only the R process that created a future that can
resolve it. If a non-resolved future is queried by another R process,
then an informative error is generated explaining that this is not
possible.
o ROBUSTNESS: Now value() for multicore futures detects if the underlying
forked R process was terminated before completing and if so generates
an informative error messages.
PERFORMANCE:
o Adjusted the parameters for the schema used to wait for next available
cluster node such that nodes are polled more frequently.
GLOBALS:
o resolve() gained argument 'recursive'.
o Added option 'future.globals.resolve' for controlling whether
global variables should be resolved for futures or not. If TRUE, then
globals are searched recursively for any futures and if found such
"global" futures are resolved. If FALSE, global futures are not
located, but if they are later trying to be resolved by the parent
future, then an informative error message is generated clarifying
that only the R process that created the future can resolve it.
The default is currently FALSE.
BUG FIX:
o FIX: Exports of objects available in packages already attached
by the future were still exported.
o FIX: Now availableCores() returns 3L (=2L+1L) instead of 2L
if _R_CHECK_LIMIT_CORES_ is set.
Version: 0.11.0 [2016-01-15]
NEW FEATURES:
o Add multisession futures, which analogously to multicore ones,
use multiple cores on the local machine with the difference
that they are evaluated in separate R session running in the
background rather than separate forked R processes.
A multisession future is a special type of cluster futures that
do not require explicit setup of cluster nodes.
o Add support for cluster futures, which can make use of a cluster
of nodes created by parallel::makeCluster().
o Add futureCall(), which is for futures what do.call() is otherwise.
o Standardized how options are named, i.e. 'future.