A Future for R: Common Issues with Solutions

Missing globals (false negatives)

If a global variable is used in a future expression that conditionally overrides this global variable with a local one, the future framework fails to identify the global variable and therefore fails to export it, resulting in a run-time error. For example, although this works:

plan(multisession)

reset <- TRUE
x <- 1
y %<-% { if (reset) x <- 0; x + 1 }
y
## [1] 1

the following does not work:

reset <- FALSE
x <- 1
y %<-% { if (reset) x <- 0; x + 1 }
y
## Error: object 'x' not found

It is recommended to avoid above constructs where it is ambiguous whether a variable is global or local. To force variable x to always be global, insert it at the very being of the future expression, e.g.

reset <- FALSE
x <- 1
y %<-% { x; if (reset) x <- 0; x + 1 }
y
## [1] 2

Comment: The goal is to in a future version of the package detect globals also in expression where the local-global state of a variable is only known at run time.

do.call(file_ext, list("foo.txt"))

do.call("file_ext", list("foo.txt"))

> library(future)
> library(listenv)
> library(tools)
> plan(sequential)
> pathnames <- c("foo.txt", "bar.png", "yoo.md")
> res <- listenv()
> for (ii in seq_along(pathnames)) {
+   res[[ii]] %<-% do.call("file_ext", list(pathnames[ii])) %lazy% TRUE
+ }
> file_ext <- function(...) "haha!"
> unlist(res)
[1] "haha!" "haha!" "haha!"

> a <- 1:3
> b <- 4:6
> c <- 3:5
> my_sum <- function(var) { sum(get(var)) }
> y <- my_sum("a")
> y
[1] 6

> library(future)
> plan(multisession)
> f <- future(my_sum("a"))
> y <- value(f)
Error in get(var) : object 'a' not found

> f <- future(my_sum("a"), globals = structure(TRUE, add = "a"))
> y <- value(f)
> y
[1] 6

> f <- future({ a; my_sum("a") })
> y <- value(f)
> y
[1] 6

> a <- 1:3
> b <- 4:7
> c <- 3:5
> my_sum <- function(var) { sum(get(var)) }
> y <- sapply(c("a", "b", "c"), FUN = my_sum)
> y
 a  b  c
 6 22 12

> data <- list(a = 1:3, b = 4:7, c = 3:5)

> my_sum <- function(x) { sum(x) }
> y <- sapply(data, FUN = my_sum)
> y
 a  b  c
 6 22 12

> library(glue)
> a <- 42
> s <- glue("The value of a is {a}.")
> s
The value of a is 42.

> library(glue)
> library(future)
> plan(multisession)
> a <- 42
> s %<-% glue("The value of a is {a}.")
> s
Error in eval(parse(text = text, keep.source = FALSE), envir) : 
  object 'a' not found

> s %<-% glue("The value of a is {a}.") %globals% structure(TRUE, add = "a")
> s
The value of a is 42.

> s %<-% { a; glue("The value of a is {a}.") }
> s
The value of a is 42.

Missing packages (false negatives)

Occasionally, the static-code inspection of the future expression fails to identify packages needed to evaluated the expression. This may occur when an expression uses S3 generic functions part of one package whereas the required S3 method is in another package. For example, in the below future generic function [ is used on data.table object DT, which requires S3 method [.data.table from the data.table package. However, the future and globals packages fail to identify data.table as a required package, which results in an evaluation error:

> library(future)
> plan(multisession)

> library(data.table)
> DT <- data.table(a = LETTERS[1:3], b = 1:3)
> y %<-% DT[, sum(b)]
> y
Error: object 'b' not found

The above error occurs because, contrary to the master R process, the R worker that evaluated the future expression does not have data.table loaded. Instead the evaluation falls back to the [.data.frame method, which is not what we want.

Until the future framework manages to identify data.table as a required package (which is the goal), we can guide future by specifying additional packages needed:

> y %<-% DT[, sum(b)] %packages% "data.table"
> y
[1] 6

or equivalently

> f <- future(DT[, sum(b)], packages = "data.table")
> value(f)
[1] 6

Note, do not use library() or loadNamespace() to resolve these problems. It is always better to use the above packages approach.

Capturing errors, outputting their messages, and returning a default value

Sometimes a function call produce an error for a particular input. In such cases, we might want to return a default value, say, a missing value, instead of signaling an error. This can be done using:

res <- tryCatch({
  unstable_calc(x)
}, error = function(e) {
  NA_real_
})

Here, res takes the value of unstable_calc(x), unless it produces an error, in case it takes value NA_real_.

In addition to the above, we could produce a warning whenever we get an error and replace it with a missing value. We can do this as:

res <- tryCatch({
  unstable_calc(x)
}, error = function(e) {
  warning(conditionMessage(e))
  NA_real_
})

This will turn the error into a warning with the same message. If we want to just output the message without producing a warning, we can use message(conditionMessage(e)).

Importantly, we must not use just warning(e) or message(e), although it appears to work at a first glance. If we do, we will end up re-signaling the error but without interruption. It is an important distinction that will reveal itself if used within futures. The above example with warning(conditionMessage(e)) will work as expected, but if we use warning(e) the future framework will produce an error, not a warning.

Using source() in a future

Avoid using source() inside futures. It is always better to source external R scripts at the top of your main R script, e.g.

library(future)
source("./my-script.R")

f <- future({
  ...
})

However, if you find yourself having to source a script inside a future, or inside a function, make sure to specify argument local = TRUE, e.g.

f <- future({
  source("./my-script.R", local = TRUE)
  ...
})

This is because source() defaults to local = FALSE, which has side effects. When using local = FALSE, any functions or variables defined by the R script are assigned to the global environment - not the calling environment as we might expect. This may make little different when calling source() from the R prompt, or from another script. However, when called from inside a function, inside local(), or inside a future, it might result in unexpected behavior. It is similar to using assign("a", 42, envir = globalenv()), which is known be a bad practice. To be on the safe side, it is almost always better call source() with local = TRUE.

Clashes with other packages

Sometimes other packages have functions or operators with the same name as the future package, and if those packages are attached after the future package, their objects will mask the ones of the future package. For instance, the igraph package also defines a %<-% operator which clashes with the one in future if used at the prompt or in a script (it is not a problem inside package because there we explicitly import objects in a known order). Here is what we might get:

> library(future)
> library(igraph)

Attaching package: 'igraph'

The following objects are masked from 'package:future':

    %<-%, %->%

The following objects are masked from 'package:stats':

    decompose, spectrum

The following object is masked from 'package:base':

    union

> y %<-% { 42 }
Error in get(".igraph.from", parent.frame()) : 
  object '.igraph.from' not found

Here we get an error because %<-% is from igraph and not the future assignment operator as we wanted. This can be confirmed as:

> environment(`%<-%`)
<environment: namespace:igraph>

To avoid this problem, attach the two packages in opposite order such that future comes last and thereby overrides igraph, i.e.

> library(igraph)
> library(future)

Attaching package: 'future'

The following objects are masked from 'package:igraph':

%<-%, %->%

> y %<-% { 42 }
> y
[1] 42

An alternative is to detach the future package and re-attach it, which will achieve the same thing:

> detach("package:future")
> library(future)

Yet another alternative is to explicitly override the object by importing it to the global environment, e.g.

> `%<-%` <- future::`%<-%`
> y %<-% { 42 }
> y
[1] 42

In this case, it does not matter in what order the packages are attached because we will always use the copy of future::`%<-%`.

Syntax error: “non-numeric argument to binary operator”

The future assignment operator %<-% is a binary infix operator, which means it has higher precedence than most other binary operators but also higher than some of the unary operators in R. For instance, this explains why we get the following error:

> x %<-% 2 * runif(1)
Error in x %<-% 2 * runif(1) : non-numeric argument to binary operator

What effectively is happening here is that because of the higher priority of %<-%, we first create a future x %<-% 2 and then we try to multiply the future (not its value) with the value of runif(1) - which makes no sense. In order to properly assign the future variable, we need to put the future expression within curly brackets;

> x %<-% { 2 * runif(1) }
> x
[1] 1.030209

Parentheses will also do. For details on precedence on operators in R, see Section ‘Infix and prefix operators’ in the ‘R Language Definition’ document.

Error: invalid ‘type’ (environment) of argument with magrittr

Another example where the future assignment operator %<-% requires curly brackets is when using the magrittr infix operator %>%, e.g.

> library(magrittr)
> x %<-% 1:100 %>% sum
Error in sum(.) : invalid 'type' (environment) of argument

The reason for this error is that x %<-% 1:100 is passed to sum() by %>%. To fix this, use:

> x %<-% { 1:100 %>% sum }
> x
[1] 5050

R CMD check NOTEs

The code inspection run by R CMD check will not recognize the future assignment operator %<-% as an assignment operator, which is not surprising because %<-% is technically an infix operator. This means that if you for instance use the following code in your package:

foo <- function() {
  b <- 3.14
  a %<-% { b + 1 }
  a
}

then R CMD check will produce a NOTE saying:

* checking R code for possible problems ... NOTE
foo: no visible binding for global variable 'a'
Undefined global functions or variables:
  a

In order to avoid this, we can add a dummy assignment of the missing global at the top of the function, i.e.

foo <- function() {
  a <- NULL ## To please R CMD check
  b <- 3.14
  a %<-% { b + 1 }
  a
}